Gene I. Sher - Handbook of Neuroevolution Through Erlang [2013, PDF, ENG]

Chapter 1 Introduction: Applications & Motivations ........................................ 1
1.1 Motivations ................................................................................................... 2
1.2 Applications .................................................................................................. 5
1.2.1 Robotics ................................................................................................ 5
1.2.2 Financial Markets................................................................................ 10
1.2.3 Artificial Life ...................................................................................... 14
1.2.4 Image Analysis & Computer Vision ................................................... 18
1.2.5 Data Compression ............................................................................... 19
1.2.6 Games & Entertainment ...................................................................... 21
1.2.7 Cyber Warfare ..................................................................................... 22
1.2.8 Circuit Creation & Optimization ........................................................ 26
1.2.9 Optimizing Shapes and Structures ...................................................... 30
1.2.10 Computational Intelligence & Towards Singularity ......................... 34
1.3 A Whirlwind Overview .............................................................................. 34
1.4 Endgame ..................................................................................................... 37
1.5 References .................................................................................................. 38
PART I
FOUNDATIONS
Chapter 2 Introduction to Neural Networks ..................................................... 43
2.1 Biological Neural Network......................................................................... 45
2.2 Artificial Neural Network........................................................................... 51
2.2.1 The Neurode in Detail ......................................................................... 54
2.3 Neural Networks and Neural Network Based Systems .............................. 57
2.3.1 Recurrent Neural Networks and Memory Loops ................................ 59
2.3.2 A Neural Network Based System ....................................................... 60
2.4 Learning Vs. Training ................................................................................ 61
2.5 Neural Network Supervised “Learning” Algorithms ................................. 63
2.6 Neural Network Unsupervised Learning Algorithms ................................ 66
2.6.1 Hebbian Learning................................................................................ 67
2.6.2 Neuromodulation ................................................................................ 70
2.6.3 Competitive Learning ......................................................................... 73
2.6.4 Kohonen/Self Organizing Map ........................................................... 74
2.6.5 Putting it All Together ........................................................................ 76
2.7 Summary ..................................................................................................... 78
2.8 References .................................................................................................. 78
Chapter 3 Introduction to Evolutionary Computation .................................... 81
3.1 Evolution .................................................................................................... 81
3.2 Extracting the Essentials of Biological Evolution ...................................... 85
3.3 Formulating a Given Problem in Evolutionary Terms ............................... 88
3.4 The Different Flavors of Evolutionary Algorithms .................................... 93
3.4.1 Genetic Algorithms ............................................................................. 94
3.4.2 Genetic Programming ......................................................................... 96
3.4.3 Evolutionary Strategies ....................................................................... 97
3.4.4 Evolutionary Programming ................................................................. 98
3.5 A Few Words on Memetic Computing ...................................................... 98
3.6 The Different Sides of the Same Coin...................................................... 102
3.7 References ................................................................................................ 103
Chapter 4 Introduction to Neuroevolutionary Methods ................................ 105
4.1 Neural Network Encoding Approaches and Mutation Operators ............ 106
4.1.1 Neural Network Mutation Operators ................................................ 106
4.1.2 Neural Network Genotype Encoding ................................................ 108
4.1.3 The Neural Network Phenotype........................................................ 113
4.2 Neuroevolution Through Genetic Algorithms ......................................... 118
4.2.1 Example 1: Evolving a XOR Logical Operator ................................ 118
4.2.2 Example 2: Evolving a pole balancing neurocontroller ................... 123
4.2.3 Example 3: End Game; Evolving Intelligence ................................. 127
4.3 Neuroevolution Through Memetic Algorithms ........................................ 135
4.4 Neural Networks as Graph Based Genetic Programming Systems.......... 138
4.5 References ................................................................................................ 140
Chapter 5 The Unintentional Neural Network Programming Language .... 143
5.1 The Necessary Features ............................................................................ 144
5.2 Erlang: From Telecommunications Networks To Neural Networks ....... 146
5.3 The Conceptual Mapping of a NN to Erlang’s Architecture ................... 147
5.4 Robustness and Fault Tolerance in Computational Intelligence .............. 149
5.5 I’ve Done This Once Before; And We Are On Our Way ........................ 149
5.6 References ................................................................................................ 150
PART II
NEUROEVOLUTION: TAKING THE FIRST STEP
Chapter 6 Developing a Feed Forward Neural Network ............................... 153
6.1 Simulating A Neuron ................................................................................ 155
6.2 A One Neuron Neural Network ................................................................ 158
6.3 Planning Our Neural Network System’s Architecture ............................. 162
6.4 Developing a Genotype Representation ................................................... 166
6.5 Programming the Genotype Constructor .................................................. 168
6.6 Developing the Genotype to Phenotype Mapping Module ...................... 174
6.7 Summary ................................................................................................... 184
Chapter 7 Adding the “Stochastic Hill-Climber” Learning Algorithm ....... 187
7.1 The Learning Method ............................................................................... 188
7.1.1 Comparing EA to Random Restart SHC .......................................... 191
7.2 The Trainer ............................................................................................... 194
7.3 The Exoself ............................................................................................... 195
7.4 The Scape ................................................................................................. 197
7.5 Scapes, Sensors, Actuators, Morphologies, and Fitness .......................... 199
7.6 Developing the Extended Architecture .................................................... 202
7.6.1 Modifying the genotype Module ...................................................... 203
7.6.2 Modifying the morphology Module ................................................. 204
7.6.3 Developing the trainer Module ......................................................... 205
7.6.4 Modifying the exoself Module.......................................................... 207
7.6.5 Developing the scape Module ........................................................... 210
7.6.6 Modifying the cortex Module ........................................................... 213
7.6.7 Modifying the neuron Module .......................................................... 214
7.6.8 Modifying the sensor Module ........................................................... 218
7.6.9 Modifying the actuator Module ........................................................ 219
7.7 Compiling Modules & Simulating the XOR Operation ........................... 219
7.8 Adding the benchmarker Module ............................................................. 223
7.9 Summary ................................................................................................... 226
Chapter 8 Developing a Simple Neuroevolutionary Platform ....................... 229
8.1 The New Architecture .............................................................................. 231
8.2 The New Data Structures.......................................................................... 233
8.3 Developing the polis Module ................................................................... 236
8.4 Updating the genotype Module ................................................................ 242
8.4.1 Moving from ETS to Mnesia ............................................................ 243
8.4.2 A NN Based Adaptive Agent ............................................................ 243
8.4.3 A New Neuron Id Structure; Adding Recursive Connections .......... 244
8.4.4 Seed Computational Intelligence ...................................................... 247
8.4.5 Constraints ........................................................................................ 248
8.4.6 The Updated genotype Module......................................................... 249
8.5 Developing the genotype_mutator ........................................................... 262
8.5.1 The Precursor Connection Making and Breaking Functions ............ 266
8.5.2 mutate_weights ................................................................................. 274
8.5.3 add_bias & remove_bias ................................................................... 276
8.5.4 mutate_af ........................................................................................... 278
8.5.5 add_outlink........................................................................................ 279
8.5.6 add_inlink.......................................................................................... 281
8.5.7 add_sensorlink .................................................................................. 284
8.5.8 add_actuatorlink ................................................................................ 285
8.5.9 add_neuron ........................................................................................ 287
8.5.10 outsplice .......................................................................................... 289
8.5.11 add_sensor....................................................................................... 294
8.5.12 add_actuator .................................................................................... 296
8.5.13 Planning the Remaining Few Details of the Genotype Mutator
Module ............................................................................................ 298
8.5.14 Implementing the genotype_mutator Module................................. 301
8.5.15 Developing the population_monitor ............................................... 309
8.5.16 Updating the exoself Module .......................................................... 327
8.5.17 Updating the neuron Module .......................................................... 337
8.6 Summary ....................................................................................................345
8.7 Reference ...................................................................................................346
Chapter 9 Testing the Neuroevolutionary System .......................................... 347
9.1 Testing the Mutation Operators ................................................................ 347
9.2 Testing the Neuroevolutionary System on the Simple XOR Benchmark 369
9.2.1 The mutate_af Error .......................................................................... 375
9.2.2 Same Neuron, But Different Ids in the evo_hist List ....................... 378
9.2.3 Multiple Actuators of the Same Type ............................................... 379
9.2.4 Making Fingerprint Store Generalized Sensors & Actuators ........... 382
9.2.5 The Quizzical Topology of the Fittest NN System........................... 383
9.3 Retesting Our Neuroevolutionary System ................................................ 388
9.4 Summary ................................................................................................... 395
PART III
A CASE STUDY
Chapter 10 DXNN: A Case Study..................................................................... 399
10.1 The Reason for the Memetic Approach to Synaptic Weight
Optimization ............................................................................................. 399
10.2 The DXNN Encoding and Architecture ................................................. 402
10.3 Generational Evolution ........................................................................... 405
10.3.1 Step-1: Initialization Phase ............................................................. 407
10.3.2 Step-4: Parametric Tuning Phase .................................................... 409
10.3.3 Step-8 & 13: The Selection & Offspring Allocation Phase ............ 413
10.3.4 Step-18: The Topological Mutation Phase ..................................... 415
10.4 Steady-State Evolution ........................................................................... 417
10.5 Direct (Neural) and Indirect (Substrate) Encoding ................................ 421
10.5.1 Neural Substrates ............................................................................ 423
10.5.2 Genotype Representation ................................................................ 427
10.5.3 Substrate Phenotype Representation ............................................... 435
10.6 DXNN Research Group & the NN Research Repository ...................... 439
10.7 Currently Active Projects ....................................................................... 440
10.7.1 Cyberwarfare................................................................................... 440
10.7.2 Coevolving Unmanned Ariel Vehicle Combat Maneuvers ............ 441
10.7.3 Evolving New CPU Architectures & Optimizing Existing Ones ... 442
10.8 Summary and Future Work .................................................................... 443
10.9 References .............................................................................................. 443
PART IV
ADVANCED NEUROEVOLUTION: CREATING THE CUTTING EDGE
Chapter 11 Decoupling & Modularizing Our Neuroevolutionary Platform 447
11.1 What Can be Decoupled? ....................................................................... 448
11.2 Updating the Genotype Representation .................................................. 451
11.2.1 The sensor & actuator Records ....................................................... 452
11.2.2 The neuron Record .......................................................................... 453
11.2.3 The agent Record ............................................................................ 454
11.2.4 The population Record .................................................................... 456
11.2.5 The constraint Record ..................................................................... 457
11.3 Updating the records.hrl ......................................................................... 460
11.4 Updating the Modules ............................................................................ 465
11.4.1 Updating the genotype Module....................................................... 465
11.4.2 Updating the genome_mutator Module .......................................... 468
11.4.3 Updating the population_monitor Module ..................................... 478
11.4.4 Creating the selection_algorithm Module....................................... 485
11.4.5 Creating the fitness_postprocessor Module .................................... 489
11.4.6 Creating the steady_state Evolutionary Loop ................................. 490
11.4.7 Updating the exoself Module .......................................................... 495
11.4.8 Updating the neuron Module .......................................................... 508
11.4.9 Creating the signal_aggregator Module .......................................... 514
11.4.10 Creating the plasticity Module ...................................................... 516
11.5 Compiling & Testing the New System................................................... 518
11.6 Summary & Discussion .......................................................................... 525
11.7 References .............................................................................................. 526
Chapter 12 Keeping Track of Important Population and Evolutionary
Stats ................................................................................................. 527
12.1 The Necessary Additions to the System ................................................. 528
12.2 The Trace Format ................................................................................... 530
12.3 Implementation ....................................................................................... 532
12.3.1 Updating records.hrl ....................................................................... 532
12.3.2 Building the Topological Summary of a Neural Network.............. 533
12.3.3 Implementing the Trace Updating Cast Clause .............................. 535
12.3.4 Updating the exoself Module .......................................................... 541
12.4 Compiling & Testing .............................................................................. 542
12.5 Summary & Discussion .......................................................................... 545
Chapter 13 The Benchmarker .......................................................................... 547
13.1 The benchmarker Architecture ............................................................... 548
13.2 Adding New Records ............................................................................. 549
13.3 Updating the population_monitor Module ............................................. 555
13.4 Implementing the benchmarker .............................................................. 557
13.5 Compiling and Testing ........................................................................... 564
13.6 Summary ................................................................................................. 570
13.7 References .............................................................................................. 571
Chapter 14 Creating the Two Slightly More Complex Benchmarks ............ 573
14.1 Pole Balancing Simulation ..................................................................... 573
14.1.1 Implementing the Pole Balancing Scape ........................................ 577
14.1.2 Implementing the Pole Balancing morphology .............................. 581
14.1.3 Benchmark Results ......................................................................... 583
14.2 T-Maze Simulation ................................................................................. 590
14.2.1 T-Maze Implementation.................................................................. 598
14.2.2 Benchmark Results ......................................................................... 604
14.3 Summary & Discussion .......................................................................... 607
14.4 References .............................................................................................. 608
Chapter 15 Neural Plasticity ............................................................................. 609
15.1 Hebbian Rule .......................................................................................... 610
15.1.1 Implementing the New input_idps & pf Formats ........................... 613
15.1.2 Implementing the Simple Hebbian Learning Rule ......................... 617
15.2 Oja’s Rule ............................................................................................... 620
15.2.1 Implementing the Oja’s Learning Rule .......................................... 621
15.3 Neuromodulation .................................................................................... 623
15.3.1 The Neuromodulatory Architecture ................................................ 626
15.3.2 Implementing the self_modulation Learning Rules ....................... 631
15.3.3 Implementing the input_idps_modulation Based
Neuromodulated Plasticity .............................................................. 635
15.4 Plasticity Parameter Mutation Operators ............................................... 641
15.4.1 Implementing the Weight Parameter Mutation Operator ............... 642
15.4.2 Implementing the Neural Parameter Mutation Operator ................ 643
15.4.3 Implementing the Hybrid, Weight & Neural Parameters
Mutation Operator ........................................................................... 644
15.4.4 Updating the genome_mutator Module .......................................... 645
15.5 Tuning of a NN which has Plastic Neurons ........................................... 650
15.6 Compiling & Testing .............................................................................. 656
15.7 Summary & Discussion .......................................................................... 658
15.8 References .............................................................................................. 659
Chapter 16 Substrate Encoding ........................................................................ 661
16.1 A Brief Overview of Substrate Encoding............................................... 662
16.2 The Updated Architecture of Our NN Based Systems ........................... 669
16.3 The Genotype of the Substrate Encoded NN ......................................... 672
16.4 The SENN Phenotype............................................................................. 679
16.5 Implementing the substrate_cpps & substrate_ceps............................... 686
16.5.1 Implementing the substrate_cpp Module ........................................ 691
16.5.2 Implementing the substrate_cep Module ........................................ 694
16.6 Updating the genotype Module .............................................................. 696
16.7 Updating the exoself Module ................................................................. 701
16.8 Implementing the substrate Module ....................................................... 705
16.9 Updating the genome_mutator Module .................................................. 724
16.10 Implementing the add_cpp and add_cep Mutation Operators ............. 726
16.11 Testing the New Encoding Method ...................................................... 728
16.12 Summary and Discussion ..................................................................... 733
16.13 References ............................................................................................ 733
Chapter 17 Substrate Plasticity ........................................................................ 735
17.1 The Updated Architecture ...................................................................... 736
17.2 Implementing the abcn Learning Rule ................................................... 737
17.2.1 Updating the substrate Module ....................................................... 737
17.2.2 Updating the Morphology Module ................................................. 741
17.2.3 Updating the substrate_cpp & substrate_cep Modules ................... 743
17.2.4 Benchmarking the New Substrate Plasticity ................................... 745
17.3 Implementing the iterative Learning Rule.............................................. 746
17.3.1 Benchmarking the New iterative Substrate Plasticity .................... 750
17.4 Discussion............................................................................................... 752
PART V
APPLICATIONS
Chapter 18 Artificial Life .................................................................................. 755
18.1 Simulated Environment and Artificial Organisms: Flatland .................. 755
18.2 The Scape and the Fitness Function ...................................................... 756
18.2.1 Public Scape Architectures, Polis Interface, and Scape Sectors ..... 759
18.2.2 The Flatland Interface ..................................................................... 761
18.3 Flatland’s Avatar Encoding .................................................................... 762
18.4 Updating the Morphology, Sensor, and Actuator Modules ................... 763
18.5 Updating the exoself Module ................................................................. 766
18.6 The Simulation and Results .................................................................... 768
18.6.1 Simple Food Gathering Simulation ................................................ 769
18.6.2 Dangerous Food Gathering Simulation .......................................... 773
18.6.3 Predator Vs. Prey Simulation.......................................................... 776
18.7 Discussion............................................................................................... 781
18.8 Summary ................................................................................................. 782
18.9 References .............................................................................................. 783
Chapter 19 Evolving Currency Trading Agents ............................................. 785
19.1 Introduction to Forex .............................................................................. 786
19.2 Trading and the Objective ...................................................................... 788
19.3 The Forex Simulator ............................................................................... 794
19.4 Implementing the Forex Simulator......................................................... 797
19.5 Implementing the New Sensors and Actuators ...................................... 806
19.6 Generalization Testing............................................................................ 809
19.7 Benchmark & Results ............................................................................. 815
19.7.1 Running the Benchmark ................................................................. 817
19.8 Discussion............................................................................................... 822
19.9 Summary ................................................................................................. 823
19.10 References ............................................................................................ 824
PART VI
PROMISES KEPT
Chapter 20 Conclusion ...................................................................................... 829
20.1 References .............................................................................................. 830
Abbreviations ..................................................................................................... 831