Optimization Techniques (eBook)

Front Cover 1
Optimization Techniques 4
Copyright Page 5
Contents 6
Contributors 16
Preface 18
Chapter 1. Optimal Learning in Artificial Neural Networks: A Theoretical View 24
I. Introduction 24
II. Formulation of Learning as an Optimization Problem 27
III. Learning with No Local Minima 33
IV. Learning with Suboptimal Solutions 56
V. Advanced Techniques for Optimal Learning 67
VI. Conclusions 68
References 70
Chapter 2. Orthogonal Transformation Techniques in the Optimization of Feedforward Neural Network Systems 76
I. Introduction 76
II. Mathematical Background for the Transformations Used 78
III. Network-Size Optimization through Subset Selection 81
IV. Introduction to Illustrative Examples 84
V. Example 1: Modeling of the Mackey–Glass Series 85
VI. Example 2: Modeling of the Sunspot Series 88
VII. Example 3: Modeling of the Rocket Engine Testing Problem 94
VIII. Assessment of Convergence in Training Using Singular Value Decomposition 97
IX. Conclusions 99
Appendix A: Configuration of a Series with Nearly Repeating Periodicity for Singular Value Decomposition-Based Analysis 99
Appendix B: Singular Value Ratio Spectrum 100
References 100
Chapter 3. Sequential Constructive Techniques 104
I. Introduction 104
II. Problems in Training with Back Propagation 105
III. Constructive Training Methods 108
IV. Sequential Constructive Methods: General Structure 111
V. Sequential Constructive Methods: Specific Approaches 128
VI. Hamming Clustering Procedure 146
VII. Experimental Results 148
VIII. Conclusions 162
References 163
Chapter 4. Fast Backpropagation Training Using Optimal Learning Rate and Momentum 168
I. Introduction 168
II. Computation of Derivatives of Learning Parameters 171
III. Optimization of Dynamic Learning Rate 177
IV. Simultaneous Optimization of µ and a 181
V. Selection of the Descent Direction 183
VI. Simulation Results 184
VII. Conclusion 191
References 195
Chapter 5. Learning of Nonstationary Processes 198
I. Introduction 198
II. A Priori Limitations 200
III. Formalization of the Problem 201
IV. Transformation into an Unconstrained Minimization Problem 202
V. One-to-One Mapping D 205
VI. Learning with Minimal Degradation Algorithm 206
VII. Adaptation of Learning with Minimal Degradation for Radial Basis Function Units 209
VIII. Choosing the Coefficients of the Cost Function 211
IX. Implementation Details 213
X. Performance Measures 214
XI. Experimental Results 217
XII. Discussion 223
XIII. Conclusion 227
References 229
Chapter 6. Constraint Satisfaction Problems 232
I. Constraint Satisfaction Problems 232
II. Assessment Criteria for Constraint Satisfaction Techniques 236
III. Constraint Satisfaction Techniques 244
IV. Neural Networks for Constraint Satisfaction 250
V. Assessment 263
References 267
Chapter 7. Dominant Neuron Techniques 272
I. Introduction 272
II. Continuous Winner-Take-All Neural Networks 275
III. Iterative Winner-Take-All Neural Networks 279
IV. K-Winners-Take-All Neural Networks 291
V. Conclusions 296
References 297
Chapter 8. CMAC-Based Techniques for Adaptive Learning Control 300
I. Introduction 300
II. Neural Networks for Learning Control 301
III. Conventional Cerebellar Model Articulation Controller 307
IV. Advanced Cerebellar Model Articulation Controller-Based Techniques 313
V. Structure Composed of Small Cerebellar Model Articulation Controllers 321
VI. Conclusions 325
References 326
Chapter 9. Information Dynamics and Neural Techniques for Data Analysis 328
I. Introduction 328
II. Statistical Structure Extraction: Parametric Formulation by Unsupervised Neural Learning 330
III. Statistical Structure Extraction: Nonparametric Formulation 349
IV. Nonparametric Characterization of Dynamics: The Information Flow Concept 360
V. Conclusions 368
References 372
Chapter 10. Radial Basis Function Network Approximation and Learning in Task-Dependent Feedforward Control of Nonlinear Dynamical Systems 376
I. Introduction 376
II. Problem Statement 380
III. Radial Basis Function Approximation 389
IV. Learning Feedforward for a Given Task 396
V. On-Line Learning Update in Task-Dependent Feedforward 401
VI. Adaptive Learning of Task-Dependent Feedforward 405
VII. Conclusions 414
References 414
Index 418
Erratum 422