Computational Intelligence in Time Series Forecasting (eBook)

Series Editors’ Foreword 9
Preface 11
Contents 15
Part I Introduction 22
1 Computational Intelligence: An Introduction 24
1.1 Introduction 24
1.2 Soft Computing 24
1.3 Probabilistic Reasoning 25
1.4 Evolutionary Computation 27
1.5 Computational Intelligence 29
1.6 Hybrid Computational Technology 30
1.7 Application Areas 31
1.8 Applications in Industry 32
References 33
2 Traditional Problem Definition 38
2.1 Introduction to Time Series Analysis 38
2.2 Traditional Problem Definition 39
2.2.1 Characteristic Features 39
2.2.1.1 Stationarity 39
2.2.1.2 Linearity 41
2.2.1.3 Trend 41
2.2.1.4 Seasonality 42
2.2.1.5 Estimation and Elimination of Trend and Seasonality 42
2.3 Classification of Time Series 43
2.3.1 Linear Time Series 44
2.3.2 Nonlinear Time Series 44
2.3.3 Univariate Time Series 44
2.3.4 Multivariate Time Series 45
2.3.5 Chaotic Time Series 45
2.4 Time Series Analysis 46
2.4.1 Objectives of Analysis 46
2.4.2 Time Series Modelling 47
2.4.3 Time Series Models 47
2.5 Regressive Models 48
2.5.1 Autoregression Model 48
2.5.2 Moving-average Model 49
2.5.3 ARMA Model 49
2.5.4 ARIMA Model 50
2.5.5 CARMAX Model 53
2.5.6 Multivariate Time Series Models 54
2.5.7 Linear Time Series Models 56
2.5.8 Nonlinear Time Series Models 56
2.5.9 Chaotic Time Series Models 57
2.6 Time-domain Models 58
2.6.1 Transfer-function Models 58
2.6.2 State-space Models 59
2.7 Frequency-domain Models 60
2.8 Model Building 63
2.8.1 Model Identification 64
2.8.2 Model Estimation 66
2.8.3 Model Validation and Diagnostic Check 69
2.9 Forecasting Methods 70
2.9.1 Some Forecasting Issues 71
2.9.2 Forecasting Using Trend Analysis 72
2.9.3 Forecasting Using Regression Approaches 72
2.9.4 Forecasting Using the Box-Jenkins Method 74
2.9.5 Forecasting Using Smoothing 78
2.10 Application Examples 87
2.10.1 Forecasting Nonstationary Processes 87
2.10.2 Quality Prediction of Crude Oil 88
2.10.3 Production Monitoring and Failure Diagnosis 89
2.10.4 Tool Wear Monitoring 89
2.10.5 Minimum Variance Control 90
2.10.6 General Predictive Control 92
References 95
Selected Reading 95
Part II Basic Intelligent Computational Technologies 98
3 Neural Networks Approach 100
3.1 Introduction 100
3.2 Basic Network Architectures 101
3.3 Networks Used for Forecasting 105
3.3.1 Multilayer Perceptron Networks 105
3.3.2 Radial Basis Function Networks 106
3.3.3 Recurrent Networks 108
3.3.4 Counterpropagation Networks 113
3.3.5 Probabilistic Neural Networks 115
3.4 Network Training Methods 116
3.4.1 Accelerated Backpropagation Algorithm 120
3.5 Forecasting Methodology 124
3.5.1 Data Preparation for Forecasting 125
3.5.2 Determination of Network Architecture 127
3.5.3 Network Training Strategy 133
3.5.4 Training, Stopping and Evaluation 137
3.6 Forecasting Using Neural Networks 150
3.6.1 Neural Networks versus Traditional Forecasting 150
3.6.2. Combining Neural Networks and Traditional Approaches 152
3.6.3 Nonlinear Combination of Forecasts Using Neural Networks 153
3.6.4 Forecasting of Multivariate Time Series 157
References 158
Selected Reading 163
4 Fuzzy Logic Approach 164
4.1 Introduction 164
4.2 Fuzzy Sets and Membership Functions 165
4.3 Fuzzy Logic Systems 167
4.3.1 Mamdani Type of Fuzzy Logic Systems 169
4.3.2 Takagi-Sugeno Type of Fuzzy Logic Systems 169
4.3.3 Relational Fuzzy Logic System of Pedrycz 170
4.4 Inferencing the Fuzzy Logic System 171
4.4.1 Inferencing a Mamdani-type Fuzzy Model 171
4.4.2 Inferencing a Takagi-Sugeno type Fuzzy Model 174
4.4.3 Inferencing a (Pedrycz) Relational Fuzzy Model 175
4.5 Automated Generation of Fuzzy Rule Base 178
4.5.1 The Rules Generation Algorithm 178
4.5.2 Modifications Proposed for Automated Rules Generation 183
4.5.3 Estimation of Takagi-Sugeno Rule’s Consequent Parameters 187
4.6 Forecasting Time Series Using the Fuzzy Logic Approach 190
4.6.1 Forecasting Chaotic Time Series: An Example 190
4.7 Rules Generation by Clustering 194
4.7.1 Fuzzy Clustering Algorithms for Rules Generation 194
4.7.2 Fuzzy c-means Clustering 199
4.7.2.1 Fuzzy c-means Algorithm 200
4.7.2.1.1 Parameters of Fuzzy c-means Algorithm 201
4.7.3 Gustafson - Kessel Algorithm 204
4.7.3.1 Gustafson-Kessel Clustering Algorithm 205
4.7.3.1.1 Parameters of Gustafson-Kessel Algorithm 206
4.7.3.1.2 Interpretation of Cluster Covariance Matrix 206
4.7.4 Identification of Antecedent Parameters by Fuzzy Clustering 206
4.7.5 Modelling of a Nonlinear Plant 208
4.8 Fuzzy Model as Nonlinear Forecasts Combiner 211
4.9 Concluding Remarks 214
References 214
5 Evolutionary Computation 216
5.1 Introduction 216
5.1.1 The Mechanisms of Evolution 217
5.1.2 Evolutionary Algorithms 217
5.2 Genetic Algorithms 218
5.2.1 Genetic Operators 219
5.2.1.1 Selection 220
5.2.1.2 Reproduction 220
5.2.1.3 Mutation 220
5.2.1.4 Crossover 222
5.2.2 Auxiliary Genetic Operators 222
5.2.2.1 Fitness Windowing or Scaling 222
5.2.3 Real-coded Genetic Algorithms 224
5.2.3.1 Real Genetic Operators 225
5.2.3.1.1 Selection Function 225
5.2.3.1.2 Crossover Operators for Real-coded Genetic Algorithms 226
5.2.3.1.3 Mutation Operators 226
5.2.4 Forecasting Example 227
5.3 Genetic Programming 230
5.3.1 Initialization 231
5.3.2 Execution of Algorithm 232
5.3.3 Fitness Measure 232
5.3.4 Improved Genetic Versions 232
5.3.5 Applications 233
5.4 Evolutionary Strategies 233
5.4.1 Applications to Real-world Problems 234
5.5 Evolutionary Programming 235
5.5.1 Evolutionary Programming Mechanism 236
5.6 Differential Evolution 236
5.6.1 First Variant of Differential Evolution (DE1) 237
5.6.2 Second Variant of Differential Evolution (DE2) 239
References 239
Part III Hybrid Computational Technologies 242
6 Neuro-fuzzy Approach 244
6.1 Motivation for Technology Merging 244
6.2 Neuro-fuzzy Modelling 245
6.2.1 Fuzzy Neurons 248
6.2.1.1 AND Fuzzy Neuron 249
6.2.1.2 OR Fuzzy Neuron 250
6.3 Neuro-fuzzy System Selection for Forecasting 251
6.4 Takagi-Sugeno-type Neuro-fuzzy Network 253
6.4.1 Neural Network Representation of Fuzzy Logic Systems 254
6.4.2 Training Algorithm for Neuro-fuzzy Network 255
6.4.2.1 Backpropagation Training of Takagi-Sugeno-type Neuro-fuzzy Network 255
6.4.2.2 Improved Backpropagation Training Algorithm 259
6.4.2.3 Levenberg-Marquardt Training Algorithm 260
6.4.2.3.1 Computation of Jacobian Matrix 262
6.4.2.4 Adaptive Learning Rate and Oscillation Control 267
6.5 Comparison of Radial Basis Function Network and Neurofuzzy Network 268
6.6 Comparison of Neural Network and Neuro-fuzzy Network Training 269
6.7 Modelling and Identification of Nonlinear Dynamics 270
6.7.1 Short-term Forecasting of Electrical Load 270
6.7.2 Prediction of Chaotic Time Series 274
6.7.3 Modelling and Prediction of Wang Data 279
6.8 Other Engineering Application Examples 285
6.8.1 Application of Neuro-fuzzy Modelling to Material Property Prediction 286
6.8.2 Correction of Pyrometer Reading 287
6.8.3 Application for Tool Wear Monitoring 289
6.9 Concluding Remarks 291
References 292
7 Transparent Fuzzy/Neuro-fuzzy Modelling 296
7.1 Introduction 296
7.2 Model Transparency and Compactness 297
7.3 Fuzzy Modelling with Enhanced Transparency 298
7.3.1 Redundancy in Numerical Data-driven Modelling 298
7.3.2 Compact and Transparent Modelling Scheme 300
7.4 Similarity Between Fuzzy Sets 302
7.4.1 Similarity Measure 303
7.4.2 Similarity-based Rule Base Simplification 303
7.5 Simplification of Rule Base 306
7.5.1 Merging Similar Fuzzy Sets 308
7.5.2 Removing Irrelevant Fuzzy Sets 310
7.5.3 Removing Redundant Inputs 311
7.5.4 Merging Rules 311
7.6 Rule Base Simplification Algorithms 312
7.6.1 Iterative Merging 313
7.6.2 Similarity Relations 315
7.7 Model Competitive Issues: Accuracy versus Complexity 317
7.8 Application Examples 320
7.9 Concluding Remarks 323
References 323
8 Evolving Neural and Fuzzy Systems 326
8.1 Introduction 326
8.1.1 Evolving Neural Networks 326
8.1.1.1 Evolving Connection Weights 327
8.1.1.2 Evolving the Network Architecture 330
8.1.1.3 Evolving the Pure Network Architecture 331
8.1.1.4 Evolving Complete Network 332
8.1.1.5 Evolving the Activation Function 333
8.1.1.6 Application Examples 334
8.1.2 Evolving Fuzzy Logic Systems 334
References 338
9 Adaptive Genetic Algorithms 342
9.1 Introduction 342
9.2 Genetic Algorithms Parameters to Be Adapted 343
9.3 Probabilistic Control of Genetic Algorithms Parameters 344
9.4 Adaptation of Population Size 348
9.5 Fuzzy Logic Controlled Genetic Algoithms 350
9.6 Concluding Remarks 351
References 351
Part IV Recent Developments 354
10 State of the Art and Development Trend 355
10.1 Introduction 355
10.2 Support Vector Machines 358
10.2.1 Data-dependent Representation 363
10.2.2 Machine Implementation 364
10.2.3 Applications 365
10.3 Wavelet Networks 366
10.3.1 Wavelet Theory 366
10.3.2 Wavelet Neural Networks 367
10.3.3 Applications 370
10.4 Fractally Configured Neural Networks 371
10.5 Fuzzy Clustering 373
10.5.1 Fuzzy Clustering Using Kohonen Networks 374
10.5.2 Entropy-based Fuzzy Clustering 376
10.5.2.1 Entropy Measure for Cluster Estimation 377
10.5.2.1.1 The Entropy Measure 377
10.5.2.2 Fuzzy Clustering Based on Entropy Measure 379
10.5.2.3 Fuzzy Model Identification Using Entropy-based Fuzzy Clustering 380
References 381
Index 384

6 Neuro-fuzzy Approach (p.223)

6.1 Motivation for Technology Merging

Contemporary intelligent technologies have various characteristic features that can be used to implement systems that mimic the behaviour of human beings. For example, expert systems are capable of reasoning about the facts and situations using the rules out of a specific domain, etc. The outstanding feature of neural networks is their capability of learning, which can help in building artificial systems for pattern recognition, classification, etc. Fuzzy logic systems, again, are capable of interpreting the imprecise data that can be helpful in making possible decisions. On the other hand, genetic algorithms provide implementation of random, parallel solution search procedures within a large search space.

Therefore, in fact, the complementary features of individual categories of intelligent technologies make them ideal for isolated use in solving some specific problems, but not well suited for solving other kinds of intelligent problem. For example, the black-box modelling approach through neural networks is evidently well suited for process modelling or for intelligent control, but less suitable for decision making. On the other hand, the fuzzy logic systems can easily handle imprecise data, and explain their decisions in the context of the available facts in linguistic form; however, they cannot automatically acquire the linguistic rules to make those decisions. Such capabilities and restrictions of individual intelligent technologies have actually been a central driving force behind their fusion for creation of hybrid intelligent systems capable of solving many complex problems.

The permanent growing interest in intelligent technology merging, particularly in merging of neural and fuzzy technology, the two technologies that complement each other (Bezdek, 1993), to create neuro-fuzzy or fuzzy-neural structures, has largely extended the capabilities of both technologies in hybrid intelligent systems. The advantages of neural networks in learning and adaptation and those of fuzzy logic systems in dealing with the issues of human-like reasoning on a linguistic level, transparency and interpretability of the generated model, and handling of uncertain or imprecise data, enable building of higher level intelligent systems. The synergism of integrating neural networks with fuzzy logic technology into a hybrid functional system with low-level learning and high-level reasoning transforms the burden of the tedious design problems of the fuzzy logic decision systems to the learning of connectionist neural networks. In this way the approximation capability and the overall performance of the resulting system are enhanced.

A number of different schemes and architectures of this hybrid system have been proposed, such as fuzzy-logic-based neurons (Pedrycz, 1995), fuzzy neurons (Gupta, 1994), neural networks with fuzzy weights (Buckley and Hayashi, 1994), neuro-fuzzy adaptive models (Brown and Harris, 1994), etc. The proposed architectures have been successful in solving various engineering and real-world problems, such as in applications like system identification and modelling, process control, systems diagnosis, cognitive simulation, classification, pattern recognition, image processing, engineering design, financial trading, signal processing, time series prediction and forecasting, etc.