Natural Image Statistics (eBook)

Preface 6
Aims and Scope 6
Targeted Audience and Prerequisites 6
Structure of the Book and Its Use as a Textbook 7
Referencing and Exercises 8
Code for Reproducing Experiments 8
Acknowledgements 8
Contents 9
Abbreviations 19
Introduction 20
What this Book Is All About 20
What Is Vision? 21
The Magic of Your Visual System 22
Importance of Prior Information 26
Ecological Adaptation Provides Prior Information 26
Generative Models and Latent Quantities 27
Projection onto the Retina Loses Information 28
Bayesian Inference and Priors 28
Natural Images 29
The Image Space 29
Definition of Natural Images 30
Redundancy and Information 32
Information Theory and Image Coding 32
Redundancy Reduction and Neural Coding 33
Statistical Modeling of the Visual System 34
Connecting Information Theory and Bayesian Inference 34
Normative vs. Descriptive Modeling of Visual System 34
Toward Predictive Theoretical Neuroscience 35
Features and Statistical Models of Natural Images 36
Image Representations and Features 36
Statistics of Features 37
From Features to Statistical Models 38
The Statistical-Ecological Approach Recapitulated 39
References 40
Background 41
Linear Filters and Frequency Analysis 42
Linear Filtering 42
Definition 42
Impulse Response and Convolution 45
Frequency-Based Representation 46
Motivation 46
Representation in One and Two Dimensions 46
Note on Terminology 50
Frequency-Based Representation and Linear Filtering 51
Computation and Mathematical Details 54
Representation Using Linear Basis 55
Basic Idea 55
Frequency-Based Representation as a Basis 57
Space-Frequency Analysis 58
Introduction 58
Space-Frequency Analysis and Gabor Filters 60
Spatial Localization vs. Spectral Accuracy 63
References 65
Exercises 65
Mathematical Exercises 65
Computer Assignments 65
Outline of the Visual System 67
Neurons and Firing Rates 67
Neurons 67
Axons 67
Action Potentials 67
Signal Reception and Processing 67
Firing Rate 69
Computation by the Neuron 69
From the Eye to the Cortex 69
Linear Models of Visual Neurons 70
Responses to Visual Stimulation 70
Simple Cells and Linear Models 72
Gabor Models and Selectivities of Simple Cells 73
Frequency Channels 74
Non-linear Models of Visual Neurons 75
Non-linearities in Simple-Cell Responses 75
Complex Cells and Energy Models 77
Interactions between Visual Neurons 78
Topographic Organization 80
Processing after the Primary Visual Cortex 80
References 81
Exercises 81
Mathematical Exercises 81
Computer Assignments 82
Multivariate Probability and Statistics 83
Natural Images Patches as Random Vectors 83
Multivariate Probability Distributions 84
Notation and Motivation 84
Probability Density Function 85
Marginal and Joint Probabilities 86
Conditional Probabilities 89
Generalization to Many Dimensions 90
Discrete-Valued Variables 91
Independence 91
Expectation and Covariance 93
Expectation 93
Variance and Covariance in One Dimension 94
Covariance Matrix 94
Independence and Covariances 95
Bayesian Inference 97
Motivating Example 97
Bayes' Rule 99
Non-informative Priors 99
Bayesian Inference as an Incremental Learning Process 100
Parameter Estimation and Likelihood 102
Models, Estimation, and Samples 102
Maximum Likelihood and Maximum a Posteriori 103
Prior and Large Samples 105
References 105
Exercises 105
Mathematical Exercises 105
Computer Assignments 106
Statistics of Linear Features 107
Principal Components and Whitening 108
DC Component or Mean Grey-Scale Value 108
Principal Component Analysis 109
A Basic Dependency of Pixels in Natural Images 109
Learning One Feature by Maximization of Variance 111
Principal Component as Variance-Maximizing Feature 111
Learning One Feature from Natural Images 113
Learning Many Features by PCA 113
Defining Many Principal Components 113
Definition 113
Critique of the Definition 114
All Principal Components of Natural Images 115
Computational Implementation of PCA 116
The Implications of Translation-Invariance 117
PCA as a Preprocessing Tool 118
Dimension Reduction by PCA 118
Whitening by PCA 119
Whitening as Normalized Decorrelation 119
Whitening Transformations and Orthogonality 120
Anti-aliasing by PCA 121
Oblique Gratings Can Have Higher Frequencies 121
Highest Frequencies Can Have only Two Different Phases 122
Dimension Selection to Avoid Aliasing 123
Canonical Preprocessing Used in This Book 124
Notation 124
Gaussianity as the Basis for PCA 124
The Probability Model Related to PCA 124
PCA as a Generative Model 125
Image Synthesis Results 126
Power Spectrum of Natural Images 126
The 1/f Fourier Amplitude or 1/f2 Power Spectrum 126
Connection between Power Spectrum and Covariances 128
Relative Importance of Amplitude and Phase 129
Anisotropy in Natural Images 130
Mathematics of Principal Component Analysis* 131
Eigenvalue Decomposition of the Covariance Matrix 132
Eigenvectors and Translation-Invariance 134
Decorrelation Models of Retina and LGN * 135
Whitening and Redundancy Reduction 135
Patch-Based Decorrelation 136
Matrix Square Root 138
Symmetric Whitening Matrix 139
Application to Natural Images 139
Filter-Based Decorrelation 139
Concluding Remarks and References 143
Exercises 144
Mathematical Exercises 144
Computer Assignments 145
Sparse Coding and Simple Cells 146
Definition of Sparseness 146
Learning One Feature by Maximization of Sparseness 147
Measuring Sparseness: General Framework 148
Measuring Sparseness Using Kurtosis 148
Measuring Sparseness Using Convex Functions of Square 149
Convexity and Sparseness 149
An Example Distribution 150
Suitable Convex Functions 151
Summary 153
The Case of Canonically Preprocessed Data 153
One Feature Learned from Natural Images 153
Learning Many Features by Maximization of Sparseness 154
Deflationary Decorrelation 155
Symmetric Decorrelation 156
Sparseness of Feature vs. Sparseness of Representation 156
Sparse Coding Features for Natural Images 158
Full Set of Features 158
Analysis of Tuning Properties 159
How Is Sparseness Useful? 162
Bayesian Modeling 162
Neural Modeling 163
Metabolic Economy 163
Concluding Remarks and References 163
Exercises 164
Mathematical Exercises 164
Computer Assignments 165
Independent Component Analysis 166
Limitations of the Sparse Coding Approach 166
Definition of ICA 167
Independence 167
Generative Model 167
Model for Preprocessed Data 169
Insufficiency of Second-Order Information 169
Why Whitening Does Not Find Independent Components 169
Why Components Have to Be Non-Gaussian 171
Whitened Gaussian pdf is Spherically Symmetric 171
Uncorrelated Gaussian Variables Are Independent 172
The Probability Density Defined by ICA 173
Short Digression to Probability Theory 173
Maximum Likelihood Estimation in ICA 174
Results on Natural Images 175
Estimation of Features 175
Image Synthesis Using ICA 175
Connection to Maximization of Sparseness 176
Likelihood as a Measure of Sparseness 176
Optimal Sparseness Measures 178
Why Are Independent Components Sparse? 181
Different Forms of Non-Gaussianity 182
Non-Gaussianity in Natural Images 182
Why Is Sparseness Dominant? 183
General ICA as Maximization of Non-Gaussianity 183
Central Limit Theorem 184
``Non-Gaussian Is Independent'' 184
Sparse Coding as a Special Case of ICA 185
Receptive Fields vs. Feature Vectors 186
Problem of Inversion of Preprocessing 187
Frequency Channels and ICA 188
Concluding Remarks and References 188
Exercises 189
Mathematical Exercises 189
Computer Assignments 189
Information-Theoretic Interpretations 191
Basic Motivation for Information Theory 191
Compression 191
Transmission 192
Entropy as a Measure of Uncertainty 193
Definition of Entropy 193
Entropy as Minimum Coding Length 194
Redundancy 195
Differential Entropy 196
Maximum Entropy 197
Mutual Information 198
Minimum Entropy Coding of Natural Images 199
Image Compression and Sparse Coding 199
Mutual Information and Sparse Coding 201
Minimum Entropy Coding in the Cortex 201
Information Transmission in the Nervous System 202
Definition of Information Flow and Infomax 202
Basic Infomax with Linear Neurons 202
Infomax with Non-linear Neurons 203
Definition of Model 203
Infomax with Non-constant Noise Variance 204
Problems with Non-linear Neuron Model 204
Using Neurons with Non-constant Variance 205
Caveats in Application of Information Theory 207
Concluding Remarks and References 209
Exercises 209
Mathematical Exercises 209
Computer Assignments 210
Nonlinear Features and Dependency of Linear Features 211
Energy Correlation of Linear Features and Normalization 212
Why Estimated Independent Components Are Not Independent 212
Estimates vs. Theoretical Components 212
Counting the Number of Free Parameters 213
Correlations of Squares of Components in Natural Images 214
Modeling Using a Variance Variable 214
Normalization of Variance and Contrast Gain Control 216
Physical and Neurophysiological Interpretations 218
Canceling the Effect of Changing Lighting Conditions 218
Uniform Surfaces 219
Saturation of Cell Responses 219
Effect of Normalization on ICA 220
Concluding Remarks and References 223
Exercises 224
Mathematical Exercises 224
Computer Assignments 224
Energy Detectors and Complex Cells 225
Subspace Model of Invariant Features 225
Why Linear Features Are Insufficient 225
Subspaces or Groups of Linear Features 225
Energy Model of Feature Detection 226
Canonically Preprocessed Data 228
Maximizing Sparseness in the Energy Model 228
Definition of Sparseness of Output 228
One Feature Learned from Natural Images 229
Model of Independent Subspace Analysis 231
Dependency as Energy Correlation 232
Why Energy Correlations Are Related to Sparseness 232
Spherical Symmetry and Changing Variance 233
Correlation of Squares and Convexity of Non-linearity 234
Connection to Contrast Gain Control 235
ISA as a Non-linear Version of ICA 236
Results on Natural Images 237
Emergence of Invariance to Phase 237
Data and Preprocessing 237
Features Obtained 237
Analysis of Tuning and Invariance 238
Image Synthesis Results 242
The Importance of Being Invariant 242
Grouping of Dependencies 244
Superiority of the Model over ICA 244
Analysis of Convexity and Energy Correlations* 246
Variance Variable Model Gives Convex h 246
Convex h Typically Implies Positive Energy Correlations 247
Concluding Remarks and References 248
Exercises 248
Mathematical Exercises 248
Computer Assignments 249
Energy Correlations and Topographic Organization 250
Topography in the Cortex 250
Modeling Topography by Statistical Dependence 251
Topographic Grid 251
Defining Topography by Statistical Dependencies 251
Definition of Topographic ICA 253
Connection to Independent Subspaces and Invariant Features 254
Utility of Topography 255
Estimation of Topographic ICA 256
Topographic ICA of Natural Images 257
Emergence of V1-like Topography 257
Data and Preprocessing 257
Results and Analysis 258
Image Synthesis Results and Sketch of Generative Model 262
Comparison with Other Models 264
Learning Both Layers in a Two-Layer Model * 264
Generative vs. Energy-Based Approach 264
Definition of the Generative Model 265
Basic Properties of the Generative Model 266
The Components si Are Uncorrelated 266
The Components si Are Sparse 267
Topographic Organization Can Be Modeled 267
Independent Subspaces Are a Special Case 267
Estimation of the Generative Model 267
Integrating Out 267
Approximating the Likelihood 268
Difficulty of Estimating the Model 270
Energy-Based Two-Layer Models 270
Concluding Remarks and References 271
Dependencies of Energy Detectors: Beyond V1 273
Predictive Modeling of Extrastriate Cortex 273
Simulation of V1 by a Fixed Two-Layer Model 273
Learning the Third Layer by Another ICA Model 275
Methods for Analyzing Higher-Order Components 276
Results on Natural Images 278
Emergence of Collinear Contour Units 278
Emergence of Pooling over Frequencies 279
Discussion of Results 283
Why Coding of Contours? 283
Frequency Channels and Edges 284
Toward Predictive Modeling 284
References and Related Work 285
Conclusion 286
Overcomplete and Non-negative Models 287
Overcomplete Bases 287
Motivation 287
Definition of Generative Model 288
Nonlinear Computation of the Basis Coefficients 289
Estimation of the Basis 291
Approach Using Energy-Based Models 292
Results on Natural Images 295
Markov Random Field Models * 295
Non-negative Models 298
Motivation 298
Definition 298
Adding Sparseness Constraints 300
Conclusion 303
Lateral Interactions and Feedback 304
Feedback as Bayesian Inference 304
Example: Contour Integrator Units 305
Thresholding (Shrinkage) of a Sparse Code 307
Decoupling of Estimates 307
Sparseness Leads to Shrinkage 309
Categorization and Top-Down Feedback 311
Overcomplete Basis and End-stopping 311
Predictive Coding 313
Conclusion 314
Time, Color, and Stereo 316
Color and Stereo Images 317
Color Image Experiments 317
Choice of Data 317
Preprocessing and PCA 318
ICA Results and Discussion 321
Stereo Image Experiments 323
Choice of Data 323
Preprocessing and PCA 324
ICA Results and Discussion 325
Further References 330
Color and Stereo Images 330
Other Modalities, Including Audition 331
Conclusion 331
Temporal Sequences of Natural Images 332
Natural Image Sequences and Spatiotemporal Filtering 332
Temporal and Spatiotemporal Receptive Fields 333
Second-Order Statistics 335
Average Spatiotemporal Power Spectrum 335
The Temporally Decorrelating Filter 339
Sparse Coding and ICA of Natural Image Sequences 340
Temporal Coherence in Spatial Features 343
Temporal Coherence and Invariant Representation 343
Quantifying Temporal Coherence 344
Interpretation as Generative Model * 345
Experiments on Natural Image Sequences 346
Data and Preprocessing 346
Results and Analysis 347
Why Gabor-Like Features Maximize Temporal Coherence 348
Control Experiments 351
Spatiotemporal Energy Correlations in Linear Features 352
Definition of the Model 352
Estimation of the Model 354
Experiments on Natural Images 355
Intuitive Explanation of Results 357
Unifying Model of Spatiotemporal Dependencies 359
Features with Minimal Average Temporal Change 361
Slow Feature Analysis 361
Motivation and History 361
SFA in a Linear Neuron Model 363
Quadratic Slow Feature Analysis 364
Sparse Slow Feature Analysis 366
Conclusion 368
Conclusion 369
Conclusion and Future Prospects 370
Short Overview 370
Open, or Frequently Asked, Questions 372
What Is the Real Learning Principle in the Brain? 372
Nature vs. Nurture 373
How to Model Whole Images 374
Are There Clear-Cut Cell Types? 374
How Far Can We Go? 376
Other Mathematical Models of Images 376
Scaling Laws 377
Wavelet Theory 377
Physically Inspired Models 378
Future Work 379
Appendix: Supplementary Mathematical Tools 380
Optimization Theory and Algorithms 381
Levels of Modeling 381
Gradient Method 382
Definition and Meaning of Gradient 382
Gradient and Optimization 384
Optimization of Function of Matrix 385
Constrained Optimization 385
Projecting Back to Constraint Set 386
Projection of the Gradient 387
Global and Local Maxima 387
Hebb's Rule and Gradient Methods 388
Hebb's Rule 388
Hebb's Rule and Optimization 389
Stochastic Gradient Methods 390
Role of the Hebbian Non-linearity 391
Receptive Fields vs. Synaptic Strengths 392
The Problem of Feedback 392
Optimization in Topographic ICA * 393
Beyond Basic Gradient Methods * 394
Newton's Method 395
Conjugate Gradient Methods 397
FastICA, a Fixed-Point Algorithm for ICA 398
The FastICA Algorithm 398
Choice of the FastICA Non-linearity 399
Mathematics of FastICA * 399
Derivation of the Fixed-Point Iteration 399
Connection to Gradient Methods 400
Crash Course on Linear Algebra 402
Vectors 402
Linear Transformations 403
Matrices 404
Determinant 405
Inverse 405
Basis Representations 406
Orthogonality 407
Pseudo-Inverse * 408
The Discrete Fourier Transform 409
Linear Shift-Invariant Systems 409
One-Dimensional Discrete Fourier Transform 410
Euler's Formula 410
Representation in Complex Exponentials 410
The Discrete Fourier Transform and Its Inverse 413
Negative Frequencies and Periodicity in the DFT 415
Periodicity of the IDFT and the Convolution Theorem 416
Real- and Complex-Valued DFT Coefficients 417
The Sinusoidal Representation from the DFT 418
The Basis is Orthogonal, Perhaps up to Scaling 418
DFT Can Be Computed by the Fast Fourier Transformation 419
Two- and Three-Dimensional Discrete Fourier Transforms 419
Estimation of Non-normalized Statistical Models 421
Non-normalized Statistical Models 421
Estimation by Score Matching 422
Example 1: Multivariate Gaussian Density 424
Example 2: Estimation of Basic ICA Model 426
Example 3: Estimation of an Overcomplete ICA Model 427
Conclusion 427
References 429
Index 443