Multimedia Content Analysis (eBook)

Jens Rainer Ohm graduated in Electrical Engineering at TU Berlin. After his habilitations he became project coordinator at Fraunhofer Heinrich Hertz Institute, Berlin. Since 2000 Jens Ohm is Chair for Communications Engineering and Head of the Institute for Communications Engineering at Aachen University.

Preface 6
Table of Contents 7
1 Introduction 11
1.1 Context 11
1.2 Applications 13
2 Preprocessing 19
2.1 Nonlinear filters 21
2.1.1 Median filters and rank-order filters 21
2.1.2 Morphological filters 25
2.1.3 Polynomial filters 29
2.2 Amplitude-value transformations 30
2.2.1 Amplitude mapping characteristics 31
2.2.2 Probability distribution modification and equalization 32
2.3 Interpolation 34
2.3.1 Zero and first order interpolation basis functions 35
2.3.2 LTI systems as interpolators 37
2.3.3 Spline, Lagrangian and polynomial interpolation 38
2.3.4 Interpolation on 2D grids 43
2.4 Multi-resolution representation 47
2.5 Locally adaptive filters 53
2.5.1 Steerable smoothing filters 53
2.5.2 Iterative smoothing (diffusion filters) 55
2.6 Problems 58
3 Signal and Parameter Estimation 61
3.1 Expected values and probability description 61
3.2 Observation and degradation models 66
3.3 Estimation based on linear filters 67
3.3.1 Inverse filters 67
3.3.2 Wiener filters 68
3.4 Least-squares estimation 70
3.5 Singular value decomposition 75
3.6 ML and MAP estimation 77
3.7 Parameter estimation and fitting 79
3.8 Outlier rejection 81
3.9 Correspondence analysis 84
3.10 State modeling and estimation 87
3.10.1 Markov processes and random fields 87
3.10.2 Hidden Markov models 90
3.10.3 Kalman filters 91
3.10.4 Particle filters 94
3.11 Problems 94
4 Features of Multimedia Signals 97
4.1 Color 97
4.1.1 Color space transformations 98
4.1.2 Representation of color features 107
4.2 Texture 112
4.2.1 Texture analysis based on occurrence counts 114
4.2.2 Texture analysis based on statistical models 117
4.2.3 Spectral features of texture 120
4.2.4 Inhomogeneous texture analysis 124
4.3 Edge analysis 125
4.3.1 Edge detection by gradient operators 125
4.3.2 Edge characterization by second derivative 129
4.3.3 Edge finding and consistency analysis 131
4.3.4 Edge model fitting 134
4.3.5 Description and analysis of edge properties 135
4.4 Salient feature detection 137
4.5 Contour and shape analysis 142
4.5.1 Contour fitting 142
4.5.2 Contour description by orientation and curvature 146
4.5.3 Geometric features and binary shape features 150
4.5.4 Projection and geometric mapping 154
4.5.5 Moment analysis of region shapes 164
4.5.6 Region shape analysis by basis functions 168
4.6 Motion analysis 169
4.6.1 Projection of 3D motion into the image plane 169
4.6.2 Motion estimation by the optical flow principle 173
4.6.3 Motion estimation by matching 178
4.6.4 Estimation of non-translational motion parameters 188
4.6.5 Estimation of motion vector fields at object boundaries 190
4.7 Disparity and depth analysis 193
4.7.1 Coplanar stereoscopy 193
4.7.2 Epipolar geometry 196
4.7.3 Camera calibration 199
4.8 Audio signal features 203
4.8.1 Audio feature extraction on the timeline 204
4.8.2 Time domain features 206
4.8.3 Spectral domain features 212
4.8.4 Cepstral domain features 216
4.8.5 Harmonic features 217
4.8.6 Multi-channel features 222
4.8.7 Perceptual features 223
4.8.8 Semantic features 225
4.9 Problems 227
5 Feature Transforms and Classification 233
5.1 Feature value normalization and transforms 233
5.2 Distance metrics 248
5.3 Compressed representation of feature data 261
5.4 Feature-based comparison 263
5.5 Reliability 267
5.6 Classification methods 274
5.7 Belief, plausibility and evidence 299
5.8 Problems 302
6 Signal Decomposition 305
6.1 Spatial segmentation of pictures 306
6.1.1 Segmentation based on sample classification 307
6.1.2 Region-based methods 312
6.1.3 Contour-based methods 314
6.1.4 Segmentation based on ‘energy minimization’ 315
6.2 Segmentation of video signals 321
6.2.1 Key picture and shot transition detection 322
6.2.2 Segmentation by background differencing 323
6.2.3 Object tracking and spatio-temporal segmentation 324
6.2.4 Combined segmentation and motion estimation 330
6.3 3D surface and volume reconstruction 331
6.3.1 3D point cloud generation 332
6.3.2 3D surface reconstruction 333
6.3.3 3D volume reconstruction 335
6.3.4 Projection based description of 3D shapes 336
6.4 Decomposition of audio signals 339
6.4.1 Temporal segmentation of audio 339
6.4.2 Audio source separation 339
6.5 Problems 341
7 Signal Composition, Rendering and Presentation 343
7.1 Composition and mixing of multimedia signals 343
7.2 Mosaicking and stitching 348
7.3 Synthesis of picture content 351
7.4 Warping and morphing 355
7.5 Virtual view synthesis 357
7.6 Frame rate conversion 362
7.7 View-adaptive and stereoscopic rendering of image and video signals 366
7.8 Composition and rendering of audio signals 369
7.8.1 Sound effects 371
7.8.2 Spatial (room) features 374
A Fundamentals and definitions 377
A.1 Fundamentals of signal processing and signal analysis 377
A.2 Fundamentals of stochastic analysis and description 386
A.3 Vector and matrix algebra 395
B Symbols and Variables 401
C Glossary and Acronyms 406
D References 408
E Index 421