3D Computer Vision (eBook)

Christian Wöhler is Professor of Image Analysis at the Department of Electrical Engineering and Information Technology of TU Dortmund, Germany. His scientific interests are in the domains of computer vision, photogrammetry, remote sensing, and pattern classification, with applications in various fields including machine vision, robotics, advanced driver assistance systems, and planetary science.

Preface 6
Acknowledgements 9
Contents 11
Part I: Methods of 3D Computer Vision 16
Chapter 1: Triangulation-Based Approaches to Three-Dimensional Scene Reconstruction 17
1.1 The Pinhole Model 17
1.2 Geometric Aspects of Stereo Image Analysis 20
1.2.1 Euclidean Formulation of Stereo Image Analysis 20
1.2.2 Stereo Image Analysis in Terms of Projective Geometry 22
1.2.2.1 De nition of Coordinates and Camera Properties 22
1.2.2.2 The Essential Matrix 23
1.2.2.3 The Fundamental Matrix 24
1.2.2.4 Projective Reconstruction of the Scene 25
1.3 The Bundle Adjustment Approach 28
1.4 Geometric Calibration of Single and Multiple Cameras 29
1.4.1 Methods for Intrinsic Camera Calibration 29
1.4.2 The Direct Linear Transform (DLT) Method 30
1.4.3 The Camera Calibration Method by Tsai (1987) 33
1.4.4 The Camera Calibration Method by Zhang (1999a) 34
1.4.5 The Camera Calibration Toolbox by Bouguet (2007) 37
1.4.6 Self-calibration of Camera Systems from Multiple Views of a Static Scene 37
1.4.6.1 Projective Reconstruction: Determination of the Fundamental Matrix 37
1.4.6.2 Metric Self-calibration 40
The Basic Equations for Self-calibration and Methods for Their Solution 41
1.4.6.3 Self-calibration Based on Vanishing Points 43
1.4.7 Semi-automatic Calibration of Multiocular Camera Systems 44
1.4.7.1 The Calibration Rig 45
1.4.7.2 Existing Algorithms for Extracting the Calibration Rig 46
1.4.7.3 A Graph-Based Rig Extraction Algorithm 47
Outline of the Rig Finding Algorithm 47
De nition of the Graph 49
Extraction of Corner Candidates 49
Candidate Filter and Graph Construction 50
Non-bidirectional Edge Elimination 50
Edge Circle Filter 51
Edge Length Filter 51
Corner Enumeration 52
Notch Direction Detector 52
Rig Direction 52
1.4.7.4 Discussion 52
1.4.8 Accurate Localisation of Chequerboard Corners 53
1.4.8.1 Different Types of Calibration Targets and Their Localisationin Images 54
1.4.8.2 A Model-Based Method for Chequerboard Corner Localisation 57
1.4.8.3 Experimental Evaluation 60
1.4.8.4 Discussion 65
1.5 Stereo Image Analysis in Standard Geometry 66
1.5.1 Image Recti cation According to Standard Geometry 66
1.5.2 The Determination of Corresponding Points 69
1.5.2.1 Correlation-Based Blockmatching Stereo Vision Algorithms 70
1.5.2.2 Feature-Based Stereo Vision Algorithms 71
General Overview 71
A Contour-Based Stereo Vision Algorithm 73
1.5.2.3 Dense Stereo Vision Algorithms 79
1.5.2.4 Model-Based Stereo Vision Algorithms 80
1.5.2.5 Spacetime Stereo Vision and Scene Flow Algorithms 81
General Overview 81
Local Intensity Modelling 83
1.6 Resolving Stereo Matching Errors due to Repetitive Structures Using Model Information 88
1.6.1 Plane Model 90
1.6.1.1 Detection and Characterisation of Repetitive Structures 90
1.6.1.2 Determination of Model Parameters 91
1.6.2 Multiple-plane Hand-Arm Model 93
1.6.3 Decision Feedback 93
1.6.4 Experimental Evaluation 95
1.6.5 Discussion 101
Chapter 2: Three-Dimensional Pose Estimation and Segmentation Methods 102
2.1 Pose Estimation of Rigid Objects 102
2.1.1 General Overview 103
2.1.1.1 Pose Estimation Methods Based on Explicit Feature Matching 103
2.1.1.2 Appearance-Based Pose Estimation Methods 104
Methods Based on Monocular Image Data 105
Methods Based on Multiocular Image Data 106
2.1.2 Template-Based Pose Estimation 107
2.2 Pose Estimation of Non-rigid and Articulated Objects 110
2.2.1 General Overview 110
2.2.1.1 Non-rigid Objects 110
2.2.1.2 Articulated Objects 112
2.2.2 Three-Dimensional Active Contours 117
2.2.2.1 Active Contours 117
2.2.2.2 Three-Dimensional Multiple-View Active Contours 118
2.2.2.3 Experimental Results on Synthetic Image Data 120
2.2.3 Three-Dimensional Spatio-Temporal Curve Fitting 122
2.2.3.1 Modelling the Hand-Forearm Limb 122
2.2.3.2 Principles and Extensions of the CCD Algorithm 124
Step 1: Learning Local Probability Distributions 125
Step 2: Re nement of the Estimate (MAP Estimation) 127
2.2.3.3 The Multiocular Extension of the CCD Algorithm 129
Step 1: Extraction and Projection of the Three-Dimensional Model 129
Step 2: Learning Local Probability Distributions from all Nc Images 129
Step 3: Re nement of the Estimate (MAP Estimation) 129
2.2.3.4 The Shape Flow Algorithm 130
Step 1: Projection of the Spatio-Temporal Three-Dimensional Contour Model 131
Step 2: Learn Local Probability Distributions from all Nc Images 132
Step 3: Re ne the Estimate (MAP Estimation) 132
2.2.3.5 Veri cation and Recovery of the Pose Estimation Results 133
Pose Veri cation 133
Pose Recovery on Loss of Object 134
2.3 Point Cloud Segmentation Approaches 135
2.3.1 General Overview 136
2.3.1.1 The k-Means Clustering Algorithm 136
2.3.1.2 Agglomerative Clustering 136
2.3.1.3 Mean-Shift Clustering 137
2.3.1.4 Graph Cut and Spectral Clustering 137
2.3.1.5 The ICP Algorithm 138
2.3.1.6 Photogrammetric Approaches 139
2.3.2 Mean-Shift Tracking of Human Body Parts 139
2.3.2.1 Clustering and Object Detection 139
2.3.2.2 Target Model 140
2.3.2.3 Image-Based Mean-Shift 141
2.3.2.4 Point Cloud-Based Mean-Shift 141
2.3.3 Segmentation and Spatio-Temporal Pose Estimation 142
2.3.3.1 Scene Clustering and Model-Based Pose Estimation 143
2.3.3.2 Estimation of the Temporal Pose Derivatives 144
2.3.4 Object Detection and Tracking in Point Clouds 147
2.3.4.1 Motion-Attributed Point Cloud 147
2.3.4.2 Over-Segmentation for Motion-Attributed Clusters 148
2.3.4.3 Generation and Tracking of Object Hypotheses 149
Chapter 3: Intensity-Based and Polarisation-Based Approaches to Three-Dimensional Scene Reconstruction 151
3.1 Shape from Shadow 151
3.1.1 Extraction of Shadows from Image Pairs 152
3.1.2 Shadow-Based Surface Reconstruction from Dense Sets of Images 154
3.2 Shape from Shading 155
3.2.1 The Bidirectional Re ectance Distribution Function (BRDF) 156
3.2.2 Determination of Surface Gradients 160
3.2.2.1 Photoclinometry 160
3.2.2.2 Single-Image Approaches with Regularisation Constraints 162
3.2.3 Reconstruction of Height from Gradients 165
3.2.4 Surface Reconstruction Based on Partial Differential Equations 167
3.3 Photometric Stereo 170
3.3.1 Photometric Stereo: Principle and Extensions 170
3.3.2 Photometric Stereo Approaches Based on Ratio Images 172
3.3.2.1 Ratio-Based Photoclinometry of Surfaces with Non-uniform Albedo 173
3.3.2.2 Ratio-Based Variational Photometric Stereo Approach 174
3.4 Shape from Polarisation 175
3.4.1 Surface Orientation from Dielectric Polarisation Models 175
3.4.2 Determination of Polarimetric Properties of Rough Metallic Surfaces for Three-Dimensional Reconstruction Purposes 178
Chapter 4: Point Spread Function-Based Approaches to Three-Dimensional Scene Reconstruction 183
4.1 The Point Spread Function 183
4.2 Reconstruction of Depth from Defocus 184
4.2.1 Basic Principles 184
4.2.2 Determination of Small Depth Differences 188
4.2.3 Determination of Absolute Depth Across Broad Ranges 191
4.2.3.1 De nition of the Depth-Defocus Function 192
4.2.3.2 Calibration of the Depth-Defocus Function 192
Stationary Camera 192
Moving Camera 193
4.2.3.3 Determination of the Depth Map 194
Stationary Camera 194
Moving Camera 195
4.2.3.4 Estimation of the Useful Depth Range 197
4.3 Reconstruction of Depth from Focus 198
Chapter 5: Integrated Frameworks for Three-Dimensional Scene Reconstruction 200
5.1 Monocular Three-Dimensional Scene Reconstruction at Absolute Scale 201
5.1.1 Combining Motion, Structure, and Defocus 202
5.1.2 Online Version of the Algorithm 203
5.1.3 Experimental Evaluation Based on Tabletop Scenes 203
5.1.3.1 Evaluation of the Of ine Algorithm 204
Cuboid Sequence 207
Bottle Sequence 207
Lava Stone Sequence 208
5.1.3.2 Evaluation of the Online Algorithm 209
5.1.3.3 Random Errors vs. Systematic Deviations 210
5.1.4 Discussion 212
5.2 Self-consistent Combination of Shadow and Shading Features 213
5.2.1 Selection of a Shape from Shading Solution Based on Shadow Analysis 214
5.2.2 Accounting for the Detailed Shadow Structure in the Shape from Shading Formalism 217
5.2.3 Initialisation of the Shape from Shading Algorithm Based on Shadow Analysis 218
5.2.4 Experimental Evaluation Based on Synthetic Data 220
5.2.5 Discussion 221
5.3 Shape from Photopolarimetric Re ectance and Depth 222
5.3.1 Shape from Photopolarimetric Re ectance 224
5.3.1.1 Global Optimisation Scheme 225
5.3.1.2 Local Optimisation Scheme 227
5.3.2 Estimation of the Surface Albedo 228
5.3.3 Integration of Depth Information 229
5.3.3.1 Fusion of SfPR with Depth from Defocus 230
5.3.3.2 Integration of Accurate but Sparse Depth Information 231
5.3.4 Experimental Evaluation Based on Synthetic Data 233
5.3.5 Discussion 238
5.4 Stereo Image Analysis of Non-Lambertian Surfaces 239
5.4.1 Iterative Scheme for Disparity Estimation 242
5.4.2 Qualitative Behaviour of the Specular Stereo Algorithm 245
5.5 Combination of Shape from Shading and Active Range Scanning Data 246
5.6 Three-Dimensional Pose Estimation Based on Combinations of Monocular Cues 249
5.6.1 Photometric and Polarimetric Information 250
5.6.2 Edge Information 251
5.6.3 Defocus Information 252
5.6.4 Total Error Optimisation 252
5.6.5 Experimental Evaluation Based on a Simple Real-World Object 253
5.6.6 Discussion 255
Part II: Application Scenarios 256
Chapter 6: Applications to Industrial Quality Inspection 257
6.1 Inspection of Rigid Parts 258
6.1.1 Object Detection by Pose Estimation 258
Comparison with Other Pose Estimation Methods 260
6.1.2 Pose Re nement 262
Comparison with Other Pose Re nement Methods 266
6.2 Inspection of Non-rigid Parts 267
6.3 Inspection of Metallic Surfaces 270
6.3.1 Inspection Based on Integration of Shadow and Shading Features 271
6.3.2 Inspection of Surfaces with Non-uniform Albedo 271
6.3.3 Inspection Based on SfPR and SfPRD 273
6.3.3.1 Results Obtained with the SfPR Technique 274
6.3.3.2 Results Obtained with the SfPRD Technique 277
6.3.4 Inspection Based on Specular Stereo 280
6.3.4.1 Qualitative Discussion of the Three-Dimensional Reconstruction Results 280
6.3.4.2 Comparison to Ground Truth Data 282
6.3.4.3 Self-consistency Measures for Three-Dimensional Reconstruction Accuracy 283
6.3.4.4 Consequences of Poorly Known Re ectance Parameters 285
6.3.5 Inspection Based on Integration of Photometric Image Information and Active Range Scanning Data 287
6.3.6 Discussion 289
Chapter 7: Applications to Safe Human-Robot Interaction 291
7.1 Vision-Based Human-Robot Interaction 291
7.1.1 Vision-Based Safe Human-Robot Interaction 292
7.1.2 Pose Estimation of Articulated Objects in the Context of Human-Robot Interaction 295
7.1.2.1 The Role of Gestures in Human-Robot Interaction 296
7.1.2.2 Recognition of Gestures 296
7.1.2.3 Including Context Information: Pointing Gestures and Interactions with Objects 297
7.1.2.4 Discussion in the Context of Industrial Safety Systems 298
7.2 Object Detection and Tracking in Three-Dimensional Point Clouds 299
7.3 Detection and Spatio-Temporal Pose Estimation of Human Body Parts 301
7.4 Three-Dimensional Tracking of Human Body Parts 304
7.4.1 Image Acquisition 304
7.4.2 Data Set Used for Evaluation 305
7.4.3 Fusion of the ICP and MOCCD Poses 307
7.4.4 System Con gurations Regarded for Evaluation 309
Con guration 1: Tracking Based on the MOCCD 309
Con guration 2: Tracking Based on the Shape Flow Method 309
Con guration 3: ICP-Based Tracking 309
Con guration 4: Fusion of ICP and MOCCD 310
Con guration 5: Fusion of ICP, MOCCD, and SF 310
7.4.5 Evaluation Results 310
7.4.6 Comparison with Other Methods 314
7.4.7 Evaluation of the Three-Dimensional Mean-Shift Tracking Stage 316
7.4.8 Discussion 318
7.5 Recognition of Working Actions in an Industrial Environment 318
Chapter 8: Applications to Lunar Remote Sensing 321
8.1 Three-Dimensional Surface Reconstruction Methodsfor Planetary Remote Sensing 321
8.1.1 Topographic Mapping of the Terrestrial Planets 321
8.1.1.1 Active Methods 321
8.1.1.2 Shadow Length Measurements 322
8.1.1.3 Stereo and Multi-image Photogrammetry 323
8.1.1.4 Photoclinometry and Shape from Shading 324
8.1.2 Re ectance Behaviour of Planetary Regolith Surfaces 325
8.2 Three-Dimensional Reconstruction of Lunar Impact Craters 328
8.2.1 Shadow-Based Measurement of Crater Depth 328
8.2.2 Three-Dimensional Reconstruction of Lunar Impact Craters at High Resolution 329
8.2.3 Discussion 339
8.3 Three-Dimensional Reconstructionof Lunar Wrinkle Ridges and Faults 340
8.4 Three-Dimensional Reconstruction of Lunar Domes 343
8.4.1 General Overview of Lunar Domes 343
8.4.2 Observations of Lunar Domes 344
8.4.2.1 Spacecraft Observations of Lunar Mare Domes 344
8.4.2.2 Telescopic CCD Imagery 348
8.4.3 Image-Based Determination of Morphometric Data 349
8.4.3.1 Construction of DEMs 349
8.4.3.2 Error Estimation 358
8.4.3.3 Comparison to Other Height Measurements 360
8.4.4 Discussion 363
Chapter 9: Conclusion 366
References 372