Image and Geometry Processing for 3-D Cinematography (eBook)

Acknowledgements 8
Contents 10
Image and Geometry Processing for 3-D Cinematography: An Introduction 12
1 Overview of the Field 12
2 The Geometry of Lights, Cameras and Actions 13
3 Book Contents 14
3.1 3-D Cinematography and Applications 14
3.2 Recent Developments in Geometry 16
3.3 Recent Developments in Image Processing 17
4 Final Remarks 19
References 19
Part I 3-D Cinematography and Applications 20
Stereoscopic Cinema 21
1 Introduction 21
1.1 Stereoscopic Cinema, 3-D Cinema, and Others 21
1.2 A Brief History of Stereoscopic Cinema 22
1.3 Computer Vision, Computer Graphics, and Stereoscopic Cinema 24
1.3.1 A Few Definitions 24
1.3.2 Stereo-Specific Processes 25
2 Three-Dimensional Perception and Visual Fatigue 26
2.1 Monoscopic Depth Cues 27
2.2 Stereoscopy and Stereopsis 29
2.3 Conflicting Cues 29
2.4 Inconsistent Cues 31
2.5 Sources of Visual Fatigue 31
2.5.1 Horizontal Disparity Limits 33
2.5.2 Vertical Disparity 34
2.5.3 Vergence-Accommodation Conflicts 36
3 Picking the Right Shooting Geometry 37
3.1 The Spottiswoode Point of View 37
3.2 Shooting and Viewing Geometries 39
3.3 Depth Distortions 41
3.4 Shape Distortions and the Depth Consistency Rule 42
3.5 Shooting with the Right Depth of Field 44
3.6 Remaining Issues 44
4 Lessons for Live-Action Stereoscopic Cinema from Animated 3-D 45
4.1 Proscenium Arch or Floating Window 45
4.2 Floating Windows and Audience Experience 46
4.3 Window Violations 46
4.4 Multi-rigging 47
5 Post-production of Stereoscopic Movies 47
5.1 Eliminating Vertical Disparity: Rectification of Stereoscopic Movies 48
5.2 Shifting and Scaling the Images 49
5.3 View Interpolation, View Synthesis,and Disparity Remapping 50
5.3.1 Definitions and Existing Work 50
5.3.2 Asymmetric Processing 52
5.4 Changing the Depth of Field 53
5.5 Dealing with the Proscenium Rule 54
5.6 Compositing Stereoscopic Scenes and Relighting 55
5.7 Adding Titles or Subtitles 56
6 Conclusion 57
References 57
Free-Viewpoint Television 62
1 Introduction 62
2 Ray-Space Representation 63
3 FTV System 67
3.1 Configuration of FTV System 67
3.2 Capture 69
3.3 Correction 70
3.4 MVC Encoding and Decoding 71
3.5 View Generation 71
3.6 2D/3D Display 72
4 Creation of Ray-Based Image Engineering 74
4.1 Evolution of Image Systems 74
4.2 Ray Capture and Display 74
4.3 Ray Processing 78
5 International Standardization 80
6 Conclusion 83
References 83
Free-Viewpoint Video for TV Sport Production 86
1 Introduction 86
2 Background 88
2.1 Methodologies for Free-Viewpoint Video 89
2.2 Studio Production of Free-Viewpoint Video 91
2.3 Free-Viewpoint Video in Sports and Outdoor Scenes 92
3 Specification of Requirements for TV Sports Production 94
3.1 Visual Quality for Broadcast Production 94
3.2 Production Requirements on Timing 95
3.3 Acquisition Requirements 96
3.4 Production Requirements for Use of Free-Viewpoint Video 97
4 A Free-Viewpoint System for Sports TV Production 98
4.1 System Overview 98
4.2 Video-Rate Calibration of Live Broadcast Footage 100
4.3 Foreground Segmentation 102
4.3.1 Aperture Correction in Broadcast Cameras 104
4.4 Free-Viewpoint Video Production from Match Cameras 105
4.4.1 Conservative Visual-Hull 105
4.4.2 Local View-Dependent Visual-Hull and Segmentation Refinement 106
4.5 Free-Viewpoint Rendering Results 108
5 Conclusion 112
References 113
Challenges for Multi-View Video Capture 116
1 Overview 116
2 Spatial Resolution 117
2.1 Resolution Requirements 117
2.2 Ring of Cameras 118
2.3 Active Cameras 119
3 High-Speed Multi-View Video 120
3.1 Spatiotemporal Sampling for Camera Arrays 121
3.2 Spatiotemporal View Interpolation 123
4 Realism 126
5 Real-Time Processing and Broadcast 128
5.1 Processing 128
5.2 Broadcast 130
6 Discussion 130
References 131
Part II Recent Developments in Geometry 133
Performance Capture from Multi-View Video 134
1 Introduction 134
2 Paving the Way for Performance Capture: Motion Capture, Image-Based Rendering and 3-D Video Approaches 136
3 Performance Capture Approaches 137
3.1 Garment Capture 138
3.2 Surface Capture 139
3.3 Simultaneous Surface and Skeleton Capture 140
4 Mesh-Based Performance Capture 141
4.1 Overview 143
4.2 A Deformation Toolbox 144
4.2.1 Volumetric Deformation 144
4.2.2 Deformation Transfer 145
4.2.3 Surface-Based Deformation 145
4.3 Capturing the Global Model Pose 145
4.3.1 Pose Initialization from Image Features 146
4.3.2 Refining the Pose Using Silhouette Rims 147
4.3.3 Optimizing Key Handle Positions 147
4.3.4 Capturing Surface Detail 147
4.4 Results 148
4.4.1 Validation and Discussion 150
5 Conclusion and Further Reading 152
References 154
Combining Multi-view Stereo and Bundle Adjustment for Accurate Camera Calibration 157
1 Introduction 157
1.1 Existing Approaches and Their Problems 158
1.2 Overview of the Proposed Approach 159
2 Imaging Model and Preliminaries 161
3 Algorithm 162
3.1 Initializing Feature Correspondences 162
3.2 Refining Feature Correspondences 163
4 Experimental Results and Discussions 164
4.1 Datasets 164
4.2 Experiments 166
5 Conclusion 173
References 174
Cell-Based 3D Video Capture Method with Active Cameras 176
1 Introduction 176
2 Problem Formulation 178
3 Cell-Based Active Tracking Algorithm 181
3.1 Cell Formation 182
3.2 Camera Calibration 183
3.3 Camera Control Scheduling 184
3.4 Real-Time Object Tracking and Camera Control 186
3.5 3D Video Generation 188
4 Experiments and Evaluations 188
4.1 Evaluation of the Visual Coverage Function 189
4.2 Tracking Experiments 190
4.3 Performance Evaluation 193
5 Conclusion and Future Work 195
References 195
Dense 3D Motion Capture from Synchronized Video Streams 197
1 Introduction 197
1.1 Related Work 198
1.2 Problem Statement and Proposed Approach 199
2 Spatiotemporal Surface Model 200
2.1 Local Surface Model 201
2.1.1 Local Geometric Model 201
2.1.2 Local Photometric Model 201
2.2 Shape and Motion Estimation 202
2.2.1 Initial Motion Estimation by Expansion 203
2.2.2 Shape Optimization 203
2.2.3 Motion Optimization 203
2.2.4 Visibility Estimation 204
3 Algorithm 205
3.1 Local Tracking 205
3.2 Mesh Deformation 206
3.3 Filtering 207
4 Experimental Results and Discussion 207
5 Conclusion and Future Work 214
References 214
Part III Recent Developments in Image Processing 216
Wavelet-Based Inverse Light and Reflectance from Images of a Known Object 217
1 Introduction 217
2 Related Works in Inverse Light 219
2.1 Point Light Representations 220
2.2 Basis Function Representations 222
3 Theory 223
3.1 Inverse Light Problem Definition 223
3.2 Wavelet-Based Light Model 224
3.3 Reflectance Models 226
3.4 Inverse Light from a Single Image 228
3.5 Inverse Light from a Multiple Images 228
3.6 Reflectance Estimation 230
3.7 Numerical Light Estimation and Regularization 231
4 System and Implementation Details 232
5 Experiments 233
5.1 Quality of the Reconstructed Light Basis 234
5.2 Stability of the Inverse Light Method for DifferentLight Basis 236
5.3 Augmented Reality 238
6 Conclusion and Discussion 239
References 239
3-D Lighting Environment Estimation with Shading and Shadows 241
1 Introduction 241
2 Lighting Environment Estimation with Shading and Shadows 243
2.1 Reflectance Model 243
2.2 Computational Algorithm 245
3 Skeleton Cube 245
3.1 3-D geometric Features 245
3.2 Verifying Utilities of Skeleton Cube 246
3.2.1 Evaluating Effectiveness of Lighting Environment Estimation with Shadows 246
3.2.2 Evaluating Capability of Shadow Generation of the Skeleton Cube 248
4 Estimation Algorithm for 3-D Lighting Environment 250
4.1 Problems: Underdeterminedness and Uncertainty 250
4.2 Geometric Structure based Approach 252
4.3 Overall Algorithm 252
5 Performance Evaluation 254
6 Conclusion 257
References 258
3-D Cinematography with Approximate or No Geometry 260
1 Introduction 260
2 3-D Reconstruction 261
2.1 Model Reconstruction 261
2.2 Shape-From-Silhouettes 262
2.3 Depth-From-Stereo 263
3 Texturing with Imprecise Geometry and Sparse Camera Setups 265
4 Floating Textures 268
4.1 Soft Visibility 269
4.2 GPU Implementation 271
4.3 Static Correspondence Finding 272
5 View and Time Interpolation in Image Space 273
5.1 Image Morphing and Spatial Transformations 275
5.2 Image Deformation Model for Time and View Interpolation 275
5.3 Optimizing the Image Deformation Model 279
5.4 Rendering 280
References 282
View Dependent Texturing Using a Linear Basis 286
1 Introduction 286
2 Background: Image Geometry 288
3 Texture Basis 289
3.1 Geometric Texture Variation 291
3.2 Photometric Variation 293
3.3 Estimating Composite Variability 295
3.4 Experimental Comparison for Analytical and PCA Basis 296
4 Model and Texture Capture System Implementation 297
5 Experiments 300
6 Discussion 304
References 305