Computational Imaging for Scene Understanding

Takuya Funatomi is an associate professor in Division of Information Science at the Nara Institute of Science and Technology (NAIST) in Japan. He has received a bachelor's degree in Engineering and a master’s degree and PhD in Informatics from Kyoto University in 2002, 2004 and 2007, respectively. Takahiro Okabe is a professor at Kyushu Institute of Technology in Japan. He received a bachelor's and a master's degree in Physics and a PhD in Information Science and Technology from the University of Tokyo in 1997, 1999 and 2011, respectively.

Introduction xiii
Takuya FUNATOMI and Takahiro OKABE

Part 1 Transient Imaging and Processing 1

Chapter 1 Transient Imaging 3
Adrian JARABO

1.1 Introduction 3

1.2.Mathematical formulation 5

1.2.1 Analysis of transient light transport propagation 7

1.2.2 Sparsity of the impulse response function T (x, t) 8

1.3.Capturinglight in flight 9

1.3.1 Single-photon avalanche diodes (SPAD) 11

1.4.Applications 14

1.4.1.Range imaging 14

1.4.2.Material estimation and classification 14

1.4.3 Light transport decomposition 15

1.5.Non-line-of-sight imaging 15

1.5.1.Backprojection 17

1.5.2 Confocal NLOS and the light-cone transform 17

1.5.3 Surface-based methods 18

1.5.4.Virtualwaves and phasor fields 19

1.5.5.Discussion 21

1.6.Conclusion 22

1.7.References 22

Chapter 2 Transient Convolutional Imaging 29
Felix HEIDE

2.1 Introduction 29

2.2.Time-of-flight imaging 30

2.2.1.Correlationimage sensors 32

2.2.2 Convolutional ToF depth imaging 32

2.2.3 Multi-path interference 34

2.3.Transient convolutional imaging 35

2.3.1 Global convolutional transport 37

2.3.2 Transient imaging using correlation image sensors 37

2.3.3 Spatio-temporal modulation 40

2.4.Transient imaging in scattering media 41

2.5.Present and future directions 43

2.6.References 43

Chapter 3 Time-of-Flight and Transient Rendering 45
Adithya Kumar PEDIREDLA

3.1 Introduction 45

3.2.Mathematicalmodeling 46

3.2.1 Mathematical modeling for time-of-flight cameras 47

3.3.Howto render time-of-flight cameras? 50

3.3.1 Challenges and solutions in time-of-flight rendering 51

3.4.Open-source implementations 56

3.5.Applicationsof transient rendering 57

3.6.Future directions 61

3.7.References 62

Part 2 Spectral Imaging and Processing 69

Chapter 4 Hyperspectral Imaging 71
Nathan HAGEN

4.1 Introduction 71

4.2.2D(raster scanning) architectures 75

4.2.1.Czerny-Turnergratingspectrometers 76

4.2.2 Transmission grating/prism spectrometers 78

4.2.3.Coded aperture spectrometers 79

4.2.4.Echelle spectrometers 80

4.3.1Dscanningarchitectures 81

4.3.1.Dispersive spectrometers 82

4.3.2 Interferometric methods 83

4.3.3 Interferometric filter methods 83

4.3.4 Polarization-based filter methods 86

4.3.5 Active illumination methods 88

4.4.Snapshot architectures 88

4.4.1.Bowen-Walravenimage slicer 89

4.4.2 Image slicing and imagemapping 90

4.4.3 Integral field spectrometry with coherent fiber bundles (IFS-F) 93

4.4.4 Integral field spectroscopy with lens let arrays (IFS-L) 94

4.4.5 Filter array camera (FAC) 94

4.4.6 Computed tomography imaging spectrometry (CTIS) 96

4.4.7 Coded aperture snapshot spectral imager (CASSI) 97

4.5.Comparisonof snapshot techniques 98

4.5.1.The disadvantages of snapshot 100

4.6.Conclusion 101

4.7.References 102

Chapter 5 Spectral Modeling and Separation of Reflective-Fluorescent Scenes 109
Ying FU, Antony LAM, Imari SATO, Takahiro OKABE, and Yoichi SATO

5.1 Introduction 109

5.2.RelatedWork 111

5.3.Separationof reflection and fluorescence 113

5.3.1.Reflection and fluorescence models 113

5.3.2 Separation using high-frequency illumination 114

5.3.3 Discussion on the illumination frequency 116

5.3.4.Error analysis 118

5.4.Estimating the absorption spectra 119

5.5.Experiment results and analysis 122

5.5.1.Experimental setup 122

5.5.2 Quantitative evaluation of recovered spectra 122

5.5.3.Visual separation and relighting results 126

5.5.4 Separation by using high-frequency filters 130

5.5.5 Ambient illumination 134

5.6.Limitations and conclusion 137

5.7.References 137

Chapter 6 Shape from Water 141
Yuta ASANO, Yinqiang ZHANG, Ko NISHINO, and Imari SATO

6.1 Introduction 141

6.2.Relatedworks 143

6.3.Light absorption in water 145

6.4 Bispectral light absorption for depth recovery 146

6.4.1.Bispectral depth imaging 146

6.4.2 Depth accuracy and surface reflectance 147

6.5.Practical shape from water 148

6.5.1 Non-collinear/perpendicular light-camera configuration 148

6.5.2 Perspective camera with a point source 150

6.5.3.Non-idealnarrow-bandfilters 151

6.6 Co-axial bispectral imaging system and experiment results 151

6.6.1.Systemconfigurationand calibration 151

6.6.2 Depth and shape accuracy 152

6.6.3 Complex static and dynamic objects 154

6.7 Trispectral light absorption for depth recovery 155

6.7.1.Trispectraldepthimaging 156

6.7.2 Evaluation on the reflectance spectra database 157

6.8.Discussions 157

6.9.Conclusion 158

6.10.References 158

Chapter 7 Far Infrared Light Transport Decomposition and Its Application for Thermal Photometric Stereo 161
Kenichiro TANAKA

7.1 Introduction 161

7.1.1.Contributions 162

7.2.Relatedwork 163

7.2.1 Light transport decomposition 163

7.2.2.Computational thermal imaging 164

7.2.3.Photometricstereo 165

7.3.Far infrared light transport 165

7.4 Decomposition and application 171

7.4.1 Far infrared light transport decomposition 171

7.4.2 Separating the ambient component 172

7.4.3.Separatingreflectionand radiation 172

7.4.4 Separating diffuse and global radiations 172

7.4.5.Other options 173

7.4.6 Thermal photometric stereo 173

7.5.Experiments 174

7.5.1 Decomposition result 175

7.5.2.Surfacenormal estimation 177

7.6.Conclusion 179

7.7.References 180

Chapter 8 Synthetic Wavelength Imaging: Utilizing Spectral Correlations for High-Precision Time-of-Flight Sensing 187
Florian WILLOMITZER

8.1 Introduction 187

8.2.Syntheticwavelengthimaging 189

8.3.Synthetic wavelength interferometry 193

8.4 Synthetic wavelength holography 197

8.4.1 Imaging around corners with synthetic wavelength holography 199

8.4.2 Imaging through scattering media with synthetic wavelength holography 200

8.4.3 Discussion and comparison with the state of the art 203

8.5 Fundamental performance limits of synthetic wavelength imaging 205

8.6.Conclusionand future directions 210

8.7.Acknowledgment 210

8.8.References 211

Part 3 Polarimetric Imaging and Processing 219

Chapter 9 Polarization-Based Shape Estimation 221
Daisuke MIYAZAKI

9.1 Fundamental theory of polarization 221

9.2 Reflection component separation 225

9.3.Phase angle of polarization 226

9.4 Surface normal estimation from the phase angle 228

9.5.Degree of polarization 233

9.6 Surface normal estimation from the degree of polarization 236

9.7.Stokes vector 236

9.8 Surface normal estimation from the Stokes vector 237

9.9.References 239

Chapter 10 Shape from Polarization and Shading 241
Thanh-Trung NGO, Hajime NAGAHARA, and Rin-ichiro TANIGUCHI

10.1 Introduction 241

10.2.Relatedworks 243

10.2.1.Shadingand polarization fusion 243

10.2.2 Shape estimation under uncalibrated light sources 244

10.3 Problem setting and assumptions 245

10.4.Shadingstereoscopic constraint 246

10.5.Polarizationstereoscopic constraint 248

10.6.Normal estimation with two constraints 249

10.6.1 Algorithm 1: Recovering individual surface points 250

10.6.2 Algorithm 2: Recovering shape and light directions 251

10.7.Experiments 252

10.7.1 Simulation experiments with weights for two constraints 253

10.7.2.Real-world experiments 254

10.8.Conclusionand future works 263

10.9.References 263

Chapter 11 Polarization Imaging in the Wild Beyond the Unpolarized World Assumption 269
Jérémy Maxime RIVIERE

11.1 Introduction 269

11.2.Mueller calculus 271

11.3.Polarizingfilters 273

11.3.1.Linear polarizers 273

11.3.2.Reflectors 274

11.4.Polarizationimaging 275

11.5 Image formation model 277

11.5.1 Partially linearly polarized incident illumination 277

11.5.2 Unpolarized incident illumination 279

11.5.3.Discussion 280

11.6.Polarization imaging reflectometry in the wild 282

11.7.DigitalSingle-Lens Reflex (DSLR) setup 283

11.7.1 Data acquisition 283

11.7.2.Calibration 285

11.7.3.Polarizationprocessingpipeline 285

11.8.Reflectance recovery 287

11.8.1.Surface normal estimation 287

11.8.2.Diffuse albedo estimation 288

11.8.3 Specular component estimation 288

11.9.Results and analysis 291

11.9.1.Results 291

11.9.2.Discussion and error analysis 293

11.10.References 296

Chapter 12 Multispectral Polarization Filter Array 299
Kazuma SHINODA

12.1 Introduction 299

12.2 Multispectral polarization filter array with a photonic crystal 302

12.3 Generalization of imaging and demosaicking with multispectral polarization filter arrays 306

12.4.Demonstration 311

12.5.Conclusion 313

12.6.References 313

List of Authors 317

Index 319