Smart Camera Design (eBook)

Marilyn Wolf received her bachelor's, master's, and doctoral degrees in electrical engineering from Stanford University in 1980, 1981, and 1984, respectively. She was with AT&T Bell Laboratories in Murray Hill, N.J. from 1984 to 1989 and was with Princeton University from 1989 until 2007. In July 2007, Dr. Wolf joined Georgia Tech as the Rhesa "Ray" S. Farmer, Jr. Distinguished Chair in Embedded Computing Systems and Georgia Research Alliance Eminent Scholar. She has developed a number of techniques for embedded computing, ranging from hardware/software co-design algorithms and real-time scheduling algorithms to code compression and distributed smart cameras. She is a co-founder of Verificon Corporation, which designs smart camera systems. She helped to start several technical conferences, including CODES and MPSoC. She has written four textbooks.

Preface 6
Contents 8
Chapter 1: Digital Photography 12
1.1 Introduction 12
1.2 Previsualization and Autoprevisualization 12
1.3 Enhanced Images 13
1.4 Beyond Images to Analysis 14
1.5 Still and Moving Images 14
1.6 Taking a Picture 14
1.7 How to Read this Book 16
Chapter 2: Light, Optics, and Imaging 18
2.1 Introduction 18
2.2 Image Formation 18
2.2.1 Light and Images 18
2.2.2 The Physics of Light 22
2.3 The Human Visual System 26
2.4 Color Science 30
2.4.1 Theories of Color Vision 30
2.4.2 Color Models 31
2.5 Lenses 33
2.5.1 Lenses and Image Formation 34
2.5.2 Ray Optics 37
2.5.3 Lens Design 39
2.5.4 Panoramas 44
2.5.5 Assessing Lenses 49
2.6 Geometry and the Camera Model 52
2.6.1 Projective Geometry 52
2.6.2 The Camera Model 55
2.6.3 Camera Calibration 57
2.7 Image Display 59
2.8 Practical Image Capture 61
2.8.1 Exposure Settings 62
2.8.2 Which Exposure Setting? 63
2.8.3 Color Temperature 67
2.8.4 Image Composition 69
2.8.5 Image Quality Assessment 74
2.9 Summary 77
Chapter 3: Image Capture Systems and Algorithms 78
3.1 Introduction 78
3.2 The Generic Camera Architecture 79
3.3 The Camera Design Space 80
3.3.1 Trade-Offs 80
3.3.2 Use Cases for Cameras 81
3.3.3 Four Examples of Camera Designs 81
3.4 Image Sensors 84
3.4.1 Image Sensor Architectures 84
3.4.2 Photosensors 86
3.4.3 Charge-Coupled Devices 88
3.4.4 APS CMOS Image Sensors 89
3.4.5 Advanced Image Sensors 93
3.4.6 Image Sensor Characteristics 94
3.4.7 Shutters and Irises 100
3.5 Preexposure Operations 102
3.5.1 Autofocus 103
3.5.2 Exposure 107
3.5.3 Image Stabilization 109
3.5.4 Face Detection and Tracking 110
3.6 Postexposure Operations 113
3.6.1 Color Filter Array Interpolation 113
3.6.2 White Balance 116
3.6.3 Sharpening 118
3.7 Image and Video Compression 119
3.7.1 Lossy Compression 119
3.7.2 Image Coding and JPEG 119
3.7.3 Video Coding, H.264/AVC, and HEVC/H.265 124
3.7.4 Quality Assessment of Compressed Images 131
3.8 Computing Platforms 132
3.8.1 Cameras as Heterogeneous Multiprocessors 132
3.8.2 Buffering 133
3.8.3 Input and Output 135
3.8.4 File Formats 135
3.8.5 Operating Systems and File Systems 137
3.8.6 Accelerators 137
3.9 Image Characteristics and Image Capture 141
3.10 Stereo and Multicamera Systems 142
3.11 Trade-Offs Revisited 143
Chapter 4: Image and Video Enhancement 145
4.1 Introduction 145
4.2 Useful Algorithms 146
4.3 Tonal Mapping and Color Grading 148
4.4 High-Dynamic Range Images 153
4.5 Sharpening and Superresolution 156
4.6 Bokeh Introduction 159
4.7 Lens Corrections 159
4.8 Focus Stacking 160
4.9 Keystone Correction 162
4.10 Mosaic Composition 163
4.11 Video Stabilization 166
4.11.1 Optical Flow 166
4.11.2 Stabilization Algorithms 167
4.12 Software Design for Image Enhancement 168
4.13 Practical Image Enhancement 170
Chapter 5: Image and Video Analysis 172
5.1 Introduction 172
5.2 Image Analysis Algorithms 173
5.3 Image and Video Characteristics 176
5.3.1 Image Statistics 176
5.3.2 Saliency 177
5.3.3 Key Frame Selection 179
5.4 Scene Analysis 180
5.4.1 Visual Search 180
5.4.2 Face Detection and Recognition 181
5.5 Tracking 182
5.5.1 Background Elimination 182
5.5.2 Tracking from a Fixed Camera 184
5.5.3 Appearance Models 187
5.5.4 Activity Analysis 190
5.5.5 Tracking from a Moving Camera 192
5.6 Multicamera Systems 195
5.6.1 Multicamera Systems as Distributed Computing Systems 196
5.6.2 Multicamera Calibration 198
5.6.3 Multicamera Tracking 200
5.7 Use Cases and Workflows 206
Chapter 6: Photography and Cinematography 207
6.1 Introduction 207
6.2 Photography 207
6.3 Cinematography 213
References 221