Theory and Applications of Smart Cameras (eBook)

Chong-Min Kyung (Professor, KAIST) received B.S. in Electrical Engineering (EE) from Seoul National University in 1975, M.S. and Ph.D. in EE from KAIST in 1977 and 1981, respectively. He worked at Bell Telephone Laboratories, Murray Hill during 1981-1983. Since 1983, he has been Professor at the Department of EE at KAIST. He started Center for Integrated Smart Sensors (CISS) in 2011, sponsored as Global Frontier Project by Korean government. He has published over 300 international journal/conference papers on device physics/simulation, CAD, computer graphics, and System-on-a-Chip design including RISC/CISC microprocessors, VLIW and reconfigurable DSP cores, and video CODEC.His current research focuses on applying system-level cost-energy-rate-distortion optimization techniques to the design of smart sensors, especially smart cameras.He has led, as Founding Director, IDEC (IC Design Education Center) during 1995-2011. He received Best Paper Awards in numerous international conferences including ASP-DAC 1997 and 1998, DAC in 2000, ICSPAT in 1999, ICCD in 1999, ISOCC in 2009, and ISQED in 2014. He is a member of National Academy of Engineering Korea, and Korean Academy of Science and Technology, and IEEE Fellow.

Contents 6
Part I Fundamental/Energy-related Issuesof Smart Cameras 8
1 CMOS Image Sensor for Smart Cameras 9
Abstract 9
1 Imaging Principles 9
1.1 Solid State Imaging Devices 10
1.1.1 Charge Coupled Device 10
1.1.2 CMOS Image Sensors 11
1.2 Preliminaries for CMOS Image Sensors 11
1.2.1 Readout Timing for CMOS Image Sensors 11
1.2.2 Column Noise Canceler for CMOS Image Sensors 12
1.3 Pixel Structure for CMOS Image Sensors 13
1.3.1 Three Transistor APS 13
1.3.2 Four Transistor APS 15
1.4 Dynamic Range of Image Sensors 17
2 Dynamic Range Extension 18
2.1 Nonlinear Response Type 19
2.1.1 Logarithmic Photodetector 19
2.1.2 Linear-Logarithmic Method 20
2.2 Linear Response Type 21
2.2.1 Well Capacity Adjusting Method 21
2.2.2 Method to Use a Lateral Overflow Integration Capacitor 22
2.2.3 Dual Sampling Method 22
2.2.4 Double Sampling Method 24
2.2.5 Burst Readout Multiple Exposure 24
Acknowledgments 25
References 25
2 Architectural Analysis of a Baseline ISP Pipeline 27
Abstract 27
1 Introduction 27
1.1 Embedded ISP in an Image Sensor 28
1.2 Discrete ISP Package 28
1.3 Embedded ISP Inside an AP 28
2 Primary ISP Architecture for Bayer Image Sensors 29
2.1 Anti-aliasing Noise Filter 38
2.2 Color Filter Array Interpolation 38
2.3 Noise Filter for Luma 38
2.4 Noise Filter for Chrominance: CB and CR 38
3 ISP Architecture for Color Reproduction 39
4 ISP Architecture with Pre-/Post-processing 46
5 Further Works on ISP 49
Acknowledgments 50
References 51
3 An Ultra-Low-Power Image Signal Processor for Smart Camera Applications 52
Abstract 52
1 Introduction 52
2 Background 54
2.1 Baseline ISP Design 54
2.2 Face Detection Algorithm 56
3 Evaluation Methodology 57
4 Optimizing ISP for Face Detection 59
4.1 Skipping Non-critical ISP Stages 59
4.2 Interpolating Pixel Values 61
4.3 Scaling Images 64
4.4 Putting It Together 65
5 Related Work 66
6 Conclusion and Future Work 66
Acknowledgments 67
References 67
4 Foundations and Applications of 3D Imaging 68
Abstract 68
1 Foundations of 3D Imaging 68
1.1 Passive 3D Imaging 69
1.1.1 Binocular Stereo 69
1.1.2 Multi-view Stereo 71
1.1.3 Refractive Stereo 72
1.1.4 Other Approaches 74
1.2 Active 3D Imaging 75
1.2.1 3D Scanning with Swept-Planes 75
1.2.2 3D Scanning with Structured Lighting 75
1.2.3 Photometric Stereo 76
2 Applications of Advanced 3D Imaging 78
2.1 3D Imaging Spectroscopy 79
2.1.1 The Dispersion-Based Hyperspectral Imager 79
2.1.2 3D System Integration 81
2.1.3 Measuring 3D Hyperspectral Patterns 82
2.2 Hyperspectral Photometric Stereo 82
2.2.1 Measuring Surface Normals 82
2.2.2 Combining Hyperspectral Imaging with Photometric Stereo 83
2.2.3 Removing Interreflection 84
2.2.4 Measuring a Shape with Hyperspectral Imaging 85
2.3 Stereo Fusion of Refractive and Binocular Stereo 85
3 Conclusions 89
Acknowledgements 89
References 89
5 E-R-D Optimization in Video Compression 92
Abstract 92
1 Power-Aware Design for Hardware-Based Video Compression 93
2 Implementation of Power-Aware Design 94
2.1 Power Consumption of Mobile Devices 94
2.2 Generation of a Power Level Table 95
2.3 Effect of Video Characteristics 96
2.4 Dynamic Selection of the Power Budget Target 96
2.5 Selection of a Power-Scaling Algorithm 97
3 Power Estimation Model 98
3.1 Formulation of a Power Estimation Model 98
3.2 Impact of ADF and VLC on Power Consumption 100
4 Power-Scaling Algorithms 100
4.1 Four Power-Scaling Algorithms 101
4.2 Example Power Estimation Model 103
4.3 Power Simulation of Individual Algorithms 104
4.3.1 FME Prediction Mode Reduction 104
4.3.2 IME Search Range Control 105
4.3.3 Early SKIP Mode Decision 106
4.3.4 Intra-Frame Period Control 106
4.4 Estimation of the Combined Power Saving and Derivation of the Optimal Operating Conditions 107
4.5 Generation of a Power Table 108
5 Performance of Power-Aware Design 110
5.1 Performance Estimation of the Power-Aware Design 111
5.2 Comparison with a Previous Power-Aware Design 115
Acknowledgments 118
References 118
Part II Event/Object Detectors for Smart Sensing 120
6 Low-Power Operation for Video Event Data Recorder 121
Abstract 121
1 Introduction 121
2 Operation of the Video Event Data Recorder 122
2.1 Power Consumption of Conventional Event-Driven System 124
2.2 Proposed Method 127
3 Low-Power System Design 131
3.1 Parked Mode 132
3.2 Driving Mode 134
4 Power Analysis in Parked Mode 135
5 Performance Evaluation 138
6 Conclusion 140
Acknowledgments 141
References 141
7 Low-Power Face Detection for Smart Camera 143
Abstract 143
1 Overview of Face Detection for Smart Camera 143
2 Human Face-Based Event Detection for Low Power Operation 146
3 Low-Power Face Detection Algorithms 147
3.1 The First Stage: Region-of-Interest (ROI) Selection 147
3.1.1 Pre-filtering Step for Speed-up 148
3.1.2 Feature Extraction 149
3.1.3 Multiple Template Matching 150
3.2 False Positive (FP) Reduction 151
4 Experiments 152
4.1 Face Detection Performance Evaluation 154
4.2 Face Detection Simulation for Hardware Implementation 156
5 Conclusion 157
Acknowledgments 158
References 158
8 Accurate Face and Human Detection Using Hybrid Local Transform Features 160
Abstract 160
1 Introduction 161
2 Local Gradient Patterns 163
3 Binary Histograms of Oriented Gradients 165
4 Hybridization of Local Transform Features 167
5 Experimental Results and Discussion 168
5.1 Face Detection 168
5.1.1 Data Preparation 168
5.2 Training Procedure 171
5.2.1 Cascade of Face Detectors 171
5.2.2 Detection Performance 173
5.2.3 Memory Size 175
5.2.4 Computation Time 177
5.3 Human Detection 178
5.3.1 Data Preparation 178
5.3.2 Training Procedure 178
5.3.3 Cascade of Human Detectors 179
5.3.4 Detection Performance 180
5.3.5 Computation Time 183
6 Conclusion 185
Acknowledgements 186
References 186
9 Adaptive Resource Management for Sensor Fusion in Visual Tracking 189
Abstract 189
1 Introduction 190
2 Background 193
2.1 Overview 193
2.2 Kernel-Based Bayesian Filtering 193
2.3 Discussion of Kernel-Based Bayesian Filtering 195
3 Fusion Tracking by Mixture KBF 195
3.1 Prediction Step and Proposal Distribution 196
3.2 Measurement Step 197
3.3 Update Step 199
4 Experiments 202
4.1 Sensor Fusion with Multiple Cameras 202
4.1.1 Implementation Issues 202
4.1.2 Results 203
4.2 Sensor Fusion with Multiple Features 211
5 Conclusion 213
Acknowledgment 213
References 213
10 Traffic Pattern Analysis and Anomaly Detection via Probabilistic Inference Model 216
Abstract 216
1 Introduction 217
1.1 Objective and Contribution 217
1.2 Related Works 219
2 Proposed Approach 220
2.1 Probabilistic Inference Model 221
2.2 Two-Stage Greedy Inference 225
2.2.1 Model Learning 225
2.2.2 Anomaly Detection 229
2.3 Summary of the Proposed Method 231
3 Experiments 232
3.1 Result of Scene Understanding 233
3.2 Applications in Anomaly Detection 236
3.3 Prediction Task 238
4 Conclusion 239
Acknowledgments 240
References 240
11 Event Detection Module for Low-Power Camera 242
Abstract 242
1 Event Detection Framework 242
2 Event Detection Algorithm 244
2.1 Background Subtraction Using Color and Depth Information 244
2.1.1 Introduction 244
2.1.2 Background Subtraction Method 245
Background Modeling 245
Background Subtraction 246
Depth Image De-noising 247
2.1.3 Dataset for Performance Evaluation 248
2.1.4 Experimental Results 250
2.1.5 Conclusions 250
2.2 BGS Performance Evaluation Software 252
2.2.1 Introduction 252
2.2.2 User Interface and Functionalities 252
BGS-Type Selection 252
Main Evaluation Mode Selection 252
Input Image Conversion Selection 253
Pre and Post-Processing Selection 254
2.2.3 Simulation 254
2.2.4 Conclusions 256
3 Hardware Framework 256
3.1 Introduction 256
3.2 System Overview 257
3.3 Circuit Implementation 258
3.3.1 Image Array and Analog Data Readout 258
3.3.2 ADC Design 260
3.4 Measurement Results 260
4 Conclusions 262
Acknowledgments 262
Appendix 262
References 265
12 Advanced Human Detection Using Fused Information of Depth and Intensity Images 266
Abstract 266
1 Introduction 267
2 Related Work 267
3 Proposed Method 268
3.1 ROI Generation Using Depth Images 269
3.2 Feature Extraction from Depth and Intensity Image 271
3.3 Classification with Model 273
4 Experimental Results 273
4.1 Dataset 274
4.2 Evaluation of ROI Generation Method 275
4.3 Evaluation of Feature Extraction 276
4.4 Evaluation of Overall System 278
5 Conclusions 279
Acknowledgments 279
References 279
Part III Wireless Connectivity for VideoSensor Networks 281
13 Time Synchronization for Multi-hop Surveillance Camera Systems 282
Abstract 282
1 Introduction 282
2 Related Work 283
3 Wireless Surveillance Camera System for Public Safety 284
4 Time Synchronization Over ZigBee Networks for Surveillance Camera Systems 287
4.1 Time Synchronization Methods 287
4.2 Uncertainty in Time Synchronization 291
4.3 Drift and Correction 296
4.4 Time Representation Error 298
5 Conclusion 302
Acknowledgments 302
References 302
14 Distributed Medium Access for Camera Sensor Networks: Theory and Practice 305
Abstract 305
1 Introduction 306
1.1 Camera Sensor Network 306
1.1.1 Definition and Applications 306
1.1.2 Networking and Data Delivery 308
1.2 Focus of This Chapter 309
1.3 Optimal CSMA 309
1.3.1 Motivation 309
CSMA (Carrier-Sense Multiple Access) 309
Wireless Scheduling: A Rough History 309
1.3.2 Taxonomy 310
Saturate Versus Unsaturated 310
Synchronous Versus Asynchronous 311
Continuous Versus Discrete 311
Time-Varying Channels Versus Static Channels 311
Time-scale Separation Versus not 311
Theory Versus Implementation 311
2 CSMA: A Theoretical Perspective 313
2.1 Model 313
2.2 Objectives 314
2.2.1 Unsaturated System 314
2.2.2 Saturated System 315
3 Optimal CSMA: Survey 315
3.1 Basic Results: Unsaturated 316
3.1.1 Rate-Based Approach 316
3.1.2 Queue-Based Approach 317
3.1.3 Comparison 317
3.2 Basic Results: Saturated 318
3.3 Timescale Separation Assumption 319
3.4 Continuous/Discrete and Synchronous/Asynchronous 320
3.5 Channel: Time-Varying Versus Fixed 321
3.6 Imperfect Sensing and MIMO 321
4 Optimal CSMA: Multi-channel/Multi-radio 322
4.1 Optimal CSMA for Multi-channel/Multi-radio 322
4.1.1 Model and Objective 322
Network Model 322
Feasible Schedule Set and Feasible Rate Region 323
Objective: Saturated Case 323
4.1.2 Optimal CSMA for Multi-channel/Multi-radio 323
Multi-channel/Multi-radio CSMA with (/lambda_{cl} ,b_{cl} ,c /in {/fancyscript{C}},l /in {/fancyscript{L}}) 323
Optimal Algorithm 324
4.2 Delayed CSMA: Virtual Channel Approach 325
4.2.1 Description for Delayed CSMA 325
4.2.2 Delay-Optimality of Delayed CSMA 326
4.2.3 Related Work on Delay Reduction 328
5 Practical Protocol and Implementation 328
5.1 Research on Optimal CSMA Practice 328
5.2 O-DCF 329
5.2.1 System Architecture of O-DCF 329
5.2.2 Key Mechanisms of O-DCF 330
Rate Control 330
Access Aggressiveness Control 331
Adaptive Combination 331
5.2.3 Performance Evaluation 332
6 Summary 334
Acknowledgments 334
References 334
15 Wireless Sensor Network for Video Sensors 337
Abstract 337
1 Introduction to Wireless Sensor Networks 338
1.1 Applications of Wireless Sensor Networks 338
1.2 Review of Existing Wireless Networks 338
1.3 Demands for New Wireless Sensor Networks 341
2 Wireless Sensor Network Topology and Routing Protocols 341
2.1 Review of Wireless Sensor Network Topology 341
2.2 Routing Protocols of Wireless Networks 343
3 Low Power Multi-hop Routing Algorithm for Video Sensor Networks 344
3.1 Properties of Wireless Video Sensor Networks 345
3.2 Utilization-Based Routing and Channel Allocation 346
3.3 Experimental Results of Routing and Channel Allocation 352
3.3.1 Summary of Routing and Channel Allocation of WVSN 355
4 Multi-channel Allocation for Low Power Sensor Networks 355
4.1 Properties of Channel Utilization 355
4.2 Multi-channel Allocation Based on Channel Utilization 358
4.3 Experimental Results of Multi-channel Allocation 361
4.3.1 Summary of Multi-channel Allocation for WVSN 362
Acknowledgments 363
References 363