Computer Vision and Imaging in Intelligent Transportation Systems

Robert P. Loce, Conduent Labs, USA Dr. Robert P. Loce is a Fellow of SPIE and a Senior Member of IEEE. His publications include a book on enhancement and restoration of digital documents, and 8 book chapters on digital halftoning and digital document processing, 28 refereed journal publications, and 53 conference proceedings. He is currently an associate editor for Journal of Electronic Imaging, where he recently guest-edited a special topic issue on the subject matter of the proposed book.  He also chairs a conference within the SPIE/IS&T Electronic Imaging symposium on the subject matter of the proposed book.  He has also been an associate editor for Real-Time Imaging, and IEEE Transactions on Image Processing. Raja Bala, Samsung Research America, USA Dr. Bala has authored over 100 publications, including several book chapters, and holds over 120 U.S. patents in the field of digital and color imaging. He has served as adjunct faculty member at the Rochester Institute of Technology, and has taught many short courses and guest lectures on a variety of topics in digital imaging. From 2008-12, he served as Vice President of Publications for the Society for Imaging Science and Technology, where he led the Editorial Board for the IS&T/Wiley Book Series. He has served as Associate Editor of the Journal of Imaging Science and Technology, and is a frequent reviewer for IEEE Transactions on Image Processing, Journal of Electronic Imaging, and Journal of Imaging Science and Technology. Dr. Bala is a Fellow of IS&T and Senior Member of IEEE. Mohan Trivedi, Jacobs School of Engineering, University of California, San Diego, USA Prof. Mohan Trivedi is the Head of UCSD's Computer Vision and Robotics Research laboratory, overseeing projects such as a robotic, sensor-based traffic-incident monitoring and response system (sponsored by Caltrans). Prof. Trivedi is leading an interdisciplinary effort, as UCSD layer leader for intelligent transportation and telematics for the California Institute for Telecommunications and Information Technology [Cal-(IT)2]. Prof. Trivedi is a recipient of the Pioneer Award and the Meritorious Service Award from the IEEE Computer Society; and the Distinguished Alumnus Award from Utah State University. He is a Fellow of the International Society for Optical Engineering (SPIE). He is a founding member of the Executive Committee of the UC System-wide Digital Media Innovation Program (DiMI). He is also Editor-in-Chief of Machine Vision & Applications (Springer).   

List of Contributors xiii

Preface xvii

Acknowledgments xxi

About the Companion Website xxiii

1 Introduction 1

Raja Bala and Robert P. Loce

1.1 Law Enforcement and Security 1

1.2 Efficiency 4

1.3 Driver Safety and Comfort 5

1.4 A Computer Vision Framework for Transportation Applications 7

1.4.1 Image and Video Capture 8

1.4.2 Data Preprocessing 8

1.4.3 Feature Extraction 9

1.4.4 Inference Engine 10

1.4.5 Data Presentation and Feedback 11

Part I Imaging from the Roadway Infrastructure 15

2 Automated License Plate Recognition 17

Aaron Burry and Vladimir Kozitsky

2.1 Introduction 17

2.2 Core ALPR Technologies 18

2.2.1 License Plate Localization 19

2.2.2 Character Segmentation 24

2.2.3 Character Recognition 28

2.2.4 State Identification 38

3 Vehicle Classification 47

Shashank Deshpande, Wiktor Muron and Yang Cai

3.1 Introduction 47

3.2 Overview of the Algorithms 48

3.3 Existing AVC Methods 48

3.4 LiDAR Imaging-Based 49

3.4.1 LiDAR Sensors 49

3.4.2 Fusion of LiDAR and Vision Sensors 50

3.5 Thermal Imaging-Based 53

3.5.1 Thermal Signatures 53

3.5.2 Intensity Shape-Based 56

3.6 Shape- and Profile-Based 58

3.6.1 Silhouette Measurements 60

3.6.2 Edge-Based Classification 65

3.6.3 Histogram of Oriented Gradients 67

3.6.4 Haar Features 68

3.6.5 Principal Component Analysis 69

3.7 Intrinsic Proportion Model 72

3.8 3D Model-Based Classification 74

3.9 SIFT-Based Classification 74

3.10 Summary 75

4 Detection of Passenger Compartment Violations 81

Orhan Bulan, Beilei Xu, Robert P. Loce and Peter Paul

4.1 Introduction 81

4.2 Sensing within the Passenger Compartment 82

4.2.1 Seat Belt Usage Detection 82

4.2.2 Cell Phone Usage Detection 83

4.2.3 Occupancy Detection 83

4.3 Roadside Imaging 84

4.3.1 Image Acquisition Setup 84

4.3.2 Image Classification Methods 85

4.3.3 Detection-Based Methods 94

5 Detection of Moving Violations 101

Wencheng Wu, Orhan Bulan, Edgar A. Bernal and Robert P. Loce

5.1 Introduction 101

5.2 Detection of Speed Violations 101

5.2.1 Speed Estimation from Monocular Cameras 102

5.2.2 Speed Estimation from Stereo Cameras 108

5.2.3 Discussion 115

5.3 Stop Violations 115

5.3.1 Red Light Cameras 115

5.4 Other Violations 125

5.4.1 Wrong-Way Driver Detection 125

5.4.2 Crossing Solid Lines 126

6 Traffic Flow Analysis 131

Rodrigo Fernandez, Muhammad Haroon Yousaf, Timothy J. Ellis, Zezhi Chen and Sergio A. Velastin

6.1 What is Traffic Flow Analysis? 131

6.1.1 Traffic Conflicts and Traffic Analysis 131

6.1.2 Time Observation 132

6.1.3 Space Observation 133

6.1.4 The Fundamental Equation 133

6.1.5 The Fundamental Diagram 133

6.1.6 Measuring Traffic Variables 134

6.1.7 Road Counts 135

6.1.8 Junction Counts 135

6.1.9 Passenger Counts 136

6.1.10 Pedestrian Counts 136

6.1.11 Speed Measurement 136

6.2 The Use of Video Analysis in Intelligent Transportation Systems 137

6.2.1 Introduction 137

6.2.2 General Framework for Traffic Flow Analysis 137

6.2.3 Application Domains 143

6.3 Measuring Traffic Flow from Roadside CCTV Video 144

6.3.1 Video Analysis Framework 144

6.3.2 Vehicle Detection 146

6.3.3 Background Model 146

6.3.4 Counting Vehicles 149

6.3.5 Tracking 150

6.3.6 Camera Calibration 150

6.3.7 Feature Extraction and Vehicle Classification 152

6.3.8 Lane Detection 153

6.3.9 Results 155

6.4 Some Challenges 156

7 Intersection Monitoring Using Computer Vision Techniques for Capacity, Delay, and Safety Analysis 163

Brendan Tran Morris and Mohammad Shokrolah Shirazi

7.1 Vision-Based Intersection Analysis: Capacity, Delay, and Safety 163

7.1.1 Intersection Monitoring 163

7.1.2 Computer Vision Application 164

7.2 System Overview 165

7.2.1 Tracking Road Users 166

7.2.2 Camera Calibration 169

7.3 Count Analysis 171

7.3.1 Vehicular Counts 171

7.3.2 Nonvehicular Counts 173

7.4 Queue Length Estimation 173

7.4.1 Detection-Based Methods 174

7.4.2 Tracking-Based Methods 175

7.5 Safety Analysis 177

7.5.1 Behaviors 178

7.5.2 Accidents 182

7.5.3 Conflicts 185

7.6 Challenging Problems and Perspectives 187

7.6.1 Robust Detection and Tracking 187

7.6.2 Validity of Prediction Models for Conflict and Collisions 188

7.6.3 Cooperating Sensing Modalities 189

7.6.4 Networked Traffic Monitoring Systems 189

7.7 Conclusion 189

8 Video-Based Parking Management 195

Oliver Sidla and Yuriy Lipetski

8.1 Introduction 195

8.2 Overview of Parking Sensors 197

8.3 Introduction to Vehicle Occupancy Detection Methods 200

8.4 Monocular Vehicle Detection 200

8.4.1 Advantages of Simple 2D Vehicle Detection 200

8.4.2 Background Model–Based Approaches 200

8.4.3 Vehicle Detection Using Local Feature Descriptors 202

8.4.4 Appearance-Based Vehicle Detection 203

8.4.5 Histograms of Oriented Gradients 204

8.4.6 LBP Features and LBP Histograms 207

8.4.7 Combining Detectors into Cascades and Complex Descriptors 208

8.4.8 Case Study: Parking Space Monitoring Using a Combined Feature Detector 208

8.4.9 Detection Using Artificial Neural Networks 211

8.5 Introduction to Vehicle Detection with 3D Methods 213

8.6 Stereo Vision Methods 215

8.6.1 Introduction to Stereo Methods 215

8.6.2 Limits on the Accuracy of Stereo Reconstruction 216

8.6.3 Computing the Stereo Correspondence 217

8.6.4 Simple Stereo for Volume Occupation Measurement 218

8.6.5 A Practical System for Parking Space Monitoring Using a Stereo System 218

8.6.6 Detection Methods Using Sparse 3D Reconstruction 220

9 Video Anomaly Detection 227

Raja Bala and Vishal Monga

9.1 Introduction 227

9.2 Event Encoding 228

9.2.1 Trajectory Descriptors 229

9.2.2 Spatiotemporal Descriptors 231

9.3 Anomaly Detection Models 233

9.3.1 Classification Methods 233

9.3.2 Hidden Markov Models 234

9.3.3 Contextual Methods 234

9.4 Sparse Representation Methods for Robust Video Anomaly Detection 236

9.4.1 Structured Anomaly Detection 237

9.4.2 Unstructured Video Anomaly Detection 243

9.4.3 Experimental Setup and Results 245

9.5 Conclusion and Future Research 253

Part II Imaging from and within the Vehicle 257

10 Pedestrian Detection 259

Shashank Deshpande and Yang Cai

10.1 Introduction 259

10.2 Overview of the Algorithms 259

10.3 Thermal Imaging 260

10.4 Background Subtraction Methods 261

10.4.1 Frame Subtraction 261

10.4.2 Approximate Median 262

10.4.3 Gaussian Mixture Model 263

10.5 Polar Coordinate Profile 263

10.6 Image-Based Features 265

10.6.1 Histogram of Oriented Gradients 265

10.6.2 Deformable Parts Model 266

10.6.3 LiDAR and Camera Fusion–Based Detection 266

10.7 LiDAR Features 268

10.7.1 Preprocessing Module 268

10.7.2 Feature Extraction Module 268

10.7.3 Fusion Module 268

10.7.4 LIPD Dataset 270

10.7.5 Overview of the Algorithm 270

10.7.6 LiDAR Module 272

10.7.7 Vision Module 275

10.7.8 Results and Discussion 276

10.7.8.1 LiDAR Module 276

10.7.8.2 Vision Module 276

10.8 Summary 280

11 Lane Detection and Tracking Problems in Lane Departure Warning Systems 283

Gianni Cario, Alessandro Casavola and Marco Lupia

11.1 Introduction 283

11.2 LD: Algorithms for a Single Frame 285

11.2.1 Image Preprocessing 285

11.2.2 Edge Extraction 287

11.2.3 Stripe Identification 291

11.2.4 Line Fitting 294

11.3 LT Algorithms 297

11.3.1 Recursive Filters on Subsequent N frames 298

11.3.2 Kalman Filter 298

11.4 Implementation of an LD and LT Algorithm 299

11.4.1 Simulations 300

11.4.2 Test Driving Scenario 300

11.4.3 Driving Scenario: Lane Departures at Increasing Longitudinal Speed 300

11.4.4 The Proposed Algorithm 302

11.4.5 Conclusions 303

12 Vision-Based Integrated Techniques for Collision Avoidance Systems 305

Ravi Satzoda and Mohan Trivedi

12.1 Introduction 305

12.2 Related Work 307

12.3 Context Definition for Integrated Approach 307

12.4 ELVIS: Proposed Integrated Approach 308

12.4.1 Vehicle Detection Using Lane Information 309

12.4.2 Improving Lane Detection using On-Road Vehicle Information 312

12.5 Performance Evaluation 313

12.5.1 Vehicle Detection in ELVIS 313

12.5.2 Lane Detection in ELVIS 316

12.6 Concluding Remarks 319

13 Driver Monitoring 321

Raja Bala and Edgar A. Bernal

13.1 Introduction 321

13.2 Video Acquisition 322

13.3 Face Detection and Alignment 323

13.4 Eye Detection and Analysis 325

13.5 Head Pose and Gaze Estimation 326

13.5.1 Head Pose Estimation 326

13.5.2 Gaze Estimation 328

13.6 Facial Expression Analysis 332

13.7 Multimodal Sensing and Fusion 334

13.8 Conclusions and Future Directions 336

14 Traffic Sign Detection and Recognition 343

Hasan Fleyeh

14.1 Introduction 343

14.2 Traffic Signs 344

14.2.1 The European Road and Traffic Signs 344

14.2.2 The American Road and Traffic Signs 347

14.3 Traffic Sign Recognition 347

14.4 Traffic Sign Recognition Applications 348

14.5 Potential Challenges 349

14.6 Traffic Sign Recognition System Design 349

14.6.1 Traffic Signs Datasets 352

14.6.2 Colour Segmentation 354

14.6.3 Traffic Sign's Rim Analysis 359

14.6.4 Pictogram Extraction 364

14.6.5 Pictogram Classification Using Features 365

14.7 Working Systems 369

15 Road Condition Monitoring 375

Matti Kutila, Pasi Pyykonen, Johan Casselgren and Patrik Jonsson

15.1 Introduction 375

15.2 Measurement Principles 376

15.3 Sensor Solutions 377

15.3.1 Camera-Based Friction Estimation Systems 377

15.3.2 Pavement Sensors 379

15.3.3 Spectroscopy 380

15.3.4 Roadside Fog Sensing 382

15.3.5 In-Vehicle Sensors 383

15.4 Classification and Sensor Fusion 386

15.5 Field Studies 390

15.6 Cooperative Road Weather Services 394

15.7 Discussion and Future Work 395

Index 399