Handbook of Camera Monitor Systems (eBook)

Professor Dr. Anestis Terzis is the head of the Automotive Electronics study course at Ulm University of Applied Sciences in Germany and the coordinator of the university’s International Electrical Engineering Program (IEEP). Prior to this, he was with Daimler AG for ten years and worked in the “Group Research & Mercedes-Benz Cars Development,” with a main focus on future advanced driver assistance systems.Born 1978 in Heidenheim, Germany, Professor Terzis received his diploma in Communications Engineering from Ulm University of Applied Sciences. He also holds a doctoral degree in Electrical Engineering from the University of Erlangen-Nuremberg, Germany. He is one of the authors of the ISO 16505 content and an expert member of the standardization and regulation committees in the field of camera monitor systems.He left the Daimler AG Research Center in 2012 to become a professor for Digital Systems Design at Ulm University of Applied Sciences. His lectures include digital technology with VHDL and FPGAs, fundamentals of electrical engineering, and automotive systems. His primary research field is advanced camera-based driver assistance systems.Professor Terzis combines academic and industrial experience. He is the founder and the director of the Steinbeis Transfer Center “DSI - Digital Systems and Innovations,” a company in the Steinbeis Network and based in Ulm, Germany that offers consulting, as well as courses, prototype development and measurement technologies (www.stw.de/su/1637). His contact information can be found at www.hs-ulm.de/terzis.

Preface 7
Acknowledgments 9
Contents 11
Editor and Contributors 13
Part ICMS System Design and Standardizationand Regulation Aspects 16
1 Automotive Mirror-Replacement by Camera Monitor Systems 17
Abstract 17
1 Introduction 18
1.1 Normative and Regulatory Framework 19
1.1.1 Scope and Structure of ISO 16505 20
1.1.2 The Way of the Amendment for UN Regulation No. 46 21
1.2 Characteristics of Conventional Vehicle Mirrors 22
2 Potential for Mirror-Replacement 27
2.1 Potential Benefits 28
2.1.1 Aerodynamic and Fuel Economy Aspects 28
2.1.2 Potentials for Vision 32
2.2 Potential Challenges 35
3 Overview of Camera Monitor Systems 37
3.1 Base Architecture of a CMS 37
3.2 Advanced Architecture of a CMS 40
4 CMS Requirements 42
4.1 Relevant Standards and Regulations 43
4.2 Specific CMS Requirements Based on ISO 16505 and UN R.46 47
5 CMS System Design Based on ISO 16505 51
5.1 General Considerations and Definitions 51
5.2 Parameters for a Class III CMS 53
5.3 Architecture and Components of a CMS 57
5.3.1 Camera of a CMS 57
5.3.2 Display of a CMS 59
5.3.3 Processing Components for a CMS 60
6 Conclusion 63
References 63
2 Standardization and Vehicle Regulation Aspects of Camera Monitor Systems 65
Abstract 65
1 Standards and Regulations 68
1.1 Requirements for Vehicles 68
1.2 Standards 69
1.3 Developing ISO Standards 70
1.4 Developing ISO Standard 16505 72
1.5 Vehicle Regulations and Certification 74
1.6 Developing UN Regulations 75
1.7 Developing UN Regulation No. 46 Towards CMS 78
2 Regulatory Situation for CMS in Regulations Other Than UN-R 46 81
2.1 General Aspects 81
2.2 Camera Monitor Systems in China 82
2.3 Camera Monitor Systems in Japan 82
2.4 Camera Monitor Systems in South Korea 83
2.5 Camera Monitor Systems in the USA and in Canada 83
2.6 Exemption Procedures 84
2.6.1 Europe 84
2.6.2 USA 86
2.7 Summary Current Regulatory Situation 87
3 Requirements for CMS in UN Regulation No. 46 87
3.1 General Aspects 87
3.2 Requirements for Class I–IV Camera Monitor Systems 91
3.2.1 Mechanical Requirements 92
3.2.2 Functional Component Requirements 93
3.2.3 Installation Requirements 96
4 Other Relevant UN Regulations for CMS 100
4.1 UN Regulation No. 10 (Electromagnetic Compatibility) 100
4.2 UN Regulation No. 21 (Interior Fittings) 100
4.3 UN Regulation No. 48 (Installation of Lighting and Light-Signaling Devices) 101
4.4 UN Regulation No. 95 (Lateral Collision Protection) 101
4.5 UN Regulation No. 125 (Forward Field of Vision) 102
5 Conclusion and Future Activities 105
Annex A 107
Annex B 109
References 111
3 Resolution and Sharpness Requirements for CMS 115
Abstract 115
1 Introduction 115
2 Modulation Transfer Function, Definition of MTF10 and MTF50 117
3 Image Height and Measuring Units 118
4 Specific Use of the [LW/PH] Unit Within ISO16505:2015 120
5 CMS Resolution (MTF) Requirement 123
6 Measuring the Spatial Frequency 125
7 Evaluation Procedure Depending on the System’s Performance 130
8 Measurement of Resolution and Sharpness 132
8.1 Single Hyperbolic Chart Versus ISO 12233:2014 Standard Chart 132
8.2 Introduction of the Single Hyperbolic Chart 133
8.3 Spatial Frequency of the Chart at Reference Position 1 and Measurement Procedure 135
8.4 Preparation and Use of the Parallel Line Chart 140
8.5 Using SFR Method for Measuring the Resolution on Image Displayed on the Monitor 141
9 Sharpness 143
10 Others 145
11 Conclusions 146
References 146
4 Vision in Commercial Vehicles with Respect to Camera Monitor Systems 147
Abstract 147
1 Introduction 148
2 Relation Between Direct and Indirect Vision in Commercial Vehicles 149
3 Use Cases for Commercial Vehicles 152
3.1 General Categories of Use Cases 152
3.2 Driving-Related Use Cases 153
3.3 Use Cases not Related to Driving 156
3.3.1 Ordinary Entry in Order to Take Off 157
3.3.2 Ordinary Exit After Coming to a Halt 157
3.3.3 Getting Out After a Longer Time Period at Standstill 158
3.3.4 Taking off Directly After a Longer Time Period at Standstill 158
3.3.5 Getting Out of the Vehicle While the Engine Is Still Running 158
4 Categories of Commercial Vehicles in Relation to Requirement Fulfilment 159
5 Fields of View—Direct and Indirect Vision 162
5.1 Fields of View Provided by Indirect Vision—A Global Overview 163
5.2 Minimum Areas on Ground and Vertical Fields of View 163
5.2.1 Minimum Fields of View According to UN Regulation No 46 [8] 163
5.2.2 Minimum Fields of View for Commercial Vehicles in Japan [9] 163
6 Certain Driving Situations that Require Different Fields of View 164
6.1 Procedure for Establishing the Required Changed Fields of View 167
6.2 Aspects of the Changed Fields of View in Need of Requirements 170
6.2.1 Total Size of the Changed Fields of View 170
6.2.2 Area Coverage by Changed Magnification and Panning 170
6.2.3 Timing Considerations and Changed Fields of View 172
6.2.4 User Interaction and Interface When Changed Fields of View are Provided 174
7 Aspects Behind Indirect Vision View Integration in Commercial Vehicles 175
7.1 Background of Actual Mirror Positions on Commercial Vehicles 175
7.2 Considerations for Positioning Cameras onto Commercial Vehicles 177
7.2.1 Provided Fields of View 177
7.2.2 Positions Compatible with Different Vehicle Variants 177
7.2.3 Driver’s Understanding of Depth and Speed 178
7.2.4 Negative Effects from Direct Light into the Cameras 178
7.2.5 Problems of Soiling 178
7.2.6 Common Rule that Solves Most Problems 178
7.3 Monitor Integration Inside the Vehicle 179
7.3.1 Need for Eye Movements While Monitoring Vision Views 180
7.3.2 Obstruction of Direct Vision 180
7.3.3 Sensitivity to Incoming Light 180
7.3.4 Risk of Glare and Other Disturbances in Dark Conditions 181
8 Key Considerations for Future Development of CMS Replacing Rear-View Mirrors 182
8.1 Field of View Integration and Simplified HMI 182
8.2 Opportunity for Common Requirements that Can Be Covered by Either Indirect or Direct Vision 182
8.3 Communication with and Information to Other Road Users 183
8.4 Available Locations for Camera Installations 184
8.5 Developed Content of Driver Tasks 185
9 Conclusion 185
References 186
Part IIFundamentals of AutomotiveTechnology for CMS 187
5 Image Sensors for Camera Monitor Systems 188
Abstract 188
1 Introduction 189
2 Image Capture 189
3 Image Processing 197
4 Imaging System Measurements 201
5 High Dynamic Range Imaging 204
6 Light Source Flicker 208
7 Representative System Solution 210
8 Safety 212
9 Conclusion 213
References 213
6 Optical Effects in Camera Monitor Systems 215
Abstract 215
1 Introduction and Groundwork 216
1.1 Photometry Versus Radiometry 216
1.2 Review of Photometric Quantities 217
1.3 Notation Conventions 220
1.4 Luminance as the Crucial Quantity 220
1.5 Purpose of a Camera-Monitor-System 222
1.6 Recap—Summary of the Fundamentals 223
2 Stray Light Artifacts 224
2.1 Origins and Occurrence 224
2.2 Types of Flares 226
2.2.1 Veiling Glare 227
2.2.2 Directed Flares 229
2.2.3 Aperture Ghosts 231
2.2.4 Ghost Images 232
2.3 Possible Remedies for Some Types of Flares 233
2.4 Test Scenarios 234
2.5 Recap—Types of Flares and Test Setup Requirements 237
3 Propositions for Test Setups and Procedures 237
3.1 Specifications 237
3.1.1 System Accessibility and Boundary Conditions 238
3.1.2 Interfacing 238
3.1.3 Display Properties 238
3.1.4 Design of the Diagnostic Camera 240
3.1.5 Target Quantities 240
3.1.6 Elements of the Setup 242
3.1.7 Glare Sources 242
3.2 Sample Setup 243
3.3 Positions to Measure Lblack 244
3.4 Test Procedures 244
4 Summary and Conclusion 245
Acknowledgments 246
References 246
7 Camera-Monitor-Systems as Solution for the Automotive Market 247
Abstract 247
1 Introduction 247
2 A Single Chip for Camera-Based Driver Assistance 248
3 Remote Graphic Processing Unit for Scalable Systems 251
4 Graphic Controller with Control Functions 252
5 Flexible System Architecture 254
6 Integrated Systems 256
7 Socionext Unveils a Video and Communication Bridge Providing Interconnection Between an Application Processor and Automotive Interfaces 257
8 Multiple Applications with One APIX® Connection 259
9 Conclusion 264
References 264
8 Video Interface Technology 265
Abstract 265
1 Introduction 267
2 Video Interface Basics 268
3 Automotive Solutions 275
3.1 Analog Cameras 275
3.2 Serializer- and Deserializer-Based Camera Interfaces 276
3.3 Network-Based Camera Interfaces 280
3.4 Camera API 284
4 Conclusion 284
Annex: Standards 286
References 286
Part IIIHuman Visual Perceptionand Ergonomic Design 288
9 Human Visual Perception 289
Abstract 289
1 Introduction 289
2 The Human Eye 291
3 Human Visual System 292
3.1 The Retina 292
3.2 The Lateral Geniculate Nuclei 297
3.3 The Visual Cortex 297
3.4 Summary 300
4 Motion Blur in Human Vision 301
4.1 Inertia of Photoisomerization 301
4.2 Neuronal Principles of Motion Sensation 302
4.3 Sensation of Movement in Still Images 303
5 Feature and Object Perception 303
5.1 Feature Extraction 304
5.2 Image Segmentation 304
5.3 Object Representation 305
5.4 From Object Perception to Mental Scene Representation 305
6 Visual Search-and-Find 306
6.1 Feature Integration Theory 306
6.2 Guided Search 308
6.3 Speeding up Visual Search-and-Find 309
7 Human Depth Perception 309
7.1 Binocular Disparity 309
7.2 Monocular Depth Perception 311
7.3 Cue Combination 317
8 Conclusion 318
References 318
10 Camera Monitor Systems Optimized on Human Cognition—Fundamentals of Optical Perception and Requirements for Mirror Replacements in Commercial Vehicles 323
Abstract 323
1 Fields of View in Commercial Vehicles 324
2 Foveal and Peripheral Vision: The Cornerstones of Human Sight 325
3 Human Perception as Direct and Indirect Vision in Commercial Vehicles 329
4 Design of Camera Monitor Systems on Commercial Vehicles 332
5 Situation-Adapted Imaging—The “Maneuvering View” 335
6 Summary 336
References 337
11 Ergonomic Design of Camera-Monitor Systems in Heavy Commercial Vehicles 339
Abstract 339
1 Introduction 340
1.1 Motivation 340
1.1.1 Transportation Task of Commercial Vehicles 341
1.1.2 CO2 Reduction Targets as a Trailblazer for New Technologies 344
1.1.3 Establishment of Camera-Monitor Systems 345
1.1.4 Dynamic Vision Interaction with Conventional Mirror Systems 347
1.2 Scope of the Study Content 350
2 Fundamentals 351
2.1 Ergonomic Aspects of the Vision from the Vehicle 351
2.1.1 Information Acquisition and Processing 352
2.1.2 Characteristics of the Human Eye 353
2.1.3 Vision Task and Vision Restriction for Commercial Vehicles 354
2.2 Design of Camera-Monitor Systems 357
2.2.1 Ergonomic Design Process 359
2.2.2 Technology-Related Design Parameters 360
2.2.3 Defining the Conceptual Design Parameters 360
2.2.4 Viewing Angle-Dependent Display Quality of the Monitor 364
2.2.5 Situation-Dependent Adaptation of the Displayed Vision Area 364
3 Ergonomic Design of Camera-Monitor Systems 367
3.1 Monitor Alignment on the Dynamic Eye Point Positions 367
3.2 Panning Characteristic of the Camera-Monitor Systems 368
4 Conclusion 371
References 371
Standards, Laws and Regulations 375
Part IVCMS Tests and Conceptsfor Passenger Cars andfor Commercial Vehicles 376
12 Camera-Monitor Systems as a Replacement for Exterior Mirrors in Cars and Trucks 377
Abstract 377
1 Introduction 378
2 Literature Analysis 378
2.1 Technical Background 378
2.2 Human-Machine Interaction 379
2.2.1 Glance Behaviour in Real Traffic Situations 379
2.2.2 Glance Behaviour During Lane-Changing 380
2.2.3 Distance and Speed Perception in Road Traffic 381
2.2.4 Distance and Speed Perception for Exterior Mirrors and Displays 382
3 Technical Aspects 382
3.1 Test Vehicles 383
3.1.1 Cars 383
3.1.2 Trucks 385
3.2 Test Concept 386
3.3 Mirror and CMS Properties 387
3.4 Fundamentals of Optical Image Effects 388
3.5 Tests and Results 389
3.5.1 Rear Field of Vision and Direct View Forward 389
3.5.2 General Day and Night Properties 391
3.5.3 Image Reproduction 398
3.5.4 Behaviour in Glare 403
3.5.5 Reflection on the Display and Screen Glare 406
3.5.6 Adjustability of the Camera and Display 407
3.5.7 Failure Safety 408
3.5.8 Behaviour in Extreme Cold and Heat 409
3.5.9 Effects of Soiling 413
4 Aspects of Human-Machine Interaction 415
4.1 Car Study 415
4.1.1 Sample 416
4.1.2 Test Procedure 416
4.1.3 Experiment I: Distance and Speed Perception 416
4.1.4 Experiment II: Glance Behaviour During Real Drives 419
4.1.5 Results 421
4.2 Truck Study 427
4.2.1 Sample 427
4.2.2 Test Procedure 427
4.2.3 Experiment I: Distance Estimation 428
4.2.4 Experiment II: Commented Drives in Real Traffic 429
4.2.5 Results 430
5 Discussion of the Results 435
5.1 Technical Aspects 435
5.2 Aspects of Human-Machine Interaction 436
6 Conclusions and Recommendations 439
References 441
13 CMS Concept for Commercial Vehicles: Optimized Fuel Efficiency and Increased Safe Mobility 444
Abstract 444
1 Automotive Megatrends 445
1.1 Safe Mobility 445
1.2 Clean Mobility 447
2 Challenges 447
2.1 Providing the Right Information at the Right Time 447
2.2 Technology 448
3 Solution 450
3.1 System Architecture 450
3.2 Display 451
3.3 Camera 452
3.4 Features 453
3.5 Variations 454
4 Conclusion 455
References 455
Part VAdvanced Topics 457
14 Optimization of Demanding Scenarios in CMS and Image Quality Criteria 458
Abstract 458
1 Introduction 459
2 HDR Imaging and Display 460
2.1 HDR Image Acquisition 461
2.1.1 Capturing Multiple Exposures 462
2.1.2 Image Matching and Alignment 462
2.1.3 Camera Response Curve 462
2.1.4 Weight Factoring 463
2.1.5 The HDR Radiance Map 463
2.2 HDR Image Display 464
2.2.1 Photoreceptor Adaptations 464
2.2.2 Perception Based Tone Reproduction 465
3 Prototype Camera Setup 465
3.1 Image Sensor 466
3.1.1 Image Sensor Architecture 466
3.2 Vehicle Prototype Setup 469
3.2.1 ADTF 469
3.2.2 ADTF Toolchain 470
4 Image Quality Issues 471
4.1 Noise 471
4.1.1 Photon Shot Noise 471
4.1.2 Dark Current Noise 472
4.1.3 Reset and Thermal Noise 472
4.1.4 Quantization Noise 472
4.2 Color Reproduction 472
4.3 The Point Light Sources Issue 473
5 Image Quality Criteria 474
5.1 Ground Truth of an Image 474
5.2 Average Entropy 475
5.3 Peak Signal-to-Noise Ratio 476
5.4 Average Luminance 476
5.5 Average Contrast 477
5.6 Color Rendering 478
5.7 Point Light Sources Measurement 478
5.8 Sharpness 479
5.8.1 Exemplary Results 480
5.9 Examples 480
6 Evaluation and Results 482
6.1 Optimization Approaches 482
6.1.1 Noise and Color Rendering 483
6.1.2 Point Light Sources Optimization Techniques 484
6.2 Manual Camera Settings Results and Optimization 484
6.2.1 Linear Light Increase 484
6.2.2 Linear Gain Increase 485
6.3 Automatic Camera Settings Results and Optimization 486
6.3.1 Noise and Color Rendering 486
6.3.2 Point Light Sources Optimization 487
7 Conclusion 488
References 489
15 Intuitive Motion and Depth Visualization for Rear-View Camera Applications 490
Abstract 490
1 Monocular Spatial and Temporal Scene Reconstruction on Highways 491
1.1 Related Work 492
1.2 Road Course Reconstruction 496
1.3 Vehicle Detection and Segmentation 497
1.4 Distance and Velocity Estimation 500
1.5 Monocular Scene Reconstruction Results and Evaluation 501
2 Artificial Depth-of-Field Rendering 502
2.1 Mathematical Description of Approximated Defocus Blur 502
3 Distance-Adaptive Desaturation 506
4 Motion and Distance Visualization for Rear-View Camera Applications 506
4.1 Exaggerated Artificial Motion Blur Rendering 508
4.2 Risk Potential Visualization 509
4.3 Perceptual Study on Exaggerated Motion Blur 510
5 Conclusion 512
References 513
16 Functional Safety of Camera Monitor Systems 516
Abstract 516
1 Safety Relevance of Camera Monitor Systems 517
2 Particularities of CMS in Comparison to Other Safety-Related Systems 519
3 Interplay of Nominal Performance and Safety 521
4 ISO 26262 Requirements for the Development of Safety Relevant Systems 522
5 Contents of an Item Definition 524
6 Identifying and Classifying Hazards 526
7 Typical Hazards of CMS 529
8 Recommended Types of Safety Analyses 530
9 Recommended Content and Structure of a Safety Concept 531
10 Design and Implementation of Software and Hardware 534
11 Particularities Regarding Verification and Validation 535
12 Additional Aspects for CMS Connected to Driver Assistance Systems 536
13 Conclusion 538
References 539