Image, Video and 3D Data Registration

Vasileios Argyriou, School of Computing and Information Systems, Faculty of Science, Engineering and Computing University of Kingston, UK Dr. Argyriou is a Senior Lecturer of Computing Information Systems and Mathematics at Kingston University. His research focuses on computer vision and computer games. Dr. Argyriou is a committee member of BSI British Standards working on the new global standard for biometric IDs and 3D face information. Dr. Argyriou conceived, ?organised and chaired the first, second and third international workshops on Computer Vision for Computer Games (CVCG) part of the CVPR 2010-2012 conferences to bring together game industry and researchers on video processing, action recognition, real time systems and 3D scene reconstruction. Georgios Tzimiropoulos, School of Computer Science, University of Lincoln, UK Dr. Tzimiropoulos is a Senior Lecturer in the School of Computer Science at the University of Lincoln, U.K. He is currently an Associate Editor of the Image and Vision Computing Journal. He also serves as an Area Chair for the tenth IEEE Conference on Automatic Face and Gesture Recognition. His research interests include face and object recognition, alignment and tracking, and facial expression analysis. He is a member of the IEEE. Barbara Villarini, Advanced Research Technology ART Group, Italy Dr. Villarini is a researcher and developer at Advanced Research Technology ART group s.r.l working on innovative technologies to manage and optimize digital cinema from small cinema to multiplexing. She has participated in a number of EU and national projects, including the European Project EDCine in IST FP6, working on the improvement and achievement of the interoperability of Digital Cinema based on JPEG 2000 coding.

Preface xi

Acknowledgements xiii

1 Introduction 1

1.1 The History of Image Registration 1

1.2 Definition of Registration 2

1.3 What is Motion Estimation 3

1.4 Video Quality Assessment 5

1.5 Applications 5

1.5.1 Video Processing 5

1.5.2 Medical Applications 7

1.5.3 Security Applications 8

1.5.4 Military and Satellite Applications 10

1.5.5 Reconstruction Applications 11

1.6 Organization of the Book 12

References 13

2 Registration for Video Coding 15

2.1 Introduction 15

2.2 Motion Estimation Technique 16

2.2.1 Block-Based Motion Estimation Techniques 16

2.3 Registration and Standards for Video Coding 30

2.3.1 H.264 30

2.3.2 H.265 34

2.4 Evaluation Criteria 35

2.4.1 Dataset 35

2.4.2 Motion-Compensated Prediction Error (MCPE) in dB 38

2.4.3 Entropy in bpp 39

2.4.4 Angular Error in Degrees 40

2.5 Objective Quality Assessment 41

2.5.1 Full-Reference Quality Assessment 41

2.5.2 No-Reference and Reduced-Reference Quality Metrics 44

2.5.3 Temporal Masking in Video Quality Assessment 46

2.6 Conclusion 48

2.7 Exercises 49

References 49

3 Registration for Motion Estimation and Object Tracking 53

3.1 Introduction 53

3.1.1 Mathematical Notation 54

3.2 Optical Flow 55

3.2.1 Horn–Schunk Method 56

3.2.2 Lukas–Kanade Method 56

3.2.3 Applications of Optical Flow for Motion Estimation 57

3.3 Efficient Discriminative Features for Motion Estimation 61

3.3.1 Invariant Features 62

3.3.2 Optimization Stage 64

3.4 Object Tracking 64

3.4.1 KLT Tracking 64

3.4.2 Motion Filtering 66

3.4.3 Multiple Object Tracking 67

3.5 Evaluating Motion Estimation and Tracking 68

3.5.1 Metrics for Motion Detection 68

3.5.2 Metrics for Motion Tracking 69

3.5.3 Metrics for Efficiency 70

3.5.4 Datasets 70

3.6 Conclusion 70

3.7 Exercise 75

References 75

4 Face Alignment and Recognition Using Registration 79

4.1 Introduction 79

4.2 Unsupervised Alignment Methods 80

4.2.1 Natural Features: Gradient Features 81

4.2.2 Dense Grids: Non-rigid Non-affine Transformations 81

4.3 Supervised Alignment Methods 83

4.3.1 Generative Models 84

4.3.2 Discriminative Approaches 86

4.4 3D Alignment 88

4.4.1 Hausdorff Distance Matching 88

4.4.2 Iterative Closest Point (ICP) 89

4.4.3 Multistage Alignment 89

4.5 Metrics for Evaluation 90

4.5.1 Evaluating Face Recognition 90

4.5.2 Evaluating Face Alignment 90

4.5.3 Testing Protocols and Benchmarks 91

4.5.4 Datasets 92

4.6 Conclusion 94

4.7 Exercise 94

References 94

5 Remote Sensing Image Registration in the Frequency Domain 97

5.1 Introduction 97

5.2 Challenges in Remote Sensing Imaging 100

5.3 Satellite Image Registration in the Fourier Domain 102

5.3.1 Translation Estimation Using Correlation 102

5.4 Correlation Methods 103

5.5 Subpixel Shift Estimation in the Fourier Domain 107

5.6 FFT-Based Scale-Invariant Image Registration 111

5.7 Motion Estimation in the Frequency Domain for Remote Sensing Image Sequences 115

5.7.1 Quad-Tree Phase Correlation 116

5.7.2 Shape Adaptive Motion Estimation in the Frequency Domain 119

5.7.3 Optical Flow in the Fourier Domain 120

5.8 Evaluation Process and Related Datasets 122

5.8.1 Remote Sensing Image Datasets 123

5.9 Conclusion 123

5.10 Exercise – Practice 124

References 124

6 Structure from Motion 129

6.1 Introduction 129

6.2 Pinhole Model 131

6.3 Camera Calibration 133

6.4 Correspondence Problem 135

6.5 Epipolar Geometry 136

6.6 Projection Matrix Recovery 140

6.6.1 Triangulation 141

6.7 Feature Detection and Registration 141

6.7.1 Auto-correlation 143

6.7.2 Harris Detector 143

6.7.3 SIFT Feature Detector 146

6.8 Reconstruction of 3D Structure and Motion 148

6.8.1 Simultaneous Localization and Mapping 149

6.8.2 Registration for Panoramic View 150

6.9 Metrics and Datasets 152

6.9.1 Datasets for Performance Evaluation 154

6.10 Conclusion 155

6.11 Exercise – Practice 155

References 155

7 Medical Image Registration Measures 162

7.1 Introduction 162

7.2 Feature-Based Registration 163

7.2.1 Generalized Iterative Closest Point Algorithm 164

7.2.2 Hierarchical Maximization 165

7.3 Intensity-Based Registration 165

7.3.1 Voxels as Features 166

7.3.2 Special Case: Spatially Determined Correspondences 168

7.3.3 Intensity Difference Measures 169

7.3.4 Correlation Coefficient 170

7.3.5 Pseudo-likelihood Measures 171

7.3.6 General Implementation Using Joint Histograms 181

7.4 Transformation Spaces and Optimization 184

7.4.1 Rigid Transformations 185

7.4.2 Similarity Transformations 186

7.4.3 Affine Transformations 186

7.4.4 Projective Transformations 187

7.4.5 Polyaffine Transformations 187

7.4.6 Free-Form Transformations: ‘Small Deformation’ Model 188

7.4.7 Free-Form Transformations: ‘Large Deformation’ Models 189

7.5 Conclusion 193

7.6 Exercise 193

7.6.1 Implementation Guidelines 195

References 196

8 Video Restoration Using Motion Information 201

8.1 Introduction 201

8.2 History of Video and Film Restoration 203

8.3 Restoration of Video Noise and Grain 206

8.4 Restoration Algorithms for Video Noise 208

8.5 Instability Correction Using Registration 211

8.6 Estimating and Removing Flickering 214

8.7 Dirt Removal in Video Sequences 217

8.8 Metrics in Video Restoration 221

8.9 Conclusions 225

8.10 Exercise – Practice 225

References 225

Index 229