Facial Analysis from Continuous Video with Applications to Human-Computer Interface (eBook)

Contents 7
List of Figures 11
List of Tables 17
Preface 22
Acknowledgments 23
Chapter 1 INTRODUCTION 25
1. Facial Analysis for Human-Computer Interface 26
Chapter 2 INFORMATION-BASED MAXIMUM DISCRIMINATION 29
1. The Bayes Classifier 29
1.1 Two-class discrimination problem 30
2. Information Theoretic-Based Learning 31
2.1 Kullback-Leibler divergence 31
2.2 Nonparametric probability models 32
2.3 Learning procedure 33
2.4 Error bootstrapping 37
3. Fast Classification 38
Chapter 3 FACE AND FACIAL FEATURE DETECTION 39
1. Previous Approaches 40
2. IBMD Face and Facial Feature Detection 41
2.1 Multi-scale detection of faces 41
2.2 Facial feature detection 44
3. Discussion 46
Chapter 4 FACE AND FACIAL FEATURE TRACKING 49
1. Visual Object Tracking and Motion Analysis 49
2. Previous Approaches 51
3. Face and Facial Feature Tracking 52
3.1 Feature matching 52
3.2 Model fitting 53
3.3 Initialization 53
4. Global Head Position Analysis 56
Chapter 5 FACE AND FACIAL EXPRESSION RECOGNITION 59
1. Previous Approaches 60
1.1 Face recognition 60
1.2 Facial expression analysis 61
2. Modelling Faces by Facial Features 62
2.1 Class-conditional probability of the feature images 64
3. Modeling Temporal Information of Facial Expressions 65
Chapter 6 3-D MODEL-BASED IMAGE COMMUNICATION 67
1. Introduction 67
1.1 The Model-Based Approach 67
1.2 Model-Based Analysis 68
1.3 Model-Based Coding 69
1.4 Virtual Agent 69
2. Modelling and Analysis 70
2.1 Geometric Face Modelling 71
2.1.1 3D Surface Modelling 71
2.1.2 Generic Face Model from MRI Data 72
2.1.3 Face Geometric Model Fitting 73
2.1.4 Face Geometric Model Compression 75
2.2 Facial Articulation Modelling 76
2.2.1 Articulation Parameters 76
2.2.2 Model Deformation 77
2.2.3 Articulation Parameter Stream Compression 79
2.3 Synthesis 79
2.3.1 Environment Model 79
2.3.2 Texture Mapping 80
3. Analysis 81
3.1 Review of Past and Current Work 81
3.1.1 MBVC systems 82
3.1.2 Rigid Motion 82
3.1.3 Non-Rigid Motion 83
3.2 Tracking Rigid Global Motion 84
3.2.1 2D-3D Pose Estimation 84
3.2.2 Pose Optimization 86
3.2.3 Filtering and Prediction 87
3.3 Tracking Non-Rigid Local Motion 88
3.3.1 2D Motion + Constraints 89
3.3.2 Model-Based Non-Rigid Local Motion Estimation 91
4. Model-based Video System Implementation 92
4.1 Tracking Non-Rigid Local Motion 93
4.2 Tracking System 94
4.2.1 Initialization 95
4.2.2 Tracking Loop 95
4.2.3 Head Modelling 96
4.3 Practical Implementation Issues 96
Chapter 7 IMPLEMENTATIONS, EXPERIMENTS AND RESULTS 99
1. Image and Video Databases 99
1.1 FERET database 99
1.2 CMU/MIT database for face detection 100
1.3 Face video database 101
2. Face Detection in Complex Backgrounds 103
2.1 Further comparison issues 106
3. Facial Feature Detection and Tracking 107
3.1 Facial feature detection 107
3.2 Face and facial feature tracking 118
4. Face and Facial Expression Recognition 118
4.1 Face recognition 122
4.2 Facial expression recognition 127
4.3 Model-based Video Coding 130
Chapter 8 APPLICATION IN AN AUDIO-VISUAL PERSON RECOGNITION SYSTEM 141
1. Speaker Identification Based on MFCC and GMM 142
2. Online Training of both Face Model and Speaker Model 143
3. Experimental Results 143
Chapter 9 CONCLUSIONS 145
References 149
Index 157
More eBooks at www.ciando.com 0

2. Previous Approaches (p. 27-28)

In early tracking systems [26, 27, 28], the feature matching step was carried out from one frame to the next using optical flow computations, resulting in drifting errors accumulating over long image sequences. In later techniques, feature texture information is gathered during initialization, so the feature matching step is carried out with respect to the initialization frame to overcome drifting.

In order to deal with large out-of-plane rotations, a 3D model of the geometry of the face has been used together with the texture obtained from the initialization step to achieve 3D pose estimation simultaneously with face tracking in an analysis-by-synthesis scheme [29, 30]. In this approach, the 3D model is used to create the templates by rendering the texture given the head pose so that the feature matching step performs well on large out-of–plane rotations. However, this system requires the 3D model of the person’s head/face.

A wire-frame model capable of nonrigid motion has also been used to analyze facial expressions together with the global position of the face [31]; however, the templates used in the feature matching algorithm do not adapt according to the nonrigid deformation or global head position, resulting in poor accuracy on extreme expressions and large out-of-plane rotations when the templates and the input images do not match well.

In this approach, a piece-wise linear deformation model is used to constrain the non-rigid motion into a subspace of deformations established beforehand. In a more complex scheme [32], optical flow constraints are used together with a wire-frame model to track rigid and nonrigid motion and adapt the wire-frame model to fit the person’s head. One of the most serious limitations in the wire-frame approaches is the fitting of the wire-frame model to the face in the initialization frame; this task involve the accurate location of many facial feature points and is carried out by hand.

Other approaches of current interest are those based on "blobs," where the face and other body parts are modelled with 2D or 3D Gaussian distributions of pixels. Pixels are clustered by their intensity [33] or color [34], or even by disparity maps from stereo images [35]. Although these techniques fail to capture nonrigid facial motion, they are easily initialized and operate very efficiently, especially even in sequences with moderate occlusion.

In general, algorithms that use complex wire-frame models provide a framework for high-level motion analysis of nonrigid facial motion. However, these complex models need to be customized to the face being tracked during a similarly complex initialization procedure. At the other end of the spectrum, algorithms based on simple models, such as blobs, have proven to be feasible. Their simple initialization procedures and low computational requirements allow them to run in real-time on portable computers, but they are limited in the amount of information they extract from the object parts.

The next step would be to combine these two schemes in a hierarchical approach that would benefit from both; however, the gap between the two schemes is too wide to be bridged since the complex models still must be initialized with person-independent accurate location of facial features. The face and facial feature tracking algorithm described here stands somewhere in between these two schemes. Faces and facial features are detected and tracked using person-independent appearance and geometry models that can be easily initialized and efficiently implemented to perform in real time. Nine facial features of multiple people are tracked; these account for global positions of the heads as well as for nonrigid facial deformations.