Face Analysis Under Uncontrolled Conditions

Preface xi
Romain BELMONTE and Benjamin ALLAERT

Part 1. Facial Landmark Detection 1

Introduction to Part 1 3
Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA

Chapter 1. Facial Landmark Detection 13
Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA

1.1. Facial landmark detection in still images 14

1.1.1.Generativeapproaches 14

1.1.2.Discriminative approaches 18

1.1.3.Deep learningapproaches 24

1.1.4.Handlingchallenges 34

1.1.5.Summary 40

1.2.Extendingfacial landmarkdetectionto videos 41

1.2.1.Trackingby detection 41

1.2.2.Box, landmarkand pose tracking 43

1.2.3.Adaptive approaches 45

1.2.4. Joint approaches 46

1.2.5. Temporal constrained approaches 47

1.2.6.Summary 49

1.3.Discussion 50

1.4.References 52

Chapter 2. Effectiveness of Facial Landmark Detection 67
Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA

2.1.Overview 68

2.2.Datasets and evaluationmetrics 69

2.2.1. Image and videodatasets 69

2.2.2. Face preprocessing and data augmentation 73

2.2.3.Evaluationmetrics 75

2.2.4.Summary 77

2.3. Image andvideobenchmarks 77

2.3.1. Compiled results on 300W 77

2.3.2. Compiled results on 300VW 79

2.4.Cross-dataset benchmark 80

2.4.1.Evaluationprotocol 80

2.4.2.Comparisonof selected approaches 82

2.5.Discussion 86

2.6.References 88

Chapter 3. Facial Landmark Detection with Spatio-temporal Modeling 93
Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA

3.1.Overview 94

3.2.Spatio-temporalmodelingreview 95

3.2.1.Hand-craftedapproaches 95

3.2.2.Deep learningapproaches 97

3.2.3.Summary 103

3.3.Architecturedesign 104

3.3.1. Coordinate regression networks 104

3.3.2.Heatmapregressionnetworks 106

3.4.Experiments 107

3.4.1.Datasets andevaluationprotocols 107

3.4.2. Implementationdetails 108

3.4.3.EvaluationonSNaP-2DFe 109

3.4.4. Evaluation on 300VW 111

3.4.5.Comparisonwith existingmodels 112

3.4.6. Qualitative results 112

3.4.7.Propertiesof the networks 114

3.5.Design investigations 114

3.5.1.Encoder-decoder 115

3.5.2. Complementarity between spatial and temporal information 117

3.5.3. Complementarity between local and global motion 119

3.6.Discussion 122

3.7.References 123

Conclusion to Part 1 133
Romain BELMONTE, Pierre TIRILLY, IoanMarius BILASCO, Nacim IHADDADENE and Chaabane DJERABA

Part 2. Facial Expression Analysis 147

Introduction to Part 2 149
Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA

Chapter 4. Extraction of Facial Features 157
Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA

4.1. Introduction 157

4.2.Face detection 158

4.2.1.Point-of-interestdetectionalgorithms 160

4.2.2.Face alignment approaches 162

4.2.3.Synthesis 166

4.3.Face normalization 166

4.3.1.Dealingwith headpose variations 167

4.3.2.Dealingwith facial occlusions 170

4.3.3.Synthesis 172

4.4.Extractionof visual features 172

4.4.1.Facial appearancefeatures 172

4.4.2.Facial geometric features 174

4.4.3. Facial dynamics features 175

4.4.4.Facial segmentationmodels 177

4.4.5.Synthesis 179

4.5. Learning methods 179

4.5.1.Classification versus regression 180

4.5.2.Fusionmodel 182

4.5.3.Synthesis 184

4.6.Conclusion 185

4.7.References 186

Chapter 5. Facial Expression Modeling 191
Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA

5.1. Introduction 191

5.2.Modelingof the affective state 192

5.2.1.Categoricalmodeling 192

5.2.2.Dimensionalmodeling 194

5.2.3.Synthesis 196

5.3. The challenges of facial expression recognition 197

5.3.1. The variation of the intensity of the expressions 197

5.3.2.Variationof facialmovement 199

5.3.3.Synthesis 200

5.4.The learningdatabases 201

5.4.1. Improvementof learningdata 201

5.4.2. Comparison of learning databases 203

5.4.3.Synthesis 205

5.5. Invariance to facial expression intensities 206

5.5.1.Macro-expression 206

5.5.2.Micro-expression 208

5.5.3.Synthesis 209

5.6. Invarianceto facialmovements 211

5.6.1. Pose variations (PV) and large displacements (LD) 211

5.6.2.Synthesis 214

5.7.Conclusion 215

5.8.References 216

Chapter 6. Facial Motion Characteristics 223
Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA

6.1. Introduction 223

6.2.Characteristics of the facialmovement 225

6.2.1. Local constraint of magnitude and direction 226

6.2.2. Local constraint of the motion distribution 228

6.2.3.Motionpropagationconstraint 230

6.3.LMP 232

6.3.1. Local consistency of the movement 233

6.3.2.Consistencyof local distribution 236

6.3.3. Coherence in the propagationof themovement 238

6.4.Conclusion 241

6.5.References 242

Chapter 7. Micro- and Macro-Expression Analysis 243
Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA

7.1. Introduction 243

7.2. Definition of a facial segmentation model 244

7.3.Feature vector construction 247

7.3.1.Motionfeaturesvector 247

7.3.2.Geometric featuresvector 248

7.3.3.Features fusion 249

7.4. Recognition process 250

7.5. Evaluation on micro- and macro-expressions 251

7.5.1.Learningdatabases 252

7.5.2. Micro-expression recognition 253

7.5.3. Macro-expressions recognition 255

7.5.4. Synthesis of experiments on micro- and macro-expressions 258

7.6. Same expression with different intensities 260

7.6.1.Data preparation 260

7.6.2.Fractional time analysis 263

7.6.3.Analysis on a different time frame 264

7.6.4. Synthesis of experiments on activation segments 265

7.7.Conclusion 265

7.8.References 266

Chapter 8. Towards Adaptation to Head Pose Variations 271
Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA

8.1. Introduction 271

8.2.Learningdatabase challenges 273

8.3. Innovative acquisition system (SNaP-2DFe) 274

8.4. Evaluation of face normalization methods 276

8.4.1. Does the normalization preserve the facial geometry? 277

8.4.2. Does normalization preserve facial expressions? 280

8.5.Conclusion 283

8.6.References 284

Conclusion to Part 2 287
Benjamin ALLAERT, IoanMarius BILASCO and Chaabane DJERABA

List of Authors 293

Index 295