1 2D-To-3D Face Recognition Systems

Michael I. Miller, Marc Vaillant, William Hoffman and Paul Schuepp

Animetrics, Conway, New Hampshire

1.1 Intelligent Video Systems

1.1.1 The Need for Intelligent Video Systems

Intelligent image and video interpretation systems of the future will be highly integrated with emergent video databases interacting with real-time access control and surveillance. The intelligent video surveillance software market, including video analysis, is experiencing meteoric growth. Airports, borders, ports, energy plants, historical buildings, monuments, manufacturing plants, retail establishments, and businesses all require access control and surveillance video solutions. Forrester predicts that 40% of businesses will need integrated security. The access control market is expected to reach nearly 14 billion dollars in 2009 [1]. Ultimately, these systems will integrate with and allow for the retrieval and cueing of the massive data stores such as the FBI's archives that contain both annotated as well as un-annotated video resources.

Figure 1.1 depicts an access control and video surveillance system handling the identities and monitoring dynamically the locations of individuals. According to ABI Research [2], the video surveillance market, already $13.5 billion as of 2006, will grow to $46 billion by 2012. The goal of spotting individuals of particular identities, and indexing and analyzing video archives is a fundamental challenge to the future of noncooperative FR. Solving the problem of dynamically identifying faces from live streaming or stored video will enable integrated, intelligent security solutions of the future.

Figure 1.20 depicts the FR performance achievable with 2D-to-3D technologies when attempting to recognize face images at 40° of pose compared to that of a leading 2D conventional FR system [17]. The lower light solid shows a 94% verification rate at the FAR of 0.1%. This means that 94% of the time the system was able to recognize and verify that the person is who he says he is. At the same time, the system only accepted people incorrectly 1 out of 1000 times (i.e. the FAR). If the FAR is constrained to approximately 1 out of 100, or a 1% FAR, then it follows that the verification rate goes up.

The principal focus of this chapter is to describe the emerging 2D-to-3D technologies for extending FR systems historically applied to document verification and access control to the uncontrolled environments, which require pose and lighting invariant ID such as is required for tracking and recognizing faces in noncooperative surveillance and video applications.

1.1.2 The Barrier to Face

The surveillance environment has posed great challenges for FR technologies. Due to FR's nonintrusive nature, it has been pushed to the forefront as the biometric of choice. The International Civil Aviation Organization has embraced FR as its biometric standard. Yet, we must emphasize that, to date, the facial biometric has not adequately penetrated this marketplace. Presently, most systems do not incorporate FR biometrics. The major difficulty is that its advantage—its noninvasive, touch-free nature—is also its daunting challenge. Comparing it to fingerprint for the moment, imagine a “touch-free” fingerprint system in which any random medium—perspiration, oil, sand, grit—you name it, could be between the finger and the sensor. Alternatively, imagine that the finger could be at any “distance to the sensor”, and could be occluded by gloves or bandages. Could we expect to deploy fingerprint biometrics in such conditions and require “constant performance independent of environmental variables”—hardly.

Such issues directly confound the successful deployment of FR technologies including the huge range of camera qualities (CCTV, Webcams, high resolution imagery, etc.), the infinite variety and schema of environmental lightings, arbitrary subject's positioning, facial hair, ornaments such as eyeglasses and jewelry, and complex backgrounds. The left column of Figure 1.2 depicts control photographs associated with access and document verification. The right column shows uncontrol photographs with the confounding variations of lighting and complex backgrounds. Figure 1.3 shows the results from the Facial Recognition Grand Challenge (FRGC) 2005 report [3] depicting the gap in performance between controlled and uncontrolled data by FR systems. FR performance is studied worldwide by examining false reject rate (FRR) or verification rate (1-FRR) as a function of false accept rate (FAR). The table lists out the FRRs in percentages at an FAR of 0.001 meaning the systems only accepted people incorrectly 1 out of 1000 times. The difference in median performance of participants is 60% between the control and uncontrol.

The confounding challenges of the uncontrolled environment resulting in this gap must be solved in order to make effective use of existing video recordings and to embark upon “action oriented” intelligent analytical systems that will provide next generation security methods. This chapter examines how 2D-to-3D technologies provide the crucial technological link from the historical application of controlled front facial recognition for access control and document verification to the intelligent systems of video surveillance.

1.1.3 2D-to-3D Bridges the Performance Gap for Intelligent Video Systems

FR today works well in a “controlled setting” where the camera-person interaction is cooperative, illumination is monitored, and backgrounds are simple and uncluttered. Facial identification systems and face trackers have been in use for at least a decade. Typically, facial identification systems comprise detection and identification systems based on the manipulation of 2D likenesses of faces, which represent photometric and geometric variation robustly as manifest in the 2D likeness.

The “uncontrolled” surveillance environment introduces uncooperative subjects where facial pose is relatively arbitrary, lighting is infinitely variable, and backgrounds are arbitrarily complex. Next generation FR must accommodate these kinds of variations for successful transition from the controlled “checkpoint” access application to the “uncontrolled” surveillance video application. Purely 2D legacy FR technologies are limited in their deployment to the more controlled environments of access control and document verification. Figure 1.4 shows an example of 2D representational lattice model used in many of the legacy 2D FR systems.

Since 2D systems are limited to the manipulation of 2D geometric variations of in-plane geometric variation, they can be used for tracking and/or identification of faces while accommodating in-plane variation. However, they degrade as the target subjects are viewed out of plane. Since they rely on 2D likeness they cannot be robust to changes in both photometric and geometric variation, which depend on the 3D shape of the face and its interaction with variations in the external lighting illumination. 2D-to-3D technology provides a unified technological infrastructure for addressing all of these technical challenges, accommodating simultaneous high performance for imaging volumes at a physical access checkpoint (a more “controlled” scenario), as well as the “image at a distance” surveillance scenario.

Figure 1.5 depicts a 3D geometric model and texture representing the 2D photograph resulting from the 2D-to-3D technology. It is the core 3D data structure generated from a 2D image that can provide the opportunity for FR systems to pass between controlled frontal interactions between camera probing and arbitrary positions of uncalibrated surveillance cameras. The 3D geometric data structures unify (i) the uncalibrated nature of camera position to viewer allowing for identification to be invariant to pose (position and rotation), (ii) the infinite variety of variations introduced via the external variability of the lighting world can be accommodated via the representation of the light field associated with the observed luminance on the geometry, and (iii) the dynamic coherence of video is directly encoded into the rigid motions associated with the 3D object representation.

1.2 Computational Anatomy and Diffeomorphisms for 2D-to-3D Model Generation

Geometry based 3D facial ID systems are robust to geometric variation of the pose of individual faces and efficiently detects and identifies faces from projective imagery such as measured by conventional video cameras. The key technological advance required for their application using conventional projective imaging cameras lies in a system to automatically determine the 3D geometry of a person's face with a finite set of images or photographs or through the analysis of a persistent set of images from video data. The technological term being used to describe...