2
Facial Recognition: Underpinning Concepts and Concerns

Introduction

At this point, it is worth taking a closer look at the technological advances and public concerns introduced in chapter 1. In particular, we can consider the various logics and concepts underpinning the integration of facial recognition technology into the activities of contemporary society. There are three distinct areas of thought to consider here. First there are the basic computational logics, ideas and concepts that underpin the technical challenge of 'recognizing' faces from digital images. From this perspective, facial recognition is a complex mathematical problem that has benefited from some neat statistical and computational advances. Here the face is conceived primarily as a data-information object, with any act of 'recognition' or 'identification' more accurately described as an act of statistics-driven pattern matching. Secondly, are the various psychological and physiological theories of the face that have guided some of the recent advances in facial analysis and facial detection. In particular, we consider how computational analysis of the face is seen by some people as a ready means of 'affect detection', as well as of inferring characteristics such as race and gender - thus revitalizing long-standing ideas of physiognomy and other forms of 'reading' human faces.

Finally, we need to begin to get to grips with the various socially focused critical concerns beginning to be levelled against facial recognition. In particular, it is worth considering how the development of facial recognition technology is increasingly bumping up against contemporary theorizations of surveillance. Themes covered in this section include facial recognition as a form of post-panoptic 'monitoring at a distance', as well as issues of all-encompassing algorithmic surveillance. All told, a range of different critical perspectives have long presumed the face to be a particularly revealing object. Face scanning can take place unobtrusively, at a distance, but it is also a deeply intimate and invasive act. This combination requires our sustained consideration.

Fundamental computational concepts and logics

As described in chapter 1, facial recognition essentially involves making measurements of key features from someone's face. These features might include how far their nose protrudes, how their eyes align, or how their chin is dimpled. As few as five carefully chosen facial features can comprise a dataset that is unique to an individual, although the most sophisticated facial recognition systems might measure over one hundred features. These measurements are commonly taken from photographic images but can also be made by sensors that shine light beams onto the face, thus measuring the key facial features without technically photographing the face.

The process of 'recognition' sees these data points from the newly measured face compared with data-points already extracted from existing photographs in order to look for a match. This can lead to two distinct forms of 'facial recognition'. The most straightforward process involves verification - addressing the question of whether someone is who they say that they are. This requires a process of 'one-to-one' (1:1) matching. Here the individual's face is scanned and then matched against one existing image of the target person. This 'authentic' image might be already held in a database, such as employee records in a workplace, or from national records of passport photos. In some cases, the image is simply pre-supplied, such as a smartphone owner uploading a photo on the device so they can unlock their phone using their face. This process will confirm whether someone presenting themselves to be Jane Doe is actually Jane Doe or not. This form of authentication allows Jane Doe to unlock her smartphone, access her bank account, or walk through the entry doors at her workplace. If no match is made, then the system simply decides that this is not Jane Doe and the process stops.

A second more complicated process is identification - addressing the question of who someone is. This requires a process of 'one-to-many' (1:n) matching. Here the matched image comes from existing photographs found in comprehensive (sometimes population-wide) datasets, such as drivers' licences or national ID cards. This process allows the identification of otherwise unknown faces, usually unbeknown to the target individuals involved. Here individuals' faces are compared across a large database of faces until matches are found. This will identify whether we are looking at Jane Doe, John Doe or anyone else in the database.

A third process associated with facial recognition is more accurately referred to as 'facial analysis', given that the target individual is not necessarily being identified. This involves inference - i.e. answering the question of what might be known about someone. This might involve inferring demographic characteristics such as likely age, gender and race, or perhaps someone's emotional state or even intentions. Here the data from a scanned face is correlated against measurements that are derived from pre-designated faces with the particular characteristic - for example, 'smiling' or 'depressed'. While the veracity of such calculations remain highly contested, such systems are sold on the basis as being able to provide plausible indications without necessarily identifying the person concerned.

As described in chapter 1, the accuracy of facial verification and facial identification technologies has been improving rapidly, often now reaching claimed levels of 99 per cent and above. Of course, as with all data-driven processes, this still leaves some room for error, which is usually described as taking one of two forms. A false positive involves facial recognition software erroneously judging two images to be of the same individual. A false negative involves failure to match two images that are actually of the same person. Clearly, the consequences of each form of error can be very different depending on the circumstances and rationales behind why the match is being made. False positives can lead, for example, to arrests of innocent people or incorrect risk evaluations that lead to over-aggressive responses on the part of authorities. False negatives can lead to denial of access or other benefits to which people are rightly entitled.

Working to reduce these errors is a fundamental challenge for all facial recognition developers. In ideal circumstances, a facial recognition system will have a database of well-lit, well-posed, high-definition photographs which can be compared with a new image of the face that is similarly well lit, well posed and in high definition. The best-case scenario involves a flat angle, with no face coverings or other intrusions in the images. In this manner, the gold standard for FRTs is often reckoned to be the facial recognition set-up seen in many airports where head-height images from well-lit airport departure gate cameras are compared with similarly posed passport photographs.

One ongoing technical challenge for facial recognition developers is being able to make accurate and reliable matches in less than ideal circumstances. The accuracy of any facial recognition system is understandably compromised by poorly lit images taken from highly placed cameras - situations that result in what are sometimes referred to as 'non-compliant' photos or 'in the wild' images. Another long-standing challenge is partial face identification, when people's faces are obscured by hats, glasses, niqabs, burkas or face masks. A further complication is the challenge of 'liveness detection' - put simply, how the system can be certain that the newly acquired image is of a genuine living person in front of the camera, rather than a high-definition photograph of a face or an impostor wearing a lifelike latex mask. Such challenges might not seem serious for a human observer glancing up and quickly recognizing someone that they know, but can be incredibly difficult to address in computational terms.

The computer science of facial recognition

Next, then, it is worth exploring the basic computational logics, ideas and concepts that underpin FRT as an area of technology research and development. In computer science terms, facial recognition is part of the broader fields of 'computer vision' and 'machine vision' which focus on engineering computers to extract information from images in a similar manner to the human visual system. Computer vision is therefore interested in programming computers to identify objects, perceive distance and motion, recognize patterns and so on. More specifically, much of the technical development of facial recognition is related to the computer vision subfields of 'object detection' and 'object recognition'. In this sense, human faces are just one of thousands of different objects that computers are being trained to detect - from red traffic lights to boxes in a warehouse.

To accomplish this detection and recognition of facial 'objects', facial recognition systems perform four steps. Key concepts here include programming computers to visually detect the specific 'landmark' features that constitute the object class of a...