Chapter 6 – Facial Recognition

Below: A few thoughts before you begin this chapter’s readings and media:

“Before integrating any new technologies into American life, we must be absolutely sure that those innovations are imbued with our values.” – U.S. Senator Edward Markey, 2018.

Before you begin reading about this topic, here is a collection of some encounters with facial recognition in the real world.

facial recognition for home
facial recognition on twitter
youtube facial recognition policy


Do any of today’s most common technologies make you feel as though your privacy or liberty are threatened?

For example, it is commonplace today to see people taking photos with their smartphones. It has become so ubiquitous now that an observation was made that the sound people make when watching Tiger Woods strike a golf ball has changed in the past 15+ years because of smartphones. In 2002, they applauded after his shot. Today, everyone is taking a picture or video with a smartphone in their hands, so they shout instead of applaud.

But portable photographic technology was not always perceived as an acceptable presence in society. When the first Kodak portable cameras were sold to the public around 1888, a cry of outrage ensued about the breach of privacy that was being inflicted by “all those Kodakers” taking people’s pictures in public. Over time, however, society became accustomed to it while social norms adjusted to define what is considered acceptable behavior for individuals using camera technology.

The lesson here is that people tend to see new technologies as a threat at first encounter, but then relax once they get used to it.

Now let’s look at another case of an emerging technology that is coming to the forefront of mainstream attention.

Enter: facial recognition.

Several companies (see below) have developed facial recognition software systems that can be licensed to developers for use in law enforcement, customs enforcement, property management for landlords, and for passenger checks by the Transportation Safety Administration (TSA).

However, strong objections have been levied by the ACLU, congressional leaders, privacy advocates, and others who feel that facial recognition is not only an invasion of privacy, but a potential weapon to be used to advance discrimination. In fact, IBM – a global technology company – has pledged never to pursue FR technology development because of the negative ethical implications.

Despite these concerns, the presence of facial recognition technology has not had any unsettling effect on Facebook users who upload 350,000,000 images a day – all processed in Facebook’s DeepFace facial recognition system to conveniently identify their friends in each image. Nor is there much concern from iPhone users about Apple’s FaceID feature for biometric authentication. And then there is Snapchat’s Face Lenses accessory for animating facial images which can acquire several dimensions of a face within a second. (UPDATE: As of November, 2021, Facebook has decommissioned its FR system due to “uncertainty” about its use in society. Read the complete announcement.)

In this chapter, we will examine the affordances of facial recognition systems, where they are used, and then explore some of the ethical issues surrounding their use.

Key Terms

Physiognomy – A pseudoscientific study of a person’s facial features or expression as indicative of character or ethnic origin; the supposed art of judging character from facial characteristics. Here is more on the history of physiognomy dating back several centuries before classical Greek culture. Findings of these efforts have often been associated with proponents of eugenics theories (Valla, J. M., Ceci, S. J., & Williams, W. M., 2011).

    Required     How facial recognition it works

Below (fig. 1) is a graphical representation of a basic facial recognition system. Each phase of the process is defined below.

facial recognition process
Fig. 1 – AFER Process Flow – © Corneanu, C et al (2016)


Localization: The process of identifying the location of key facial features in an image, or in simpler terms, finding where the face is.

facial registration
Fig. 2 – Facial Registration Points – © Xuehan Xiong, Fernando De la Torre (2013)

Registration: The affixing of points on the localized face that correspond with key registration points on a standardized matrix (see Fig. 2).

Feature extraction: Once the registration points have been established, a set of features can be extracted from the matrix that corresponds to a model that has been “taught” to the computer through the input of examples. The section of the graphic referring to “other modalities” means that it is possible to combine facial data with data sets from other forms of acquisition, such as audio, posture, gait, or other physiological data.

Classification: A facial recognition system can be trained to classify input data into prescribed sets that correspond to patterns.  For example, a system can classify input into a set associated with stress and then flag an image if it matches the preset “stress” pattern.

Regression: A statistical process that attempts to produce a “best fit” trend to a set of data. In this case, it is used to provide a match to a database within a statistical degree of certainty, such as “we are 90% certain that the input image matches against the existing profile identified as [ person “X” ].”

Below is a link to a slideshow with a simplified visual representation of the facial recognition process. It is produced by the Electronic Frontier Foundation (EFF), a non-profit organization that advocates for privacy and legal issues related to emerging technologies.

Facial Recognition What is it and how does it work?” slideshow.

What should you be focusing on?

Your objectives in this module are:

  • Identify the current facial recognition systems in use today.
  • Identify the various contexts of deployment, by whom, for what purpose, and under what moral/ethical presumptions.
  • Explain how facial recognition might or might not be a valuable aspect of your project idea.

Readings & Media

Thematic narrative in this chapter

In the following readings and media, the authors will present the following themes:

  1. The history of facial recognition and its uses span to the extremes: from enhancing one’s social media experience to catching international terrorists.
  2. Facial recognition quietly and discretely keeps us all safe.
  3. Facial recognition is a genie that cannot be put back in its bottle. Once it is in the hands of agents with bad intent, it will lead to oppression and tyranny.

    Required     Article: A brief history of facial recognition technology (9 pages)

FaceFirst, a corporation that develops facial recognition software and hardware systems, offers a brief description of the key points in the historical development of facial recognition technologies: “A brief history of facial recognition technology.”

Galton portraits
Francis Galton, Frontispiece from Inquiries into Human Faculties, 1883

Optional:  For a glimpse at a very early attempt to formulate archetypes of criminality, sickliness, and other general characteristics based on facial appearance, look at the work of Victorian era (late 1800s) statistician Francis Galton who postulated that by combining photographs of multiple individuals, a composite archetype would be revealed, such as a “pianistic face” of a piano player. He did not consider it successful, though the effort is noteworthy.


    Required     Articles: List of the most common uses of facial recognition

Required: Alexis AliThe GlobalMe Blog – “Facing the Future: Facial Recognition Uses Today” (Oct 25, 2018). The GlobalMe Blog is produced by a commercial venture that specializes in language localization – a service that enables companies to present themselves in multiple languages. This involves analysis of massive data sets of speech patterns in a way that is similar to facial recognition.

Optional: is a facial recognition product for use by law enforcement. “We believe law enforcement should have the best tools available to help investigate and solve crimes. Quicker identifications and apprehensions help make our communities safer.”

Optional: Hong Kong-based startup Find Solution AI is developing the 4 Little Tree system to monitor children’s faces as they learn to determine their emotional state. Another company, Yoti, has developed a Facial Age Estimation system that can be integrated into mobile and desktop applications to control services to consumers under age 13.

Optional: NEC corporation published an easy-to-read whitepaper that explains common use cases for facial recognition with a few FAQ about the technology and data that is collected. This is a useful document to acquaint yourself with some of the basics, but keep in mind that it is a corporate publication – not an objective report.

Optional: A more scientific document can be accessed through the Granite State College online library entitled, “Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications.”

Optional:With Safety in Mind, Schools Turn to Facial Recognition Technology. But at What Cost?” by Emily Tate, Jan 31, 2019. Describes how a pre-K-8 school called St. Therese Catholic Academy has implemented a FR system to promote a sense of safety. Questions remain whether it actually improves safety given the amount of continuous maintenance of the FR database.

“A more surveillant society is a safer one.”

    Required     Video (5:56): How Facial Recognition Technologies are Enhancing Australia’s Public Safety

NEC’s General Manager of Smart Systems, Paul Howie describes the various benefits of facial recognition.

    Required     List: Companies that develop facial recognition systems (15:00)

FaceReader software is used to record data of people’s facial expressions for research purposes and emotional profiling.

NtechLab software .”..detects and identifies people’s faces in live video streams and video footage addressing a wide range of business tasks, such as precise people count, demographic information, people flow and client behavior.” Coming soon: tracking a person’s path and recognizing a person’s ethnicity.

Faception is software that claims to detect personality types. Here are excerpts from their website:

Faception is a [technology] for profiling people and revealing their personality based only on their facial image.

Faception can analyze faces from video streams (recorded and live), cameras, or online/offline databases, encode the faces in proprietary image descriptors and match an individual with various personality traits and types with a high level of accuracy. We develop proprietary classifiers, each describing a certain personality type or trait such as an Extrovert, a person with High IQ, Professional Poker Player or a Terrorist. Ultimately, we can score facial images on a set of classifiers and provide our clients with a better understanding of their customers, the people in front of them or in front of their cameras.

“Coding bodies leads to discrimination.”

    Required     Essay: Chris Gilliard’s “Friction-Free Racism” from Real Life magazine (11 pages)

Gilliard’s essay recounts incidences where the perception of his racial identity caused people to react as if his presence was somehow out of place. He connects this experience to the reality that facial recognition is only marginally helpful for things like unlocking smartphones or identifying a single criminal fugitive. Rather, he states, facial recognition is deployed substantially for the purpose of assigning a person to an identity category which, in a historical context, has served systems of discrimination. (A Harvard University article documents how FR has been used unevenly in law enforcement).

He then projects a future use of facial recognition technology into a form of optics we can wear that, like augmented reality, to help identify “others” according to some identity algorithm so that we can avoid categories of people with whom we don’t want “friction.”

What to look for as you read:

  • What are the connections Gilliard makes between social constructs in a society and the technologies a society embraces?
  • Gilliard says that coding biometric difference – or defining “who is what” – is a form of “biometric determinism”, which is to say that “what you are classified as will determine what you will be or where you belong.” What motivates Gilliard’s concern about this?
  • Given the power we already have in SM systems to select and block people that we do not want to be engaged with, why is Gilliard more concerned that facial recognition system will allow us to avoid friction with people in the real world?

About Real Life magazine: Real Life publishes essays, arguments, and narratives about living with technology. It was founded and edited by Nathan Jurgenson.

Gilliard, G. (2018, October 15). Friction-Free Racism [ Blog post ]. Retrieved from
false arrest tweet

    Required     Article: Letter to Jeff Bezos from Congress (4 pages)

BuzzFeed – “Bipartisan Lawmakers Want To Talk To Amazon About Its Facial Recognition Tech.” Davey Alba, a BuzzFeed reporter on technology issues, provides background surrounding the letter written to Amazon CEO Jeff Bezos by a bipartisan group of U.S. congressional lawmakers about the affordances and risks associated with Amazon’s Rekognition facial recognition technology. The actual letter is embedded at the bottom of the article. A PDF of the primary resource can be downloaded here.

What to look for as you read:

  • What do these lawmakers perceive as the risks of using facial recognition technology, and why?
  • What do they believe is the solution for controlling the technology?


Optional: Supplemental resources that are relevant to Facial Recognition

WIRED Magazine: “The Secret History of Facial Recognition” describes one of the key individuals, Woody Bledsoe who, in the 1960s, produced the foundation principles for pattern recognition through programming a computer. A quote:

“Many of the biases that we may write off as being relics of Woody’s time—the sample sets skewed almost entirely toward white men; the seemingly blithe trust in government authority; the temptation to use facial recognition to discriminate between races—continue to dog the technology today.”

NPR: Read an interview of the documentary filmmaker and researchers Joy Buolamwini and Shalini Kantayya about their work related to coded bias in AI systems.

TechCrunch:3D-printed heads let hackers – and cops – unlock your phone.”

Wall Street Journal (paywall): “The quiet growth of race-detection software sparks concerns over bias” – This would be a required article if it were not behind a paywall. It extends the discussion presented by Gilliard.

Crain’s New York Business: “Facial-recognition startup pushes ahead despite controversy.” This is the story of an inventor who created an FR system that can alert landlords when there is suspicious activity in an apartment building and surrounding area. While this system has caused a great deal of stress about privacy issues and when/how to notify residents when it is being used, it has been embraced wholeheartedly by operators of nursing homes that have wandering patients with dementia. “It has definitely improved patient care,” they say.


Valla, J. M., Ceci, S. J., & Williams, W. M. (2011). The accuracy of inferences about criminality based on facial appearance. Journal of Social, Evolutionary, and Cultural Psychology5(1), 66.Corneanu, C., Oliu, M., Cohn, J. F., & Escalera, S. (2016). Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications. Retrieved from

Xiong, X., & De la Torre, F. (2013). Supervised descent method and its applications to face alignment. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 532-539).


Icon for the Creative Commons Attribution 4.0 International License

Chapter 6 – Facial Recognition by Steve Covello is licensed under a Creative Commons Attribution 4.0 International License, except where otherwise noted.

Share This Book