To find out, I asked Luuk Spreeuwers, a researcher in biometrics and facial recognition at the University of Twente.
How does facial recognition work?
To understand how facial recognition works, Luuk first distinguishes between two methods: verification and identification. Your phone uses verification. The system already knows who you are supposed to be: the owner of the phone. Luckily, your phone doesn’t have to search through millions of faces to find yours. It only needs to determine whether the face in front of the camera matches that of the owner. To do this, the system compares features from the face in front of the camera with previously stored features from one specific person. Luuk calls this a 1:1 comparison. The result is either a match or a non-match.
Identification works differently. In this case, the system doesn’t yet know who the person in front of the camera is and is trying to find out. It compares the features of one face with those of multiple people stored in its database. Luuk calls this a 1:N comparison. The system then looks for the best match or concludes that there is no match. So, the main difference lies in the question the system is trying to answer. Verification asks, ‘Are you this person?’, while identification asks, ‘Who are you?’
You can see a similar principle to verification at automated passport control. There, the system can compare features from a live camera image with features from the passport photo stored on the chip in your passport.
But what does facial recognition actually look at?
Your phone doesn’t simply look at your eye colour or the distance between your nose and mouth. Modern facial recognition systems use deep learning, a form of artificial intelligence, to extract distinctive features from a face. ‘These systems are trained on thousands to millions of faces,’ Luuk explains. During this training process, the system learns which features are most useful for telling faces apart. These include information about the shape and texture of the face and different parts of it.
The system then compares these features and calculates the distance between them. If that distance is small enough, it produces a match. Surprisingly, even researchers cannot always say exactly which features ultimately make the difference. The system learns these features during the training process. Facial recognition can therefore work extremely well, while it can still be difficult to explain exactly why a system reaches a particular decision. Luuk and his colleagues are therefore researching ways to make facial recognition more explainable, including in collaboration with the Netherlands Forensic Institute.
So why does my phone recognise me with glasses?
Back to my glasses. For modern facial recognition, they don’t necessarily pose much of a problem. Luuk explains that these systems train on millions of examples. These include many images of the same people with and without glasses, wearing different glasses, with and without beards, and with all kinds of facial expressions. This teaches the system which features are important for recognising someone and which changes it can largely ignore. So glasses do change what the camera sees, but the system has learnt that this doesn’t necessarily mean there is a different person in front of it.
Facial recognition isn’t perfect, though. Image quality also plays a role. Low resolution or a partially covered face, for example, can make it harder for a system to recognise someone correctly. And sometimes it goes wrong in surprising ways. ‘Sometimes they make “stupid” mistakes,’ says Luuk. A system might match two faces that a human can clearly see belong to different people. It isn’t always easy to predict when or why such an error will occur. That is another question Luuk is investigating.
How secure is facial recognition?
If your phone uses your face to check that it really is you, you might wonder whether someone else could pretend to be you. Could they simply hold up a photograph in front of the camera? Such attempts do exist. Luuk calls this spoofing or a presentation attack. The simplest example is using a photograph of someone. More advanced attempts might use a video playing on a screen or even a 3D mask.
Facial recognition systems therefore try to spot these tricks. This is known as liveness detection. A system might, for example, check whether someone blinks or whether the face in front of the camera actually has depth. Some modern phones use 3D facial recognition and also analyse the three-dimensional shape of the face.
But that isn’t the only challenge. Luuk also researches other forms of image manipulation, such as morphing and deepfakes. With morphing, two different faces are digitally combined to create a new face. A deepfake uses AI to alter an image or video so that it appears that someone else is in the image or is doing something that never actually happened.
Facial recognition therefore needs to do more than determine who is in an image. Systems also need to recognise when someone has tampered with that image. Luuk is investigating how systems can become better at detecting these kinds of manipulated images.
So, how does your phone know it’s you?
Ultimately, the image your phone sees of your face doesn’t have to look exactly the same every time. The system compares features from your face with features it has stored previously. If they are similar enough, it produces a match. Glasses, different lighting or facial hair don’t necessarily have to be a problem.
That explains why my phone still recognised me with my glasses on. What felt like nothing more than a quick glance at my screen was, for the facial recognition system, a rapid comparison designed to answer one simple question: are you really you?
