Point 1(c) of Annex III classifies emotion recognition systems as high-risk. Article 3(39) AI Act defines ‘emotion recognition systems’ as AI systems ‘ for the purpose of identifying or inferring emotions or intentions of natural persons on the basis of their biometric data’.
Serious concerns exist about the scientific basis of AI systems aiming to identify or infer emotions or intentions, particularly since the expression of emotions vary considerably across cultures and situations and even within a single individual. Emotion recognition systems are intrusive and present key shortcomings that may affect individuals’ fundamental rights . Among the key shortcomings of such systems are their limited reliability, their lack of specificity, and their limited generalisability.
While language-based AI social companions are not specifically classified as high-risk under the AI Act, it cannot be excluded that, in certain cases, they might be intended to be used for emotion recognition (identifying and inferring emotions or intentions based on biometric data from, e.g., voice or video calls) and therefore fall to be classified as high-risk pursuant to point 1(b) of Annex III AI Act. i. Identification and inference of emotions or intentions
The identification of emotions or intentions occurs where the processing of the biometric data (for example, the voice or a facial expression) of a natural person allows to directly compare and identify an emotion or intention with one that has been previously programmed in an emotion recognition system. Inferring occurs by deducing information generated by analytical and other processes by the system itself. In such a case, the information about the emotion is not solely based on data collected on the natu ral person, but is inferred from other data, including machine learning approaches that learn from data how to detect emotions or intentions24. Inferring generally encompasses identifying (of emotions or intentions) as a prerequisite, so an AI system intended to be used for inferring emotions or intentions should be understood as also identifying emotions or intentions25. ii. Emotions or intentions
The concept of emotions or intentions should be understood in a wide sense and not interpreted restrictively. Recital 18 AI Act provides some clarifications on this concept, listing emotions ‘such as happiness, sadness, anger, surprise, disgust, embarrassment, excitement, shame, contempt, satisfaction and amusement’. These examples are not exhaustive.
The AI Act should not be circumvented by referring to attitudes and therefore the high-risk use case in point 1(c) of Annex III includes cases where an AI system is intended to be used to find, based on biometric data, that a person is showing, for example, an angry attitude.
Recital 18 AI Act clarifies that emotions or intentions do not include ‘physical states, such as pain or fatigue, including, for example, systems used in detecting the state of fatigue of professional pilots or drivers for the purpose of preventing accidents’ (for example, identifying whether a person is sick is not emotion recognition). It further clarifies that emotion recognition systems do not include ‘the mere detection of readily apparent expressions, gestures or movements, unless they are used for identifying or inferring emotions’. Those expressions can be basic facial expressions, such as a frown or a smile, or gestures such as the movement of hands, arms or head, or characteristics of a person’s voice, such as a raised voice or whispering. However, when these readily apparent expressions or gestures are used for identifying or inferring emotions or intentions, they are covered by the concept of ‘emotions or intentions’.
a) Practical examples of AI systems falling within the high-risk use case of point 1(c)
- The integration of AI with body -worn cameras or remote surveillance systems (so - called police bodycams): e.g. AI-enhanced body-worn cameras used across patrol units that enables automated behavioural pattern detection, flagging ‘aggressive posture ’ through gait, body posture, movement and facial analysisto assess whether an individual is likely to engage in fight during police encounters. This qualifies as an emotion recognition system because it interprets physical cues to infer an individual’s emotional or mental state-in this case, anger.
- An AI system for the gaming industry that is intended to measure a gaming experience to further improve the product (real - life gaming experience) . The AI system tracks body posture, facial expressions, eye closures and gazes to measure the reaction of the player, including their excitement, anger, frustration, amusement; analyses the above-mentioned recorded data to identify relevant elements of th e gaming experience that needs improvement. Such a system involves processing of biometric data (e.g., facial expressions which are a source of biometric data). It also identifies or infers gamers’ emotions based on their biometric data. Excitement, anger, frustrations are examples of emotions.
- An AI-based solution to maintain order during concert events. The AI system is installed in cameras at walls or ceilings; screens the mood of the audience, e.g., voices, faces and movements; when the system finds that the mood is getting aggressive in a certain area of the stadium, more security personnel is sent to that area. The AI system is intended for emotion recognition : i t screens emotions of individual participants to assess the mood of the audience.
- An AI system used in call centres to infer emotions of customers. The AI system analyses customers’ voices and evaluates vocal tone, pitch and volume to gauge customer satisfaction level for statistical reasons, to identify the moment to route them to a human agent (e.g., anger), for troubleshooting purposes, or to enhance customer care business unit. These AI systems deduce from the customer’s voice their emotions, e.g., if they are happyor angry. It does not matter whether the biometric system is intended to identify the customer.
- Smart watch mood monitor/wearable device. The AI system is integrated into a smart watch to measure biometric data (e.g., voice, heart rate) and monitor user emotions (e.g., sad, curious, happy, bored) to assist users in recognising their emotional state and provide emotional improvement suggestions. Such AI systems infer user emotions, which is intrusive . It is not relevant whether the emotion recognition results are disclosed solely to the specific user.
b) Practical examples of AI systems falling outside the high-risk use case of point 1(c)
- An AI system intended to be used by the automotive industry to help prevent accidents by detecting when a driver loses focus (i.e., drowsiness, falling asleep, experiencing sudden health issues). The AI system tracks body posture, facial expressions, eye closures and gazes via installed cameras and motion detectors in a vehicle; analyses the collected data to alert the driver or assume control of the vehicle, and safely bring it to a stop. Such AI systems are not considered high-risk AI systems because emotions or intentions do not include pain o r fatigue, which are physical states.
- The observation of readily apparent expressions. The mere o bservation that a person is smiling is not emotional recognition. A TV broadcaster using a device that allows to track how many times its news presenters smile at the camera is not emotion recognition.
- See Recital 12 AI Act. Inferred data is hence also often the result of probability-based analytical (big data) processes aimed at finding correlations and finding patterns in data sets.
- Recital 18 AI Act.