The prohibition in Article 5(1)(f) AI Act contains an explicit exception for emotion recognition systems used in the area of the workplace and education institutions for medical or safety reasons, such as systems for therapeutical use . 166 In light of the AI Act’s objectiveto ensure a high-level of fundamental rights protection, this exception should be narrowly interpreted.
In particular, therapeutic uses should be understood to mean uses of CE-marked medical devices. Moreover, t his exception does not comprise the use of emotion recognition systems to detect general aspects of wellbeing. The general monitoring of stress levels at the workplace is not permitted under health or safety aspects. For example, an AI system intended to detect burnout or depression at the workplace or in education institutions would not be covered by the ex ception and would remain prohibited.
The notion of safety reasons within this exception should be understood to apply only in relation to the protection of life and health and not to protect other interests, for example property against theft or fraud.
It follows from this narrow interpretation of the exception that any use for medical and safety reasons sh ould always remain limited to what is strictly necessary and proportionate, including limits in time, personal application and scale, and should be accompanied by sufficient safeguards. Such safeguards could include, for example, prior written and motivated expert opinion relating to t he specific use case. The necessity should be assessed on an objective basis in relation to the medical and safety purpose, and not refer to the employer’s or educational institution’s ‘needs’. This assessment should inquire whether less intrusive alternative means exist which would achieve the same purpose.
Employers and educators should only deploy emotion recognition systems for medical and safety reasons in case of an explicit need.167 Data collected and processed in this context may not be used for any other purpose.This is particularly important given that the use of AI management software at work has proven topotentially negatively impact workers’ health and safety.Continuous monitoring via wearables, for instance, may increase work-stress while affecting productivity.168
Since Recital 18 AI Act excludes from the definition of emotion recognition systems physical states, such as pain or fatigue, a number of AI systems used for safety reasons would already not fall under that definition, including, for example, systems used in detecting the state of fatigue of professional pilots or drivers for the purpose of preventing accidents.
Other laws, including data protection rules, remain applicable to emotion recognition systems that fulfil the conditions of the exception in Article 5(1)(f) AI Act.169
Emotion recognition systems that classify as high-risk systems pursuant to Article 6(2) and Annex III(1)(c) AI Act will need to comply with the high-risk requirements in Chapter III Section 2 AI Act and the transparency obligation of Article 50(3) AI Act. For example, Emotion recognition may be deployed for medical reasons to assist employees or students with autism and improve accessibility for those who are blind or deaf.170 Such uses would fall within the exception for medical reasons in Article 5(1)(f) AI Act. By contrast, e motion recognition for assessing students’ or employees ’ wellbeing, motivation levels, and job or learning satisfaction do not qualify as ‘use for medical reasons’and would be prohibited. An employer would be prohibited from deploying AI-enabled devices or digital assistants at the workplace for measuring anxiety based on measured stress levels or for measuring boredom of employees, unless the elevated stress level/lack of concentration would pose a specific danger, for example when deploying dangerous machines or dealing with dangerous chemicals. In the latter case, the employer may not use the data for other purposes, such as assessing the employee’s work performance.