(134)
An essential condition for the prohibitions in Article 5(1)(a) and (b) AI Act to apply is that the AI-enabled manipulation and exploitation of vulnerabilities should cause or be reasonably likely to cause significant harm. All manipulative, deceptive and exploitative AI applications that are not reasonably likely tocause significant harms are in principle outside the scope of the prohibitions without prejudice to other Union law that still applies (see section 3.6. below). Examples of AI systems that are not likely to cause significant harm include:
- An AI companionship system is designed in an anthropomorphic way and with affective computing to make the system more appealing and effectively makes users more engaged, but is not engaging in other manipulative or deceptive practices in a manner that is reasonably likelyto cause them serious psychological, physical or other harms, unhealthy attachment and dependency.
- A therapeutic chatbot uses subliminal techniques to steer users towards a healthier lifestyle and to quit bad habits , such as smoking . Even if the users who follow the chatbot’s adviceand subliminal therapy experience some physical discomfort and psychological stress due to the effort made to quit smoking, the AI-enabled chatbot cannot be considered likely to cause significant harm. Such temporary discomfort is unavoidable and outweighed by the long-term benefits for users’ health. There are no hidden attempts to influence decision-making beyond promoting healthy habits.
- An online music platform uses an emotion recognition system to infer users’ emotions and automatically recommends them songs in line with their moods , while avoiding excessive exposure to depressive songs. Since users are just listening to music and are not otherwise harmed or led to depression and anxiety, the system is not reasonably likely to cause significant harm.
- AI-enabled manipulative and deceptive techniques used in security training and other learning simulations that mimic phishing attempts to educate users on cybersecurity threats. These systems may deploy purposefully manipulative techniques (e.g., exploiting cognitive biases) without users’ awareness that distort the behaviour, but this is done temporarily for beneficial training and awareness raising purposes and without causing significant harms.