Finally, for the prohibition in Article 5(1)(b) AI Act to apply, the distortion of the behaviour of the vulnerable person or group of persons must cause or be reasonably likely to cause that person or another person significant harm . Article 5(1)(a) and (b) AI Act make use of the same concepts and should therefore be interpreted in the same way. The explanations provided in section 3.2.3. in relation to the types of harms, the threshold for significance of the harm , and the causal link and its reasonableness are thus equally relevant for the interpretation of Article 5(1)(b) AI Act.
As explained in section 3.2.3., significant harm encompasses a range of significant adverse impacts, including physical, psychological, financial, and economic harms that must be reasonably likely to occurfor the prohibition in Article 5(1)(b) AI Act to apply. For vulnerable groups — children, older persons, persons with disabilities, and socio-economically disadvantaged populations — these harms may be particularly severe and multifaceted due to their heightened susceptibility to exploitation. What may be considered an acceptable risk of harm for adults often represents an unacceptable harm for children and these other vulnerable groups. A precautionary approach is therefore particularly warranted in case of uncertainty and potential for significant harms.
For instance, children are highly impressionable and may not possess the cognitive maturity to evaluate persuasive content critically or resist certain exploitative practices that aim to keep them dependent on the AI-enabled services . This could, in turn, contribute to the shap ing of their values , beliefs and steer behaviours in potentially harmful ways . The significant harm here is both physical and psychological, exacerbated by the children ’s inability to discern and resist the exploitation and the harmful effects to their development and well-being that may have a long-term impact. For instance,
- AI systems used for the generation of child sexual abuse material (or manipulating existing material depicting real children to create further novel content featuring them) and the development of strategies for grooming and sexually extorting children are likely to cause serious harms and abuses of the affected children and often result in long-term physical, psychological and social consequences for survivors.88
- AI systems may target the vulnerabilities of young users and use addictive reinforced schedules with the objective to keep them dependent on the service are particularly harmful for young persons and girls. They may cause serious psychological and physical harms, including anxiety and depression, body dissatisfaction, eating disorders and mental health problems, including in some cases self-harm and suicidal behaviour.89 This may also have lo n g term harmful consequences for child development, including i mpaired cognitive development and learning and reduced social skills and displacement of experiences such as physical play, sleep, and face-to-face social interactions that are essential for the emotional and physical well-being of the child.90
- An AI system that is designed in an anthropomorphic way and simulates human-like emotional responses in its interactions with children can exploit children’s vulnerabilities in a manner that fosters unhealthy emotional attachment, manipulate s engagement time and distorts children’s understanding of authentic human relationships. This may hamper their normal social and emotional development and relations with other human beings and socio-emotional skills like empathy, emotional regulation, and social understanding and adaptability.91 As a result, this may lead to psychological harms such as increased anxiety and dependency of the children on the service and longer-term harms to the child’s well-being. Such deliberate addictive and exploitative design features of the AI-enabled services that can lead to combined significant harms , as described above, should be distinguished from other legitimate behaviour of the providers and deployers to pursue user engagement that respects individual autonomy and the safety of children and do not lead to significant harms which are outside the scope of Article 5(1)(b) AI Act (see section out of scope 5.3).
Similarly, older people may face cognitive decline and reduced digital literacy, making them prime targets for AI-driven scams or manipulative marketing. The harm in this case is often financial and psychological, compounded by the frustration and isolation many older people experience, which may be exploited to amplify the manipulative impact. For instance, an AI system that exploits the reduced cognitive vulnerabilities of older people by particularly targeting expensive medical treatments , unnecessary insurance policies or deceptive investments schemes to older persons may lead to significant loss of savings, increased debt, and emotional distress for older people. Certain AI-enabled differential pricing practices in key services such as insurance that exploit the specific soci al or economic situation and provide higher prices to lower income consumers can lead to a significant financial burden to pay more for the same coverage, leaving them vulnerable to shocks.92
Persons with disabilities also represent a vulnerable group that exploitative and manipulative AI systems may significantly harm. For instance, an AI system that uses emotion recognition to support mentally disabled individuals in their daily life may also manipulate them into making harmful decisions, like purchasing products promising unrealistic mental health benefits . This is likely to worsen their mental health condition and financially exploit them through the purchase of i neffective and expensive products , which is likely to cause them significant psychological and financial harms.
Socially or economically disadvantaged individuals are particularly susceptible to AI systems exploiting their financial desperation and precarious social situation and are often less informed and digitally literate. For instance, an AI chatbot could target specific socially or economically disadvantaged groups inciting them to commit acts of violence or injuries of other persons, by identifying their heightened susceptibility to certain types of content, fear-based narratives, or exploitative offers. The system’s targeted approach exacerbates the existing vulnerabilities of these socio-economically disadvantaged individuals, deepening their challenges . In certain cases this may lead to increased anxiety, depression, feelings of helplessness, social isolation, or self-harm and radicalisation to a point that reaches the threshold of significant harm under Article 5(1)(b) AI Act.
Unlike Article 5(1)(a) AI Act, Article 5(1)(b) AI Act does not explicitly refer to group harms, while recital 29 AI Act refers for both pr o hibitions to harms suffered by both specific persons and groups of individuals . The two prohibitions should thus be interpreted in a consistent manner aligned also with the safety logic of the AI Act and the objective of the prohibition in Article 5(1)(b) to protect all individuals belonging to the specific vulnerable groups due to age, disability and specific social or-economic situation. Harms that can be extern alised and affect other persons, even if not directly affected by the system, should therefore also be taken into account in the assessment of the significance of the harm under Article 5(1)(b) AI Act. For instance,
- The AI-enabled exploitation of children’s vulnerabilities may have long-term societal impacts, including increased prevalence of mental health concerns, healthcare costs, and reduced productivity due to chronic health issues.
- An AI system exploiting the financial vulnerabilities of economically disadvantaged people may lead to financial exclusion and creat e a downward spiral of socio-economic hardship for those disadvantaged groups . Such exploitation may cause societal harms with broader negative impacts on societal structures and values, including the perpetuation and exacerbation of discrimination and social inequality and the exclusion of those groups.
- A chatbot targeting certain vulnerable social or economic groups with misinformation or hate speech may lead to social polarisation and radicalisation, possibly igniting violence and even injuries and deaths of other persons.
These examples of exploitative AI practices should be distinguished from numerous other AI systems which do not exploit the vulnerabilities of children, persons with disabilities, or persons in specific social or economic situations and are not reasonably likely to cause significant harms, but aim to benefit those persons when properly designed and used (see also section 3.5. out of scope). For instance,
- AI systems that support children in their learning and in games;
- AI systems that help older persons in their daily life and improve their health and medical treatment, such as personal assistants or assistive robots , or improve their digital skills;
- AI systems that support the economic and other integration of socio-disadvantaged persons in the society, improve their skills, etc.;
- AI systems and devices that support visually or hearing impaired persons or pro vide adapted and personalised learning;
- AI systems that generate accessible solutions removing barriers for the use by persons with disabilities of products and services;
- AI-enabled prosthetics etc. t hat help disabled persons in their daily life and enable their integration and full participation in society.