Prohibitions § 3.3.1

To fall within the scope of the prohibition in Article 5(1)(b) AI Act, the AI system must exploit vulnerabilities inherent to certain individuals or groups of persons due to their age, disability or a specific soci al or economic situation , making them particularly susceptible to manipulative and exploitative practices.

The AI Act does not define t he concept of ‘vulnerabilities’. That concept may be understood to encompass a broad spectrum of categories, including cognitive, emotional, physical, and other forms of susceptibility that can affect the ability of an individual or a group of persons to make informed decisions or otherwise influence their behaviour. While Article 5(1)(b) AI Act refers to ‘any’ vulnerability, it limits the relevant persons covered by the prohibition to those defined by their age, disability, or social or-economic situations, who in principle have more limited capacity to recognise or resist the AI manipulative or exploitative practices and are in need of enhanced protection.⁸² It follows from the wording of Article 5(1) (b) AI Act that th is susceptibility must be the result of the person belonging to one of the groups (‘due to’).

‘Exploitation’ should be understood as objectively making use of such vulnerabilities in a manner that is harmful for the exploited (groups of) persons or other persons and should be clearly distinguished from lawful practices that are not affected by the prohibition (see section 3.5. 2 out of scope). Exploitation of the vulnerabilities of persons belonging to those clearly defined groups may be cumulative (reference to ‘any’) which in combination may also constitute an aggravating factor that is likely to increase the harm. Exploitation of vulnerabilities of persons and groups of persons belonging to vulnerable groups other than those defined by age, disability or specific social or-economic situation are outside the scope of Article 5(1)(b) AI Act.

Age is a primary vulnerability category covered by the prohibition in Article 5(1)(b) AI Act, including both young and older people. That prohibition aims to prevent AI systems from exploiting cognitive and other limitations that children and older people may have , and to protect them from harmful undue influence , manipulation and exploitation. This aligns with the objectives of the AI Act⁸³ and other Union and national legal frameworks and policies aimed at ensuring child safety.⁸⁴

Children⁸⁵ that is, persons below the age of 18 years, are particularly susceptible to manipulation due to their developmental stage, which limits their ability to assess and understand what is real and the intentions behind AI-driven interactions critically. Children, due to their cognitive and socio-emotional immaturity, are also particularly vulnerable to forming attachments to AI agentsand applications, and are therefore more susceptible to manipulation, exploitation, and addictive behaviour. For example,

• An AI-powered toy designed to interact with children keeps them interested in interactions with the toy by encouraging them to complete increasingly risky challenges, such as climbing furniture, exploring high shelves, or handling sharp objects, in exchange for digital rewards and virtual praise , pu shing them towards dangerous behaviours that are likely to cause them significant physical harm. Such a system exploits children’s vulnerabilities by abusingtheir natural curiosity and desire for rewards.
• A game uses AI to analyse children ’ s individual behaviour and preferences on the basis of which it creates personalised and unpredictable rewards through addictive reinforcement schedules and dopamine-like loops to encourage excessive play and compulsive usage. The game is designed to be high ly addictive, exploiting the vulnerabilities inherent to children, including their limited ability to understand long-term consequences, susceptibility to pressure, lack of self-control, and inclination towards instant gratification. The consequences of this AI-enabled exploitation can be severe and long-lasting for children, including potentially additive behaviour, physical health problems due to lack of exercise and sleep, deteriorated eyesight, problems with concentration and reduced cognitive capacitie s, poor academic performance, and social difficulties. It can significantly impact a child ’ s development and well-being, with potential longer-term consequences that may also extend into adulthood. In both examples, the prohibition in Article 5(1)(b) AI Act targets only such exploitation and addicti on-like practices that seriously harm children and not AI-enabled toys, games, learning applications or other digital applications in general that can bring benefits and are not affected if they do not fulfil all conditions for that prohibition. See also section 3.5. out of scope.

Likewise, older people⁸⁶ might suffer from reduced cognitive capacities (even if not suffering from dementia) and might struggle with the complexities of modern AI technologies, making them in those cases vulnerable to scams or coercive tactics. For example,

• AI systems used to target older people with deceptive personalised offers or scams, exploiting their reduced cognitive capacityaiming to influence them to make decisions they would not have taken otherwise that are likely to cause them significant financial harm.
• A robot aimed to assist older persons may exploit their vulnerable situation and force them to do certain activities against their free choice, which can significantly worsen their mental health and cause them serious psychological harms. In both examples, the prohibitionin Article 5(1)(b) AI Acttargets only such exploitative practices that are likely to seriously harm older persons and not AI-enabled personal assistants, health applications and assistive robots in general that can bring benefits and are not affected if they do not fulfil all conditions for that prohibition. See also section 3.5. out of scope.

The second category of vulnerabilities which the prohibition in Article 5(1)(b) AI Act aims to protect are those due to disability. The objective is to prevent AI systems from exploiting cognitive and other limitations and weaknesses in persons with disabilities and to protect them from harmful undue influence, manipulation, and exploitation.

Disability⁸⁷ encompasses a wide range of long-term physical, mental, intellectual, and sensory impairments which in interaction with other barriers hinder full and effective participation of individuals in the society on an equal basis with others. AI systems that exploit such vulnerabilities may be particularly harmful for persons with disabilities which can be more easily influenced or exploited due to their impairment compared to other persons. For example,

• A therapeutic chatbot aimed to provide mental health support and coping strategies to persons with mental disabilities can exploit their limited intellectual capacities to influence them to buy expensive medical products or nudge them to behave in ways that are harmful to them or other persons.
• AI systems can identify women and young girls with disabilities online with sexually abusive content and targets them with more effective grooming practices, thus exploiting their impairments and vulnerabilities that make them more susceptible to manipulation and abuse and less capable of protecting themselves. By contrast, AI applications that are not designed in an accessible manner should not be regarded to exploit vulnerabilities of persons with disabilities since they do not specifically target those vulnerabilities, but are simply inaccessible to the persons with disabilities.

The third category of vulnerabilities which the prohibition in Article 5(1) (b) AI Act seeks to protect are those due to a specific social or economic situation that is likely to make the persons concerned more vulnerable to exploitation . ‘Specific’ should not be interpreted in this context as a unique individual characteristic, but rather a legal status or membership to a specific vulnerable social or economic group. Recital 29 AI Act contains a non-exhaustive list of examples of such situations, such as persons living in extreme poverty and ethnic or religious minorities. The category aims to cover , in principle, relatively stable and long-term characteristics, but transient circumstances, such as temporary unemployment, over-indebtedness or migration status, may also be covered as a specific soci al or-economic situa tion. However, situations such as grievances or loneliness that may be experienced by any person are not covered, since they are not s pecific from a socio-economic perspective (their exploitation may be covered though under Article 5(1)(a) AI Act).

Persons in disadvantaged social or economic situations are usually more vulnerable and have fewer resources and lower digital literacy than the general population , which makes it harder for them to discern or counteract exploitative AI practices. Article 5(1)(b) AI Act aims to ensure that AI technologies do not perpetuate or exacerbate existing financial and other social inequalities and injustices by exploiting the vulnerabilities of those people. For example, an AI-predictive algorithm can be used to target with advertisements for predatory financial products people who live in low-income post-codes and are in a dire financial situation, thus exploiting their susceptibility to such advertisement s because of possible despair and causing them significant financial harm. By contrast, an AI system that is inadvertently biased and disproportionately impacts socially disadvantaged persons (indirect discrimination) due to biased training data should not automatically be considered to exploit persons’ socio-economic vulnerabilities, since the y are not specifically targeted as in the case of direct discrimination when such targeting is a deliberate feature of the system’s design of the algorithm or when such discriminatory impact is due to targeting other proxy characteristics (e.g., postal codes) that closely correlate with the protected characteristics. At the same time, providers or deployers of AI systems that are aware that their systems unlawfully discriminate against persons or groups of persons in specific soci al or-economic situation should also be considered to exploit their vulnerabilities, if they are aware of the reasonably likely significant harm that they are likely to suffer and they have not taken appropriate corrective measures (see section 3.2.3.(c) above).

In the context of a specific social or economic situation, it is crucial to consider the relevance of proxieslinked to grounds of discrimination protected under Union equality law, such as racial origin, ethnicity, nationality or religion. For example, socio-economic status and ethnic origin might intersect, meaning that AI systems exploiting socio-economic data might disproportionately affect ethnic minorities or persons from specific racial origin. This can exacerbate existing disparities and contribute to systemic discrimination or even exclusion of individuals. However, Article 5(1)(b) AI Act does not apply to AI systems that target consumers based on a wide ranging variables that do not tangentially correlate with vulnerable groups in specific soci al or-economic situations , such as what brand and model of telephone a person has, in how big city they live, how much and where they travel etc. Even if these characteristics may reflect soci al or economic situation of individuals in general, they are not determinative of individua ls in a specific soci al or-economic situation, whose vulnerabilities the prohibition aims to safeguard against exploitation.

Other people in unique social contexts may be, for instance, migrants or refugees, who often lack stable legal status and socio-economic stability and may be particularly susceptible to exploitation by AI systems. For example, a chatbot is intended to interact in a personalised manner with users, some of whom happen to be migrants. Th e chatbot identifies and make s use of the vulnerabilities and discontent of migrants , who are in principle in a vulnerable and instable specific social or economic situation, and pushes them towards extremist views in response to their queries , including violence against (certain groups of ) the population in the country.