Prohibitions § 3.2.3

Finally, for the prohibition in Article 5(1) (a) AI Act to apply , the distortion of the behaviour of a person or group of persons must cause or be reasonably likely to cause that person, another person , or group of persons significant harm. In this context, important concepts that require clarification are the types of harm s covered by the prohibition, the threshold of significance of the harm, and its reasonable likelihood and causal link between the harm and the manipulative or deceptive technique and the person’s behaviour.

The AI Act addresses various types of harmful effects associated with manipulative and deceptive AI systems, each with distinct implications for individual personsand groups of persons that may be affected⁷⁴. The main types of harms relevant for Article 5(1)(a) AI Act include physical, psychological, financial, and economic harms⁷⁵ that may be compound with broader societal harms in certain cases⁷⁶.

Physical harm encompasses any injury or damage to a person’s life, health and material damage to property. Physical harm to a person’s life and healthhave, in many cases, immediate, serious, and irreversible consequences. In line with its product safety logic, the AI Act aims to prohibit AI-enabled manipulation and deception resulting in significant physical harm. For example, an AI chatbot promotes self-harm to users or incentivises them to commit suicide or harm other persons or groups of persons by promoting terrorist content or incentivising violence against certain persons or groups of persons (i.e. minorities).

Psychological harm is particularly relevant in the context of AI systems deploying manipulative techniques that exploit cognitive and emotional vulnerabilities and influence an individual’s behaviours in ways that can cause significant harm. Psychological harm s encompass adverse effects on a person’s mental health and psychological and emotional well-being. Such harms are particularly significant because they may accumulate over time and may not be immediately apparent, but may produce long-lasting and severe consequences.However, it is more difficult to measure them, which requires a case-by-case assessment, in particular to determine their severity, taking into account all relevant circumstances of the case. For example, an AI companionship application designed to emulate human speech patterns, behaviours and emotions uses anthropomorphic features and emotional cues to influence users’ feelings, dispositions, and opinions, making those users emotionally dependent on the service incentivising addiction-like behaviour and potentially causing significant harms, such as suicidal behaviours and risks of harming other persons.⁷⁷

Financial and economic harm may encompass a range of adverse effects, including financial loss, financial exclusion, economic instability. For example, a chatbot that offers fraudulent products that cause significant financial harms.

In assessing the harms caused by AI systems when applying Article 5(1)(a) AI Act, it is important to highlight that the harms are often not isolated, but manifest themselves in combination, leading to compounded and multifaceted negative impacts. Understanding the combination of harm is crucial to effectively assess their significance (see also see section 3.2.3. b) below), whereby physical, psychological, financial and economic harm may be combined and exacerbate the overall impact on individuals and communities and may even have broader adverse impacts. For example,

• An AI system that causes physical harm may also lead to psychological trauma, stress, and anxiety and vice versus. For example, addictive design of AI systems used in products and other AI-enabled applications may lead to psychological harm by fostering addictive behaviours, anxiety, and depression . The psychological distress may subsequently result in physical harm, such as insomnia and other stress-related health issues and physical problems.
• AI-driven harassment may lead to both psychological distress and physical manifestations of stress, such as insomnia , deteriorated physical health or weakened immune system.
• Psychological harm resulting from the use of AI may also lead to physical harm, including death. For example, AI systems used online may facilitate gender-based violence through harassment, stalking, cyberbullying and sexual extortion.
• Individual psychological harms for example due to AI-enabled generation of ‘deep fakes’ impersonating actual persons to deceive and undermine the decision-making, individual autonomy and free choices of individuals can also be combined with significant harms for groups of persons (e.g. sharing the same ethnic or racial origin or gender as the victims depicted on the deep fakes).

The prohibition in Article 5(1) (a) AI Act only applies if the harm caused by the subliminal, manipulative and deceptive techniques is ‘significant’. The AI Act does not provide a definition for the concept of ‘significant harm’, but it should be understood as implying significant adverse impacts on physical, psychological health or financial and economic interests of persons and groups of persons . ⁷⁸ The determination of ‘significant harm ’ is fact-specific, requiring careful consideration of each case ’ s individual circumstances and a case-by-case assessment , but the individual effects should be always significant in each case.

In other U nion laws, the concept of ‘significant harm’ is also used as a nuanced and context-dependent concept guided by high-level protection and preventive action goals.⁷⁹ By analogy, the following key considerations can be derived and could be taken into account when assessing what constitutes significant harm within the meaning of Article 5(1)(a) AI Act:

• The severity of harm refers to the degree of harm that has resulted or is reasonably likely to result from using an AI system with objective and observable effects for the significant harms. It is particularly important to consider in this context the AI system's interdependencies , the combination of various types of harms , and the adverse effects on individuals or groups of persons.
• Context and cumulative effects:⁸⁰ The specific context, including the existing state, and the cumulative effects of multiple actions, play a n important role in assessing the severity of the harm.
• Scale and intensity : The extent of the harm and the intensity of adverse effects are critical in evaluating whether the harm is significant. Whether the harm impacts a large number of people is also relevant for assessing its significance.
• Affected persons' vulnerability: Certain groups, such as c hildren, the elderly, or persons with disabilities may be more susceptible to harm from specific AI systems. What may be considered less significant harm for persons in general might be considered significant and unacceptable for such vulnerable groups, especially children.
• Duration and reversibility: Long-lasting or irreversible harm typically meets the threshold for significant harm. Short-term and reversible effects might be considered less significant, unless they occur frequently.

The objective of the AI Act to ensure ‘a high level of protection’,in conjunction with Article 191(2) TFEU, suggests a comprehensive approach to protectionwhen assessing the significance of the harm. This means considering both immediate and direct harms and systemic , indirect adverse impacts associated with AI systems deploying subliminal, purposefully manipulative or deceptive techniques that are intended to or capable of impairing individual autonomy, decision-making and free choices of persons and groups of persons. For example, significant physical harm that is reasonably likely to be caused by an AI system includes injuries or fatalities or a sufficiently serious impact on individuals’ health or the destruction of property. AI systems that suggest to an individual to commit criminal acts such as sexual abuse and exploitation, extreme violent or terrorist content or incentivise individuals to commit crimes, self-harm or harm to other persons should be considered to reach such a threshold. By contrast, minor physical harms may include less severe injuries, such as bruises or temporary discomfort, which do not have significant or lasting consequences and will therefore not reach the threshold of significance within the meaning of Article 5(1)(a) AI Act. Whether the physical harm specifically concerns vulnerable groups, such as children, should be assessed, as should the scale of the harm and whether it is compounded with other types of harms, such as psychological , financial etc. This will require a case-by-case assessment , taking into account the circumstances and the criteria presented above to guide that assessment. There are numerous cases where the threshold of significant harm will likely not be reached even if the systems may be deploying subliminal, purposefully manipulative, or deceptive techniques (see for examples section 3.5. below).

The concept of ‘reasonably likely’ is used in Article 5(1)(a) AI Act to determine whether there is a plausible/reasonably likely causal link between the manipulative or deceptive technique capable of distorting the person’s behaviourin a manner that undermines their free choices and the potential significant harm . This concept allows the application of the prohibition not only in cases where the harm has occurred , but also where it is reasonably likely to occur in line with the safety logic of the AI Act. In this context, it is particularly relevant to assess whether the provider or deployer of the AI system could have reasonably foreseen the significant harm that is reasonably likely to result from the subliminal, purposefully manipulative or the deceptive techniques deployed and whether they implemented appropriate preventive and mitigating measures to avoid or mitigate the risk of such significant harm s . This implies a judgement of reasonableness on an objective basis and according to universally accepted criteria ( e.g., technical and scientific), including a criterion of rationality in establishing plausible causality between the AI practice and the signi ficant harm that may arise. The opacity or transparency of the AI system and its functioning may affect the conclusion regarding this causal link and, hence, the application of the prohibition.

To avoid providing or using AI system s that are likely to be prohibited, providers and deployers of AI systems that deploy such manipulative or deceptive techniques are encouraged to take appropriate measures such as: 1. Transparency and individual autonomy: provide transparency in how the AI system operates, clear disclosures about its capabilities and limitations, and relevant information to ensure informed decisions; respect individual autonomy and avoid engaging in exploitative or deceptive practices that are likely to appreciably impair an individual’sautonomy, decision-making and free choices in potentially harmful ways; integrate appropriate user control and safeguard measures to ensure that the system is not deceptive and operates within the boundaries of lawful persuasion that is outside the scope of the prohibition (see section 3.5.1). 2. Compliance with relevant applicable legislation: in many cases compliance with relevant applicable legislation will mitigate the risks of harm and indicate that the practice does not constitute a purposefully manipulative or deceptive practice and that mitigating measures have been put in place to prevent likely significant harms (see section 3.4. and 3.5.1.). 3. State of the art practices and industry standards: adherence to professional due diligence practices and industry standards for the responsible development and use of safe and ethical AI systems and measures to mitigate the harms can help to pre-empt and mitigate unintended significant harms.

By contrast, harms and distortion of the behaviour of individuals which result from factors external to the AI system and which are not within the control and reasonably foreseeable by the provider or the deployer to pre-empt and mitigate risks would not be relevant for the assessment whether there is a plausible causal /reasonably likely link between the distorted behaviour of the persons interacting with the system and the significant harm.⁸¹ For example, the provider of a n AI system may assess and try to mitigate potential harmful manipulative effects in the design of the system and the interactions with humans through the design, prior testing, and other proportionate mitigating measures, but it may not be in a position to foresee if a person may get depressed or change their behaviour due to other external factors in the ir personal life that are not known and beyond their interactions with the system.

Other examples that fall outside the scope of the prohibition as not fu l filling all conditions (e.g., in case of lawful persuasion) are provided in section 3.5. below.