Prohibitions § 3.2.2

A third condition for the prohibition in Article 5(1) (a) AI Act to apply is that the deployed subliminal, purposefully manipulative, or deceptive technique must have ‘the objective, or the effect of materially distorting the behaviour of a person or a group of persons’. This implies a substantial impacton the behaviour where a person’s autonomy and free choices are undermined, rather than a minor influence. However, intent is not a necessary requirement , since Article 5(1)(a) AI Act also covers practices that may only have the ‘effect’ of causing material distortion. There should be a plausible/reasonably likely causal link between the potential material distortion of the behaviour and the subliminal, purposefully manipulative or deceptive technique deployed by the AI system.

The concept of ‘material distortion of the behaviour’ of a person or a group of persons is central to Article 5(1)(a) AI Act . It involves the deployment of subliminal, purposefully manipulative or deceptive techniques that are capable of influencing people’s behaviour in a manner that appreciably impairs their ability to make an informed decision, thereby causing them to behave in a way or to take a decision that they would otherwise not have taken.

‘A ppreciable impairment’ refers to a substantially reduced ability to make informed and autonomous decisions , thereby causing individuals to behave in a way or to take a decision that they would otherwise not have taken . It goes beyond minor or negligible impacts and involves a significant distortion or hindrance in decision-making and free choice, including in relation to opinion-and belief-formation. This suggests that ‘material distortion’ involves a degree of coercion, manipulation, or deception that goes beyond lawful persuasion, which falls outside the scope of the prohibition (see section 3.5.1. below).

An ‘informed decision ’ requires a n understanding and knowledge of the relevant information, including the available options, the risks and benefits of each choice, the possible effects of the AI system on their behaviour , and, as appropriate, other contextual information that is important for the decision-making or the behaviour of the person.

For the interpretation of the concept of ‘material distortionof behaviour ’, Union consumer protection law , in particular, Directive 2005/29/EC (Unfair Commercial Practices Directive or ‘UCPD’), may constitute a valid source of inspiration. The UCPD prohibits various unfair, misleading, and aggressivecommercial practices (Articles 5 to 9 UCPD) capable of causing consumers to make transactional decisionsthat they would otherwise not have made. According to the CJEU and the Commission guidance on the UCPD,⁶⁷ there is no need to prove that a consumer ’s economic behaviour has been distorted, it suffices to establish that a commercial practice is ‘likely’ (i.e. capable) of impacting an average consumer ’ s transactional decision.⁶⁸ The CJEU has also underscored that even accurate information may be misleading if presented in a way that distorts the consumer’s decisionmaking process. - ⁶⁹ National enforcement authorities are tasked to investigate the specific f acts and circumstances of each case (in concreto) and to evaluate the potential impact of the practice on the average consumer’s decision-making process (in abstract).⁷⁰ For that purpose, the y must take the point of view of the ‘average’ consumer, which is the benchmark developed by the CJEU, now integrated in the UCPD.⁷¹

In the context of the prohibition in Article 5(1) (a) AI Act , market surveillance authorities must also investigate each case’s specific facts and circumstances, assessing whether the subliminal, purposefully manipulative or deceptive technique deployed by the AI system is likely to appreciably impair the decision-making, individual autonomy and free choice of an ‘average’ individual within a targeted group when the system affects a group of persons in a manner that is reasonably likely to cause significant harm. Such an interpretation seems justified given that the AI Act intends to complement the UCPD⁷² and must be applied in a consistent manner. At the same time, given that Article 5(1)(a) AI Act also refers to the possibility to distort the ‘behaviour of a natural person’and, if the perspective of the ‘average’ individual proves difficult or ineffective to assess in certain contexts (for example due to very tailored or ‘personalised’ manipulation or harmful effects on specific vulnerable groups), specific cases may be examined also from the perspective of specific individuals by assessing to what extent an AI system deploying subliminal, purposefully manipulate or deceptive techniques is capable of undermining their individual autonomy in concrete cases and significant harm has occurred or is likely to occur.

Article 5(1)(a) AI Act applies to AI systems deploying the above-mentioned techniques and having as a first scenario ‘the objective to materially distort the behaviour of a person or a group of persons’. Such an objective may be pursued by the provider or the deployer of the AI system, or by the system itself within the implicit objectives it may pursue.⁷³ This objective should be distinguished from the ‘intended purpose’ of the AI system (Article 3(12) AI Act). Even if intended by the provider, the manipulative objective is in most cases not the purpose of the use for which the system is offered and it is often neither t ransparent, nor specified as such in the information supplied by the provider (e.g., in the instructions for use, promotional or sales materials and statements, as well as in the technical documentation). For example, a chatbot that may be used in different contexts is designed to use subliminal messaging techniques, such as flashing brief visual cues and embedding inaudible auditory signals or to exploit emotional dependency or specific vulnerabilities of users in advertisements. These techniques are deployed ‘with the objective to’ materially distort users’ behaviour, since theyare objectively a design feature that aims to influence consumers ’ purchasing decisions without their conscious awareness, to push people to take significantly harmful financial decisions. Deploying AI systems to impersonate other persons could also be considered as an AI system deployed ‘with the objective’ to deceive and materially distort the behaviour of persons if the person is effectively deceived, thus substantially affecting their ability to make informed decisions about the identity of the person. If in both examples the other conditions in Article 5(1) (a) AI Act are fulfilled , in particular regarding the significant harm, those systems are likely to fall within the scope of the prohibition, but this will require a case-by-case assessment.

The intent of the provider or deployer to materially distort the behaviour of a person or group of persons is a sufficient, but not a necessary , condition for the prohibition in Article 5(1)(a) AI Act to apply. That prohibition also applies where no such intent is present, but the effect of the technique(s) deployed by an AI system is capable of materially distorting the behaviour of a person or a group of persons to a point that undermines their individual autonomy and free choices.

A plausible/reasonably likely causal link between the subliminal, purposefully manipulative or deceptive technique deployed by the AI system and its effects on the behaviour is, however, always necessary for the prohibition to apply. In consistency with consumer protection law, these effects do not need to have fully materialised, but there must be sufficient indication that they are likely or capable to materialise and undermine individual autonomy based on an objective assessment of all circumstances of the case and existing scientific knowledge and methods , as well as available information about the impact of the system on the individuals’ behaviour in real life. In this context, the fact that a system is capable of triggering behaviours that appreciably impair individuals ’ ability to make an informed decision and undermine their free choices suffice to fulfil that condition and is not dependent on considerations relating to the ‘timing’ for the harm to materialize(e.g., in case of addiction-like behaviours) as long as it is reasonably likely to occur. For example, an AI-powered well-being chatbot is intended by the provider to support and steer users in maintaining a healthy life style and provide tailored advice for psychological and physical exercises. However, if the chatbot exploits individual vulnerabilities to adopt unhealthy habits or to engage in dangerous activities (e.g., engage in excessive sports without rest or drinking water) where it can reasonably be expected that certain users will follow that advice, which they wouldotherwise not have done, and suffer significant harm (e.g., a heart attack, or other serious health problem), that AI system would fall under the prohibition in Article 5(1)(a) AI Act, even if the provider might not have intended this behaviour and harmful consequences for the persons. The mere fact that the chatbot is capable of appreciably impairing individual autonomy and materially distorting the behaviour of certain users in a significantly harmful way and that the provider has not taken appropriate preventive and mitigating measures to avoid those significantly harmful effects suffices for the prohibition to apply (see more for relevant considerations of the reasonablylikeliness of the harm in section 3.2.3. and out of scope section 3.5.).