Prohibitions § 3.2.1

Article 5(1)(a) AI Act prohibits three alternative types of manipulative techniques: a) subliminal techniques beyond a person’s consciousness: b) purposefully manipulative techniques; and c) deceptive techniques. To fall within scope of Article 5(1)(a) AI Act, an AI system must deploy one or more of these techniques.

While t he AI Act does not define ‘subliminal techniques ’, Article 5(1) (a) AI Act specifies that subliminal techniques operate beyond (below or above) the threshold of conscious awareness . Because subliminal techniques and the way they operate are inherently covert , such techniques bypass a person’s rational defences against manipulation and are capable of influencing decisions without the conscious awareness of the person, raising significant ethical concerns and impairing individual autonomy, agency and free choice.⁵³

The subliminal techniques must be capable of influencing behaviour in ways in which the person remains unaware of such influence , how it works, or its effects on the person’s decision-making or value-and opinion-formation. In particular, s ubliminal techniques may use stimuli delivered through audio, visual, or tactile media that are too brief or subtle to be noticed and that have been traditionally known and prohibited in other sectors , such as media advertising.⁵⁴ These stimuli, while not consciously perceived, may still be processed by the brain and influence behaviour. Examples of subliminal techniques (not necessarily prohibited unless all other conditions listed in Article 5(1)(a) AI Act are fulfilled) include:

• Visual Subliminal Messages : an AI system may show or embed images or text flashed briefly during video playback which are technically visible , but flashed too quickly for the conscious mind to register , while still being capable of influencing attitudes or behaviours.
• Auditory Subliminal Messages: an AI system may deploy sounds or verbal messages at low volumes or masked by other sounds, influencing the listener without conscious awareness. These sounds are still technically within the range of hearing , but are not consciously noticed by the listener due to their subtlety or masking by other audio.
• Tactile Subliminal Stimuli: an AI system may stimulate subtle physical sensations that are perceived unconsciously, capable of influencing emotional states or behaviour.
• Subvisual and Subaudible Cueing : an AI system may deploy stimuli that are not just subtle or masked, but are presented in a way that makes them entirely undetectable by the human senses under normal conditions, for example flashing visual stimuli (e.g., flashing images) too quickly for the human eye to detectconsciously or playing sounds at volumes imperceptible to the human ear.
• Embedded Images: an AI system may hide images within other visual contentwhich are not consciously perceived , but may still be processed by the brain and influence behaviour.
• Misdirection: an AI system may draw attention to specific stimuli or content to prevent noticing other content, often by exploiting cognitive biases and vulnerabilities in attention.
• Temporal manipulation: an AI system may alter the perception of time in user interactions, thus influencing their behaviour and causing impatience and dependence.

The rapid development of AI and related technologies, such as big data analytics, neuro technologies, brain-computer interfaces and virtual realityheightens the risk of , sophisticated subliminal manipulation and its capability to effectively influence human behaviourin a subconscious manner.⁵⁵ AI can also extend to emerging machine-brain interfaces and advanced techniques like dream-hacking and brain spyware. For example, a game can leverage AI-enabled neuro technologies and machine-brain interfaces that permit users to control (parts of) a game with headgear that detects brain activity. AI may be used to train the user’s brain surreptitiouslyand without their awareness to reveal or infer from the neural data information that can be very intrusive and sensitive (e.g., personal bank information, intimate information, etc.) in a manner that can cause them significant harm. The prohibition in Article 5(1)(a) AI Act targets only cases of such significantly harmful subliminal manipulation and not machine-brain interface applications in general when designed in a safe and secure manner and respectful of privacy and individual autonomy.

‘Purposefully manipulative techniques’ are not defined in the AI Act, but they should be understood as techniques that are designed or objectively aim to influence, alter, or control an individual’s behaviour in a manner that undermines their individual autonomy and free choices. Manipulative techniques are typically designed to exploit cognitive biases, psychological vulnerabilities, or situational factors that make individuals more susceptible to influence. Because of their adaptability, AI systems are also able to respond well to a person ’s individual circumstances or vulnerabilities and increase the effectiveness and impact of manipulation at scale. While the manipulative capability is an important element to determine the nature of the technique, it is not necessary that the provider or deployer or the system itself deploying the manipulative techniques also intends to cause harm.⁵⁶

While not all manipulative techniques operate beyond the threshold of conscious awareness, many doand there may be an overlap with subliminal techniques, since such techniques also ultimately have manipulative effects. Recital 29AI Actclarifies that the prohibitionin Article 5(1)(a) also covers techniques where individuals, even ifthey are aware of the influence attempt, may notbe able tocontrol or resistitsmanipulative effect.⁵⁷ As a result, individuals are influenced or pushed into behaviour and decisions they would normally not have made if they were not subjected to the manipulative techniques to a point that undermines their individual autonomy or free choice. An example of purposefully manipulative techniques is sensory manipulation where an AI system deploys background audio or images that lead to mood alterations, for example increasing anxiety and mental distress that influence users ’ behaviour to the point of creating significant harm. Another example is personalised manipulation where an AI system that creates and tailors highly persuasive messages based on an individual’s personal dataor exploits other individual vulnerabilities influences their behaviour or choices to a point of creating significant harm.

The prohibition against purposefully manipulative techniques also cover s AI systems that manipulate individuals without any human intending them to do so. Article 5(1)(a) AI Act prohibits AI systems that deploy certain techniques or exhibit a specific manipulative behaviour. Therefore, it could also be the AI system that deploys such manipulative techniques, rather than the provider or the deployer that has designed or used the system in this way. For example, regardless of whether the provider intend s it, an AI system may learn manipulative techniques because the data on which it is trained contain many instances of manipulative techniques,⁵⁸ or because reinforcement learning from human feedback can be ‘gamed’ through manipulative techniques.⁵⁹ By contrast, if the manipulative behaviour of the system is merely incidental, the system should not be considered deploying purposefully manipulative techniques as long as the provider has taken appropriate preventive and mitigating measures in case significant harms are reasonably likely to occur (see section 3.2.3.c) below).

The AI Actdoes not define ‘deceptive techniques’. Recital 29 AI Act clarifies that these are techniques that subvert or impair a person’s autonomy, decisionmaking or free-choice in ways that the person is not consciously aware or, where it is aware, can still be deceived or is not able to control or resist them. ‘Deceptive techniques’ deployed by AI systems should be understood to involve presenting false or misleading information with the objective or the effect ofdeceiving individuals and influencing their behaviour in a manner that undermines their autonomy, decision-making and free choices.

In this context, the interplay between the prohibition in Article 5(1)(a) AI Act and the deployer’s obligations in Article 50(4) AI Act to label ‘deep fakes’ and certain AI-generated text publications on matters of public interest , ⁶⁰ as well as the provider’s obligation to ensure AI systems interacting with people are designed in a way to inform people that they are interacting with AI and not a human , ⁶¹ should be clarifie d. Such visible disclosure constitutes a mitigating measure that should also be enabled through design features embedded in the AI system provided by the provider, including technical measures enabling the detection of AI-generated and manipulated content .⁶² The visible labelling of ‘deep fakes’ and chatbots reduces the risk of deception that is likely to arise once the AI-generated content is disseminated to the public and reduces the risk of harmful distorting effects on the individual’s opinionand belief- -formation and behaviour.

By contrast, the prohibition in Article 5(1)(a) AI Act has a much more limited scope. It may, for example , cover cases where a chatbot or deceptive AI-generated content presents false or misleading information in ways that aim to or have the effect of deceiving individuals and distorting their behaviour that would not have happened if they were not exposed to the interaction with the AI system or the deceptive AI generated content, in particular if this has not been visibly disclosed.⁶³

As with purposefully manipulative techniques, the prohibition of deceptive techniques may also cover AI systems that deceive individuals without any human intending them to do so (see section 3.2.1.b)above). For example, regardless of whether their providers intend such an outcome , AI systems may learn deceptive techniques simply because this increases their performance for the task for which they were developed, for example by reinforced learning.⁶⁴ An example of deceptive techniques that may be deployed by AI is an AI chatbot that impersonates a friend of a person or a relative with synthetic voice and tries to pretend it is the person causing scams and significant harms. Another example is an AI system that learns to identify when it is under evaluation and temporarily halt s any undesired behaviour, only to resume such behaviour once the evaluation period is over.⁶⁵ Such deceptive behaviour is particularly dangerous, since it defies any external human oversight over the system and may be prohibited if it is reasonably likely to cause significant harms. By contrast, a generative AI system that incidentally presents false or misleading information and hallucinates⁶⁶ may not be considered to deploy deceptive techniques within the meaning of Article 5(1)(a) AI Act , taking into account the limitations and the state of the art of generative AI . In particular , this may be the case where the provider of the system has properly informed users about the system’s limitations and integrated appropriate safeguards into the system to minimise such outcomes and provided that the system is not intended for, nor deployed in, sensitive contexts (e.g., health, education, elections) where serious harmful consequences are likely to occur (see also considerations in section 3.2.3.c) below).

Article 5(1)(a) AI Act applies to subliminal, purposefully manipulative , or deceptive techniques, or to combination s of such techniques that can have a compound impact. As stated above, purposefully manipulative techniques may be also subliminal in nature, if they operate beyond the threshold of conscious awareness.

Furthermore, when purposefully manipulative and deceptive techniques are applied in combination, this may significantly influence the behaviour of individuals , leading them to make decisions based on unconscious manipulations and false beliefs . This combination may create a feedback loop where individuals are less likely to question or critically evaluate the information received, since the manipulative elements have already primed their cognitive biases and emotional responses.