Distinguishing manipulation from persuasion is crucial to delineate the scope of the prohibition in Article 5(1) (a) AI Act , which does not apply to lawful per suasion practices. While both manipulation and persuasion influence individuals’ decisions and behaviours, they differ significantly in methods and ethical implications.
Manipulation involves , in most cases , covert techniques undermining autonomy, leading individuals to make decisions they might not have otherwise made if they were fully aware of the influences at play. These techniques often exploit psychological weaknesses or cognitive biases. By contrast, persuasion operates within the bounds of transparency and respect for individual autonomy. It involves presenting arguments or information in a way that appeals to reason and emotions, but explains the AI system’s objectives and functioning , provide relevan t and accurate information to ensure informed decision-making and supports the individual ’ s ability to evaluate the information and make free and autonomous choices. For example, an AI system using personalised recommendations based on transparent algorithms and user preferences and controls engages in persuasion. By contrast, a system that uses subliminal clues (e.g., imperceptible images ) to influence users towards specific choices without their knowledge and understanding constitutes manipulation.
The objective and impact of these techniques also differ. Manipulation often aims at benefitting the manipulator at the expense of the individual’s autonomy and well-being. By contrast, persuasion aims to inform and convince, aligning interests and benefits for both parties. Ethical persuasion respects an individual ’ s autonomy to make informed choices and avoids exploiting vulnerabilities. For example, an AI system which operates in a transparent manner and analyses customers’ emotions to improve customer interactions and provide support with the knowledge of the users engages in persuasion and aligns with their interests . By contrast, an emotion recognition system used for targeted advertising that infers emotions of consumers in a hidden manner to offer products of higher prices at a specific moment when the user is more likely to buy them engages in manipulation and is to the detriment of the consumers.
Consent also plays an important role in certain c ases . In persuasive interactions, individuals are aware of the influence attempt and can freely and autonomously choose it. In manipulative interactions, the lack of awareness of the techniques or their impact negates the freedom of choice and informed and autonomous decision-making. For example, an AI system that aims to help users learn a foreign language better and faster through the deployment of subliminal techniquesis not manipulative if itoperates in a transparent manner and respects individual autonomy and user’s free and informed choice to consent to the use of the system or not.
Compliance with legal and regulatory frameworks also play s an important role in measuring manipulation as compared to lawful persuasion. AI practices that comply with applicable laws that uphold transparency, fairness, and individuals’ rights and autonomy are therefore more likely not to be prohibited under the AI Act. For example, compliance with data protection laws, such as the GDPR, which mandates transparency obligations in data processing, namely that the information to be provided to the data subjects should avoid deceptive or manipulative language . 93 In some instances, consent may be required for personal data processing to be lawful, as for certain online personalised advertisement s based on off-service users’ data in social networks.94 That consent must be, amongst others, free and informed. AI systems that meet these legal standards are more likely to engage in lawful persuasion. Conversely, systems that circumvent these requirements to influence behaviour are likely engaging in manipulation.
In particular, recital 29 AI Act clarifies that the prohibitions in Article 5(1)(a) and (b) AI Act do not affect lawful practices in the context of medical treatment under certain conditions. For example, AI-enabled subliminal techniques may be used in the psychological treatment of a mental disease or physical rehabilitation when carried out in accordance with the applicable law and medical standards, including obtaining the explicit consent of the individual or its legal representatives as a condition for use.
Furthermore, recital 29 AI Act clarifies that common and legitimate commercial practices, such as advertising, should not be regarded ‘in themselves’ or by their very nature as harmful manipulative, deceptive or exploitative AI-enabled practices. For example,
- Advertising techniques that use AI to personalise content based on user preferences are not inherently manipulative if they do not deploy subliminal, purposefully manipulative or deceptive techniques that subvert individual autonomy or exploit vulnerabilities in harmfu l ways as prohibited under Article 5(1)(a) and (b) AI Act. Compliance with the relevant obligations under the GDPR, consumer protection law and Regulation (EU) 2022/2065 (‘the DSA’)help to mitigate such risks.
- The development of AI models and classifiers to detect child sexual abuse online in accordance with applicable law are legitimate practices that are not exploitative of children’s vulnerabilities and are, on the contrary, essential to improve child safety online.
- AI systems used for providing banking services, such as mortgages and loans, that use the age or the specific socio-economic situation of the clientas an input, in compliance with Union legislation on financial service, consumer protection, data protection and non-discrimination, do not qualify as the exploitation of vulnerabilities within the meaning of Article 5(1)( b ) AI Act when they are designed to protect and support people identified as vulnerable due to their age, disability or specific socio-economic circumstances and are beneficial for those groups, contributing also to fairer and more sustainable financial services for those groups.
- AI systems t hat detect drowsiness and fatigue in drivers and alert them to rest in compliance with safety law s are beneficial and do not qualify as exploitation of vulnerabilities within the meaning of Article 5(1)(b) AI Act.