Prohibitions § 4.2.2

For the prohibition in Article 5(1)(c) AI Act to apply, the social score created by or with the assistance of an AI system must lead to a detrimental or unfavourable treatment for the evaluated person or group of persons. In other words, the treatment must be the consequence of the score, and the score the cause of the treatmen t. Such a plausible causal link may also exist in cases where the harmful consequences have not yet materialised but the AI system is intended to or capable of producing such an adverse outcome. This is particular ly relevant given that the prohibited practice in Article 5(1)(c) AI Act also covers the ‘placing on the market’ of such AI systems.

Article 5(1)(c) AI Act does not requirethe evaluation or classification performed by the AI system to be the sole cause of the detrimental or unfavourable treatment. It therefore also covers AI-enabled scoring practices that may also be subject to or combined with other human assessments. At the same time, the AI output must play a sufficiently important role in producing the social score. For example, in the case where a public authority deploys an AI system to assess the trustworthiness of persons and combines its output with a human assessment of additional facts, this AI-enabled social scoring practice will fall within the scope of the prohibition onlyif the AI-generated score plays a sufficiently important role in the final decision, provided the other conditions for detrimental or unfavourable treatment are fulfilled as described below (see section 4.2.2. b).

A score may lead to detrimental or unfavourable treatment even if it is produced by an organisation(s) different from the one that uses the score .¹²⁰ For example, a public authority may obtain a score for a natural person ’s creditworthiness assessment produced by another company specialised in creditworthiness and risk assessments , which are based on information about the individuals and their behaviour from a variety of sources.

The final condition for the prohibition in Article 5(1)(c) AI Act to apply is that the use of the social score must result (or be capable of resulting) in detrimental or unfavourable treatment either: i. in social context(s) unrelated to the contexts in which the data was originally generated or collected, or ii. unjustified or disproportionate to the social behaviour or its gravity.

These conditions are alternative and may apply also in combination. A n analysis on a case-by-case basis is necessary to assess if at least one of them is fulfilled, since many AI-enabled scoring and evaluation practices may not f u lfil them and therefore be outside the scope of the prohibition . In particular, this may not be the case where the AI-enabled scoring practices are for a specific legitimate evaluation purpose and comply with applicable Union and national laws that specify the data considered as relevant for the purposes of evaluation and ensure that the detrimental or unfavourable treatment is justified and proportionate to the social behaviour (see section 4.3. out of scope).

‘Unfavourable treatment’ means that as a result from the scoring, the person or group of persons must be treated less favourably compared to other s without necessarily requiring a particular harm or damage (for example, in the case of scoring practices where people are singled out for additional inspections in case of fraud suspicious). By contrast, ‘ detrimental’ treatment requires the person or group of persons to suffer certain harm and detriment from the treatment. Unfavourable or detrimental treatment may also be discriminatory and prohibited under EU non-discrimination law or imply the exclusion of certain individuals or groups ,¹²¹ but that is not a necessary condition for the prohibition to apply . Article 5(1) ( c) AI Act could therefore cover unfair treatment beyond EU non-discrimination law that applies only to certain protected groups (e.g., age, ethnic and racial origin, sex, religion). Scenario 1: Detrimental or unfavourable treatment in unrelated social contexts

In the first scenario described under Article 5(1)(c)(i) AI Act , the detrimental or unfavourable treatment resulting from the score must take place in social context(s) unrelated to the contexts in which the data was originally generated or collected. This implies not only that the persons may be treated in a n unfavourable or detrimental manner due to the social score , but also that the data about their social behaviour or their known, inferred or predicted personal or personality characteristics are generated or collected in social contexts unrelated to the one in which the scoring takes place. The data collected or generated from these unrelated contexts must be subsequently used by the AI system for the scoring of the persons without an apparent connection for the purpose of the evaluation or classification or in a manner that leads to generalised surveillance of the persons or the groups of persons. In most cases, this happens against the reasonable expectations of the personsand in violation of Union data protection law and possibly other applicable rules that specify the types of data and sources considered relevant and necessary for the evaluation or classification. Whether this condition is fulfilled will require a case-by-case assessment, taking into account the purpose of the evaluation and the contexts from which the data has been collected and generated. Examples of detrimental or unfavourable treatment in unrelated social contexts prohibited under Article 5(1)(c)(i) AI Act

• National tax authorities use an AI predictive tool on all taxpayers’ tax returns in a country to select tax returns for closer inspection. The AI tool uses relevant variables, such as yearly income, assets (real estate property, cars etc.), data on family members of beneficiaries, but also unrelated data , such as taxpayers’ social habits or internet connections, to single out specific individuals for inspections.
• A social welfare agency uses an AI system to estimate the probability of fraud by beneficiaries of household allowance s that relies on characteristics collected or inferred from social contexts with no apparent connection or relevance for the assessment of fraud, such as having a spouse of a certain nationality or ethnic origin, having an internet connection, behaviour on social platforms , or performance at the workplace, etc.¹²² By contrast, data that is relevant for the allocation of the benefits and lawfully collected could be used to determine the risk of fraud , since public authorities pursue a legitimate aim in verifying if social benefits are correctly allocated.
• A public labour agency uses an AI system to score unemployed individuals based on an interview and a n AI-based assessment for determining whether an individual should benefit from state support for employment . That score is based on relevant personal characteristics , such as age and education, but also variables collected or inferred from data and contexts with no apparent connection to the purpose of evaluation, such as marital status, health data for chronic diseases, addiction, etc.¹²³ These unacceptable scoring practices may be distinguished from l awful practices that evaluate persons for specific purpose in compliance with Union and national law , in particular when such laws, in compliance with EU law, specify the data considered as relevant and necessary for the purposes of evaluation (see section 4.3. out of scope). Scenario 2: Unfavourable or detrimental treatment disproportionate to the social behaviour

Another alternative scenario under Article 5(1)(c)(ii) AI Act where an AI scoring system may be prohibited is if the treatment resulting from the score is unjustified or disproportionate to the gravity of the social behaviour. The severity of the impact and the interference with the fundamental rights of the person concerned resulting from the social scoring compared to the gravity of the social behaviour of the person should determine whether such treatment is disproportionate for the legitimate aim pursued , taking into account the general principle of proportionality. This requires a case-by-case assessment, which should consider all relevant circumstances of the case, as well as general ethical considerations and principles for fairness and social justice related to the assessment of the social behaviour and the proportionality of the detrimental treatment. The treatment may also be ‘unjustified’, such as lacking a legitimate aim. Sectoral Union or national legislation setting specific criteria and procedures that regulate such potential detrimental or unfavourable treatment may also be relevant as part of this assessment. Examples of unjustified or disproportionate treatment compared to the social behaviour prohibited under Article 5(1)(c) ii) AI Act

• A public agency uses an AI system to profile families for early detection of children at risk based on criteria such as parental mental health and unemployment, but also information on parents’ social behaviour derived from multiple contexts. Based on the resulting score, families are singled out for inspection and children considered ‘at risk’ are taken from their families, including in casesof minor transgressions by theparents, such as occasionally missing doctors’ appointments or receiving traffic fines.
• A municipality use s an AI system to score trustworthiness of residents based on multiple data points related to their social behaviour in a variety of contexts. The generated score for residents considered ‘less trustworthy’ is used for blacklisting, i.e. withdrawal of public benefits, other serious punitive measures, and increased control or surveillance. Among the factors considered in the assessment are insufficient volunteering and minor misbehaviour, such as not returning books to the library on time, leaving rubbish on the street outside the day of collection , and a delay in the payment of local taxes. These unacceptable social scoring practices may be distinguished from lawful practices that evaluate persons for a legitimate specific purpose in compliance with Union and national law, in particular where those laws ensure that detrimental or unfavourable treatment is justified and proportionate to the social behaviour (see section 4.3. out of scope).

Both alternatives under Article 5(1) (c)(i) and (ii) AI Act may also be fulfilled simultaneously. Examples of unjustified or disproportionate treatment under Article 5(1)(c)) i) and ii) AI Act

• A tax authorit y uses an AI system to detect child benefit fraud by profiling and assigning beneficiaries suspected of fraud to categories such as ‘deliberate intent/gross negligence’ using criteria such as low income, dual nationality, social behaviour, etc. Based on the risk score, a ben eficiary’s file is inspected and, in many cases, their childcare benefit ceased, they receive notice to repay the received benefits, and no longer qualify for standard debt collection arrangements. Such scoring causes many families to be heavily indebted and lead s to unjust, discriminatory and detrimental treatment of individuals and groups of individuals , ¹²⁴ driving many families into severe financial hardship.
• A public authority uses an AI system to control fraud in the student housing grant process that considers among the indicators the internet connections, family status or level of education of beneficiaries as distinguishing factor s for fraud risk, which do not seem relevant, nor justified.
• A government introduces a comprehensive AI-based system that monitors and rates citizens based on their behaviour in various aspects of life, such as social interactions, online activit ies, purchasing habits , and punctuality in paying bills . People with a lower score face restricted access to public services, higher interest rates on loans, and difficulty in traveling, renting apartments, and even finding jobs. The system leads to excessive surveillance of individuals and detrimental treatment in contexts unrelated to the social behaviour used to determine the social score (e.g., job opportunities are influenced by social media activity), while also imposing excessive penalties for minor infractions ( e.g., social and financial disadvantages for relatively minor offences). These unacceptable social scoring practices may be distinguished from lawful practices evaluating persons for legitimate specific purposes that do not fulfil th ese conditions and are in compliance with Union and national law, in particular whenthose laws ensure that detrimental or unfavourable treatment is justified and proportionate and data from related social contexts is used (see section 4.3. out of scope).

The prohibition under Article 5(1)(c) AI Act may also cover cases where awards or preferential treatment are given to certain individuals or groups of persons , since this implies less favourable treatment of other individuals (e.g., in cases of support employment programmes, (de-)prioritisation for housing or resettlement).