The prohibition in Article 5(1)(c) AI Act only applies to the scoring of natural persons or groups of persons, thus excluding in principle scoring of legal entities where the evaluation is not based on personal or personality characteristics or social behaviour of individuals, even if in some cases individuals may be indirectly impacted by the score (e.g., all citizens in a municipality in case of allocation of budget). However, if legal entities have been evaluated based on an overall score that aggregates the evaluation or classification of a group of natural persons based on their social behaviour or personal or personality characteristics and this score directly affects th ose persons (e.g. all employees in a company, students in a specific school whose behaviour has been evaluated), the practice may fall within the scope of Article 5(1) (c) AI Act if all other conditions are fulfilled. This will depend on a case-by-case assessment.
AI-based social scoring as a ‘probabilistic value ’ and prognosis should also be distinguished from individual ratings by users which assess the quality of a service (such as a driver in an online car-sharing platform or a host in an online platform for accommodation). Such ratings are the mere aggregation of individual human scores that do not necessarily involve AI, unless the data are combined with other information and analysed by the AI system for evaluating or classifying individuals fulfilling all conditions in Article 5(1)(c) AI Act.
Furthermore, the scoring of natural persons is not at all times prohibited, but only in the limited cases where all of the conditions of Article 5(1)(c) AI Act are cumulatively fulfilled, as analysed above . Recital 31 AI Act , in particular mentions that the prohibition ‘should not affect lawful evaluation practices of natural persons that are carried out for a spec i fic purpose in accordance wi t h Union and national law’. For example, credit scoring and risk scoring and underwriting are essential aspects of the services of financial and insurance businesses. Such practices , as well as other legitimate practices (i.e. to improve the quality and efficiency of services , to ensure more efficient claims handling , to perform specific employee evaluations , fraud prevention and detection, law enforcement or scoring of users’ behaviour on online platforms), are not per se prohibited, if lawful and undertaken in line with the AI Act and other applicable Union law and national law, which must comply with Union law.
In other words, AI systems which evaluate or classify individuals for the purposes of generating a social score in a lawful manner and for a specific purpose in the related context as that in which the personal data used for the score were collected are not prohibited, provided that any detrimental or unfavourable treatment from using the score is justified and proportionate to the gravity of the social behaviour.125
Compliance with sectoral Union legislation, such as in the field of credit-scoring, anti-money laundering, etc., that specifies the type of data that can be used as relevant and necessary for the specific legitimate purpose of evaluation and ensures that the treatment is justified and proportionate to the social behaviour may thus ensure that the AI practice falls outside the scope of the prohibition in Article 5(1)(c) AI Act. Examples of legitimate scoring practices in line with Union and national law that are outside the scope of Article 5(1)(c) AI Act:
- Financial credit scoring systems used by creditors or credit information agencies to assess a customer’s financial creditworthiness or outstanding debts, providing a credit score or determining their creditworthiness assessment , which are based on the customer’s income and expenses and other financial and economic circumstances, are out of scope of Article 5(1)(c) AI Act if they are relevant for the legitimate purpose of the credit scoring and if they comply with consumer protection laws126 specifying the type of data and the necessary safeguards to ensure the fair treatment of consumers in creditworthiness assessments.
- Companies have a legitimate interest to evaluate customers for financial fraud and those practices are not affected by the prohibition, if the evaluation is based on relevant data such as transactional behaviour and metadata in the context of the services, past history and other factors from sources that are objectively relevant to determine the risk of fraud and if the detrimental treatment is justified and proportionate as a consequence of the fraudulent behaviour.
- Information collected through telematic devices that show that a driver is speeding or not maintaining safe driving practices used by an insurer that offers telematics-based tariffs in relation to a policyholder ’s high-risk driving behaviour may be used to increase the premium of that policyholder due to the higher risk of an accident caused by that driving behaviour, provided the increase in the premium is proportionate to the risky behaviour of the driver.
- The collection and processing of data that is relevant and necessary for the intended legitimate purpose of the AI systems (e.g. hea l th and schizophrenic data collected from various sources to diagnose patients) is out of scope of Article 5(1)(c) AI Act, in particular because it process relevant and necessary data and typically does not entail unjustified detrimental or unfavourable treatment of certain natural persons.
- Online platforms profiling users for safety reasons on their services based on data which is relevant for the context and purpose of assessment is out of scope of Article 5(1)(c) AI Act , when the evaluation does not result in detrimental treatment that is disproportionate to the gravity of the user’s misbehaviour.
- AI-enabled targeted commercial advertising is out of scope , where it is based on relevant data (e.g., users’ preferences), it is done in line with Union law on consumer protection, data protection and digital services, and it does not result in detrimental or unfavourable treatment disproportionate to the gravity of the user’s social behaviour (e.g., exploitative and unfair differential pricing).
- AI systems using data collected in refugee camps ( e.g., behavioural compliance) for decisions about resettlement or employment is not affected by the prohibition , given that this data is relevant for the purpose of assessmentand provided that the procedures under applicable Union migration law are fulfilled to ensure the treatment is justified and proportionate.
- AI-enabled scoring by an online shopping platform which offers privileges to users with a strong purchase history and a low rate of product returns,such as a faster returns application process or returnless refunds are out of scope of Article 5(1)(c) AI Act , given that the advantages are justified and proportionate to reward positive behaviour and other users continue to have access to the standard return process.
- AI-evaluation and scoring of individuals by police and other law enforcement authorities that collect data about individuals’ social behaviour from multiple contexts are out of scope of Article 5(1)(c) AI Act where those data are relevant for the specific purposes of the prevention, detection, prosecution and punishment of criminal offences, and where the detrimental treatment is justified and proportionate in accordance with substantive and procedural Union and national criminal and police law. It is also relevant to consider in this context the prohibition in Article 5(1)(d) AI Act, which imposes additional and more specific conditions for AI-enabled risk assessments and prediction s of the likelihood of a person committing a criminal offence which must not be solely based on profiling or the assessment of personality traits (see section 5).