Prohibitions § 4.2.1

The second condition for the prohibition in Article 5(1)(c) AI Act to apply is that the AI system is intended or used for the evaluation or classification of natural persons or groups of persons and assigns them scores based on their social behaviour or their personal or personality characteristics. The score produced by the system may take various forms, such as a mathematical number (for example, from 0 to 1), a ranking, or a label.

The scope of the prohibition is broad covering evaluation and classification practices in both the public and the private sector ( see section 4.2.3.). At the same time the evaluation or classification concerns only natural persons or groups of natural persons, thereby excluding in principle legal entities (see section 4.3. out of scope).

While ‘ evaluation’ suggests the involvement of some form of an assessment or judgement about a person or group of persons , a simple classification of persons or groups of persons based on characteristics, such as their age, sex, and height, need not necessarily lead to evaluation.¹⁰⁸ The scope of ‘classification’ is therefore broader than ‘evaluation’ and can also cover other types of classifications or categorisation s of natural persons or groups of persons based on criteria that do not necessarily involve a particular assessment or judgement about those persons or group s of persons and their characteristics or behaviour.

The term ‘evaluation’ also relates to the concept of ‘profiling’, which is regulated by Union data protection legislation¹⁰⁹ and constitutes a specific form of evaluation. While no direct reference is made in Article 5(1)(c) AI Actto that concept or thatlegislation,¹¹⁰ they may also be relevant for the prohibition contained in that provision, as well as for other prohibitions in the AI Act,¹¹¹ when the evaluation occurs in an automated fashion by an AI system based on personal data. Profiling means the use of information about an individual (or group of individuals) and evaluating their characteristics or behaviour patterns in order to place them into a certain category or group, in particularto analyse and/or make predictions about, for example, their ability to perform a task; interests; or likely behaviour’.¹¹² Profiling of natural persons under EU data protection law, when conducted through AI systems, may therefore also be covered by Article 5(1)(c) AI Act. For example, in the SCHUFA I judgment, t he CJEU examined a creditworthiness scoring system used in Germany.¹¹³ In that case, the ‘score’ generated by the computer programme was a ‘probability value’concerning the ability of a person to meet payment commitments, which was qualified by the CJEU as ‘profiling’. More specifically, the system established ‘a prognosis on the probability of a future behaviour of a person (‘score’), such as the repayment of a loan, based on certain characteristics of that person. The establishment of scores (‘scoring’) is based on the assumption that, by assigning a person to a group of other persons with comparable characteristics who have behaved in a certain w ay, similar behaviour can be predicted.¹¹⁴ According to the CJEU , this activity met the definition of ‘profiling’ within the meaning of Article 4(4) GDPR.¹¹⁵ This form of profiling may also be considered to constitute an evaluation of persons based on their personal characteristics within the meaning of Article 5(1) (c) AI Act, which will be prohibited if done with AI systems andprovided that the other conditions for the application of that provision are fulfilled.

The prohibition in Article 5(1)(c) AI Act requires that the evaluation or classification is based on data that spans over ‘ a certain period of time ’. This suggests that the assessment should not be limited to a one-time or an at once rating or gradingwith data or behaviour from a very specific individual context. At the same time, it is important that this condition is assessed, taking all circumstances of the case into account to avoid circumvention of the scope of the prohibition. For example, an authority for m igration and asylum implement s a partly automated surveillance system at refugee camps built on a range of surveillance infrastructure, including cameras and motion sensors. If the analysed data spans a period of time and specific individuals are evaluated (such as migrants) for example to ascertain whether they are at risk of trying to abscond, then this would qualify as ‘over a certain period of time’ and the prohibition in Article 5(1)(c) AI Act may apply if all other conditions are fulfilled.

The practices of ‘evaluation’ and ‘classification’prohibited under Article 5(1)(c) AI Act must be based on the AI-enabled processing of data (often extensive) in relation to either i) the social behaviour of individuals or groups of persons or ii) their known, inferred or predicted personal and personality characteristics, or both. The data may be directly provided by the persons or indirectly collected , i.e. through surveillance, obtained from third parties or through inferences from other information.

Regarding the first scenario, ‘ social behaviour ’ is a broad term that can generally include actions, behaviour, habits, interaction s within society, etc., and usually covers behaviour related data points from multiple sources.¹¹⁶ This could include behaviour of individuals and groups of individuals in social and private contexts, such as participation in cultural events, volunteering, etc., but also social behaviour in business contexts, for example the payment of debts, behaviour when using certain services, as well as relations with public and private entities, government, police, and the law (for example, whether a person obeys traffic rules ). Social behaviour data from multiple contexts and data points may be collected in a centralised way by the same entity, but is most often collected in a distributed wayand combined from different sources, which may involve increased monitoring and the tracking of individuals (so called ‘dataveillance’).

The second scenario is where the scoring is based on personal or personality characteristics, which may or may not involve specific social behavioural aspects. ‘Personal characteristics’ may include a variety of information relating to a person, for example sex, sexual orientation or sexual characteristics, gender, gender identity, race, ethnicity, family situation, address, income, household members, profession, employment or other legal status, performance at work, economic situation, financial liquidity, health, personal preferences, interests, reliability, behaviour, location or movement, level of debt , type of car etc.¹¹⁷ ‘Personality characteristics’ should be in principle interpreted as synonymous with personal characteristics, but may also imply the creation of specific profiles of individuals as personalities . Personality characteristics may be also based on a number of factors and imply a judgement, which may be made by the individuals themselves, other persons, or generated by AI systems. In the AI Act, personality characteristics are sometimes referred to as personality traits and characteristics;¹¹⁸ those concepts should be interpreted consistently.

‘Known, inferred or predicted’ personal or personality characteristics are different types of information and personal data that need to be distinguished. ‘Known characteristics’ are based on information which has been provided to the AI system as an input, and which is in most cases verifiable information . By contrast, ‘inferred characteristics’ are based on information which has been inferred from other information, with the inference usually made by an AI system. ‘Predicted characteristics’ are those which are estimated based on patterns with less than 100% accuracy. The concepts of ‘inferred’ (or derived) data are also used in the context of profiling in Union data protection law and may therefore be a source of inspiration for interpreting those concepts used in Article 5(1)(c) AI Act.¹¹⁹ The use of these different types of data may have different implications for the accuracy and the fairness of the scoring practices and therefore may be taken into account, in particular where the processing is opaque or relies on data points whose accuracy is more difficult to be verified.