Prohibitions § 5.2.2

The third condition for the prohibition in Article 5(1)(d) AI Act to apply is that the risk assessment to assess or predict the risk of a natural person committing a crime must be based solely on a) the profiling of the person or b) on the assessment of their personality traits and characteristics.

The prohibition in Article 5(1)(d) AI Act applies, irrespective of whether the AI system profiles or assesses the personality traits and characteristics of only one natural person or a group of natural persons simultaneously, since the prohibition aims to protect every individual in respect of whom the risk of committing a criminal offence is being predicted or assessed.

Unlike Article 5(1)(c) AI Act, Article 5(1)(d) explicitly uses the term ‘profiling’. Article 3(52) AI Act defines that term by reference to its definition in Article 4(4) GDPR . ¹³⁵ The concept of profiling includes the objective to ‘evaluate certain personal aspects’ as one of its core elements.¹³⁶ In the context ofArticle 5(1)(d) AI Act, the profiling is done for the purposes of assessing or predicting the risk of a person committing a crime.

The concept of so-called group profiling¹³⁷ may also be relevant in this context. That concept refers to the construction and the application of a descriptive profile for a given group, for example categories of perpetrators of criminal offences (e.g., terrorists, gangsters etc.) constructed on historic data about previously committed crimes by other persons. Those group profiles may be used later to assess and predict the risk of other persons committing similar offences. Whenever an AI system makes prediction and applies such a (group) profile to a specific individual, this constitutes profiling of the person and may therefore fall within the prohibition of Article 5(1)(d) AI Act.

The prohibition also applies if the risk assessment to assess or predict the risk of the person committing a crimi nal offence is only based on assessing the person’s personality traits and characteristics. Such an assessment or prediction is often included in the concept of profiling , but it could also be seen as an alternative, if profiling as defined in Article 4(4) GDPR cannot be established.

As mentioned in section 4.2.1.c), personality traits and characteristics constitute a broad category of characteristics related to a particular natural person, for which there is no generally agreed taxonomy. Recital 42 AI Act provides examples of personality traits and characteristics which may be assessed for predicting the risk of a person committing an offence, such as ‘nationality, place of birth, place of residence, number of children, level of debt or type of car’. This is only an illustrative and not an exhaustive list.

Article 5(1)(d) AI Act provides that the risk assessments covered by that provision are only prohibited where they are based ‘solely’ on the profiling of a person or the assessment of their personality traits and characteristics. It is clear from Recital 42 AI Act that ‘solely’ is intended to apply both to profiling or to the assessment of personality traits and characteristics.

The condition that the risk assessment must be based ‘solely’ on profiling or assessing personality traits and characteristics may not be fulfilled in a number of situations.

As is evident from the last phrase of Article 5(1)(d) AI Act, such a situation arises, in any event, where the AI system is used to support the human assessment of the involvement of a person in a criminal activity, which is already based on objective and verifiable facts directly linked to a criminal activity. As Recital 42 makes clear, in this context one should think, in particular but not necessarily exclusively, of a situation in which a reasonable suspicion in respect of the natural person concerned already exists. After all, in such cases there will normally have been a human assessment, which will normally be based on relevant objective and verifiable facts.

However, there can also be other situations, which will always need to be assess ed on a case-by-case basis. On the one hand, the use of the term ‘solely’ leaves open the possibility of various other elements being taken into account in the risk assessment, which makes that it is no longer based on profiling or assessing personality tr aits or characteristics alone. On the other hand, in order to avoid circumvention of the prohibition and ensure its effectiveness, any such other elements will have to be real, substantial and meaningful for them to be able to justify the conclusion that t he prohibition does not apply. A reading of the prohibition of Article 5(1)(d) AI Act together with the exclusion contained in the last phrase thereof suggests that, in particular, the existence of certain pre-established objective and verifiable facts may justify that conclusion. For example,

• A law enforcement authority uses an AI system to predict criminal behaviour for crimes such as terrorism solely based on individuals’age, nationality, address, type of car, and marital status . With that system, individuals are deemed more likely to commit future offences that they have not yet committed solely based on their personal characteristics. Such a system may be assumed to be prohibited under Article 5(1)(d) AI Act.
• National tax authorities use an AI predictive tool to review all taxpayers’ tax returns to predict potential criminal tax offences to identify cases requiring further investigation. This is done solely on the basis of the profile built by the AI system, which uses for its assessment personal ity traits, such as double nationality, place of birth, number of children, and opaque variables, especially inferred information that is predictive and therefore non-objective and hard to verify. Such a system will normally fall under the prohibition of Article 5(1)(d) AI Act, since there is no reasonable suspicion of the involvement of a particular person in a criminal activity or other objective and verifiable facts linking that to that criminal activity. This is also an example that falls within the scope of social scoring prohibited under Article 5(1)(c) AI Act involving unfavourable treatment with data from unrelated social contexts.
• A police department uses AI-based risk assessment tool to assess the risk of young children and adolescents being involv ed in ‘future violent and property offending ’ . The system assesses children based on their relationships with other people and their supposed risk levels, meaning that children may be deemed at a higher risk of offending simply by being linked to another individual with a high-risk assessment, such as a sibling or a friend. The parents’ risk levels may also impact a child’s risk level. The risk assessments result in police ‘registering’ these children in their systems, monitoring them with addition al inspections, and referring them to youth ‘care’ services. Such a system is also likely to fall under the prohibition of Article 5(1)(d) AI Act.