The prohibition in Article 5(1)(g) AI Act does not cover AI systems engaged in the labelling or filtering of lawfully acquired biometric datasets, such as images, based on biometric data, including in the area of law enforcement. This is further explained in recital 30 AI Act.177
The labelling or filtering of biometric datasets may be doneby biometric categorisation systems precisely to guarantee that the data equally represent all demographic groups, and not, for example, over-represent one specific group. If the data used for training an algorithm are biased against a specific group (i.e. systematic differences in the data exist between groups due to the way the data are collected, or data is historically biased), the algorithm may replicate this bias, possibly resulting in unlawful discrimination against persons or groups of persons.178 For this reason, labelling on the basis of some protected sensitive information may be necessary for high-quality data, precisely to prevent discrimination. The AI Act may even require labelling operations to conform to the AI Act’s requirements for high-risk AI systems.179 Such labelling or filtering of biometric data is therefore explicitly exempted from the prohibition in Article 5(1)(g) AI Act. The prohibition only applies where biometric data is categorised to infer race, political opinions, trade union membership, religious or philosophical beliefs, sex-life or sexual orientation. Examples of permissible labelling or filtering include:
- the labelling of biometric data to avoid cases where a member of an ethnic group has a lower chance of being invited to a job interview because the algorithm was ‘trained’ based on data where that particular group performs worse, i.e. has worse outcomes than other groups.180
- the categorisation of patients using images according to their skin or eye colour may be important for medical diagnosis, for example cancer diagnoses.
Article 5(1)(g) AI Act also provides that the prohibition in that provision does not apply to the l abelling or filtering of lawfully acquired datasets in the area of law enforcement.181 For example, this covers the use by a law enforcement authority of an AI system that allows labelling and filtering of a dataset suspected of containing child sexual abuse material. In a first step, law enforcement would use the support of AI systems to detect and redact sensitive data from images. Furthermore, filtering and labelling according to gender, age, biometric data such as eye and hair colour, scars and marking could help with identifying the victim s or creating links with other cases. Similarly filtering and labelling abusers’ hands based on specific characteristics such as length of fingersor any distinguishing markings or tattoos to help with identifying possible suspects is permitted.