Article 5(1)(d) AI Act provides, in its last phrase, that the prohibition does not apply to AI systems used to support the human assessment of the involvement of a person in a criminal activity, which is already based on objective and verifiable facts directly linked to a criminal activity. Although, as noted, the situation described in this express exclusion is not necessarily the only one in which the prohibition does not apply , including that situation expressly in that provision offers legal certainty by delineating the scope of the prohibition and by making it clear that, where that situation is at issue, the prohibition does not in any event apply.
Where the system falls within the scope of theexclusion and is therefore not prohibited, it will be classified as a high-risk AI system (as referred to in Annex III, point 6(d), AI Act) if intended to be used by law enforcement authorities or on their behalf and therefore subject to the requirements and safeguards, including human oversight (Article 14 and Article 26 AI Act). These requirements include that the human oversight must be assigned to personswith the necessary competence, training and authority who should be able to properly understand the capabilities and limitations of the AI system, correctly interpret its output and address the risk of automation bias. Those persons should have clear procedures, training and the necessary competence and authority to meaningfully assess the outputs of the AI system . In this specific case, their human assessment should ensure that any AI prediction or assessment of the risk of a person committing a crime is based on objective and verifiable facts linked to a criminal activity. Those persons should also intervene in order to avoid negative consequences or risks, or stop the use of the AI system if it does not perform as intended.
Furthermore, the concept of ‘human intervention’ has been subject to CJEU case-law, in particular in the context of solely automated decision-making predicting the risk of air passengers being involved in serious crimes. That case-law may also be relevant for the application of the concept of ‘human assessment’ as used in Article 5(1)(d) AI Act. In the Ligue des droits humains case,138 the CJEU examined the legality of the use of an advanced AI system for the systematic processing of passenger name record (PNR) data of air travellers to assess their likelihood of being involved in terrorism and other serious crimes. The CJEU interpreted the rule in Directive (EU) 2016/681 ( ‘ PNR Directive ’) that prohibits adverse legal decisions based solely on automated processing and required individual human assessment and review for any positive matches by non-automated means to identify false positives and ensure non-discriminatory results. According to the CJEU, that human assessment , subject to which any results of automated processing of PNR data must be based , must rely on objective criteria to evaluate whether a positive match concerns someone who might be involved in this specific case in terrorist offenses or serious crime, and to ensure the non-discriminatory nature of automated processing.
As to the content of the excl usion, one of its central elements is that the AI system is used to support human assessment, rather than involving the AI system itself making the risk assessment as occurs in the situations covered by the prohibition. However, for the exclusion to apply, that human assessment must, in addition, already be based on objective and verifiable facts directly linked to a criminal activity.