In accordance with Article 5(3) AI Act , even where the deployer of a real-time RBI system obtains an authorisation, no decision that produces an adverse legal effect on a person may be taken based solely on the output of the ‘real-time’ RBI system. For example,
- A person is arrested and imprisoned for a serious crime solely based on identification by a facial recognition system, without any further checks. This comes on top of the requirement under Article 14 AI Act for human oversight. Checks could relate for example to the question whether a given person has been at a different place or also whether there are other reasons for that the person cannot be the person searched. Requirements of Article 14 AI Act for Human Oversight
The use of real-time RBI that is permitted because it pursues one of the objectives listed in Article 5(1)(h) and complies with Articles 5(2) -(6) AI Act still falls under the rules for high-risk systems. In accordance with Article 14 AI Act, high-risk AI systems ‘shall be designed and developed in such a way, including with appropriate human-machine interface tools, that they can be effectively overseen by natural persons during the period in which they are in use.’ Pursuant to Article 14(5) AI Act, no action or decision may be taken by the deployer on the basis of the identification resulting from the system, ‘unless that identification has been separately verified and confirmed by at least two natural persons with the necessary competence, training and a uthority’or unless ‘Union or national law considers the application of this requirement to be disproportionate’. Article 4 AI Act prescribes AI literacy measures for providers and users of AI systems to ensure ‘a sufficient level of AI literacof their staff and other y persons dealing with the operation and use of AI systems’and considering the persons on whom the systems are to be used.
As stated by the EDPB in the data protection context , for human oversight to be effective it is crucial ‘to enable the person to understand the (in that case facial recognition) system and its limits as well as to interpret its results properly. It is also necessary to establish a workplace and organisation that counteracts the effects of automation bias, and av oids fostering the uncritical acceptance of the results e.g., by time pressure, burdensome procedures, potential detrimental career effects etc .’ 232 Similar considerations may apply in the context of the AI Act.