Point 3(d) of Annex III classifies as high-risk AI systems intended to be usedto monitor and detect prohibited behaviour of students during tests in the context of or within educational and vocational training institutions at all levels.
This use case encompasses AI systems intended to be used to monitor and identify prohibited behaviours of students during tests , both on-site and remote or online, synchronously and asynchronously.
Prohibited behaviour of students should be understood, in this context, to refer to any action or activities by students that compromise the integrity of the testing process. This may include a broader range of behaviours, such as plagiarism, collusion, and tampering with exam materials.AI systems falling within this use case are typically used to prevent forms of compromising academic integrity in test settings (e.g. cheating). They may encompass a range of AI technologies, including facial recognition, keystroke analysis, screen monitoring, audio and video recording.
The AI system must both monitor and detect the prohibited behaviour. Monitoring and detecting are interconnected processes, where monitoring is often a prerequisite for detection.
The reference to tests in point 3(d) of Annex III means that the prohibited behaviour must occur during a process leading to an evaluation or a grade of a student, thus in the context of a summative assessment. Key elements to assess whether the monitoring and detection occurs during tests is the timing and the context in which the AI system is intended to be used. The system must be intended to be used during a controlled testing situation, such as written exams, oral exams, or other real-time assessments under supervision, by educational and vocationaltraining institutions at all levels.
Activities like checking homework, essays, or other assignments for plagiarism or improper collaboration fall outside the scope of this use case , since they typically take place in an unsupervised environment, after the work has been completed and submitted , and do not involve live monitoring or real-time behavioural analysis during the testing process itself.
a) Practical examples of AI systems falling within the high-risk use case of point 3(d)
- AI-enabled proctoring system for exam certification intended to be used by an educational or vocational training institution to monitor test-takers for prohibited behaviour during a certification exam, such as accessing unauthorized materials or communicating with someone else during the exam,by using a combination of facial recognition, keystroke analysis, and screen monitoring.
- Real-time behaviour analysis system using machine learning algorithms intended to be used by an educational institution to analyse student behaviour and detect patterns that may indicate cheating during online exams.
- AI-enabled e xam monitoring system intended to be used by an educational institution to monitor and detect prohibited behaviour during in-person exams, such as the use of unauthorized materials or devices, by using facial recognition and object detection.
b) Practical examples of AI systems falling outside the high-risk use case of point 3(d)
- AI system intended to be used by an educational institution to check homework or assignments for plagiarismagainst a database of existing content. Such a systemis not intended to be used to monitor and detect prohibited behaviour during tests, but to analyse submitted work which has been produced in a non-supervised environment and does not involve live monitoring or real-time behavioural analysis during the testing process.
- AI system for non - academic student behaviour monitoring intended to be used by an educational institution to monitor and detect student behaviour in the cafeteria, hallways, and other non-testing environments and to alert staff to potential incidents of bullying, harassment, or other undesirable behaviour . Such a system is not used to monitor and detect prohibited behaviour during tests, but for non-academic purposes, such as maintaining a safe and orderly school environment, and it does not involve the eva luation or assessment of student academic performance. However, this AI system could be classified as high-risk under point 1 of Annex III, if the system fulfils the definition of a remote biometric identification system.
c) Practical examples of AI systems falling within theuse case but exempted by the filter mechanism in Article 6(3)
- AI-enabled confirmation of suspicious behaviour: During an online test, a human proctor observes a student exhibiting suspicious behaviour, such as consistently looking away from the screen or typing in a pattern that suggests they may be copying from a hidden source and uses an AI system to confirm or d eny his suspicions. The AI system is trained on a dataset of known cheating behaviours and can identify patterns that are indicative of academic misconduct. The AI system analyses the student’s data and provides a report to the human proctor, indicating that the student's behaviour is consistent with cheating. The report highlights specific patterns and anomalies that the AI system has identified, such as unusual keystroke patterns or excessive mouse movements. The human proctor reviews the report and may decide to investigate further and take appropriate action, such as alerting the student's instructor or initiating a formal investigation. Such a system would fall under the exception for systems intended to improve the result of a previously completed human activity listed in Article 6(3)(b) AI Act.
- Identity verification: An AI system intended to be used during a test toautomatically processes students’ identity verification documents, such as IDs or biometric data like facial recognition scans, to confirm that the person taking the test matches the registered candidate . Since the system helps streamline identity verification, but it does not directly decide if the student can proceed with the exam or if any sanctions should be applied in the case of misrepresentation, as these decisions are made by the human proctor, it should fall under the exception for systems intended to perform a procedural task listed in Article 6(3)(a) AI Act.