Point 4 of Annex III lists as high risk two use cases of AI systems intended to be used in employment, worker’s management and access to self-employment. These include:
AI systems intended to be used for the recruitment or selection of natural persons, [and] in particular:
- to place targeted job advertisements;
- to analyse and filter job applications; and
- to evaluate candidates.
AI systems intended to be used:
- to make decisions affecting:
- terms of work-related relationships;
- the promotion; or
- termination of work-related contractual relationships.
- to allocate tasks based on individual behaviour or personal traits or characteristics; or
- to monito r and evaluate the performance and behaviour of persons in work-related relationships.
Recital 44 AI Act clarifies that, in the specific context of emotion recognition (particularly in the workplace), the use of AI systems may aggravate structural imbalances of power32 . Such imbalances relate to the unequal bargaining position of workers and job candidates vis-à-vis employers. Recital 57 AI Act justifies classifying AI systems in the area of employment as high-risk on the ground that such systems may have ‘an appreciable impact on future career prospects, livelihoods of those persons and workers’ rights.’ That recital also makes explicit that work-related contractual relationships should, in a meaningful manner, include both employees and persons providing services through platforms. Moreover, the same recital indicates that ‘such systems may perpetuate historical patterns of discrimination, for example against women, certain age groups, persons with disabilities, or persons of certain racial or ethnic origins or sexual orientation (…) and may also undermine their fundamental rights to data protection and privacy’.
- The recital addresses, in specific, AI systems intended to be used to detect the emotional state of individuals in situations related to the workplace and education . However, its logic can be transposed to the wid er set of high-risk AI systems falling within point 4 of Annex IIIAI Act.