As already noted , Article 5(1)(c) AI Act prohibits unacceptable AI-enabled social scoring practices regardless of whether the AI system or the score are provided or used by public or private persons. While scoring in the public sector may have very significant consequences for people due to an imbalance of power and a dependence on public services, simil arly harmful consequences may also occur in the private sector, where scoring practices are also increasingly undertaken by companies and other entities. For example,
- An insurance company collects spending and other financial information from a bank which is unrelated to the determination of eligibility of candidates for life insurance and which is used to determine the price of the premium to be paid for such insurance. An AI system analyses this information and recommends, on that basis, whether to refuse a contract or set higher life insurance premiums for a particular individual or a group of customers.
- A private credit agency uses an AI system to determine the creditworthiness of people and decide whether an individual should obtain a loan for housing based on unrelated personal characteristics. These unacceptable social scoring practices may be distinguished from lawful practices evaluating persons for spec ific legitimate purposes that do not fulfil these conditions and are in compliance with Union and national law, in particular whenthose laws ensure that detrimental or unfavourable treatment is justified and proportionate and data from related social contexts is used (see section 4.3. out of scope).
In the case of checks by competent market surveillance authorities, it is on the provider and the deployer of the AI system, each within their responsibilities,to demonstrate that the AI practice is legitimate and justified, including by providing transparency of the functioning of the AI system, and information about the types of data and data sources, ensuring that only data related to the social context in which the score is used are processed for the purpose of the evaluation or classification, the system is performing as intended, and any resulting detrimental or unfavourable treatment is justified and proportionate to the social behaviour. Compliance with applicable legislation and appropriate and proportionate safeguards built in the system and applied during its operation will help to avoid the prohibition from applying, while enabling the use of AI systems for the evaluation or classification of persons for legitimate and beneficial purposes (e.g., to improve the effectiveness of processes, quality of service, safety, etc.) (see section 4.3. out of scope).
Compliance with the requirements for high-risk AI systems (e.g. in the area of essential public services and benefits, credit-scoring and creditworthiness assessment, migration etc.) may also help to ensure that AI systems used for evaluation and classification purposes in those high-risk areas do not constitute unacceptable social scoring practices that providers and deployers should consider when implementi ng their respective obligations (e.g. on risk management, transparency, data governance, fundamental rights impact assessment, human oversight, monitoring, etc.).