High-risk § 3.5.3

Point 5(b) of Annex III classifies as high-risk AI systems intended to be used to evaluate the creditworthiness of natural persons or establish their credit score. The rationale for the inclusion of this use case in Annex III is that the determination of creditworthiness or a credit score often determines access to essential private services, so that the use of an AI system in that context may give rise to risks for the health, safety or fundamental rights of the natural person seeking such access, especially in cases of discrimination as described in Recital 58 AI Act.

Point 5(b) AI Act covers AI systems intended to be used for two distinct use cases: the evaluation of creditworthiness and the establishment of a credit score. The AI Act defines neither use case.

The evaluation of creditworthiness refers to the assessment of a natural person’s ability and willingness to fulfil its contractual obligations to pay for the services provided or the credit granted. Such an evaluation may be based e.g. on demographic data (such as age, the level of education or the place of residence ) and/or on financial data including payment behaviour, credit history and income and financial variables . A creditworthiness evaluation will normally be carried out by financial institutions for the purpose of obtaining a loan, but it may be carried out in other sectors and for other essential private or public services. A creditworthiness assessment may, forexample, be carried out to assess a person’s eligibility or access to public assistance benefits and services. In such a case, the AI system should be classified as high-risk pursuant to point 5(a) of Annex III. A credit score may be established for several purposes.

The establishment of a credit score refers to the creation and building of a representation of a natural person’s creditworthiness. The establishment of such a score may be based on e.g. demographic data and/or financial data including payment behaviour, credit history and income and financial variables. The score produced by the system may take various forms, such as a number, a ranking, or a label. A credit score may be established for several purposes. Where an AI system is intended to be used to establish such a score for the purpose of determining the access to financial resources or essential services such as housing, electricity, and telecommunication services , it should be classified as high-risk regardless of other purpose the AI system may have. In that case, the assessment for compliance with the requirements for high-risk AI systems should be limited to the intended high-risk purpose and should not extend to other purposes.

It should be noted that point 5(b) of Annex III only classifies AI systems as high-risk when they are intended to be used for the assessment of creditworthiness of and the establishment of a credit score for natural persons; it does not establish as such a right to obtain credit or access to financial resources or any other essential private service. The conditions that an AI system must fulfil to be classified as high-risk pursuant to point 5(b) of Annex III and related concepts are analysed in more detail in the following subsections.

To classify as high-risk pursuant to Article 5(b) of Annex III, an AI system must be intended to be used either to evaluate creditworthiness or to establish a credit score of natural persons , or both . Although establishing a credit score is often part of evaluating creditworthiness, it is sufficient that an AI system is intended to be used for one of those purposesto be classified as high-risk. Each use case in itself may influence the decision-making process and have an impact on the access to the essential private service at issue. For example, both the creditworthiness evaluation and the credit score establishme nt may directly influence how interest rates are priced or calculated for a borrower. Nevertheless, to qualify as high-risk pursuant to point 5(b) of Annex III, it is not necessary that the system is also intended to be used to calculate a certain price or interest rates.

By contrast, an AI system intended to be used for pricing is not covered by the use case listed in point 5(b) of Annex III , even if it relies on data obtained from a creditworthiness assessment or credit-scoring. However, an AI system which combines the two functionalities in one system in an integrated process (assessing the creditworthiness of natural persons and the credit pricing) will qualify as such a high-risk use case.

A private service is essential if it is necessary for people to fully participate in society or to improve their standard of living. Recital 58 AI Act clarifies that essential private services include access to financial resources and provides, asexamples, housing, electricity, and telecommunication services. For other services to be considered an essential private service, they must be equally important to natural persons and the improvement of their standard of living and the exclusion of natural persons from access to such services m ust have a comparable significant impact on their life, health, livelihood and participation in society as the services mentioned in that recital. Examples of such services may include health and long-term care, other utilities, such as gas and water services, and transport services.

When it comes to the type of financial services⁶³ that could constitute essential private services, it may be necessary to draw further distinctions based on the general reference to access to financial resources and examples of essential private services mentioned in Recital 58 AI Act. Besides the determination of access to financial resources, o nly financial services equal or similar to the following services should be considered as essential private services: providing a bank account⁶⁴; payment services , including when credit (e.g. temporary overdraft) is provided as an ancillary feature to support a transaction; the offering of loans and credit; the offering of extension of a credit line or of credit card limit; the offering of mortgage; and public financial services as further described in paragraph 297. For the purpose of the credit-scoring and creditworthiness assessment use case in point 5(b), this list of essential private services does no t give rise to an entitlement of the natural persons to those services.

Some public authorities or public financial institutions offer financial services, such as financial support schemes with a credit element (e.g. subsidised loans, guarantees, repayable advances) where creditworthiness may be assessed beforehand. Those public financial services may include publicly backed low-interest or zero-interest loan s for first time home buyers or for home renovations, for certain groups of people at risk of exclusion, or be provided in the form of student support schemes or social microcredit schemes. Those public financial services are also considered essential private services within the meaning of point 5(b) of Annex III.

In contrast, t he following financial services are not considered essential private service s : the acquisition of stocks and securities; access to margin trading; access to complex financial instruments; premium credit cards; and special loans, such as leisure/travel loans.

Since it is the intended use of an AI system that matters when assessing whether it should be classified as high-risk under the use case listed in point 5(b) of Annex III, it is for the provider to demonstrate to the competent market surveillance authority that its system is solely intended to be used for non-essential private services if it wishes that system not to be so classified. If the system is intended to be used for both essential and non-essential private services, it will be classified as high-risk.

Point 5(b) of Annex III specifies that AI systems intended to be used to evaluate the creditworthiness of natural persons or establish their credit score should not be considered high-risk if they are used for the purpose of detecting financial fraud. AI systems may detect anomalous behaviour in credit applications or suspicious patterns in applicant data that could indicate fraud. A typical example of such a system isa tool that uses AI to detect misused, forged, or altered personal identification documents (such as ID cards, driver’s licenses, etc.) or other forged or modified documents (such as income statements, bank statements, and the like).

For the exception to apply, fraud detection must be the main intended use of the AI system , preceding all other purposes for which the AI system may be used . This will be the case, for example, where the AI system is mainly used for pattern recognition and anomaly detection, rather than for assessing a person’s ability or willingness to repay a loan.

If an AI system is intended to be used for fraud detection, that system should be considered to fall outside the use case of point 5(b) of Annex III , even if its output may be used for the creditworthiness assessment or establishing the credit score.

AI systems intended to be used for anti-money laundering and for countering the financing of terrorism (‘AML/CFT’) are not covered by the exception, since those activities are regulated by other EU legislation and they are not related to financial fraud. S uch systems are nevertheless out of scope to the extent their intended use does not cover the assessment of creditworthiness. Nevertheless, AI systems intended to be used for AML/CFT may classify as high-risk AI systems under Point 5(b), if the AI system is functionally linked and simultaneously intended to be used for the evaluation of creditworthiness or to establish a credit-score.