An important step in the deployment of AI systems is hardware and software testing. In this case, too, the general requirements of DORA apply to AI systems.
The RTS RMF describes in detail the measures to be taken for testing ICT systems (Article 16(2) of the RTS RMF). For example, security testing is required as part of the source code review for internet-exposed systems and applications (Article 16(3) of the RTS RMF).
Scope of testing. Financial entities are required to develop, document and implement tests. The level of testing must be commensurate to the criticality of the business procedures and ICT assets concerned. The testing must be designed to verify that new ICT systems – and hence also AI systems – are appropriate for their intended purpose. This also includes the quality of software developed internally (Article 16(2) of the RTS RMF).
Source code. The RTS RMF contains specific requirements for working with source code from application development. For example, the source code must be checked for anomalies using static and dynamic testing procedures before it is deployed in the production environment (Article 16(3) of the RTS RMF).
In particular when integrating open source libraries, entities are encouraged to ensure that the functions used do not contain any AI-based functionalities whose risks are unknown to the financial entity or cannot be mitigated (Article 16(3) of the RTS RMF).
The test also requires proprietary software (compiled source code) and, where feasible, source code provided by ICT third-party service providers or coming from open source projects to be analysed and tested before being deployed in the production environment (Article 16(8) of the RTS RMF).
One challenge in testing is to identify whether applications integrate external AI models, for example via APIs (application programming interfaces). These can transform software originally planned as a non-AI application into an AI system.
Generative AI. The use of generative AI poses particular challenges for testing. The large language models (LLMs) typically used have a complex internal structure (known as a “transformer” structure), with a large number of peripheral models that are connected to the actual model. The transformer structure is based on deep neural networks with several billion parameters (known as “weights”). LLMs are trained by the provider on the basis of large data sets (e.g. Wikipedia). Users can retrain (“fine-tune”) the model, but this requires a large amount of computing power.
Since generative AI is suitable for general purposes, testing is more difficult than with internally developed software that has been designed for a specific application. General testing procedures for LLMs can take the internal structure of the models into account (e.g. by considering the probabilities for generating a specific token stream for a given input prompt) or can be designed to be agnostic. In the latter case, the model answers test questions, for example, and the quality of the answers is assessed by a human or another model. Another challenge for testing arises from unannounced model changes where an AI model obtained from third parties is used.
Several approaches have proven suitable for meeting the protection objectives of DORA when developing and modifying AI systems. The following measures may be appropriate, depending on criticality:
- simulating attacks on AI systems (adversarial testing, e.g. data poisoning, evasion attacks);
- conducting adversarial penetration testing (e.g. in coordination with the cloud provider) to simulate AI-specific attacks;
- stress-testing the performance of AI systems, e.g. with significantly modified data distributions or in overload scenarios;
- In the case of AI systems not developed internally, it may also be appropriate to involve the manufacturer of the system in the tests and, if necessary, to obtain appropriate evidence of their performance.