If financial entities develop AI systems themselves, they usually do so within the standard processes designed for this purpose. This gives them full control over the software and mathematical models. Software development not only requires sufficient specialised knowledge and skills, but also places demands on the design of the development process. In this case, too, the DORA requirements apply to AI systems.
Knowledge and skills. The development of AI systems places high demands on the skills of employees and management. Employees tasked with AI-related responsibilities face the challenge of acquiring appropriate and relevant skills and knowledge about how the AI systems work, the risks associated with them, and the technical and organisational characteristics of cloud and on-premise operations.
In particular, the ability to develop software with the support of AI assistants can enable users outside the ICT function to create even complex applications themselves. The requirements for knowledge and skills referred to above (Article 16(9) of the RTS RMF) also apply here, including with regard to testing.
Management, and in particular the members of the management body, can also contribute to assessing and managing the risks associated with software development by having a sufficiently deep understanding of AI systems (Article 5(4) and Article 13(6) of DORA).
The extent of the knowledge required will depend on the tasks involved: the more technical the tasks, the more specific the knowledge will have to be. Due to the rapid pace of technological progress in the field of artificial intelligence, it is necessary to regularly train employees and continuously monitor the pace of development. Typically, training measures are defined for employees involved in developing, operating and maintaining AI systems (Article 13(6) of DORA).
Development process. When introducing and further developing AI systems, it helps to establish robust project management that covers all phases (planning, development, testing, rollout and operation). This includes clear objectives, the establishment of governance structures, detailed project risk assessments and testing procedures (see below) (Article 15 of the RTS RMF).
Standard software engineering practices such as unit testing, integration testing and source code reviews have proven their worth in the development and validation of AI systems. Regular, ideally automated, testing and validation procedures are suitable for ensuring accuracy and reliability. A secure and separate environment for the experimental development and testing of AI systems is essential.
Comprehensive documentation in the form of technical specifications can contribute to the reproducibility and transparency of AI system development. In addition to the technical ICT specifications, it is also useful to describe the algorithms, data and parameters used (Article 16 of the RTS RMF).
AI systems are procured, developed and maintained in accordance with security and functionality requirements, in the same way as ICT systems. This typically involves defining technical specifications that also include security requirements, such as protection against manipulation (Article 16 of the RTS RMF).
Any changes to AI systems (whether to the software or hardware) are generally subject to strict change management, which includes an independent review, documented testing and specifying fall-back procedures. In addition, procedures, protocols, and tools must be established to manage emergency changes that provide adequate safeguards. The goal is to mitigate the risk of unintended security breaches or manipulation (Article 17 of the RTS RMF).
Version control systems are useful for tracking changes to AI models and applications. They make it possible to trace errors and reproduce results. Systematic archiving of all model versions and parameters ensures reusability and reproducibility (Article 17(1) to (2) of the RTS RMF). DORA does not distinguish between applications developed within the ICT function and those developed outside the ICT function (often referred to as “end-user computing” (EUC)) (Article 16(9) of the RTS RMF). If it becomes necessary to develop AI systems outside the ICT function, this must also be carried out in accordance with the processes described above.
Several approaches have proven helpful in practice in developing and modifying AI systems to meet DORA requirements. For the training process, these measures include:
- ensuring that training and testing data comes from trustworthy sources;
- ensuring that (open-source) libraries and software packages used for training do not contain any known vulnerabilities;
- documenting and versioning the entire training process to guarantee transparency and reproducibility;
- using security analyses to identify hidden manipulations in the model or training data.
Use of open source libraries. Internally developed and purchased software for AI systems can also use open source libraries. In extreme cases, this can even include the use of an open source model that is implemented, trained and operated on the hardware of a financial entity itself. The aspects of the development process described above have also proven to be relevant here. There is also a risk with open source software that malicious code could be introduced via these libraries. Additionally, there is a risk that the libraries will no longer be maintained after a few years. In this case, the AI system may contain identified but unresolved vulnerabilities and no longer correspond to the latest state of the art.
Generating source code using AI assistants. Software developers are increasingly using AI assistants to create applications more efficiently. In principle, the same rules apply to all types of source code creation, whether by humans or machines. However, one challenge is to review machine-generated source code to determine the extent to which AI-based functions unknown to the user are being invoked. Among other things, this can be identified through static code analysis (Article 16(3) of the RTS RMF). Deeper source code analyses and reviews of quality, reproducibility and understandability can provide additional assurance. Appropriate documentation of the source code has proven to be advisable for downstream testing.