A financial services entity plans to implement an LLM-based AI assistant to support employees in creating texts, emails and presentations. As the entity works with sensitive financial and customer data, strict security and compliance requirements must be met.
This case study describes typical risks involved in implementing this kind of AI application. Three infrastructure variants are examined: the on-premise solution and two variants with cloud operation – first an AI application in the entity’s own tenant, and second an AI application outside the entity’s own tenant. Figure 2 illustrates the concepts behind these two variants.
The case study presents an illustrative discussion of selected ICT risks that can be observed when AI assistants are used. It does not express any supervisory expectations. All measures presented in the case study are illustrative. By contrast, financial entities must implement the measures that are most appropriate for their specific risk situation.
The specific ICT assets and infrastructure used at financial entities may also represent a combination of elements from the three variants. The following discussion of risks should then be modified accordingly.
Independently of this, there are risks that are significant in every infrastructure variant. Key examples along the AI lifecycle are as follows:
1. Data acquisition and processing: A significant risk is that the AI application will process data that has not been approved for use in the application. This is why it is essential for the data to be comprehensively classified by confidentiality level. Selected measures can be categorised in greater detail as follows.
Data classification and access control:
- automated detection and classification of confidential data wherever possible;
- in-depth training on how to use the AI application;
- implementing a zero-trust policy, under which the AI application can only retrieve authorised data, as well as introduction of a role-based model for data access for AI application users.
Data minimisation and masking:
- using differential privacy wherever possible to protect personal data from possible inferences in the context of multiple queries (maximising the accuracy of responses to database queries while minimising the likelihood that the data sets used to answer the queries can be identified);
- tokenisation of sensitive data before processing by the AI assistant.
Identification and avoidance of data bias:
- data validation and cleansing prior to use, including automatic validation or manual sampling;
- training users to select and process data.
2. Model development and training: When training or fine-tuning LLMs, there is a risk that manipulated data will be used, which can lead to unexpected or altered model behaviour. The following measures can be considered here:
- ensuring that the AI assistant is only trained/fine-tuned with verified and validated entity data to avoid “data poisoning”. It may also be helpful to establish a data governance team to control which data is released for AI processing;
- ensuring that the data provided to the LLM as a knowledge base in Retrieval Augmented Generation (known as “context” or “grounding”) is also verified and validated to avoid knowledge poisoning;
- version control systems for models to enable rollback in the event of e.g. malfunction or faulty training;
- for each type of generative AI application, entities are encouraged to develop and use their own testing procedures that reflect the specific requirements of the use case.
Similar to data poisoning, there is also something known as “model poisoning”. This risk occurs particularly with open source models or models obtained from shared repositories. In this case, attackers attempt to modify the behaviour of a trained model or even the structure of the model itself. Best practice measures for risk mitigation include:
- reviewing the source code of publicly available models for backdoors and malicious code;
- ensuring the integrity of repositories and model providers.
In addition, even when a pre-trained model is used, attackers can attempt to implement backdoor attacks or adversarial manipulations. Examples of potential countermeasures include:
- implementing an auditing framework that logs all queries to the AI assistant and makes them verifiable. Explainable AI techniques for explaining the AI assistant's outputs are useful here;
- deploying anomaly-detecting security mechanisms to identify unusual model responses or suspicious queries;
- conducting regular penetration tests for the AI assistant environment to identify vulnerabilities at an early stage, as well as a red teaming approach in which security experts attempt to manipulate the model and discover vulnerabilities.
3. Model deployment and integration: ICT risks also arise in this phase of the AI lifecycle. In particular, if the AI assistant is not implemented securely, attackers can gain unauthorised access to internal systems or exfiltrate model information. The following measures have proven to be effective here.
Deployment strategy:
- deploying the AI assistant in an isolated cloud environment to prevent direct internet access;
- using containerisation to isolate the AI environment from other ICT systems.
Access control and authentication:
- implementing multi-factor authentication (MFA) for AI assistant users;
- using conditional access policies so that only authorised users with the entity's own devices can use the AI assistant;
- critical examination of interfaces and access rules to other systems within the entity, incorporating the human-in-the-loop principle (e.g. approval by the data owner).
API security:
- All API queries between the entity’s systems and the AI assistant are conducted through encrypted interfaces.
- Rate limiting and DDoS protection through cloud security mechanisms help to prevent overload attacks.
4. Operation and use: Integrating LLMs into business processes entails a wide range of risks. Besides challenges that are not directly related to ICT (such as “hallucinations”, which is when the model makes convincing statements that appear to be correct but are actually wrong), various ICT risks are relevant. These include the following major topics:
First, there is the extraction/disclosure of sensitive business information from fine-tuned, publicly accessible LLMs. In this case, selected confidential information from the training data set can be reproduced using appropriate prompts. There is a risk here that malicious prompts (known as “prompt injection”) could force the LLM to behave in an unintended way, e.g. by linking to websites that contain hidden instructions for the LLM. The following measures, among others, can be taken to counter these risks:
- using explainability tools that enable users to understand why the AI is giving a particular answer;
- providing AI safety training for employees to minimise misinterpretation and careless use;
- investigating which prompts lead to what effects and defining appropriate restrictions.
Second, a compromised or malfunctioning AI assistant may disclose confidential information to users who do not actually have access to this data. In this context, processes for monitoring the AI application and detecting anomalies can be used, e.g.
- implementing situation-specific monitoring of AI interactions;
- identifying abnormal activities within the AI assistance environment.
Third, API access to LLMs can give attackers an opportunity to access systems connected to the LLM and thereby extract confidential information. In addition to assigning appropriate access rights, financial entities should also consider isolating LLMs. Other potential measures include:
- mitigating this risk by restricting the use of LLM for AI systems that support critical or important functions;
- introducing a human review requirement (human-in-the-loop) for security-critical AI responses/recommendations.
5. Maintenance, updates and incident response: In particular when using open source LLMs, entities must consider that outdated or incorrectly configured software versions can lead to security vulnerabilities. Selected measures to counter this risk include the following.
Regular security updates:
- automatic updates for the AI assistant controlled automatically using a suitable tool, as well as using patch management tools to quickly close security gaps;
- introducing a security incident response plan (SIRP) for AI assistant-specific security incidents;
- simulating cyberattacks to test the entity’s response times.
Governance requirements:
- regular external audits to ensure that the AI assistant continues to comply with the entity's policies;
- centralised recording of governance aspects in the form of a compliance dashboard that provides transparency about usage (e.g. model version, risk assessments, responsibilities).
- In addition to generally applicable requirements, it may also be advisable to consider technological features for AI systems, such as on-premises and/or cloud-based systems.
6. End-of-life management: The uncontrolled continued use or unsafe retirement of AI assistants can lead to the misuse of historical data and models or their unintentional public disclosure (leaks). As with all ICT assets, financial entities must therefore also plan for the retirement of LLMs and associated data sources. Potential best practice measures for risk mitigation include the following.
Secure retirement of AI assistants:
- deleting all data used and historical AI interactions in accordance with GDPR guidelines;
- using cryptographic wiping to securely remove all stored data of the entity;
- blocking the AI assistant for expired user accounts or former employees;
The specific features of the selected infrastructure models will now be examined in the following. Investment costs and operating costs are not considered here. Such costs will arise to a greater extent in each variant, as either the establishment and operation of a separate infrastructure will be necessary, or the secure configuration of the model may lead to increased cloud spending.
Variant 1: On-premise solution
In this variant, the financial entity has its own storage and processing capacities and operates AI software developed (partially or completely) internally on its own hardware. This is the case when the entity implements, maintains, trains, and operates an LLM itself. This means that all data flows remain within the entity’s own infrastructure. The financial entity has full control over the ICT assets of the AI application, but bears all the risks associated with the development, operation and, above all, maintenance of the hardware and software: both the strategic risk of the business model and the operational risk of the ICT system are significant here.
The specific risks of this approach are reflected in the AI lifecycle, particularly in model development and operation. In the first instance, implementing and maintaining open source models in particular demands sufficient knowledge and skills, meaning that the availability of suitable employees is a strategic risk.
There is a risk of vulnerability to cyberattacks, particularly when an entity operates models it has implemented itself or open source models. If the AI assistant is not implemented securely and is not continuously updated, attackers can gain unauthorised access to internal systems or exfiltrate model information. A suitable protective measure is closely controlled access management, as the effects of uncontrolled access are much greater when an entity uses AI than without it.
Additionally, capacity management is increasingly important in this infrastructure variant. Unlike cloud variants, the AI application cannot be scaled at will without expanding the hardware. The infrastructure must therefore be tailored to the expected computing power, regularly checked and adapted if necessary.
Variant 2: Cloud with AI application in the entity’s own tenant
In this case, the financial entity has its own tenant in a cloud environment. An AI assistant (such as an internally implemented open source model) is operated inside this tenant. This means that although data flows happen in the cloud, they are still exclusively inside the entity’s tenant.
The main risk in this variant is strategic in nature, as there is a high degree of dependency on the model operator if no open source implementation is used. One mitigating measure could be to use several models from different providers, although this would lead to increased operating and maintenance costs. In addition, employees must have the appropriate skills and knowledge, especially if they are using open source software.
There are also challenges with regard to capacity management, as it is not always possible to scale the AI application inside the tenant. One possible countermeasure is early capacity planning.
Variant 3: Cloud with AI application outside the entity’s own tenant
In this variant, the financial entity has a tenant in a cloud. However, the AI application is located outside this tenant, meaning that data flows extend beyond the boundaries of the tenant. This occurs typically when a large language model from the cloud provider is accessed via an API or called up as an AI assistant from standard software (possibly without the user’s knowledge).
The main risk in this scenario is that data leaves the tenant and flows to the model provider. This should be counteracted by contractual and technical measures. Selected technical measures for this include:
- restricting functionality (including limiting uploads) for certain user groups;
- operating a filter that checks whether input data is confidential (Governance Shield);
- presetting the conditions for AI application in advance of each use.
Another potential mitigating measure is to assign the AI application a lower data clearance level, meaning e.g. that it can only process less sensitive data. It should be noted that users cannot exceed these restrictions of their own accord.
Technical control of IT operations, information security, etc. should be carried out in line with the Cloud supervisory statement.