A clear guide to the risks that generative AI poses for financial institutions, from privacy and pipeline security to human oversight, fraud defense, and governance. What can financial institutions do to address these risks?
Introduction – A Whole New Dimension
I’ve always been fascinated by stories in which a perfectly ordinary object is opened up, only to reveal an entire new world we weren’t previously aware of. In the science fiction comedy Men in Black, for instance, a small ornament dangling from a cat’s collar is revealed to contain an entire galaxy. The children’s classic Chronicles of Narnia opens as four children discover the far-off land of Narnia by walking into and through an apparently ordinary wardrobe in the spare bedroom of Professor Kirke’s house.
What’s fascinating about these newly discovered worlds is that they represent far more than simple objects or forces to contend with – they represent whole new constellations (whole new galaxies sometimes) of activity that must now be accounted for and factored into the surrounding reality.
The exploding world of generative AI (or genAI, as some call it) is like that. Continuing the astronomical metaphor, we might be tempted to liken its extremely rapid development to the explosion of a supernova, scattering new cosmic elements everywhere, from black holes to nascent stars to entire planetary systems. How is one to confront such a swirling mass – much of it still pure potential – which is throwing off new capabilities and new risks almost every day? In this article, we will try to get a sense of what those risks are, and how they impact the banking world.
First, let’s survey how the world has changed with the explosion of genAI, and then discuss the new risk factors that have emerged as a result of this change. We will see how these new risk factors impact every one of the traditional functions and activities of a bank. Of course, we will also need to outline the steps that any financial institution should begin taking to prepare themselves for the world ahead.
Where It All Began – Symbolic AI
The disruptive power of technology has been with us for a long time, ever since banking evolved out of paper-based recordkeeping and began using computers. Possibly even before that. Technology has offered many benefits to the banking business. New and more sophisticated banking products. More information about customers and prospects. A larger virtual footprint, even for small banks. Classical AI, in the form of “machine learning” algorithms, expanded these horizons. New and more sophisticated models could be rigorously tested and deployed in applications like credit scoring, loan underwriting, and payment fraud detection.
But technology has always been a two-edged sword. Increased complexity allows for more points of failure, not to mention more avenues for compromise or other adversarial actions. The recent explosion in genAI and large language models (LLM’s) has added a whole new dimension to the complexity, and the risks that come along with it.
Let’s take a quick look at how the AI world evolved to get us to where we are today. Up until recently, most AI work in the banking world was based on an approach called symbolic AI, where the required knowledge was explicitly constructed out of building blocks which were recognizable in terms of the vocabulary of the banking world. A early credit decisioning system, for example, might have a rule like:
IF the applicant has ever declared bankruptcy THEN decline the application.
Rules of this sort could be used in many ways to classify the cases presented to them and facilitate decisions that needed to be made, from granting a loan to retrieving the appropriate legal document, to translating legal contracts from one language to another.
This simple rule-based approach evolved into what came to be called “machine learning” – algorithms that statistically evaluated the impact of many different characteristics on the success or failure of a decision and organized them in sophisticated ways to provide numeric scores to clarify the decision thresholds. Simple yes/no conclusions evolved into scores, e.g.
IF the applicant has ever declared bankruptcy THEN assign 100 points to the risk score.
The scores themselves were determined by statistical algorithms that estimated the impact each characteristic should have on a successful or unsuccessful decision. Scoring rules allowed many more characteristics to be incorporated into flexible decision models that could be “re-tuned” as needed to adapt to changes in the decision-making environment.
Most symbolic AI was “deterministic”. Symbolic AI, whether rule-based or statistical, had two key advantages from the perspective of a closely regulated industry like banking. They were “deterministic”, in the sense that, given the same inputs, they would always produce the same outputs. This regularity helped ensure that decisions were made based only on specific selected characteristics, and other characteristics could be clearly excluded, as many banking regulations would require. Given the same inputs, the same decision would be made over and over again. Arbitrary interference could be avoided.
Symbolic AI was also “explainable”. The second key advantage was that the decisions could be explained to humans, making them suitable for human review. Every factor that went into a specific decision could be extracted and analyzed. Each business rule or statistical characteristic had a clear and explicit definition, and its impact on the final decision could be measured and explained. A banking customer could be notified about the top 3, or 5, or 7 reasons for a decision, as mandated by the Equal Credit Opportunity Act (1974) and subsequent regulations.
The Evolution of AI – From Symbolic AI to Generative AI
Ever since AI was founded, the symbolic approach was always (implicitly at least) in competition with a completely different approach which is often termed the “neural network” or “connectionist” model.
The neural approach was initially dismissed as inadequate. In the neural approach, knowledge is not represented explicitly, but coded into a network of small calculating elements that are connected into complex networks, where the calculations accumulate into a final answer at the end. Early investigations of this approach found it to be lacking in power, primarily because of its limitation to a single level of calculation.
Multi-level networks resolved the initial problems. In the mid-1980’s, Geoffrey Hinton and his colleagues discovered that by adding additional levels, connected by “hidden nodes”, the computational power of these networks would explode. It became possible to formulate training algorithms that incrementally adjust the calculation parameters based on a sequence of example after example. This development ushered in the era of “deep learning” systems that could classify cases presented according to very complex real-world criteria. Complex up-front engineering was no longer required.
Generative AI delivered a host of new capabilities. As massive computational power and correspondingly massive data storage capabilities came on-stream in the 21st century, enhanced neural networks evolved from simple classification engines into machines that could synthesize a surprising variety of digital artifacts. Large language models (or LLM’s) proved capable of constructing coherent text, from fictional stories, to summarizations of up-to-date knowledge,
to legal contracts. Static visual images could be created, from pictures of imagined scenes to pictures of known individuals in unusual settings, to faces of entirely new, non-existent humans. Formal artifacts could be made to seem legitimate, from legal documents to government ID’s. Computer code and the algorithms that drive them could be constructed. Convincing conversations could be generated. Interacting humans could be emulated in dynamic videos to an amazing degree of realism.
Generative AI operates on entirely different principles. All of this new power comes at a cost however, in terms of greater risks and the loss of some of the critical capabilities enjoyed by traditional AI. The computational networks involved are massive. As of this writing, the current generation of generative AI platforms contain from 7 billion (7×109) to around 1.7 trillion (1.7×1012) calculation parameters that are “learned” during the training process. Clearly, intermediate calculations based on so many parameters cannot be broken down into components that are comprehensible in human terms, forfeiting any ability to explain the rationale behind its decisions.
Generative AI capabilities are open to just about anybody. The rule systems and statistical models of the symbolic AI era required a certain level of skill in order to build them. Too much skill in some cases, which made them economically unfeasible for those purposes. But the complex objects of genAI can be created by anybody with a computer (or smart phone!) and access to a genAI platform, many of which are still available at no cost.
In particular, potential bad actors who may not have had the skill (or the staff) to hand-craft sophisticated fraud schemes or other attacks can now create them with a few keystrokes. This “democratization of AI” will lead to a proliferation of AI capabilities in the hands of many more actors with adversarial intent. Fraud attempts are likely to multiply exponentially. Distinguishing the real from the fake will be come an even greater challenge.
Generative AI usage involves a loss of control. If there’s anything that banking IT people distrust and try to avoid, it’s dependence on third parties for mission-critical capabilities. In the symbolic era, dependence could be minimized. Rule sets could be crafted in ordinary system languages like C or C++. Models could be trained with in-house data using standard data science languages like Python or (previously) R. But when it came time to deploy the models, the development tools were stripped out, and the model left to run as a stand-alone algorithm with no external dependencies.
Such an arrangement is simply impossible with genAI. No single financial institution, however large, has the resources to build a genAI platform from scratch. The capital costs to develop each of the GPT-4 generation of models has been estimated between $100M and $200M, and those models are already a generation or so behind. FI’s wishing to use genAI are thus lashed to the mast of whatever platform they have chosen to utilize.
An FI has no control over the training data. GenAI requires the use of large platforms that have been trained with enormous volumes of data, from the Greek classics to the latest corporate memos. Once this data has been baked into an LLM, It’s impossible to remove selected portions. The FI is stuck with whatever training material has been used, which can be a source of bias or misinformation that bleeds into the bank’s work products. The full implications of this fact are as yet undetermined, but the uncertainty, and hence the risks, should be obvious.
In extreme cases, bias and misinformation can render these work products invalid, or problematic from a regulatory perspective. Yet due to the way in which genAI produces its results, there is no systematic method to test for bias or misinformation, or guarantee its elimination.
Deploying Generative AI
Viewed from the perspective of its systems, banking activities can be seen as a collection of personnel and applications that process data in various ways, from recording a check deposit, to assisting a new customer. But supporting these tasks requires a complex network of pipelines which collect and move the necessary data, as well as humans to review the work products and intervene if necessary. Generative AI opens up new possibility for risks anywhere along these pipelines.
Generative AI is not suitable for every task. Some applications are simply not suitable for genAI. Tasks requiring decisions that are binary (yes/no) or quantitative in nature, such as transaction approval or pricing a loan, tend to be better suited to symbolic AI, which has the characteristics required for effective management and regulatory compliance, such as the repeatability and explainability we have already discussed.
GenAI results are not designed to be repeatable. A certain amount of randomness is often built directly into the architecture at prescribed levels, in order to create an impression of natural spontaneity. Moreover, the fast-paced nature of genAI development means that the platforms are constantly changing, so the results are subject to change outside the control of any financial institution.
It is a practical impossibility to “freeze” the potentially trillions of an LLM’s calculation parameters for each decision task just in order to guarantee repeatability. Further, as we’ve already noted, these trillions of parameters are not subject to human explanation, as effective management and regulatory compliance require. Traditional symbolic techniques, such as statistical machine learning, can be a better choice for tasks where the calculations employed must be systematically repeatable, or an explanation of the results is required, such as credit decisioning, loan underwriting, and funds transfer approval.
This still leaves a galaxy of possible uses for genAI in areas like the following:
· Marketing – creative collateral, illustrations, artwork
· Research/analysis – knowledge retrieval, summarization, scenario generation
· Help and assistance – knowledge retrieval, instruction sequences
· Communications – legal contracts, document editing
· Quantitative – synthetic data sets, analytic models, computer code
This last category in particular helps us see clearly that the determination of what is suitable for genAi does not rest solely on the difference between quantitative and qualitative. It has at least as much to do with characteristics like the finality of the output and the presence of humans in the workflow who can review and shape the work products, possibly through several iterations.
The data must be protected. The data used for any decision task must be kept confidential, even as some of it needs to be presented to an LLM in order to complete the task itself. It is especially important to distinguish personal identification information (PII data) that can be used to pick out an individual customer, from customer characteristics that are not PII. PII data must always be redacted before any data is passed to any genAI platform. Guarantees of confidentiality by third-party vendors cannot be relied upon.
Deployment of a genAI application requires both pre-processing and post-processing steps to be deployed as well. PII data is stripped out in the pre-processing step, and replaced if necessary with internally generated tokens. Post-processing reverses the redactions, backing out the tokens and substituting the original PII.
Data that is not PII must also be protected, as it can yield valuable information about a bank’s internal policies. Techniques such as retrieval augmented generation, or RAG, incorporate customer data into the genAI request itself, restricting ingestion by the LLM. This helps to minimize the possibility of data capture on the LLM’s part.
The pipelines must be protected too. A primary objective of any FI is to keep its pipelines flowing smoothly, without any possibility of interruption or intrusion. Incorporating genAI anywhere in this network makes things more complicated. For example, when large, unstructured data is passed through a legacy network, there is a constant risk of “buffer overflow”, causing excess data to bleed out into areas where it doesn’t belong. Not only are such vulnerabilities hard to detect because they are intermittent, they can play havoc with system operations, even possibly bringing the entire system to a halt.
Clearly, the bank has no control over the internal structure of its chosen genAI platform. Trusting that the vendor has adequately protected their own pipelines is not a solution. Excuses like “It’s the vendor’s problem” are unlikely to carry much weight with top management or with regulators.
This is particularly true with respect to the risk of “injection” attacks, where bad actors attempt to insert spurious information, or worse, executable code, into complex data streams in hopes of exploiting a buffer overflow or otherwise inserting them into the target banks’ environment, and possibly executing the code to create many all of mischief. The relatively small number of genAI platforms practically guarantees that they will be carefully studied by those who contemplate injection attacks, because their economic payoff is potentially enormous. It is absolutely necessary to ensure that genAI artifacts are properly isolated and validated.
To summarize, significant resources are required to verify the safety, reliability, and integrity of any pipeline that carries genAI data. Steps must be taken to contain, isolate, and where necessary, bypass genAI applications whenever a problem occurs. Contingency plans must be in place for continuing operations when a bypass occurs. In short, accommodating a genAI application into the bank’s pipeline is not a simple task.
Even the content produced by generative AI must be carefully curated. We’ve already discussed the risks associated with overflow and injection attacks on genAI data. But even after these risks have been mitigated, the content itself must be curated. When a legal document is created, it must be verified to ensure that all the i’s are dotted and the t’s are crossed.
Even with less formal artifacts, care must be taken to ensure that the information presented is complete and appropriate. Marketing collateral needs to be reviewed for exaggeration and excessive claims. Visual content needs to checked to make sure it does not convey a biased or inappropriate image of the institution itself. A script for new account openings, for example, must include all relevant restrictions, such as minimum balance, whether it is delivered via a chatbot or via instructions to call center employees.
Failure to make these disclosures can and do happen even in face-to-face branch interactions, of course. It happened to me at a major national bank just about a year ago. But with genAI artifacts, a record is created that can become “discoverable” if failures of this sort escalate into legal action.
Chatbot interactions have their own specific risks of possibly alienating customers. A different national bank that I use is constantly interrupting my task flow with inappropriate questions like “Tell me about the level of your salary.” Even the linguistic style of the chatbot must walk a razor’s edge to avoid being overly obsequious on one hand, or excessively terse (e.g. “Sorry, you cannot do that.”) and unhelpfully rude on the other hand. The final interpretation rests with the customer of course, and may be influenced by factors such as age, background, and education level.
Dissatisfaction with poor customer service alternatives (e.g. offshore call centers) is already widespread. Some customers are pushing back by demanding immediate escalation of their call, or even returning to the branch for face-to-face contact. The economic implications here should be obvious.
Customer dissatisfaction has the potential to get much worse as genAI comes on-stream. The very complexity of genAI solutions makes it essentially impossible to guarantee up-front that a trained LLM will have all the knowledge necessary to make appropriate recommendations, at the right level of politeness.
Constant oversight of generative AI interaction is required. When up-front correctness of genAI responses cannot be guaranteed, FI’s are likely to find it necessary to retain a “human in the middle” monitor, review, and if necessary revise them.
The degree of human intervention needed will depend on both the impact of the decision and the importance of the customer. For low-impact interactions, it may prove sufficient for the human to periodically review selected interactions against established guidelines.
For high impact cases (say, approving a large loan) or for high value customers (who demand a higher level of service), the human may be required to review every case, or even interact with the customer themselves.
The “human in the middle” should themselves be an expert. The best people to review and revise genAI outputs are the subject matter experts themselves. Contracts generated by genAI should be reviewed by legal counsel, or at least a contract administrator. Suspicious Activity Reports and other artifacts related to financial crime should be reviewed by a BSA officer. Even run-of-the-mill chatbot interactions should be periodically reviewed by experienced customer service reps for both their tone and content.
Ultimately, these potential “humans in the middle” need to be involved earlier in the development cycle, as the genAI applications are trained and prepared, even before actual deployment.
Defending Against Generative AI
Adverse attacks are probably as old as banking itself. It’s not difficult to imagine some shady Renaissance character trying to convince a Medici banker that he is someone else, in order to gain access to that someone else’s funds. Modern banks are no strangers to these sorts of attacks, ranging from identity theft to forged checks to stolen credit card numbers and many more. Impersonation, in one form or another, lies at the heart of nearly all of them.
Generative AI creates entirely new fraud opportunities. GenAI has opened the floodgates for new forms of impersonation. Most prominent is the ability to create a realistic dossier for an entirely fictitious individual or organization. Realistic simulacra of ID cards, certificates, government documents, real estate leases or purchases. Even realistic “photographs” of people who never existed. Identity synthesis, as differentiated from identity theft, will evolve to become a major fraud vector.
Surprisingly, genAI capabilities have evolved to the point where they can be enlisted to create live action sequences that appear to be initiated by a person who may or may not exist. Perhaps the most notorious incident of this sort was a fraud perpetrated on the giant UK engineering company Arup.
The attack started when a junior staffer in Hong Kong received an email that appeared to be from the head office in the UK requesting a secret transfer of 200M Hong Kong dollars (over US $25M). The staffer rightly became suspicious and balked at the request, but was then invited into a quick video conference to discuss the matter. Unbeknownst to the staffer, the video images on the call were deepfake imitations of the UK-based CFO and his staff. Satisfied by the call, the staffer proceeded to transfer the funds.
The mischief that can be engendered by these capabilities is almost unlimited. FI’s can be fooled into creating accounts for spurious individuals or organizations. Once an account is created, loans can be applied for and other transactions initiated. An account with one FI can be leveraged into accounts or lines of credit with other FI’s.
Impersonation of real people will not disappear. These same capabilities can be used to create convincing emulations of existing persons, which can be put to work in separating those individuals (or their organizations) from their funds. Even before genAI, one of the most difficult to detect fraud schemes, often called “business email compromise”, made use of impersonation techniques to send e-mail instructions from one putative member of an organization (e.g. the CFO) to another (e.g. a junior staffer in Accounts Payable) instructing them to issue an emergency wire or other funds transfer. Attacks of this sort are essentially impossible for traditional fraud platforms to detect, since they originate from a proper employee, transmitted through proper channels.
Now consider the possibility that instead of an e-mail, the junior staffer has a face-to-face video conversation with the CFO, who appears to be on vacation outside the US, and needs the transfer done quickly (there’s always a sense of urgency about these schemes). Furthermore, suppose that the transfer requested is not a wire, but an internal transfer within the same bank to a spurious account that was set up earlier for a fictitious organization and allowed to mature as it built a history of apparently innocent transactions. The difficulties of detection multiply.
Incidents of this sort have been occurring for years. We have already discussed a very recent one, but let’s consider another. In March of 2019, a similar sort of fraud was perpetrated against an unnamed UK energy subsidiary of a German firm. The target of the fraud, surprisingly, was the subsidiary’s own CEO, who might have been expected to know better. He was fooled by an AI-generated voice recording that seemed to come from his boss, CEO of the German parent. The recording was precise enough that it was even able to mimic the CEO’s German accent as he tried to speak English.
The energy CEO was duped into sending EUR 220,000 to an account in Hungary. After arriving in Hungary, the funds were immediately transferred to Mexico, from where they were distributed to various other locations. Believe it or not, it wasn’t until the UK CEO received a third call requesting another transfer that he realized his mistake. Note that all this happened seven years ago. The sophistication of these techniques has undoubtedly escalated since then.
The availability of generative AI will enable more attacks. These capabilities will no longer be limited to criminal organizations with a high degree of technical skill — they will be available to smaller and smaller fraud rings, even single individuals, since genAI platforms can be operated by almost anybody.
We don’t yet know the uses to which such techniques will be put, but it’s possible to imagine some far-out “science fiction” scenarios that include injecting entire spurious transaction histories, or even the altering of a bank’s decisioning parameters such as its underwriting rules.
What we do know, however, is that there will be thousands, perhaps millions, of potentially bad actors out there, ranging from high school pranksters to malevolent nation states, spending a lot of time and energy, not to mention creativity, in crafting new ways of using genAI to penetrate banking systems and enjoy the lucrative rewards of doing so.
Under these circumstances, the number of fraud attacks will certainly increase, perhaps almost exponentially. So will their variety, as individual fraudsters exercise their own creativity and ingenuity. FI’s will need to ramp up their detection capabilities dramatically, not to mention developing whole new methods to identify and eliminate adverse attacks at scale. You will not have a choice.
Governance and Regulation of Generative AI
In its AI Update of April 22, 2024 the UK’s Financial Conduct Authority, citing the need for regulatory resilience, issued a call for more collaboration with its regulated financial institutions, in order to build an “empirical understanding” of AI as it develops, and a “consensus on best practice.” This appeal pretty much sets the tone for how governance and regulatory issues will need to be handled in a world where new genAI capabilities seem to be coming on-stream almost daily.
Managing AI risks will necessarily become ad hoc, incremental, and ultimately experimental. Many risks that accrue from the use of genAI are already known, but perhaps not well understood. There are undoubtedly other risks we presently know little or nothing about. The situation is too complex to admit of predicting these risks in advance, or of developing “once and for all” fixes. Risks will need to be addressed immediately as soon as they are discovered. Serious risk incidents will require “all hands on deck” to diagnose the problem and devise both short-term and long-term remedies.
Internal risk management will need to descend to “street level”. Constant monitoring will be the order of the day, as internal risk management functions learn to work hand-in-hand with lines of business to spot emerging vulnerabilities early and take steps to remedy them. This will likely involve pushing risk management personnel down into the lines of business to work alongside the operational staff who will likely be the first to spot anomalies.
Regulatory relationships will follow a similar path, likewise becoming ad hoc, incremental, and iterative, at least insofar as possible. Even as regulators try to articulate new policies and guidelines, AI will introduce major gaps in the traditional risk management frameworks. How is the traditional model risk notion of “explainability” to be interpreted in light of the myriad hidden nodes in an LLM model, which almost by design are not explainable? Who is responsible for the risks posed by agentic AI? The UK Financial Conduct Authority report was in a sense prophetic. Just as internal risk managers will need to work hand-in-hand with their lines of business, regulators will need to work hand-in-hand with their FI’s just to stay abreast of the exploding capabilities.
Looking to the Future
If one thing is certain, it is that the genAI supernova will continue to explode. For FI’s this represents a good news / bad news situation. The bad news, of course, is that entirely new and undreamed of risks lie ahead.
Generative AI may help FI’s to create new weapons of their own. The good news, on the other hand, is that some AI capabilities may emerge that enable FI’s to “fight fire with fire.” For example, it might eventually prove possible to train “deep learning” AI systems to distinguish fraudulent genAI artifacts from real ones, or to recognize sophisticated fraud strategies such as cross-market manipulation.
Organizational structures will need to evolve in order to adapt. The continuing development of genAI will also require changes to a bank’s organizational structure. As already noted, AI risk management will need to operate at “street level”. But the need for centralized capabilities will also grow. FI’s will invest in highly skilled technical staff who understand the nuances around training, calibrating, and re-training genAI applications. These experts will likely cluster into “centers of excellence” that can function as central clearing houses for new information, along with distributing that information back to the lines of business as consultants and architects.
In summary, all risk management functions within an FI will need to become ad hoc, incremental, and experimental, in line with their dependence on the front-line troops for discovery and mitigation ideas. Communications up and down, as well as across, will become more frequent, informal, and transparent, as the early disclosure of each problem becomes an opportunity to learn and adapt quickly. Benjamin Franklin’s words were never more true – we must all hang together, or we will all hang separately.