Wells Fargo Foundation Model: Application to Fraud Detection at Scale

My name is Freddy Lecue.

I'm part of the Wells Fargo team.

I'm leading the Frontier AI modeling organization here.

And I'm going to show you the work we've been doing lastly on building our first foundation

model, Wells Fargo Foundation model.

So I explain why we did that, how we did it, and what we accomplished.

in doing so so so first from financial model it doesn't necessarily need to be language based i

mean we're all playing with uh claude and chat gpt and others they're all language-based

from the show models in financial services uh we also have language of course uh from a

structure unstructured representation data but we also have a lot of structure representation

a lot of these data is tabular data we have a lot of information from our customers we have

information in investment banking, consumer banking, and all of this data is data that we

have in Wells Fargo, and all big financial organizations have too. And we want to take

benefit of all of this huge amount of data that we have. And it's not like you just take it and

put it in a language model and get the best of it. Of course, you can get a lot of benefit out of it,

but it's not necessarily the best way to do it.

So what we decided to do is to look at the data we have,

all the transactions, all the customer transactions

that they're doing with their debit card, credit card, digital, check,

all of those coming from many different channels

and see if we can build a foundational model

that we can use and fine-tune for downstream tasks

like credit scoring or that you can use it for fraud detection,

anti-money laundering, do marketing.

So how we can build one backbone that you can use and streamline across downstream tasks.

So now you're moving from many models very siloed across different organizations.

In really one backbone that every organization can just reuse because you learn the representations of customers' transactions at scale.

So that's the intent.

And just spoiler alert, you may say, hey, that's tabular data.

why do you just don't use XGBoost and the usual suspect?

We did that.

This is the model we had in production.

But the model I'm going to show you is something that beats XGBoost,

even from a latency perspective, but also from a fault detection,

we get much better reduction of false positive and better detection rate.

So it was a really good lesson learned in terms of having better model

using foundation model.

And to foundation, I mean transformer blocks, and that's what we build.

So we have, as I mentioned, billions of transactions on a monthly basis.

Our customers are interacting with their debit credit card, and they do checks, and they do different sort of transactions.

And the question is really how we can use this information at scale, and not for one and every single downstream task.

So I'm going to cover a bit of the data, the representations, the architecture that we use,

what the result we get, and the lesson we learned in building this.

So left-hand side, language foundational model.

They're processing language tokens, and then the goal is to learn the representations that's going to predict the next token.

and then you have supervised fine-tuning

and then you do LHF, LRVR

and all of the great post-training process

to get better at the task that you want to, right?

And that's for languages, that's great.

But what if your data is very much more specialized

in terms of representation?

So that's where we build this transaction foundation model.

So instead of having languages as the base,

the base is now transactions.

transaction from our customers.

And you still try to predict the next thing,

and here the next thing is the next transaction,

or it's the next token in the transaction,

using all of the channels that you have.

But that's not the end goal.

I mean, similarly to language model,

where you try to predict the next token,

that's not the end of the story, right?

You learn the representation,

and then you use that representation,

and then you fine-tune it,

and you post-tune it for any other task, right?

So it could be classification tasks for languages,

but in our context for transaction foundation model it could be used for

credit line increase could be used for doing fraud detections and flagging

bad actors from

an anti-money-laundering perspective.

And the goal is to learn this relationship between transactions over time and transactions across customers. That's really the intent.

So the intent number one is you do pre-train from scratch. I'm saying it again because people are sometimes confused.

we pre-train transformer blocks at scale from scratch using our transaction data.

So the zero rate that we use from open rates, we just use the architecture,

and then we pre-train it from scratch using our GPU cluster,

and then we train it for a couple of days and weeks,

and then you do the downstream task of post-training.

but the goal here and you pre-train it

for predicting the next

token or the next transactions

and you try to do this

you try to predict the next pattern

of the customer but that's not the intent

the intent is to do the downstream

task later on and in terms of scale

we operate

at a large scale

so we have billions of transactions

a million of accounts

hundreds of transactions

per client on a monthly

basis interacting with their

debit card, credit card, or doing check.

And the five main dimensions we look at

is really who is doing,

I mean, who is processing and who is asking

and who is providing a service.

So, will it the customer or the merchant?

The what?

So, I mean, what do we buy?

What do we transact?

And how?

So, which channel?

Is it credit?

Is it debit?

Is it digital?

When is it happening?

And where is it happening?

And all of those, they're basically breaking down in hundreds of features.

And if you look at the existing model we have in production,

there's a lot of people that build those models in similar organizations

may understand that.

They build a lot of very hard-crafted, uncrafted engineering,

feature engineering efforts on that.

And that's required a lot and a lot of time.

So we step back and look at the raw features we need,

not the one that's been pre-built and post-built for doing the downstream tasks,

but try to understand which raw features.

And we ended up with basically Android features.

Even at the end of the day, we had thousands of them,

but we tried to find which ones are the raw information unit.

And then we led the architecture, the transformer blocks,

to figure out the relationship between those core features.

and as you can see

we do have a bunch of features

on those five dimensions

and the vocabulary is very unique

to the financial services setup

and that's the reason why

we didn't really use

existing pre-trained model

because the vocabulary that we use

is very specific to the work

that you want to do

and even if you merge

while all digital and checking and saving

and all of those sort of transactions,

even if you merge all of those,

you still end up with a very limited number of features and characteristics.

So at that scale, then the question is,

well, how you work with that?

So we started, everything is tabular data.

I mean, there's no secret, it's tabular data.

And then the goal is to understand

how you're going to capture the vocabulary

because you do have numerical values,

you have categorical values, you have information like zip code,

do you encode each and every zip code in a token?

Does it make sense?

So there's a lot of efforts you need to push in understanding

the vocabulary that you need and the vocabulary that you want also to scale

because at the training, if you have too huge vocabulary,

it would be kind of messy to train such a model with a vocabulary

which is far, far too big.

So there's a lot of effort in understanding this value to tokens.

So which information do you encode in a token?

And then you have the token to ID, the tokenizations,

and then the embedding lookup.

So then when we have a transaction coming up,

you do have the vocabulary.

So basically it's a lookup in aligning a transaction

and all the characteristics of that transaction

to a token and an ID and then an embedding.

So just to give you a number,

well some numbers so we have quantitative qualitative categorical of different kind

right so categories with only eight types some with much bigger types you have transaction amount

which is a big deal you have the delta time between transaction so you have anything related

to ip address postal code so just to give you example of variety and heterogeneity of each and

every characteristic could be encoded in a binary token

or could be encoded in a much larger number of tokens.

And even if you look at the zip code,

I mean, you have a much larger number of zip code

in the US than those 2,800,

but you have much better way to structure

how you're going to regroup zip code

based on distributions, for instance.

so architecture

so here just

so we use the

chinchilla

mathematics behind

to define how we build it

initially because we had

no clue how it's going to work in the tabular

space even though there are

existing

open ways and open solutions

for doing tabular

data foundation models

so we started from

what they learn and adapt it to what we needed to do and that gives you an

example of invading dimensions number of decoder blocks we use and all of the

recipe from a from an architecture perspective so we have and as you can

see we're not trying to build a trillion parameters model we don't try to compete

with any of the the leaders in the language models because we working with

and focusing on transactions, right?

Transactions, foundation models at billion level.

And one billion is because that's the model that gave us,

that was kind of the smallest and gave us the highest boost.

And now we're working on the next versions at 11 billion.

But that's good enough to give a lot of ROI

from a false positive reduction for downstream tasks

like for detections and the detection,

unit detection rate.

So, and the type of architecture,

so the decoder-only architecture,

that's basically for predicting the next token

or the next transactions.

And that's really the pre-training.

So here we initially used 20 times the number of parameters

to define the number of tokens we might need.

We ended up actually 10 times fractured than that.

So by doing a few experimentations,

we needed much more tokens per parameters.

I mean, this is kind of the initial experimentation we did.

So I'll pass on this because this is the usual suspect

from an objective function,

so causal language modeling and mass language modeling

type of optimization work.

And from an architecture perspective,

nothing really fancy here.

the most complex part was really the tokenization

and then

doing the pre-training and then how you do the fine-tuning.

So I would say the recipe or the secret sauce

is really how you do the tokenization

and how you're going to do the post-training.

But anything from a training perspective,

the question is just

how many transactions

you're going to put in your context

or in your history for doing the training.

and evaluation perplexity-based accuracy dashboard

on downstream tasks.

That's the ULs respect.

And so, yeah, so use mixed precision,

distributed data parallel,

maximize macro batch size,

get faster convergence,

yeah, using Muran,

Rope, usual,

and memory efficiency, flash attention,

activation chain pointing.

I mean, here there's nothing really new.

It's just we use the advices from the best

and from the one that built even much bigger models.

So I'll pass on this one.

Time is short.

But I wanted to spend some time on the results.

So, yeah, so when you build this foundation,

That foundation is really, is only a backbone.

It's something you can use for downstream tasks.

And the downstream task was to do fraud classification.

So you have a transaction, and then you try to predict if that transaction is going to be fraudulent.

So in the model we have in production, it's an XGBoost model,

and it's trained on large amount of transactions, which is fraudulent or non-fraudulent.

and the missing pieces

was the relationship

between whatever transaction

you try to predict, try to classify

and the relationship with this

transaction with all of the past transactions

and that information is kind of

gone when you use an XGBoost model

unless you encode it

in a very hard way which is not

so easy but if you use

transformer architectures you do have

these nice attentions

from transactions to

past transactions

And that's basically what helps, and this is what we think, this is what really helps pushing the fraud detection rate much higher.

But at the end of the day, the downstream task was to say, here's the

sequence of transactions. I have someone just using the debit card. Is that transaction fraudulent with respect of all historical information? And that's exactly what you put in at inference time when you try to do the prediction.

so that's the

architecture so nothing

really fancier you just need to have the

good engineering

in place to do

less than 250

milliseconds for doing the processing

because the

decline needed to happen within that time frame

but it's

we were

10 times slower than done from a latency perspective

so it was all good

and just in terms of

final result we get. So the main

metrics and those metrics

relate right away to

savings for the firm.

So it's detection rate

and false positive rate. Of course

you can detect much higher, but

in many cases

sometimes you get much more false positive.

And in this case, we managed to get

more fraudulent

activities and we even

reduced the false positive.

so that was a big win

because at the end of the day

if you decline a transaction

which is not fraudulent

then that person is going to call

there is an operation cost associated to it

you lose the transaction related to

that non-fraudulent action

so it's a very

well

something that you don't really want to happen

so again

that's compared to the state of the art

of the XGBoost model

using fraud detection

So it's basically huge.

It's millions of dollars savings.

So it's a very big boost.

So just to finish up on that slide,

so that foundation model has been built on billions of transactions

from Wells Fargo customers.

That's number one.

It's pre-trained from scratch.

So we don't free train from any open-rate existing model.

We use existing architecture, but the way the zero rate, we pre-training on a couple

of days.

I mean, we did a couple of weeks initially, but then that's pre-trained.

And then when it's pre-trained, it's pretty good in predicting the next transaction.

And when you're done with that, then you can use it for many downstream tasks.

We did it for fraud detection.

Now we're investigating for credit line increase, investigating for anti-money laundering,

investigating for marketing.

so now the same foundation is used

across different organizations

even Wells Fargo

and yeah

thank you for having me and thank you