Beyond the Model Card: The Jagged Frontier of AI in Finance

Hi everyone, I'm Rayan, I'm the CEO and one of the co-founders of Vals AI.

I'm going to talk a little bit about going from beyond the model card, how evaluations

can be used to measure this jagged frontier, a special focus on finance for this audience

as well. Yeah, so first off, talking about model cards themselves, we've built some of the most

important benchmarks over the last couple of years that the major foundation model labs have included

in their model releases. So people, I think, are familiar with artifacts like this. Last week,

we have Sundar talking about our VALs index on stage. And then even just yesterday, Muse models

was citing our finance agent benchmark. You know, it keeps our lives interesting to have to keep

re-updating these slides as new models come out. I had to double check level numbers. You can get

sense of how capable a model is for valuable work, but that's actually driving value for

a lot of companies.

I think two interesting stats that jump out at me, the groups that are able to actually

track AI cost and spend are much, much more likely to have inferior performance.

We also have refining the X-axis, again, it's not immediately obvious which is number one

or what will be consistently two or three.

In the X-axis, you've got cost.

In particular, it's cost per task, not just tokened Excel models.

that are core to results may be unintuitive.

When we actually build benchmarks,

one of the most important things that we need to look at

is the metric we use to report accuracy.

And so you have a different takeaway

if you pick a different metric.

A lot of benchmarks deal with partial credit,

so what this is answering is,

in expectation, if I were to give a question

or a task to a model,

what percentage of the points would I expect to see satisfied

by the work output of that model,

especially compared to humans?

And then the fully correct answers metric

is more looking at, of the percentage of questions

that I give to models, what percentage of them

will come back perfect?

And that's a very different metric.

It may mean that a model is very good at producing

an incremental response that you can work with

to build a better response,

but if you were to pass this directly to a client,

it would be a coin toss of whether it was

meeting all the criteria that you have.

And so picking the right metric is very important

to accurately representing the performance of the model

and actually forming a good intuition of what it can do.

There's another three components that we think a lot about.

So validity, are you actually evaluating what's going on in the real world?

That means are the questions or tasks reflective of actual work going on or are they contrived?

And then are the rubrics actually describing what we're looking for or the standard of

excellence we want to see?

Reliability, this is usually a function of the number of tasks or the sample count.

A lot of companies, they have maybe five or six samples and they're kind of vibe checking

new models or products as they get access.

But it's actually really important to have a rigorous approach and have enough tasks and samples to make sure your repeated runs actually show consistent results.

And then last is discriminability.

So we don't think it's that valuable to have benchmarks where all models score 100% or all models score 0%.

Ideally, we can actually discriminate between models and see what the frontier looks like.

In particular, we also want to make sure we're building benchmarks forward-looking.

So what are the new set of tasks that are even more ambitious than what models can do today

to make sure that there's room at the frontier for us to continue to see model progress as they get better?

Okay, and then some of the high-level problems that we've seen and I think others can and do run into.

The first is that agents solve the test, not the problem.

So this is talked a lot about as reward hacking.

And we've seen a lot of instances where we'll have a sandbox and models will be run with our model proxy.

and sometimes they may try and actually call other models

within that proxy to escape the sandbox

or make use of the internet.

So it's really important to build the containers

in a way that you're actually testing for what matters.

Can they do the task at hand

or are they actually leveraging other tools

or maybe even other models to get there?

We've also seen this issue in finance in particular

where a lot of the times evals are looking at a set of rubrics

or items that we want to see included in the response

And so models can hedge and give three possible answers, which has a higher chance of satisfying

all the criteria, but in practice that's way too much.

A person does not want to consider three different answers, they just want to get the first answer

and have it be right.

And so the goal here is to try and build evals that are reflective of the actual work people

are doing and the work output they want to see, not to build these contrived evals that

the models can actually gain.

The second is, I think a lot of people are starting to see

that the cost of software is coming down,

and it's actually very, very easy to produce code.

But unfortunately, we still see in practice

that running evals is incredibly challenging,

especially from an infrastructure side of things.

And in particular, as we get to longer-running agents,

it means that it's really important to have infrastructure

that can stay consistent over a long period of time,

and then, if interrupted, can be resumed and respun back up.

We have, I think, just recently crossed six trillion tokens just in evaluation.

So we've run a considerable set of evals and we run everything in as high-degree parallel as possible.

So I think this has really been load tested, but I think other people are going to run into this even on a much smaller scale.

And the last is that I think a lot of the existing benchmarks, they deal with a single prompt, single response,

but in practice everything is multi-turn.

You have initial spec which you think covers everything, but then you get questions back

or you have additional feedback once something is built, and then it takes more of this iteration

to actually drive to the end output.

And so I think benchmarks and evals need to actually cover this multi-turn component.

That's not going to go away.

So when we look at a case study in particular, when we look at our team, we did a little

token maxing experiment where we got free access to some tools and just gave it to all

the engineers, and they really made use of these unlimited tokens.

and if we actually look back for that month,

we spent $1.5 million worth of tokens.

This was like a team of I think 10 or 12.

So it was roughly 10 times the amount

we were spending on salary for tokens in that month.

And so that was pretty insane,

but we saw great output.

And so our goal was to try and maintain

the same level of output across all these engineers,

but massively cut down on cost.

And really our solution to do that

was of course to build an eval.

We built a bunch of tools internally

and then have now exposed this

as a product called Vals Smith.

This is what we are doing in coding and I think it is a sign of what is going to come

in other parts of knowledge work.

So we were able to look at the GitHub repo as a centralized way of seeing what work has

already been done

and what work is continuing to happen.

In particular, GitHub repos are comprised of PRs, which have a GitHub issue and a resolution.

So what we can do is unwind the clock and give the same tasks to agents to see if they could have done the work of humans.

In doing that, we found a much clearer understanding of which models were overkill, way too expensive and did not need to be used,

and then which ones were actually capable of getting the task done.

And I think, again, this is another place where the intuitions

that a lot of people have may not be what's actually borne out in practice.

So I think you may see we're using a lot of closed-source models.

We should try and cut costs and use our own self-hosted open-source model.

You can see the comparison here we have between GPT-5.6 Sol and Kimi K3,

and we see that 5.6 Sol is better.

It's going 2% better on this internal eval.

It's also considerably cheaper than hosting it

through a very inference-optimized stack.

So it's $2.50 instead of $11, and it's faster half the time.

And so there's no real reason why this kind of intuitive sense

of using open source would be a legitimate way

to drive down costs.

And so this has resulted in two main differences.

One, we can actually figure out which tools we're

going to procure, what the usage limits of those things

should be across the organization.

But then also, we can give recommendations for engineers

say, we think that for this GitHub issue,

you should actually use this particular coding tool with this model.

And this is able to drive the cost down considerably.

I think we're spending 90% less, but still able to get the same output.

I think the question I would have for you all,

we're continuing to expand this approach

and take this model of coding to other domains,

knowledge work, legal, finance, things like that.

But the key question is, where are you tracking the history of work

that has already been done by that firm?

Some of these law firms have been around for like 100 years.

And so where is that, I think, quality of the organization stored?

Where are you tracking the standard of excellence?

And so today may exist in places like email,

from the history of responses that people have had with their clients,

may exist in SharePoints, may exist in Databricks dumps,

but I think this is the key treasure trove that a lot of companies have

and don't realize will be the major unlock for evals

and then the unlock for AI adoption more systematically.

Once these evals are constructed, they can enable adoption to happen, but we have also

seen that there are more advantageous ways that they can be used.

First is right sizing.

Rather than just token maximizing, how do you figure out what the right usage limit should

be across the organization?

Negotiation, once you realize you have actual leverage, you can take that to the conversation

you have in bulk pricing.

Then routing, this is becoming a core primitive in a lot of AI products, but can also be done

done by users directly, and so they can build intelligent routers that actually figure out

the right model to use.

And then fine tuning or post training.

This can be the most reliable way for you to actually mint the tasks and then have an

external eval set to make sure that this approach is working.

Last thought I'd leave you with is I think for a lot of companies, they have a competitive

advantage that has existed in the heads of employees through things like institutional

knowledge.

and what evals allow you to do is actually codify the competitive advantage

and put that down in words or in scripts, in language

and that will actually be the biggest unlock

for making sure that a company succeeds the agentic era.

Okay, that's all. Thanks, everyone.