1
00:00:03,674 --> 00:00:07,314
Hi everyone, I'm Rayan, I'm the CEO and one of the co-founders of Vals AI.

2
00:00:07,734 --> 00:00:11,954
I'm going to talk a little bit about going from beyond the model card, how evaluations

3
00:00:11,954 --> 00:00:16,574
can be used to measure this jagged frontier, a special focus on finance for this audience

4
00:00:17,734 --> 00:00:22,754
as well. Yeah, so first off, talking about model cards themselves, we've built some of the most

5
00:00:22,754 --> 00:00:26,774
important benchmarks over the last couple of years that the major foundation model labs have included

6
00:00:26,774 --> 00:00:31,374
in their model releases. So people, I think, are familiar with artifacts like this. Last week,

7
00:00:31,474 --> 00:00:36,874
we have Sundar talking about our VALs index on stage. And then even just yesterday, Muse models

8
00:00:36,874 --> 00:00:41,774
was citing our finance agent benchmark. You know, it keeps our lives interesting to have to keep

9
00:00:41,774 --> 00:00:46,014
re-updating these slides as new models come out. I had to double check level numbers. You can get

10
00:00:46,014 --> 00:00:51,734
sense of how capable a model is for valuable work, but that's actually driving value for

11
00:00:51,734 --> 00:00:52,334
a lot of companies.

12
00:00:52,954 --> 00:00:57,034
I think two interesting stats that jump out at me, the groups that are able to actually

13
00:00:57,034 --> 00:01:01,134
track AI cost and spend are much, much more likely to have inferior performance.

14
00:01:01,214 --> 00:01:04,894
We also have refining the X-axis, again, it's not immediately obvious which is number one

15
00:01:04,894 --> 00:01:06,774
or what will be consistently two or three.

16
00:01:06,874 --> 00:01:08,274
In the X-axis, you've got cost.

17
00:01:08,414 --> 00:01:15,654
In particular, it's cost per task, not just tokened Excel models.

18
00:01:15,654 --> 00:01:17,634
that are core to results may be unintuitive.

19
00:01:18,274 --> 00:01:19,714
When we actually build benchmarks,

20
00:01:20,174 --> 00:01:21,834
one of the most important things that we need to look at

21
00:01:21,834 --> 00:01:23,774
is the metric we use to report accuracy.

22
00:01:24,174 --> 00:01:26,154
And so you have a different takeaway

23
00:01:26,154 --> 00:01:27,254
if you pick a different metric.

24
00:01:27,514 --> 00:01:29,374
A lot of benchmarks deal with partial credit,

25
00:01:29,934 --> 00:01:31,114
so what this is answering is,

26
00:01:31,374 --> 00:01:33,854
in expectation, if I were to give a question

27
00:01:33,854 --> 00:01:34,814
or a task to a model,

28
00:01:35,154 --> 00:01:37,994
what percentage of the points would I expect to see satisfied

29
00:01:37,994 --> 00:01:41,054
by the work output of that model,

30
00:01:41,374 --> 00:01:42,674
especially compared to humans?

31
00:01:43,074 --> 00:01:45,254
And then the fully correct answers metric

32
00:01:45,254 --> 00:01:47,574
is more looking at, of the percentage of questions

33
00:01:47,574 --> 00:01:49,514
that I give to models, what percentage of them

34
00:01:49,514 --> 00:01:50,714
will come back perfect?

35
00:01:50,974 --> 00:01:52,354
And that's a very different metric.

36
00:01:52,614 --> 00:01:55,714
It may mean that a model is very good at producing

37
00:01:55,714 --> 00:01:58,174
an incremental response that you can work with

38
00:01:58,174 --> 00:01:59,474
to build a better response,

39
00:01:59,654 --> 00:02:02,574
but if you were to pass this directly to a client,

40
00:02:02,754 --> 00:02:04,734
it would be a coin toss of whether it was

41
00:02:04,734 --> 00:02:06,454
meeting all the criteria that you have.

42
00:02:06,734 --> 00:02:08,574
And so picking the right metric is very important

43
00:02:08,574 --> 00:02:10,974
to accurately representing the performance of the model

44
00:02:10,974 --> 00:02:13,014
and actually forming a good intuition of what it can do.

45
00:02:14,814 --> 00:02:17,314
There's another three components that we think a lot about.

46
00:02:17,534 --> 00:02:20,674
So validity, are you actually evaluating what's going on in the real world?

47
00:02:20,874 --> 00:02:24,734
That means are the questions or tasks reflective of actual work going on or are they contrived?

48
00:02:25,354 --> 00:02:28,414
And then are the rubrics actually describing what we're looking for or the standard of

49
00:02:28,414 --> 00:02:29,314
excellence we want to see?

50
00:02:30,494 --> 00:02:33,994
Reliability, this is usually a function of the number of tasks or the sample count.

51
00:02:34,554 --> 00:02:38,874
A lot of companies, they have maybe five or six samples and they're kind of vibe checking

52
00:02:38,874 --> 00:02:40,634
new models or products as they get access.

53
00:02:40,634 --> 00:02:48,394
But it's actually really important to have a rigorous approach and have enough tasks and samples to make sure your repeated runs actually show consistent results.

54
00:02:49,134 --> 00:02:50,014
And then last is discriminability.

55
00:02:50,254 --> 00:02:55,734
So we don't think it's that valuable to have benchmarks where all models score 100% or all models score 0%.

56
00:02:55,734 --> 00:03:00,374
Ideally, we can actually discriminate between models and see what the frontier looks like.

57
00:03:00,494 --> 00:03:04,014
In particular, we also want to make sure we're building benchmarks forward-looking.

58
00:03:04,014 --> 00:03:08,054
So what are the new set of tasks that are even more ambitious than what models can do today

59
00:03:08,054 --> 00:03:13,414
to make sure that there's room at the frontier for us to continue to see model progress as they get better?

60
00:03:15,594 --> 00:03:22,134
Okay, and then some of the high-level problems that we've seen and I think others can and do run into.

61
00:03:22,394 --> 00:03:24,834
The first is that agents solve the test, not the problem.

62
00:03:24,974 --> 00:03:27,014
So this is talked a lot about as reward hacking.

63
00:03:27,574 --> 00:03:32,894
And we've seen a lot of instances where we'll have a sandbox and models will be run with our model proxy.

64
00:03:32,894 --> 00:03:36,214
and sometimes they may try and actually call other models

65
00:03:36,214 --> 00:03:38,694
within that proxy to escape the sandbox

66
00:03:38,694 --> 00:03:39,874
or make use of the internet.

67
00:03:40,534 --> 00:03:43,134
So it's really important to build the containers

68
00:03:43,134 --> 00:03:45,354
in a way that you're actually testing for what matters.

69
00:03:45,514 --> 00:03:46,994
Can they do the task at hand

70
00:03:46,994 --> 00:03:49,134
or are they actually leveraging other tools

71
00:03:49,134 --> 00:03:50,614
or maybe even other models to get there?

72
00:03:51,234 --> 00:03:53,534
We've also seen this issue in finance in particular

73
00:03:53,534 --> 00:03:57,114
where a lot of the times evals are looking at a set of rubrics

74
00:03:57,114 --> 00:03:59,694
or items that we want to see included in the response

75
00:03:59,694 --> 00:04:05,594
And so models can hedge and give three possible answers, which has a higher chance of satisfying

76
00:04:05,594 --> 00:04:09,294
all the criteria, but in practice that's way too much.

77
00:04:09,374 --> 00:04:13,114
A person does not want to consider three different answers, they just want to get the first answer

78
00:04:13,114 --> 00:04:14,154
and have it be right.

79
00:04:14,674 --> 00:04:19,314
And so the goal here is to try and build evals that are reflective of the actual work people

80
00:04:19,314 --> 00:04:23,594
are doing and the work output they want to see, not to build these contrived evals that

81
00:04:23,594 --> 00:04:25,014
the models can actually gain.

82
00:04:25,534 --> 00:04:28,914
The second is, I think a lot of people are starting to see

83
00:04:28,914 --> 00:04:30,834
that the cost of software is coming down,

84
00:04:30,994 --> 00:04:33,654
and it's actually very, very easy to produce code.

85
00:04:34,054 --> 00:04:36,474
But unfortunately, we still see in practice

86
00:04:36,474 --> 00:04:38,814
that running evals is incredibly challenging,

87
00:04:39,034 --> 00:04:40,714
especially from an infrastructure side of things.

88
00:04:41,074 --> 00:04:44,054
And in particular, as we get to longer-running agents,

89
00:04:44,254 --> 00:04:46,194
it means that it's really important to have infrastructure

90
00:04:46,194 --> 00:04:48,594
that can stay consistent over a long period of time,

91
00:04:48,694 --> 00:04:51,774
and then, if interrupted, can be resumed and respun back up.

92
00:04:51,774 --> 00:04:56,194
We have, I think, just recently crossed six trillion tokens just in evaluation.

93
00:04:56,914 --> 00:05:03,434
So we've run a considerable set of evals and we run everything in as high-degree parallel as possible.

94
00:05:03,974 --> 00:05:08,354
So I think this has really been load tested, but I think other people are going to run into this even on a much smaller scale.

95
00:05:09,474 --> 00:05:14,894
And the last is that I think a lot of the existing benchmarks, they deal with a single prompt, single response,

96
00:05:15,134 --> 00:05:17,234
but in practice everything is multi-turn.

97
00:05:17,234 --> 00:05:21,474
You have initial spec which you think covers everything, but then you get questions back

98
00:05:21,474 --> 00:05:25,294
or you have additional feedback once something is built, and then it takes more of this iteration

99
00:05:25,294 --> 00:05:26,694
to actually drive to the end output.

100
00:05:27,274 --> 00:05:31,374
And so I think benchmarks and evals need to actually cover this multi-turn component.

101
00:05:31,494 --> 00:05:32,294
That's not going to go away.

102
00:05:34,014 --> 00:05:37,354
So when we look at a case study in particular, when we look at our team, we did a little

103
00:05:37,354 --> 00:05:41,534
token maxing experiment where we got free access to some tools and just gave it to all

104
00:05:41,534 --> 00:05:45,474
the engineers, and they really made use of these unlimited tokens.

105
00:05:46,054 --> 00:05:48,814
and if we actually look back for that month,

106
00:05:49,034 --> 00:05:51,174
we spent $1.5 million worth of tokens.

107
00:05:51,374 --> 00:05:53,234
This was like a team of I think 10 or 12.

108
00:05:53,454 --> 00:05:55,134
So it was roughly 10 times the amount

109
00:05:55,134 --> 00:05:57,754
we were spending on salary for tokens in that month.

110
00:05:58,034 --> 00:05:59,154
And so that was pretty insane,

111
00:05:59,214 --> 00:06:00,474
but we saw great output.

112
00:06:00,654 --> 00:06:02,434
And so our goal was to try and maintain

113
00:06:02,434 --> 00:06:04,374
the same level of output across all these engineers,

114
00:06:04,654 --> 00:06:07,114
but massively cut down on cost.

115
00:06:07,434 --> 00:06:08,934
And really our solution to do that

116
00:06:08,934 --> 00:06:10,174
was of course to build an eval.

117
00:06:10,834 --> 00:06:12,774
We built a bunch of tools internally

118
00:06:12,774 --> 00:06:14,114
and then have now exposed this

119
00:06:14,114 --> 00:06:15,254
as a product called Vals Smith.

120
00:06:15,894 --> 00:06:19,994
This is what we are doing in coding and I think it is a sign of what is going to come

121
00:06:19,994 --> 00:06:21,274
in other parts of knowledge work.

122
00:06:21,914 --> 00:06:27,214
So we were able to look at the GitHub repo as a centralized way of seeing what work has

123
00:06:27,214 --> 00:06:27,934
already been done

124
00:06:28,054 --> 00:06:29,654
and what work is continuing to happen.

125
00:06:30,274 --> 00:06:35,294
In particular, GitHub repos are comprised of PRs, which have a GitHub issue and a resolution.

126
00:06:35,934 --> 00:06:42,174
So what we can do is unwind the clock and give the same tasks to agents to see if they could have done the work of humans.

127
00:06:42,994 --> 00:06:49,734
In doing that, we found a much clearer understanding of which models were overkill, way too expensive and did not need to be used,

128
00:06:49,734 --> 00:06:52,154
and then which ones were actually capable of getting the task done.

129
00:06:52,494 --> 00:06:55,154
And I think, again, this is another place where the intuitions

130
00:06:55,154 --> 00:06:58,274
that a lot of people have may not be what's actually borne out in practice.

131
00:06:58,414 --> 00:07:00,974
So I think you may see we're using a lot of closed-source models.

132
00:07:01,234 --> 00:07:04,494
We should try and cut costs and use our own self-hosted open-source model.

133
00:07:04,794 --> 00:07:09,054
You can see the comparison here we have between GPT-5.6 Sol and Kimi K3,

134
00:07:09,074 --> 00:07:11,814
and we see that 5.6 Sol is better.

135
00:07:12,014 --> 00:07:14,114
It's going 2% better on this internal eval.

136
00:07:14,274 --> 00:07:16,794
It's also considerably cheaper than hosting it

137
00:07:16,794 --> 00:07:19,854
through a very inference-optimized stack.

138
00:07:20,054 --> 00:07:23,374
So it's $2.50 instead of $11, and it's faster half the time.

139
00:07:23,614 --> 00:07:26,874
And so there's no real reason why this kind of intuitive sense

140
00:07:26,874 --> 00:07:28,994
of using open source would be a legitimate way

141
00:07:28,994 --> 00:07:29,914
to drive down costs.

142
00:07:30,294 --> 00:07:32,494
And so this has resulted in two main differences.

143
00:07:32,654 --> 00:07:35,814
One, we can actually figure out which tools we're

144
00:07:35,814 --> 00:07:37,934
going to procure, what the usage limits of those things

145
00:07:37,934 --> 00:07:39,354
should be across the organization.

146
00:07:39,774 --> 00:07:42,294
But then also, we can give recommendations for engineers

147
00:07:42,894 --> 00:07:44,934
say, we think that for this GitHub issue,

148
00:07:44,934 --> 00:07:48,994
you should actually use this particular coding tool with this model.

149
00:07:49,254 --> 00:07:51,914
And this is able to drive the cost down considerably.

150
00:07:52,094 --> 00:07:55,354
I think we're spending 90% less, but still able to get the same output.

151
00:07:57,134 --> 00:07:58,934
I think the question I would have for you all,

152
00:07:59,014 --> 00:08:00,614
we're continuing to expand this approach

153
00:08:00,614 --> 00:08:03,294
and take this model of coding to other domains,

154
00:08:03,414 --> 00:08:05,654
knowledge work, legal, finance, things like that.

155
00:08:05,854 --> 00:08:09,814
But the key question is, where are you tracking the history of work

156
00:08:09,814 --> 00:08:11,594
that has already been done by that firm?

157
00:08:11,714 --> 00:08:14,134
Some of these law firms have been around for like 100 years.

158
00:08:14,134 --> 00:08:17,994
And so where is that, I think, quality of the organization stored?

159
00:08:18,154 --> 00:08:19,834
Where are you tracking the standard of excellence?

160
00:08:20,374 --> 00:08:22,914
And so today may exist in places like email,

161
00:08:23,174 --> 00:08:26,134
from the history of responses that people have had with their clients,

162
00:08:26,374 --> 00:08:29,114
may exist in SharePoints, may exist in Databricks dumps,

163
00:08:29,174 --> 00:08:32,874
but I think this is the key treasure trove that a lot of companies have

164
00:08:32,874 --> 00:08:35,894
and don't realize will be the major unlock for evals

165
00:08:35,894 --> 00:08:38,854
and then the unlock for AI adoption more systematically.

166
00:08:44,954 --> 00:08:50,174
Once these evals are constructed, they can enable adoption to happen, but we have also

167
00:08:50,174 --> 00:08:52,794
seen that there are more advantageous ways that they can be used.

168
00:08:53,134 --> 00:08:54,434
First is right sizing.

169
00:08:55,414 --> 00:08:59,554
Rather than just token maximizing, how do you figure out what the right usage limit should

170
00:08:59,554 --> 00:09:00,534
be across the organization?

171
00:09:01,334 --> 00:09:05,974
Negotiation, once you realize you have actual leverage, you can take that to the conversation

172
00:09:05,974 --> 00:09:07,434
you have in bulk pricing.

173
00:09:08,014 --> 00:09:14,114
Then routing, this is becoming a core primitive in a lot of AI products, but can also be done

174
00:09:14,114 --> 00:09:18,574
done by users directly, and so they can build intelligent routers that actually figure out

175
00:09:18,574 --> 00:09:19,674
the right model to use.

176
00:09:20,694 --> 00:09:21,974
And then fine tuning or post training.

177
00:09:22,814 --> 00:09:26,934
This can be the most reliable way for you to actually mint the tasks and then have an

178
00:09:26,934 --> 00:09:29,454
external eval set to make sure that this approach is working.

179
00:09:31,814 --> 00:09:36,074
Last thought I'd leave you with is I think for a lot of companies, they have a competitive

180
00:09:36,074 --> 00:09:40,034
advantage that has existed in the heads of employees through things like institutional

181
00:09:40,034 --> 00:09:40,574
knowledge.

182
00:09:40,574 --> 00:09:44,734
and what evals allow you to do is actually codify the competitive advantage

183
00:09:44,734 --> 00:09:47,514
and put that down in words or in scripts, in language

184
00:09:47,514 --> 00:09:50,194
and that will actually be the biggest unlock

185
00:09:50,194 --> 00:09:53,194
for making sure that a company succeeds the agentic era.

186
00:09:53,934 --> 00:09:55,214
Okay, that's all. Thanks, everyone.