1
00:00:04,661 --> 00:00:05,101
Hey,

2
00:00:06,361 --> 00:00:08,921
All right, I precariously balanced my water over there.

3
00:00:09,561 --> 00:00:10,841
My name's John Feminella.

4
00:00:11,061 --> 00:00:14,101
I work on AI architecture and engineering at Two Sigma,

5
00:00:14,321 --> 00:00:15,801
which is a hedge fund here in New York.

6
00:00:16,581 --> 00:00:18,341
Before I get too far into things,

7
00:00:18,421 --> 00:00:21,601
I have a legal disclaimer slide that I have to show you

8
00:00:21,601 --> 00:00:22,961
that says two things.

9
00:00:23,141 --> 00:00:26,981
One, I promise not to sell or market or offer you any securities

10
00:00:26,981 --> 00:00:28,181
at any point in this talk.

11
00:00:28,181 --> 00:00:30,701
and two, everything in this is my own opinion

12
00:00:30,701 --> 00:00:32,021
and not the opinion of Two Sigma.

13
00:00:32,581 --> 00:00:33,721
Thanks for indulging me.

14
00:00:33,861 --> 00:00:35,241
If I don't say those magic words,

15
00:00:35,541 --> 00:00:38,241
the 5G microchips implanted in me will explode.

16
00:00:40,561 --> 00:00:42,481
So that slide is a good example, actually,

17
00:00:42,501 --> 00:00:44,961
of how our industry is regulated.

18
00:00:45,241 --> 00:00:47,961
And as a regulated company with lots of controls,

19
00:00:48,241 --> 00:00:50,161
I think one thing that's always interesting

20
00:00:50,161 --> 00:00:52,961
is to explore what happens when controls fail

21
00:00:52,961 --> 00:00:54,381
because that gives you a good idea

22
00:00:54,381 --> 00:00:56,681
about why the controls were there in the first place.

23
00:00:56,681 --> 00:01:00,781
And so I'd like to tell you a story about that by showing you another legal slide.

24
00:01:01,501 --> 00:01:07,141
After August 2020, a very interesting contract clause suddenly started showing up in a bunch

25
00:01:07,141 --> 00:01:08,521
of commercial loan agreements.

26
00:01:08,881 --> 00:01:12,481
And this clause essentially tries to prevent a certain scenario from happening.

27
00:01:12,641 --> 00:01:14,941
And it's complicated, but I'll explain it like this.

28
00:01:15,041 --> 00:01:20,281
Let's say that you're a lemonade stand business, and you borrow money from a lender to capitalize

29
00:01:20,281 --> 00:01:21,221
your lemonade stand.

30
00:01:21,221 --> 00:01:24,141
and the bank that's overseeing that transaction,

31
00:01:24,141 --> 00:01:27,341
that's in the middle between the borrower and the lender,

32
00:01:28,301 --> 00:01:33,821
sends the lender all of the money that you owe in a catastrophic mistake.

33
00:01:34,441 --> 00:01:36,501
You'd probably be mad and want the money back,

34
00:01:36,561 --> 00:01:38,041
but the lender might also say,

35
00:01:38,161 --> 00:01:40,201
no thanks, you paid us back and we're all good,

36
00:01:40,221 --> 00:01:41,481
so we'd rather keep the money.

37
00:01:42,081 --> 00:01:44,281
Something like this happened in August 2020.

38
00:01:45,041 --> 00:01:50,621
Revlon, the cosmetics company, had borrowed money under a big 2016 term loan.

39
00:01:50,621 --> 00:01:58,361
And that loan was split up into a bunch of pieces across 300-plus lenders, hedge funds, credit funds, CLOs, other banks.

40
00:01:58,661 --> 00:02:01,581
And in the middle is Citibank, the administrative agent.

41
00:02:01,821 --> 00:02:09,941
Being the agent in this kind of transaction means that you're performing the plumbing that connects the borrowers with the lenders and managing the lifecycle of the loan.

42
00:02:10,241 --> 00:02:13,821
So in this transaction, Revlon pays Citi, and then Citi pays the lenders.

43
00:02:14,041 --> 00:02:15,841
And Citi's not lending its own money.

44
00:02:15,961 --> 00:02:18,001
It's just in the middle, moving people's money around.

45
00:02:18,881 --> 00:02:22,281
So here's the plan for August 11th, 2020.

46
00:02:22,861 --> 00:02:26,861
Revlon would pay about $8 million of accrued interest to both groups,

47
00:02:26,901 --> 00:02:29,821
and the lender's principal would roll into this new facility.

48
00:02:29,961 --> 00:02:31,981
That's an exchange of paper, not cash.

49
00:02:32,121 --> 00:02:34,561
So the remaining lender's principal would stay put,

50
00:02:34,761 --> 00:02:38,061
and the idea here is that only interest should be going out.

51
00:02:38,161 --> 00:02:39,601
Principals should not be moving at all.

52
00:02:40,461 --> 00:02:45,241
And the bank had a piece of internal processing software called FlexCube.

53
00:02:45,361 --> 00:02:47,261
Isn't that a fun enterprise software name?

54
00:02:47,981 --> 00:02:51,841
in FlexCube, the easiest way to do this kind of transaction

55
00:02:51,841 --> 00:02:56,501
is to pretend to record it like Revlon were paying off the entire loan.

56
00:02:56,801 --> 00:03:00,481
And that triggers the correct interest payment calculations to everyone.

57
00:03:00,721 --> 00:03:02,681
You can probably guess what's about to happen next.

58
00:03:02,941 --> 00:03:05,501
So you're supposed to send the principal portion

59
00:03:05,501 --> 00:03:08,761
to an internal WASH account that you own as the bank

60
00:03:08,761 --> 00:03:11,501
so that it's booked and recorded correctly,

61
00:03:11,541 --> 00:03:13,561
but it never leaves the bank's perimeter.

62
00:03:13,761 --> 00:03:15,401
So keep your eye on that WASH account.

63
00:03:15,401 --> 00:03:21,401
What was supposed to happen is that the boxes labeled front and fund and principal should

64
00:03:21,401 --> 00:03:25,341
all include the WASH account number and should all have a check box checked next to them.

65
00:03:26,221 --> 00:03:31,061
So unfortunately it turned out that was what needed to happen, but for various arcane reasons

66
00:03:32,021 --> 00:03:33,661
the boxes were not checked.

67
00:03:33,901 --> 00:03:38,661
And although FlexCube pops up a warning notification that says, uh-oh, money is about to leave the

68
00:03:38,661 --> 00:03:40,301
bank, are you sure you want to do this?

69
00:03:40,461 --> 00:03:41,581
That was correct.

70
00:03:41,701 --> 00:03:44,501
Money was supposed to leave the bank because they were making the interest payments.

71
00:03:44,501 --> 00:03:49,041
So the three approvers who were supposed to approve that looked at it and said, yep, that's

72
00:03:49,041 --> 00:03:50,121
the right thing that's supposed to happen.

73
00:03:50,741 --> 00:03:55,381
And although that warning notification popped up, the human operators carrying out the transaction

74
00:03:55,381 --> 00:03:56,181
overwrote that.

75
00:03:57,021 --> 00:03:58,861
Remember, Revlon was making that interest payment.

76
00:03:59,281 --> 00:04:01,541
So the principal never went to the wash account.

77
00:04:01,761 --> 00:04:07,641
And instead, Flexcube sent the entire outstanding principal of the loan to all of the 300 lenders,

78
00:04:07,641 --> 00:04:10,441
about $900 million instead of $8 million.

79
00:04:11,061 --> 00:04:14,461
So every lender got the amount that they would have been owed

80
00:04:14,461 --> 00:04:16,281
if Revlon had paid the loan in full,

81
00:04:16,341 --> 00:04:17,921
but Revlon didn't want to pay off the loan.

82
00:04:18,121 --> 00:04:19,141
It wanted to hold on to its money.

83
00:04:19,341 --> 00:04:20,661
That was a great deal for the lenders,

84
00:04:20,681 --> 00:04:22,701
because remember, Revlon was in trouble at the time,

85
00:04:22,741 --> 00:04:24,961
so its loans weren't trading at face value,

86
00:04:24,981 --> 00:04:26,921
and getting all your money back on a deal

87
00:04:26,921 --> 00:04:30,001
that you thought was only worth 60 cents on the dollar

88
00:04:30,001 --> 00:04:31,021
is a great deal for you.

89
00:04:31,901 --> 00:04:33,681
So they didn't want to give the money back,

90
00:04:33,701 --> 00:04:35,321
even after Citi called them and said,

91
00:04:35,441 --> 00:04:37,061
please, that was a mistake, give us the money back,

92
00:04:37,141 --> 00:04:37,901
and their argument was,

93
00:04:37,921 --> 00:04:39,121
we shouldn't have to give the money back

94
00:04:39,121 --> 00:04:41,941
because you paid us in full, and that's how loans work.

95
00:04:42,061 --> 00:04:43,001
You eventually pay them back,

96
00:04:43,021 --> 00:04:45,361
and just because you paid it back at a different time

97
00:04:45,361 --> 00:04:46,801
than you were supposed to doesn't mean

98
00:04:46,801 --> 00:04:48,401
that we should have to give it back now.

99
00:04:48,721 --> 00:04:50,121
And so to prevent that from happening,

100
00:04:51,121 --> 00:04:52,721
commercial loan agreements everywhere started

101
00:04:52,721 --> 00:04:54,521
including a clause like that one.

102
00:04:54,981 --> 00:04:56,561
So what does a payment error at a bank

103
00:04:56,561 --> 00:04:58,201
have to do with AI agents?

104
00:04:58,361 --> 00:05:00,541
Well, the bank had this procedure, it had these controls,

105
00:05:01,281 --> 00:05:03,981
and everybody did what they thought the right thing to do was.

106
00:05:04,321 --> 00:05:06,601
That's the problem I want to talk about for the rest of this talk.

107
00:05:06,601 --> 00:05:11,621
Every day we're giving real agents real autonomy over systems like FlexCube.

108
00:05:11,781 --> 00:05:19,321
And how do we prevent accidents like that one from either from happening without locking everything down or just hoping we'll get lucky that they won't happen?

109
00:05:19,961 --> 00:05:21,321
So here's the crux of the issue.

110
00:05:21,401 --> 00:05:24,001
You can't really know in advance what an agent will do, right?

111
00:05:24,081 --> 00:05:25,721
There are these non-deterministic systems.

112
00:05:26,101 --> 00:05:31,541
And that sounds scary until you remember that humans are also non-deterministic systems and they're going to make mistakes too.

113
00:05:32,061 --> 00:05:37,261
And so most firms don't choose to say, well, in order to reduce the risk of humans making a mistake,

114
00:05:37,441 --> 00:05:39,941
we're going to lock them in rooms and prevent them from talking to each other,

115
00:05:39,961 --> 00:05:41,821
and they're all going to double-check everybody else's work.

116
00:05:41,921 --> 00:05:46,341
We allow them a certain amount of autonomy along with a certain amount of controls.

117
00:05:46,541 --> 00:05:51,201
So we give them boundaries, visibility, and we equip them with systems and controls

118
00:05:51,201 --> 00:05:52,721
that let them do their jobs effectively.

119
00:05:53,081 --> 00:05:57,701
So in this sort of model, safety and innovation are not opposite ends of a spectrum

120
00:05:57,701 --> 00:05:58,821
that you must trade off on.

121
00:05:58,921 --> 00:06:01,021
They're things that you can have at the same time.

122
00:06:01,021 --> 00:06:03,321
So I think you can have your cake and eat it too.

123
00:06:03,941 --> 00:06:06,741
And when we were thinking about what controls we wanted to apply to agents,

124
00:06:06,941 --> 00:06:09,261
we also looked at some approaches that didn't work for us.

125
00:06:09,561 --> 00:06:11,181
We don't want to sandbox or lock down everything,

126
00:06:11,381 --> 00:06:15,101
because a sandbox with no real access to systems is safe,

127
00:06:15,161 --> 00:06:16,581
but it's not very interesting or useful.

128
00:06:17,081 --> 00:06:19,481
If you let agents go wild and do whatever they want,

129
00:06:19,621 --> 00:06:23,521
that's maybe powerful and freeing from a cognitive perspective,

130
00:06:23,681 --> 00:06:26,441
but the isolation is very weak there,

131
00:06:26,641 --> 00:06:30,101
and that approach is generally not suitable for highly regulated enterprises.

132
00:06:30,101 --> 00:06:35,761
And then if you just have policy documents that agents look at or that are part of your

133
00:06:36,341 --> 00:06:40,201
specification for the prompt, that's also not very enforceable at runtime.

134
00:06:40,201 --> 00:06:43,641
You can't guarantee that that will be part of the enforcement controls.

135
00:06:43,861 --> 00:06:48,861
So what did work for us was thinking about what mix of controls would get us both safety

136
00:06:48,861 --> 00:06:50,661
and innovation at the same time.

137
00:06:50,801 --> 00:06:56,621
So how do we provide boundaries and stop problems but don't hinder folks from operating correctly?

138
00:06:56,821 --> 00:06:59,741
So we have a number of these controls, but today I'll talk about five of them.

139
00:07:00,101 --> 00:07:09,381
When I say controls here, what I mean are the guardrails and shared capabilities that let teams move in ways that don't hinder what they're trying to do.

140
00:07:09,581 --> 00:07:19,801
And we don't want to reach some chasm that we need to cross to let people get their work done and then make everyone wait while we build a bridge to get there.

141
00:07:20,421 --> 00:07:22,321
So these controls are composable.

142
00:07:22,601 --> 00:07:23,921
Each one does one job.

143
00:07:24,061 --> 00:07:29,241
And then together, they let us run agents with real autonomy at an appropriate level of safety.

144
00:07:31,081 --> 00:07:34,501
So the first problem is that agents were showing up everywhere.

145
00:07:34,741 --> 00:07:37,301
We have big teams with lots of agents running.

146
00:07:37,521 --> 00:07:38,641
We have small teams with a few.

147
00:07:38,801 --> 00:07:42,081
We have individual people running stuff in an uncontrolled way.

148
00:07:42,201 --> 00:07:43,461
And these come in every shape.

149
00:07:43,581 --> 00:07:44,741
Could be random chatbots.

150
00:07:44,861 --> 00:07:49,001
It could be MCP servers glued together in this Frankenstein hot

151
00:07:49,001 --> 00:07:49,421
podge.

152
00:07:49,421 --> 00:07:50,921
It could be cool notebook experiments.

153
00:07:51,021 --> 00:07:53,481
It could be really useful web tools for folks.

154
00:07:53,901 --> 00:07:57,821
But it's hard to answer when you look at that a few simple

155
00:07:57,821 --> 00:07:59,401
questions, like who owns this?

156
00:07:59,401 --> 00:08:03,261
what tools does it have access to, what should it be doing, what should it not be doing.

157
00:08:03,421 --> 00:08:06,001
And if you can't answer those kinds of questions, if you can't govern them,

158
00:08:06,541 --> 00:08:08,741
then you can't really also reuse that.

159
00:08:08,861 --> 00:08:13,261
So you're tempted to build multiple solutions to the same problem

160
00:08:13,261 --> 00:08:15,221
if you don't know about what exists already.

161
00:08:15,401 --> 00:08:19,141
And then the firm can't also govern those in any kind of legible way.

162
00:08:19,981 --> 00:08:23,181
So one thing the firms sometimes do here is add some kind of gate.

163
00:08:23,361 --> 00:08:26,901
Like don't let anything run until everything is registered and approved

164
00:08:26,901 --> 00:08:28,881
and has gone through this very onerous process.

165
00:08:29,401 --> 00:08:32,421
And that sounds safe, but if you look at what it does to the work,

166
00:08:32,621 --> 00:08:35,821
every idea, even some simple no-pick experiment,

167
00:08:36,001 --> 00:08:39,101
is going to wait in line behind something that's really risky.

168
00:08:39,241 --> 00:08:44,381
And we don't want to subject everybody who's given the diversity of agents

169
00:08:44,381 --> 00:08:45,761
to exactly the same process.

170
00:08:46,121 --> 00:08:47,521
So most people are going to give up,

171
00:08:47,601 --> 00:08:49,921
or they'll just get pushed to some shadow IT,

172
00:08:49,921 --> 00:08:53,221
less governed path that is less visible,

173
00:08:53,401 --> 00:08:54,881
and then we don't get to see anything at all.

174
00:08:55,581 --> 00:08:58,261
So that's what motivated the first control is an agent registry.

175
00:08:58,261 --> 00:09:02,801
You can't govern it, you can't see, so the agent registry makes agents legible, and it

176
00:09:02,801 --> 00:09:04,801
registers, retrieves, and discovers the key

177
00:09:05,021 --> 00:09:08,401
facts about agents that we want to know for any agent.

178
00:09:09,041 --> 00:09:15,061
So those are things like which tools it declares, what endpoints it's going to talk to, which product does it belong to, who owns it.

179
00:09:15,561 --> 00:09:28,121
And humans, other programs, other agents can then use the agent registry to decide what's available at the firm, whether that's useful for them, and to go use that as a source of truth for other questions.

180
00:09:28,121 --> 00:09:30,681
So this doesn't require extra effort on your part.

181
00:09:30,781 --> 00:09:34,261
It just happens when you use one of the firm's approved agent building frameworks.

182
00:09:34,421 --> 00:09:35,401
It's automatically registered.

183
00:09:35,521 --> 00:09:37,101
All that information is recorded there for you.

184
00:09:37,461 --> 00:09:40,321
And if you want to do a manual registration, you can certainly do that.

185
00:09:42,301 --> 00:09:47,121
So the second problem we have is that now that we've got all these agents doing all

186
00:09:47,121 --> 00:09:50,461
these things, we quickly notice that what they're doing is really different.

187
00:09:50,801 --> 00:09:54,581
And that means our agents carry really different risks, right?

188
00:09:54,581 --> 00:10:00,201
So a thing that summarizes Jira or linear tickets is really different than a production

189
00:10:00,201 --> 00:10:02,241
deployment agent that can touch a running system.

190
00:10:02,621 --> 00:10:08,401
And if we try to use the same review process to cover all of these agents, then every agent

191
00:10:08,401 --> 00:10:10,941
is waiting in the same queue for weeks for human review.

192
00:10:11,281 --> 00:10:16,061
And if people are going to route around slow processes, then we wind up in the same situation

193
00:10:16,061 --> 00:10:17,221
that we had before.

194
00:10:17,801 --> 00:10:21,321
So again, you might be tempted to say the answer is something like really detailed,

195
00:10:21,321 --> 00:10:26,461
heavy manual review by all of these different constituencies. Again, that feels safe, but now

196
00:10:26,461 --> 00:10:32,901
we've got a trivial summarizer waiting two months for the same sign-offs as an agent that is touching

197
00:10:32,901 --> 00:10:38,501
production systems. And most of the good agent ideas, most of the ones that have value right away,

198
00:10:38,641 --> 00:10:43,961
start out as small experiments, and you don't want those people to wait in the same line as the stuff

199
00:10:43,961 --> 00:10:48,601
that's touching production right away. So the reviewers are drowning in these low-risk requests,

200
00:10:48,601 --> 00:10:51,161
and the high-risk ones have to share that attention pool.

201
00:10:51,421 --> 00:10:53,941
So we don't want to trade away that speed if we can help it.

202
00:10:54,621 --> 00:10:57,321
So again, because not every agent needs the same review,

203
00:10:57,521 --> 00:11:01,081
and because most agents are purpose-built rather than open-ended,

204
00:11:01,181 --> 00:11:02,201
and when I say open-ended here,

205
00:11:02,221 --> 00:11:05,421
you can imagine coding agents like Pi or Claude Code.

206
00:11:05,521 --> 00:11:07,441
When you open them and you start typing,

207
00:11:08,201 --> 00:11:11,261
Claude Code or Pi doesn't know what you're about to ask.

208
00:11:11,381 --> 00:11:14,701
It doesn't circumscribe its purpose in any particular way.

209
00:11:14,881 --> 00:11:17,441
But most of the agents that are deployed in enterprises

210
00:11:17,441 --> 00:11:22,101
do have very specific purposes. When you're engaging with that agent, it's to do something

211
00:11:22,101 --> 00:11:26,981
specific that's well known and well understood. So they have a very specific reason for existing

212
00:11:26,981 --> 00:11:31,401
and a very circumscribed purpose. And you can tell these agents apart from each other by looking at

213
00:11:31,401 --> 00:11:36,601
what they declare up front, their tools, the endpoints they access, the controls that are

214
00:11:36,601 --> 00:11:44,901
applied to them, etc. So the third problem is that if you know about your agents and what they're

215
00:11:44,901 --> 00:11:48,121
doing, you'll quickly notice that many agents don't talk to each other.

216
00:11:48,401 --> 00:11:51,161
Sorry, they don't talk to just one thing or in just one capacity.

217
00:11:51,321 --> 00:11:56,501
The more open-ended the agent is, the more it looks like a Claude Code or Pi, the more

218
00:11:56,501 --> 00:12:00,001
it can do a variety and a diversity of different things.

219
00:12:00,161 --> 00:12:03,861
Call an internal API, run a tool like a web search, that sort of thing.

220
00:12:04,481 --> 00:12:08,301
And each one of those is a different opening, a different door to a different kind of risk.

221
00:12:08,701 --> 00:12:13,501
And if you give an agent an entitlement for a role account, then it can do a lot more

222
00:12:13,501 --> 00:12:17,161
damaged than something that's just going to read a public web page, for example.

223
00:12:17,421 --> 00:12:22,821
So the question is not, can this agent do X, but what's the granularity that we need

224
00:12:22,821 --> 00:12:25,341
to solve for for the agent to act successfully?

225
00:12:26,321 --> 00:12:30,941
And again, one tempting answer is to kind of be heavy-handed here, or to give every

226
00:12:30,941 --> 00:12:35,841
agent the same short allow list of stuff that's safe to do, and then block everything else.

227
00:12:36,401 --> 00:12:38,081
That's safe, but it's also useless.

228
00:12:38,281 --> 00:12:43,481
An agent that can't search, or can't call an API, or can't do any of those things without

229
00:12:43,501 --> 00:12:47,901
asking somebody first can't really do the work that it was intended for. And then

230
00:12:47,901 --> 00:12:51,441
teams either give up on those agents or they find a way around the wall and

231
00:12:51,441 --> 00:12:56,761
both outcomes are costly. So a wide kind of blanket block on everything trades

232
00:12:56,761 --> 00:13:00,521
away the innovation that we want to preserve and avoids making a real

233
00:13:00,521 --> 00:13:05,961
decision about risk. And as author John Shedd wrote, a ship and harbor is safe,

234
00:13:06,001 --> 00:13:11,081
but that is not what ships are for. So the fix here is to let agents declare

235
00:13:11,081 --> 00:13:16,921
their purpose upfront and say what tools they're going to use or what tools they

236
00:13:16,921 --> 00:13:22,761
need access to. So in the second column here we might say that a research

237
00:13:22,761 --> 00:13:29,581
summarizer is going to be allowed to use a web tool to access research, but for a

238
00:13:29,581 --> 00:13:33,261
release deployer agent we're not going to allow them to do that because they

239
00:13:33,261 --> 00:13:37,021
have no reason to use that tool. And so if they try to ask for it they won't get

240
00:13:37,021 --> 00:13:40,861
it by default. So we don't need to make this kind of decision for every tool.

241
00:13:40,861 --> 00:13:43,681
the agents just have to say which tools they want.

242
00:13:43,881 --> 00:13:47,101
And you're, as the agent author, you're in the best position to say,

243
00:13:47,261 --> 00:13:50,961
I need these four things out of the 500 that are available.

244
00:13:52,121 --> 00:13:56,881
So the way we need to enforce that is that when agents run in our managed environments,

245
00:13:57,161 --> 00:14:00,701
they look at the registry that we talked about before

246
00:14:00,701 --> 00:14:04,421
to look up what policies apply to it at initialization time.

247
00:14:04,561 --> 00:14:07,981
And that policy gets translated into enforceable statements,

248
00:14:07,981 --> 00:14:11,721
like which endpoints are you allowed to access, which hosts are you allowed to access,

249
00:14:12,401 --> 00:14:15,781
what other types of resources are available or not available to you.

250
00:14:15,941 --> 00:14:20,201
And in this way, nobody's writing a ticket every time they need to change the access.

251
00:14:20,461 --> 00:14:25,281
Nobody's going to talk to IT or some other group outside of your team.

252
00:14:25,361 --> 00:14:29,021
As long as you conform to the risk score that we talked about before

253
00:14:29,021 --> 00:14:32,821
and you're in the agent registry, you're governed, you're approved.

254
00:14:32,821 --> 00:14:39,041
So this is how we keep the cost to teams really low for deploying, building, running agents.

255
00:14:39,281 --> 00:14:41,881
And the same control architecture works in a variety of settings.

256
00:14:42,501 --> 00:14:47,421
The next thing we noticed was that teams were building a ton of different ways to evaluate agents,

257
00:14:47,541 --> 00:14:50,541
and every stack looks different, and some teams don't build anything at all.

258
00:14:50,841 --> 00:14:54,741
Sometimes you build this really detailed stack for how your models work.

259
00:14:54,981 --> 00:14:57,321
Sometimes there's nothing.

260
00:14:57,321 --> 00:15:03,081
And I think in general we'd be upset if teams were not writing unit tests, right?

261
00:15:03,141 --> 00:15:05,461
And we consider evals to be kind of the same thing.

262
00:15:05,641 --> 00:15:09,321
If models are changing, that's like a runtime dependency that your application has.

263
00:15:09,481 --> 00:15:19,481
And if your agent is going to change models, it doesn't know without the evals what's going to happen to its outputs and to the quality of those outputs.

264
00:15:19,481 --> 00:15:25,101
So, again, one tempting answer here is let's have some kind of central evaluation methodology

265
00:15:25,101 --> 00:15:29,041
that waits for a full eval suite and a committee sign-off.

266
00:15:29,221 --> 00:15:34,281
But now every prompt tweak, every one-line change is going to sit in that same queue.

267
00:15:34,421 --> 00:15:38,741
So we don't want to put more decisions on more reviewers for things like that.

268
00:15:39,221 --> 00:15:40,901
We don't want to trade away that iteration.

269
00:15:41,201 --> 00:15:45,341
So again, you wouldn't ship software without unit tests, so agents need evaluations for

270
00:15:45,341 --> 00:15:51,281
the same reason, and we want to provide the stuff at the level of the firm that's going

271
00:15:51,281 --> 00:15:52,441
to be the same for everybody.

272
00:15:52,641 --> 00:15:55,321
So Team A writing evals, Team B writing evals.

273
00:15:55,381 --> 00:15:57,561
They're both going to have many of the same features.

274
00:15:57,901 --> 00:16:02,301
So we want to have one shared approach to how we do evals, and have the shared approach

275
00:16:02,301 --> 00:16:06,561
be something that's run by the firm and available to everybody for doing their evals.

276
00:16:06,661 --> 00:16:11,201
So that covers all the common parts of evals, and all you've got to do is just write what's

277
00:16:11,201 --> 00:16:12,041
true about your agent.

278
00:16:14,481 --> 00:16:18,661
And finally, maybe you've had the experience of being asked to approve lots and lots of

279
00:16:18,661 --> 00:16:21,441
individual agent actions when there's a human in the loop.

280
00:16:21,621 --> 00:16:25,921
So that's fine on a small scale, but once you're operating agents in parallel or in

281
00:16:25,921 --> 00:16:31,341
production, this becomes completely onerous and review requests come in faster than anybody

282
00:16:31,341 --> 00:16:33,561
can be expected to approve them.

283
00:16:33,981 --> 00:16:37,501
After the first few times this is happening, you succumb to review fatigue where you just

284
00:16:37,501 --> 00:16:38,601
see yes, yes, yes, yes, yes.

285
00:16:38,601 --> 00:16:43,321
and eventually you click yes on something that's activating a nuclear bomb.

286
00:16:43,441 --> 00:16:47,241
So you don't want to click yes by reflex, but that's going to happen inevitably.

287
00:16:48,641 --> 00:16:52,801
One recent example of this is earlier this year, a director at an AI research lab

288
00:16:52,801 --> 00:16:57,741
told her open claw agent to suggest emails to delete from her inbox and wait for her approval.

289
00:16:58,001 --> 00:17:01,681
But her inbox was so large that eventually the agent compacted its context

290
00:17:01,681 --> 00:17:04,141
and lost the part that said, wait for my approval.

291
00:17:04,481 --> 00:17:08,201
And then it deleted hundreds of emails and didn't stop when she told it to stop on her phone.

292
00:17:09,041 --> 00:17:12,821
So again, you might be tempted here to make a human approve every action, but that's not

293
00:17:12,821 --> 00:17:13,781
the right way to do it.

294
00:17:14,041 --> 00:17:20,701
A better approach is to let these independent, conservative classifiers make decisions about

295
00:17:20,701 --> 00:17:25,521
routine calls, and then send a person only the requests that are most important, the

296
00:17:25,521 --> 00:17:27,281
ones that are most likely to make a difference.

297
00:17:27,521 --> 00:17:33,021
If you're saying yes, 99.99999% of the time, then that's probably just a rule that the

298
00:17:33,021 --> 00:17:36,701
agent should follow, not a request you need to ask the human for permission for.

299
00:17:36,701 --> 00:17:41,741
and the stuff that's unclear or ambiguous or potentially dangerous, yes, surface that to a human,

300
00:17:41,901 --> 00:17:48,381
or if it's definitely just dangerous, deny it outright. So this is a way to not have to do all

301
00:17:48,381 --> 00:17:53,341
the review work all the time. So we want to take the same approach for both explicit,

302
00:17:53,661 --> 00:17:59,301
when explicit rules apply, we want deterministic fast paths. When explicit rules don't apply and

303
00:17:59,301 --> 00:18:04,381
there's some ambiguity, we want a classifier to do that work instead. So putting those five pieces

304
00:18:04,381 --> 00:18:08,721
back together. You create a registry so that every agent is visible from the day it's created.

305
00:18:09,161 --> 00:18:14,481
You score each agent so human review goes in the right spot. You run agents on guardrails so that

306
00:18:14,481 --> 00:18:19,541
they only touch what they need. You share evals and metrics so everybody is reaching the same bar

307
00:18:19,541 --> 00:18:24,001
without having to reinvent the wheel. And then finally you govern your runtime to make sure that

308
00:18:24,001 --> 00:18:28,921
classification happens there and not somewhere else. So each of these controls works together.

309
00:18:29,721 --> 00:18:31,221
So I'll leave you with one question.

310
00:18:31,841 --> 00:18:32,861
Why do cars have brakes?

311
00:18:33,861 --> 00:18:37,821
It's not just so that you can stop, it's so that you can go fast and stop safely.

312
00:18:38,361 --> 00:18:40,101
So controls for our agents work the same way.

313
00:18:40,821 --> 00:18:41,741
Thank you very much.

314
00:18:42,081 --> 00:18:44,261
I'm John Feminella, and I hope you had a good time.