1
00:00:01,281 --> 00:00:09,881
Great! Sorry, I'm good. Hey everyone, I'm Akshat, CTO of Modal, and I'm going to talk

2
00:00:09,881 --> 00:00:12,301
about really need from their infrastructure.

3
00:00:15,001 --> 00:00:23,221
I'm told this audience is very finance adjacent, so I thought I'd talk about what

4
00:00:23,221 --> 00:00:27,481
if every cell in a spreadsheet could be backed by an agent?

5
00:00:29,121 --> 00:00:33,961
And imagine you're a financial analyst at a fund.

6
00:00:33,961 --> 00:00:40,401
you want to run a complex analysis on every company in your portfolio.

7
00:00:41,461 --> 00:00:43,681
Normally in the past, this would take days.

8
00:00:44,081 --> 00:00:47,421
You'd have to connect every portfolio company,

9
00:00:47,641 --> 00:00:51,761
look at their data sources, run some Python for each of them.

10
00:00:52,901 --> 00:00:54,961
now obviously with agents, this should be much easier.

11
00:00:56,101 --> 00:00:58,801
Well, actually, one of our customers, Hebbia,

12
00:00:59,141 --> 00:01:01,181
built something that's exactly this.

13
00:01:01,181 --> 00:01:04,121
you can go in right now in their product,

14
00:01:04,841 --> 00:01:07,241
you have a spreadsheet,

15
00:01:07,481 --> 00:01:10,201
and you can run a query

16
00:01:10,201 --> 00:01:13,001
where literally every cell is

17
00:01:13,021 --> 00:01:14,941
backed by a different agent.

18
00:01:16,281 --> 00:01:22,081
And I wanted to talk about what does building this really look like.

19
00:01:23,901 --> 00:01:29,641
Well, first, every agent is backed by a modal sandbox.

20
00:01:30,781 --> 00:01:36,781
Sandbox is a secure environment for running untrusted code.

21
00:01:38,921 --> 00:01:44,741
It used to be that every sandbox was one-to-one with a container or micro VM.

22
00:01:45,721 --> 00:01:47,001
That's actually no longer the case.

23
00:01:47,901 --> 00:01:54,901
Today actually, modal has this feature where you can run multiple containers or micro VMs

24
00:01:54,901 --> 00:01:55,681
inside a sandbox.

25
00:01:55,681 --> 00:02:02,581
You actually need to do that for doing something like this, where in this architecture you

26
00:02:02,581 --> 00:02:03,941
run the harness.

27
00:02:04,341 --> 00:02:10,921
The harness is the agent loop that does things like running the, calling the model API for

28
00:02:10,921 --> 00:02:14,181
inference, running the tool calls, and then looping over it.

29
00:02:15,521 --> 00:02:21,181
want to separate that from the place where the tool calls actually run for security and

30
00:02:21,181 --> 00:02:22,961
isolation and a bunch of other things.

31
00:02:23,801 --> 00:02:28,541
So you have the sandbox that's running multiple containers, including a sidecar.

32
00:02:29,421 --> 00:02:34,121
You also want shared storage, because you want your agents to be able to talk to each

33
00:02:34,121 --> 00:02:34,341
other.

34
00:02:34,621 --> 00:02:37,141
And it's up to you how you stripe it.

35
00:02:37,281 --> 00:02:43,901
Like maybe you have every row gets the same volume, and agents in that row in the spreadsheet

36
00:02:43,901 --> 00:02:44,501
can collaborate.

37
00:02:45,881 --> 00:02:50,341
And you also want to share storage between your main container and the sidecar for some

38
00:02:50,341 --> 00:02:50,641
cases.

39
00:02:51,581 --> 00:02:56,461
And finally, you need all of this to spin up really, really fast, because your spreadsheet

40
00:02:56,461 --> 00:02:57,421
can have lots of cells.

41
00:02:58,121 --> 00:03:01,841
If someone wants to query, you want to launch thousands of agents, and then you want to

42
00:03:01,841 --> 00:03:03,161
spin them back down when it's over.

43
00:03:05,141 --> 00:03:11,401
So before I talk more about that, I just wanted to take a quick aside to talk about what is

44
00:03:11,401 --> 00:03:11,641
modal.

45
00:03:12,821 --> 00:03:18,581
Modal is a platform of primitives that is built for AI applications.

46
00:03:18,581 --> 00:03:25,281
We have built this platform from scratch, built our own file systems, scheduling and

47
00:03:25,281 --> 00:03:31,641
the runtime to give you great performance and devx for use cases like inference, including

48
00:03:31,641 --> 00:03:38,161
LLM inference, sandboxes, training and functions, running a lot of batch processing and what

49
00:03:38,161 --> 00:03:38,321
not.

50
00:03:39,221 --> 00:03:43,661
Today in particular I'm going to talk about sandboxes, what this primitive is and how

51
00:03:43,661 --> 00:03:44,101
it's evolving.

52
00:03:44,101 --> 00:03:49,201
So if you want to build something like this, or even further, and we'll talk more about

53
00:03:49,201 --> 00:03:52,101
what that means, what are the primitives you need for that?

54
00:03:53,581 --> 00:03:56,721
The first thing is, if you're building an agent, you need a workspace.

55
00:03:57,201 --> 00:04:00,441
And I have some examples from some of our current customers.

56
00:04:01,101 --> 00:04:02,561
You might have heard of Ramp Inspect.

57
00:04:03,161 --> 00:04:10,541
Ramp Inspect is Ramp's internal coding agent that I think at this point pushes almost every

58
00:04:10,541 --> 00:04:12,061
single PR at Ramp.

59
00:04:12,061 --> 00:04:19,981
If you're building a coding agent, you need your repo, your GitHub repo, your dependencies,

60
00:04:20,761 --> 00:04:26,621
you need your dev services, all in the same workspace for each agent.

61
00:04:27,881 --> 00:04:29,121
Another example is basis.

62
00:04:31,621 --> 00:04:37,341
it looks like is your documents might arrive, your agent is then triggered to work

63
00:04:37,341 --> 00:04:40,901
on them, but then it's waiting for a response from the accountant.

64
00:04:41,661 --> 00:04:43,181
You don't know how long this could take.

65
00:04:43,341 --> 00:04:44,461
It could take many days.

66
00:04:45,341 --> 00:04:50,961
When you want to persist the state over time, your usage is going to change.

67
00:04:51,881 --> 00:04:55,001
In the second chart, you can see usage for each sandbox.

68
00:04:56,441 --> 00:04:59,661
There's definitely some consistent trends, but it's all over the place.

69
00:05:00,861 --> 00:05:04,141
If you were provisioning this, it doesn't make sense.

70
00:05:04,141 --> 00:05:07,101
and then reliably get them when the sandbox needs them

71
00:05:07,101 --> 00:05:09,741
and only pay for them for that duration.

72
00:05:11,661 --> 00:05:14,521
The other interesting emerging thing is

73
00:05:14,521 --> 00:05:17,681
agents need a lot more cloud science

74
00:05:17,681 --> 00:05:21,561
where pre-processed some data on CPUs

75
00:05:21,561 --> 00:05:26,881
or spin-up GPUs for running pre-processed some data on CPUs

76
00:05:26,881 --> 00:05:30,381
or spin-up GPUs for running a scientific model,

77
00:05:30,861 --> 00:05:32,861
it will offload that to a modal sandbox.

78
00:05:32,861 --> 00:05:34,921
is a scientific model,

79
00:05:35,221 --> 00:05:37,241
it will offload that to a modal sandbox.

80
00:05:37,961 --> 00:05:39,781
And we're seeing this show up in RSI

81
00:05:39,781 --> 00:05:42,301
about what happens when you give an agent

82
00:05:42,981 --> 00:05:44,261
sandboxes to Hillclient.

83
00:05:47,241 --> 00:05:49,241
The next emerging thing is

84
00:05:49,241 --> 00:05:51,501
agents are collaborating a lot more,

85
00:05:52,341 --> 00:05:56,661
and that means that you share a file system

86
00:05:56,661 --> 00:05:58,181
with your sub-agent,

87
00:05:58,381 --> 00:05:59,821
you share file system.

88
00:06:00,821 --> 00:06:09,341
The next emerging thing is agents are collaborating a lot more, and that means that you share

89
00:06:09,341 --> 00:06:15,161
a file and files with your human where you're all collaborating with the same thing.

90
00:06:16,021 --> 00:06:22,461
This means file systems that are fast, have consistency, and devx you can reason about.

91
00:06:23,161 --> 00:06:24,741
Something we're working on with modal volumes.

92
00:06:26,041 --> 00:06:29,641
The related thing is we're seeing a lot more self-improvement.

93
00:06:30,821 --> 00:06:36,461
will save and update memory, they'll run background tasks, which is now called

94
00:06:36,461 --> 00:06:38,701
dreaming, to update this memory over time.

95
00:06:40,001 --> 00:06:42,921
Actually this also just looks like files in a file system.

96
00:06:44,161 --> 00:06:47,541
But again, you want really great performance from this file system.

97
00:06:51,981 --> 00:07:01,021
and the scary thing about agents is they can mess up and they don't really know when they might

98
00:07:01,021 --> 00:07:07,741
leak your sensitive data to a german forum or something so you need to have a lot of control

99
00:07:07,741 --> 00:07:14,441
over what they can and cannot do and you need things like controlling what ips can even talk

100
00:07:14,441 --> 00:07:21,321
to the agent, how do you authenticate it, what domains can it talk to, and also if you're

101
00:07:21,321 --> 00:07:27,221
using credentials for talking to services, basically you want to inject them in a way

102
00:07:27,221 --> 00:07:30,501
that the agent cannot actually see them and then exfiltrate them somewhere later.

103
00:07:33,501 --> 00:07:40,141
And the interesting evolution of this is we've realized that domain-related allow lists are

104
00:07:40,141 --> 00:07:41,681
actually not enough.

105
00:07:41,681 --> 00:07:45,761
a lot of our users actually want very specific control

106
00:07:45,761 --> 00:07:49,461
over even like what paths can I talk to

107
00:07:49,461 --> 00:07:51,521
on let's say Datadog or GitHub.

108
00:07:51,841 --> 00:07:55,301
Like maybe I want to reason about specific operations

109
00:07:55,301 --> 00:07:56,181
that the agent is doing.

110
00:07:56,861 --> 00:07:58,721
And going back to sidecars,

111
00:07:58,821 --> 00:08:00,261
we've seen this emerging as a pattern

112
00:08:00,261 --> 00:08:04,081
where users actually write their own lightweight proxy

113
00:08:04,081 --> 00:08:06,621
that has custom rules about what exactly you can do.

114
00:08:06,881 --> 00:08:10,521
And Ramp's another example of a company

115
00:08:10,521 --> 00:08:11,721
that's been using this successfully.

116
00:08:15,201 --> 00:08:21,101
And the final underrated thing is it's really important to have really great observability.

117
00:08:22,001 --> 00:08:27,101
Going back to the example of auto research and an agent doing a lot of training, what

118
00:08:27,101 --> 00:08:28,361
happens when the training run fails?

119
00:08:29,321 --> 00:08:34,741
You want your agent to be able to easily look at the logs of the failure, reason about what

120
00:08:34,741 --> 00:08:38,881
happened and do another iteration with a different set of parameters.

121
00:08:40,261 --> 00:08:46,361
While training is running, you want to look at the metrics, GPU utilization and make sure

122
00:08:46,361 --> 00:08:47,141
it's not stalled.

123
00:08:47,941 --> 00:08:51,941
Maybe you want to look at the budget and make sure you're not spending too much on your

124
00:08:51,941 --> 00:08:52,261
sandbox.

125
00:08:53,741 --> 00:08:58,801
And obviously all of this is really important for humans, but as agents are driving a lot

126
00:08:58,801 --> 00:09:03,761
more compute, we're spending a lot of time exposing all of this to the agents themselves

127
00:09:03,761 --> 00:09:05,121
so they can reason about all of this.

128
00:09:07,101 --> 00:09:07,681
All right.

129
00:09:08,281 --> 00:09:09,781
I think we know what we need.

130
00:09:10,221 --> 00:09:13,221
Why don't we have Astra build us an agent platform?

131
00:09:14,421 --> 00:09:16,141
Well, I don't think it's a good idea.

132
00:09:16,521 --> 00:09:18,861
I don't think you should vibe code a sandbox platform.

133
00:09:19,381 --> 00:09:21,301
And there's three main reasons for that.

134
00:09:21,761 --> 00:09:23,021
Number one is reliability.

135
00:09:23,721 --> 00:09:27,901
I think we're all painfully aware how hard it is to get right.

136
00:09:29,181 --> 00:09:32,061
Certain websites we use every day keep going down.

137
00:09:32,941 --> 00:09:36,161
and the reason for that is even with AI,

138
00:09:36,501 --> 00:09:39,801
it's actually really hard to reason about the feedback loop

139
00:09:39,801 --> 00:09:43,241
of what happens in the case of this tail event

140
00:09:43,241 --> 00:09:44,281
when the system breaks

141
00:09:44,281 --> 00:09:47,581
and this is why you need to be very careful in building systems.

142
00:09:48,301 --> 00:09:49,381
Security is another one.

143
00:09:49,761 --> 00:09:51,941
You could vibe code a sandbox platform

144
00:09:51,941 --> 00:09:53,601
but who is going to do all the work

145
00:09:53,601 --> 00:09:56,881
to continually reason about the security boundaries

146
00:09:56,881 --> 00:10:00,181
and make sure it's defensible

147
00:10:00,181 --> 00:10:04,081
in an age when everything is evolving very fast.

148
00:10:05,181 --> 00:10:06,301
And finally, scale.

149
00:10:06,721 --> 00:10:10,101
We have a blog post about how we scale to a million sandboxes,

150
00:10:10,361 --> 00:10:13,861
which involves a lot of very interesting engineering design

151
00:10:13,861 --> 00:10:15,081
and reasoning about systems.

152
00:10:15,461 --> 00:10:20,361
And this is, again, something that you kind of don't want to leave up

153
00:10:20,361 --> 00:10:21,701
to a Vibe-coded platform,

154
00:10:21,881 --> 00:10:23,861
because when you get to scale, you want it to work.

155
00:10:25,641 --> 00:10:29,081
And on that point, we're actually seeing massive scale

156
00:10:29,081 --> 00:10:30,481
for a lot of things I mentioned.

157
00:10:31,021 --> 00:10:34,001
So, for example, for the snapshots I mentioned,

158
00:10:34,901 --> 00:10:36,141
it's growing quite fast.

159
00:10:36,281 --> 00:10:38,321
We're seeing 20 million snapshots a week.

160
00:10:39,001 --> 00:10:43,581
We're processing about eight petabytes of data a day

161
00:10:43,581 --> 00:10:45,821
for each of these snapshots.

162
00:10:46,541 --> 00:10:51,661
And our users are creating almost 2.6 million volumes a week,

163
00:10:51,821 --> 00:10:53,561
and all of this is growing super fast.

164
00:10:55,141 --> 00:10:58,701
And just to take a customer workload for personal assistant,

165
00:11:00,121 --> 00:11:03,041
You need to create 10,000 sandboxes a second.

166
00:11:03,541 --> 00:11:06,841
You need to run almost a million sandboxes.

167
00:11:07,301 --> 00:11:11,241
And they're not going to last very long, they'll just last for like two or three minutes.

168
00:11:11,861 --> 00:11:13,001
But when they come up,

169
00:11:13,181 --> 00:11:18,581
you want them to be interactive in less than a second. That's what this workload really demands.

170
00:11:21,701 --> 00:11:32,401
And the other interesting thing on scale is you actually cannot get all of the scale from a single cloud platform. The world is actually getting crunched on CPUs as well, not just GPUs.

171
00:11:32,401 --> 00:11:37,621
So one thing we spend a lot of time doing is actually aggregating cloud capacity from

172
00:11:37,621 --> 00:11:39,581
a bunch of hyperscalers and neoclouds.

173
00:11:40,061 --> 00:11:45,581
So our users have access to CPU capacity across the world when they need it, but we cannot

174
00:11:45,581 --> 00:11:47,481
rely on one cloud provider to do that.

175
00:11:49,201 --> 00:11:51,161
Okay, so where is all this going?

176
00:11:51,461 --> 00:11:57,841
I hinted at this earlier, but the harnesses are really good at making use of dynamic environments.

177
00:11:59,501 --> 00:12:03,821
and what this is leading to is a lot of separation of the brains,

178
00:12:04,081 --> 00:12:06,741
which is the harness, from the hands,

179
00:12:06,961 --> 00:12:09,441
which is the sandbox or the runtime where stuff is running.

180
00:12:11,361 --> 00:12:14,001
And what this means, I have an example here,

181
00:12:14,301 --> 00:12:18,661
is the harnesses are going to be able to spin up environments

182
00:12:18,661 --> 00:12:20,501
based on what they need right now.

183
00:12:21,661 --> 00:12:24,321
They're good at snapshotting them and keeping them around

184
00:12:25,601 --> 00:12:27,421
as artifacts that they need later.

185
00:12:27,841 --> 00:12:30,401
And again, adjusting all of this as necessary.

186
00:12:31,661 --> 00:12:35,181
So I think this points at an interesting evolution

187
00:12:35,181 --> 00:12:40,841
of the pets to cattle analogy we're all familiar with.

188
00:12:41,261 --> 00:12:44,281
To recap, in the olden days,

189
00:12:44,541 --> 00:12:46,741
people thought of services like pets.

190
00:12:47,141 --> 00:12:48,381
Each one was bespoke.

191
00:12:48,561 --> 00:12:49,601
You had to manage it.

192
00:12:49,921 --> 00:12:53,921
You had to reason about this individual deployment specifically.

193
00:12:54,821 --> 00:13:01,841
And then over time, with containerization and Kubernetes, people started to think about

194
00:13:01,841 --> 00:13:05,221
it as homogenous templates.

195
00:13:06,181 --> 00:13:08,921
These servers are all the same, you can scale them up and down.

196
00:13:09,201 --> 00:13:13,221
If one of them fails, it's not an issue because our system can heal from that.

197
00:13:13,861 --> 00:13:17,981
And I think this is where a lot of people deploying agents are today.

198
00:13:18,501 --> 00:13:21,621
They're kind of using these homogenous environments and scaling up these agents.

199
00:13:21,621 --> 00:13:27,441
But I think that's really limiting because the models are so good at reasoning about

200
00:13:27,441 --> 00:13:32,961
dynamic environments, snapshotting, creating lots of environments when needed and working

201
00:13:32,961 --> 00:13:33,581
across them.

202
00:13:33,941 --> 00:13:36,001
So what does the new world look like?

203
00:13:38,101 --> 00:13:43,821
analogy here is it kind of looks like an ocean of microbes, of all these organisms

204
00:13:43,821 --> 00:13:46,701
that are self-evolving and working with each other.

205
00:13:47,801 --> 00:13:49,821
credits this fix for the analogy here.

206
00:13:50,841 --> 00:13:53,921
but one thing I'm really excited about

207
00:13:53,921 --> 00:13:58,261
is as you give agents these really rock-solid primitives

208
00:13:58,261 --> 00:14:01,101
for compute and storage and networking,

209
00:14:01,441 --> 00:14:03,981
they will be able to actually do a lot more

210
00:14:03,981 --> 00:14:05,881
by doing the stuff I talked about,

211
00:14:06,101 --> 00:14:07,581
like spinning up lots of environments,

212
00:14:07,901 --> 00:14:10,521
working with each other, and being very dynamic.

213
00:14:11,801 --> 00:14:15,241
So yeah, I and us at Modal really think

214
00:14:15,241 --> 00:14:18,121
that great primitives let your agents do more,

215
00:14:19,401 --> 00:14:21,641
and yeah if you're interested in talking about this more

216
00:14:21,641 --> 00:14:22,441
reach out to us

217
00:14:22,441 --> 00:14:23,461
thank you