Great Agents Start with Great Infrastructure

Great! Sorry, I'm good. Hey everyone, I'm Akshat, CTO of Modal, and I'm going to talk

about really need from their infrastructure.

I'm told this audience is very finance adjacent, so I thought I'd talk about what

if every cell in a spreadsheet could be backed by an agent?

And imagine you're a financial analyst at a fund.

you want to run a complex analysis on every company in your portfolio.

Normally in the past, this would take days.

You'd have to connect every portfolio company,

look at their data sources, run some Python for each of them.

now obviously with agents, this should be much easier.

Well, actually, one of our customers, Hebbia,

built something that's exactly this.

you can go in right now in their product,

you have a spreadsheet,

and you can run a query

where literally every cell is

backed by a different agent.

And I wanted to talk about what does building this really look like.

Well, first, every agent is backed by a modal sandbox.

Sandbox is a secure environment for running untrusted code.

It used to be that every sandbox was one-to-one with a container or micro VM.

That's actually no longer the case.

Today actually, modal has this feature where you can run multiple containers or micro VMs

inside a sandbox.

You actually need to do that for doing something like this, where in this architecture you

run the harness.

The harness is the agent loop that does things like running the, calling the model API for

inference, running the tool calls, and then looping over it.

want to separate that from the place where the tool calls actually run for security and

isolation and a bunch of other things.

So you have the sandbox that's running multiple containers, including a sidecar.

You also want shared storage, because you want your agents to be able to talk to each

other.

And it's up to you how you stripe it.

Like maybe you have every row gets the same volume, and agents in that row in the spreadsheet

can collaborate.

And you also want to share storage between your main container and the sidecar for some

cases.

And finally, you need all of this to spin up really, really fast, because your spreadsheet

can have lots of cells.

If someone wants to query, you want to launch thousands of agents, and then you want to

spin them back down when it's over.

So before I talk more about that, I just wanted to take a quick aside to talk about what is

modal.

Modal is a platform of primitives that is built for AI applications.

We have built this platform from scratch, built our own file systems, scheduling and

the runtime to give you great performance and devx for use cases like inference, including

LLM inference, sandboxes, training and functions, running a lot of batch processing and what

not.

Today in particular I'm going to talk about sandboxes, what this primitive is and how

it's evolving.

So if you want to build something like this, or even further, and we'll talk more about

what that means, what are the primitives you need for that?

The first thing is, if you're building an agent, you need a workspace.

And I have some examples from some of our current customers.

You might have heard of Ramp Inspect.

Ramp Inspect is Ramp's internal coding agent that I think at this point pushes almost every

single PR at Ramp.

If you're building a coding agent, you need your repo, your GitHub repo, your dependencies,

you need your dev services, all in the same workspace for each agent.

Another example is basis.

it looks like is your documents might arrive, your agent is then triggered to work

on them, but then it's waiting for a response from the accountant.

You don't know how long this could take.

It could take many days.

When you want to persist the state over time, your usage is going to change.

In the second chart, you can see usage for each sandbox.

There's definitely some consistent trends, but it's all over the place.

If you were provisioning this, it doesn't make sense.

and then reliably get them when the sandbox needs them

and only pay for them for that duration.

The other interesting emerging thing is

agents need a lot more cloud science

where pre-processed some data on CPUs

or spin-up GPUs for running pre-processed some data on CPUs

or spin-up GPUs for running a scientific model,

it will offload that to a modal sandbox.

is a scientific model,

it will offload that to a modal sandbox.

And we're seeing this show up in RSI

about what happens when you give an agent

sandboxes to Hillclient.

The next emerging thing is

agents are collaborating a lot more,

and that means that you share a file system

with your sub-agent,

you share file system.

The next emerging thing is agents are collaborating a lot more, and that means that you share

a file and files with your human where you're all collaborating with the same thing.

This means file systems that are fast, have consistency, and devx you can reason about.

Something we're working on with modal volumes.

The related thing is we're seeing a lot more self-improvement.

will save and update memory, they'll run background tasks, which is now called

dreaming, to update this memory over time.

Actually this also just looks like files in a file system.

But again, you want really great performance from this file system.

and the scary thing about agents is they can mess up and they don't really know when they might

leak your sensitive data to a german forum or something so you need to have a lot of control

over what they can and cannot do and you need things like controlling what ips can even talk

to the agent, how do you authenticate it, what domains can it talk to, and also if you're

using credentials for talking to services, basically you want to inject them in a way

that the agent cannot actually see them and then exfiltrate them somewhere later.

And the interesting evolution of this is we've realized that domain-related allow lists are

actually not enough.

a lot of our users actually want very specific control

over even like what paths can I talk to

on let's say Datadog or GitHub.

Like maybe I want to reason about specific operations

that the agent is doing.

And going back to sidecars,

we've seen this emerging as a pattern

where users actually write their own lightweight proxy

that has custom rules about what exactly you can do.

And Ramp's another example of a company

that's been using this successfully.

And the final underrated thing is it's really important to have really great observability.

Going back to the example of auto research and an agent doing a lot of training, what

happens when the training run fails?

You want your agent to be able to easily look at the logs of the failure, reason about what

happened and do another iteration with a different set of parameters.

While training is running, you want to look at the metrics, GPU utilization and make sure

it's not stalled.

Maybe you want to look at the budget and make sure you're not spending too much on your

sandbox.

And obviously all of this is really important for humans, but as agents are driving a lot

more compute, we're spending a lot of time exposing all of this to the agents themselves

so they can reason about all of this.

All right.

I think we know what we need.

Why don't we have Astra build us an agent platform?

Well, I don't think it's a good idea.

I don't think you should vibe code a sandbox platform.

And there's three main reasons for that.

Number one is reliability.

I think we're all painfully aware how hard it is to get right.

Certain websites we use every day keep going down.

and the reason for that is even with AI,

it's actually really hard to reason about the feedback loop

of what happens in the case of this tail event

when the system breaks

and this is why you need to be very careful in building systems.

Security is another one.

You could vibe code a sandbox platform

but who is going to do all the work

to continually reason about the security boundaries

and make sure it's defensible

in an age when everything is evolving very fast.

And finally, scale.

We have a blog post about how we scale to a million sandboxes,

which involves a lot of very interesting engineering design

and reasoning about systems.

And this is, again, something that you kind of don't want to leave up

to a Vibe-coded platform,

because when you get to scale, you want it to work.

And on that point, we're actually seeing massive scale

for a lot of things I mentioned.

So, for example, for the snapshots I mentioned,

it's growing quite fast.

We're seeing 20 million snapshots a week.

We're processing about eight petabytes of data a day

for each of these snapshots.

And our users are creating almost 2.6 million volumes a week,

and all of this is growing super fast.

And just to take a customer workload for personal assistant,

You need to create 10,000 sandboxes a second.

You need to run almost a million sandboxes.

And they're not going to last very long, they'll just last for like two or three minutes.

But when they come up,

you want them to be interactive in less than a second. That's what this workload really demands.

And the other interesting thing on scale is you actually cannot get all of the scale from a single cloud platform. The world is actually getting crunched on CPUs as well, not just GPUs.

So one thing we spend a lot of time doing is actually aggregating cloud capacity from

a bunch of hyperscalers and neoclouds.

So our users have access to CPU capacity across the world when they need it, but we cannot

rely on one cloud provider to do that.

Okay, so where is all this going?

I hinted at this earlier, but the harnesses are really good at making use of dynamic environments.

and what this is leading to is a lot of separation of the brains,

which is the harness, from the hands,

which is the sandbox or the runtime where stuff is running.

And what this means, I have an example here,

is the harnesses are going to be able to spin up environments

based on what they need right now.

They're good at snapshotting them and keeping them around

as artifacts that they need later.

And again, adjusting all of this as necessary.

So I think this points at an interesting evolution

of the pets to cattle analogy we're all familiar with.

To recap, in the olden days,

people thought of services like pets.

Each one was bespoke.

You had to manage it.

You had to reason about this individual deployment specifically.

And then over time, with containerization and Kubernetes, people started to think about

it as homogenous templates.

These servers are all the same, you can scale them up and down.

If one of them fails, it's not an issue because our system can heal from that.

And I think this is where a lot of people deploying agents are today.

They're kind of using these homogenous environments and scaling up these agents.

But I think that's really limiting because the models are so good at reasoning about

dynamic environments, snapshotting, creating lots of environments when needed and working

across them.

So what does the new world look like?

analogy here is it kind of looks like an ocean of microbes, of all these organisms

that are self-evolving and working with each other.

credits this fix for the analogy here.

but one thing I'm really excited about

is as you give agents these really rock-solid primitives

for compute and storage and networking,

they will be able to actually do a lot more

by doing the stuff I talked about,

like spinning up lots of environments,

working with each other, and being very dynamic.

So yeah, I and us at Modal really think

that great primitives let your agents do more,

and yeah if you're interested in talking about this more

reach out to us

thank you