Frontier Models for Acting in Space and Time

Hello.

I want you to zoom out for a second and think about if somebody were to tell you today that

we would take all the information on the internet and then we would train a model on it and

we would use that in order to solve scientific problems that were not on the internet.

I think a lot of people would tell you that that's crazy.

And what we're doing has a lot of parallels to that story.

So if you think about how intelligence works, imagine for a second that you're in the movie

Frozen and you're Olaf, the snowman.

If Olaf was an LLM, and there's a scene which is one of my favorites, which is where he rolls

down a mountain and there's a little stone that pops up and Olaf spreads out, temporarily

becomes a disassembled snowman and then becomes an assembled snowman.

Olaf knows how to do that because Olaf can observe as he is acting.

Olaf can, unlike

LLMs, which autoregressively predict tokens, constantly part of that behavior is that it's constantly observing as it is acting.

And so this is the big difference between, you know, when you see LLMs try to solve robotics problems or when you look at native architectures like we're working on, it's very, very easy to think of LLMs in the demos to see on Twitter as the thing that will

Solve everything, there are fundamental limitations where purpose-built models are a much better

fit.

Today I'm going to talk a little bit about what the other side of that coin looks like.

For us, it all started with Medal.

We built the largest video game recorder in the world, and we think of that as the internet.

People use Medal to record themselves playing tens of thousands of different video games,

And if you think about it, as a human, we can teleoperate a robot with the exact same

brain that we played at video game, and in fact, in many cases, we also can use the same

controller.

So what this means is that the action space of video games, often because of game controllers,

and the action space of teleoperations or the real world, in many cases is the same.

And so every single robot, below certain degrees of freedom, ships with the game controller.

The majority of the supply chains already work this way.

And so we thought, what if you just take all of the action data that we get through the

video recorder, which associates the inputs and the actions with the actual frames, we

label the actions so we understand semantic meaning behind every action, and we use it

to give it to a vision-based model.

What you see here on the left is just an intuitive way of how the model works.

You give the model a set of keys to choose from, essentially a computer use model.

It gives frames as the input, so you can give it a text prompt as well.

And you send it through API, and you say somebody waves at you, wave back.

In the input space, the model understands that you have a grid action available, it

will output the action associated with that.

One way of intuitive understanding how this works, I've obviously simplified here because

it's predicting the next token, not the next word, but bear with me here.

LLMs took large-scale internet data, then trained it to predict the next text token,

and now use it for writing software.

These are all workloads that are really, really well fit for futures that are not simultaneously

unfolding as you are predicting.

Code is turn-based.

You can say something, the model will do something, you can say something again, the model will

do something again.

And you can use it for lots of valuable work, but at the end of the day, because it's turn-based,

it's very, very difficult for it to generalize to environments, especially that include space

and time.

So our models, the way that they work is they're continuously receiving frames or receiving

batches of information from the camera of the robot, or it could be a screen.

And it predicts everything as if it were just a video game.

So you can think of the way that Anthropic focused on just nailing coding.

So they were never frontier and just base text generation.

They always focused on being the best at generating code.

And then the bet that they made is that more people would start bringing more problems

into coding space.

If you think of general intuition, our bet is that you can, as humans, we are generally pixel-to-action converters, plus senses, which in many cases you can convert into pixels as well.

And you can output actions at the same time.

Those actions are, in this case, either keyboard and mouse or game controller.

So the output tokens of the model are essentially equivalent to that of enhanced computer use model with more actions on the output space to internalize it.

So you send frames, and you get actions.

This works, again, because every single thing that you see here can control all of these different embodiments.

So, as humans, we do not know how to predict motor torque.

We do not know what value to send to a machine.

There's this beautiful device that has been made for many years, which actually does that work for us so we can control them.

Most robotics companies today are trying to solve the problem at the end-to-end level,

trying to predict exact poses or exact values in the motor,

instead of just looking and picking up the thing that every robotics company has built

to generalize our very limited abilities to have physics understanding and predict exact values,

and solve for it.

And so our approach is, let's just use that.

And so Medal, at the moment, sees about a billion videos uploaded per year.

This year we're on track for three billion,

which lead to over a trillion and a half action tokens per month.

So just for comparison, to think about the bet,

the total amount of data that Frontier Labs use today to train robotics models,

we collect in, at the moment, half a day.

And so the type of bet, and again, think of the fact that humans can use the exact same

controllers and their eyes to play video games that they can't help upgrade robots.

It is the same thing.

We just bet on scaling pixel space the same way that anthropics killed coding space, and

betting that people will just, as we scale data compute into pixel space, that people

will bring more problems into pixel space.

And so what happens?

of a sudden you get an incredibly general agent that can go and solve problems on computers

in the real world.

in addition, similarly to how Anthropic focuses on doing coding environments, we focus

on world models, which most of you know is for, the Mirai release world models, to then

generate additional environments for these models to learn in.

One way of internalizing a big advantage of this approach against LLMs is if you're an

LLM company and you have to bet on the ability to generate environments for your models to

learn, you have two problems.

One, you actually need to spend the compute to simulate environments, and the more realistic

you try to simulate, the more expensive that gets.

and so imagine you want to learn a new thing

you want to learn that new thing

in the exact input space of the human

so if we are learning from pixels

we're learning by looking around

if I'm trying to communicate something

I want my robot partner to observe that exact same thing

and so what I don't want

is for that to have to go through an LM

for a digital twin to have to be created

for the model to have to get deployed

for it to be a bunch of verifier checks

whether or not things are equivalent

and then for the robot to learn much later.

And so one way of thinking about it is

when you can learn in the native modality of the human,

you can all of a sudden start solving a lot more problems

a lot faster.

What you'll see when these models come out

is they're significantly faster than any LLM

at doing these things.

Because games are so action-dense,

So what you see, right, is one thing that's interesting is you saw, for instance, the model predicting multiple actions at the same time.

We're able to take that action density from gamers and convert them into really complex robotics workloads.

So, for example, if you are operating a drone and you need to make lots of little adjustments, you really, really, really want to start with games.

You don't want to start with teleoperations because the human will just take a lot less actions.

It has a lot less information.

You want a model that can predict really high APM.

And so the APM of the data set puts a lot of these problems all of a sudden in reach

so that it can do a lot of these actions simultaneously.

And as I mentioned, the model doesn't just understand the key it's predicting.

It also understands the semantic meaning associated with that key it's predicting, which is really

important for things like interpretability.

So we will be releasing the models this year.

we are very happy

I think the correct word would be

that we're stunned

by the fact of what these models can do

it is

this is a

live stream so I unfortunately couldn't share

the release videos without

breaking the surprise but if you're in New York

we are also in New York

we have started doing a few demos

of the model so if you can

know somebody on the team reach out and we can start

It's showing you some of the capabilities live.

But we have essentially one model which can play video games at the highest level

and control over 10 types of robots using the exact same primitives.

So pixels in, actions out, completely real time.

And we'll be rolling out early access to these models over the next few months.

We already have some customers that are on them, but it's still private,

so you're not hearing too much about it.

But if you scan this QR code, you'll be able to get the models because we go by who signs

up on the wait list at which times.

So this is a great moment to get in touch.

As I mentioned, we are in New York.

What you'll find is similar to how Jev and a lot of these decision-making models have

kind of opened people's eyes on what you can do when you have a new architectural primitive.

People are ready

to do away with the unpredictability of an LLM and put something in that is more reliable and faster and you understand how it is going to react as opposed to having to work around it.

If that resonates, these models are exactly what you're looking for. And all of a sudden you can start exploring the environments that you generate with LLMs.

All of a sudden, one of the use cases that I'm really, really excited about is, well,

these models, they can take actions as if a human was on the interface.

And so if you're doing UX work and you're trying to understand, is this an intuitive

interface?

Well, this is a very interesting way of solving the problem.

So all these problems that LLMs can't solve because they can't actually explore like a

human, you can use these problems and start using these models for.

So the way that they're going to be released is we still expect you to use an orchestrator model.

So you may use something like Gemini or Astra in order to do long-term planning and everything.

And you'll be able to plug these into your workloads.

And we're actually, the models are so general.

It is genuinely going to, I think, feel like the early LLM days when you have a new toy

and you just want to figure out what it does and all the things you can use it for.

my reaction was like that

when I first started using the checkpoints

and I think you'll have the same reaction

and I think a lot of you will

just like with LLMs

if you're a bit early and you start understanding these primitives

I think you'll understand

that you can even build lots of businesses on top of this

think about for instance

a science lab

all of a sudden if you can reliably

orchestrate robots in space and time, and they can take text instructions and do things.

All of a sudden, it's a recursive problem, right?

Robotics is interesting because one of the big problems of solving robotics is making

more robots.

And so there are all these businesses that, because it's so bottlenecked by intelligence,

are going to emerge just from solving this one simple problem, which is pixels to actions.

And I think if you're early on getting your hands on these, it will have a similar effect

on getting your hands on early LLMs.

I am most excited also, one of the things that we're working on which is interesting

is if you have any friends or family who are in some way need accessibility help for using

their computer, because these models can use computers like a human, all of a sudden, for

For instance, if you only can use one arm, all of a sudden the model can allow you to

play video games because you can just control the mouse and the model is predicting the

actions for your keyboard.

Because the model also understands which actions you're predicting, because that's the input,

it's observing as it's acting, it will actually just not predict the things that you're doing

and it will just predict the keys.

It has all these downstream effects of things that you can build on top.

I'm an engineer.

If you're interested in building in this space, this is going to be the most interesting toy

that comes out this year.

I'm fairly certain.

I also think a lot of the initial usage will sit in simulation.

If a lot of you are into generating environments with Astra or Gemini, it's perfect.

Now all of a sudden you can have something that can explore it and you don't have to

do it yourself.

itself, and it's perfect timing because of all the environment generation capabilities

going into these models.

So really quickly before I wrap up, I want to talk about the team.

We are based in New York City.

The Medal team and the general intuition team is here in New York City, so if you're here

you should reach out to us and we can both show you the models, especially if you want

to join the team, I think we'll prioritize even higher.

But we really are looking for people who, A,

want to understand these models as much as we do.

So if you're really, really good at generating

novel environments, typically if you have some sort

of research background, but I don't have a research

background, so that's OK.

If you come from an engineering background

and you're very happy to learn, this is fine too.

And ideally what we want is somebody who has experienced shipping end-to-end AI products,

but it doesn't have to be on the research side.

So we just want to hire people that understand the entire thing.

Let's see.

We have three teams.

So one is role modeling, the other is the policy, so the agent, and the other one is

working on essentially the data generation and the environment generation.

If you are an engineer, the best one to join is the environment generation team.

And then you grow into getting access to everyone.

I'm very, very excited for November.

I really hope that you'll be on the lookout.

As I mentioned, it will be either late November or early December is the current planning.

One more thing.

We are going to organize three hackathons.

The first is in New York City, it will be mid-November.

If you sign up for the partner wait list, we will send out some emails so everybody

can get access.

We hope to see everybody at the hackathons.

I think it will be the first time to get your hands on them the way we are playing with

them internally now.

And there will be one in New York City and one in London.

So if you are in either of those cities or want to travel to either of those cities,

those would be good times to meet the team.

Yeah, let's see.

I have time for one or two questions,

and then I think we can wrap.

Let's see.

How is it different from JEPA?

Yeah, how is it different from JEPA?

So JEPA doesn't train alongside the action labels,

so they bet on inferred action labels for the most part,

which I think, one of the ways of thinking about it

is JEPA I think in the limit is the more correct approach

but it's just a lot less practical in the short term

because you need to bet on a separate

scaling law on the action inference itself

and

we really like

JEPA as a concept, we think it's a

you know, we use some of the

things in the models

and

but at the end of the day it's an approach

not a product and I think we'll

you know, borrow

concepts both ways

one way of thinking about it is

because we have so much action data

let's say you're flying a plane

and you're moving the rudder in the back

there's no way of being able to

tell that from the video frames

it's very very difficult

so our mode on having the action data

really transfers better to

action sequences than it does

to world models and to generating

the pixels and that's why

we just went all in on training these general

foundation models for action predictions

and we use all sorts of techniques to do world models.

You know, we are own.

We really like other companies building them as well.

So, great.

One more and then I've got to run, I think.

Great.

All right.

Thank you, everyone.

Have a good one.