# Frontier Models for Acting in Space and Time

Pim de Witte · AI Engineer NYC 2026 · October 13

[Watch recording](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26936s)

[07:28:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26936s) Hello.

[07:28:58](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26938s) I want you to zoom out for a second and think about if somebody were to tell you today that

[07:29:05](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26945s) we would take all the information on the internet and then we would train a model on it and

[07:29:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26951s) we would use that in order to solve scientific problems that were not on the internet.

[07:29:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26955s) I think a lot of people would tell you that that's crazy.

[07:29:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26960s) And what we're doing has a lot of parallels to that story.

[07:29:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26965s) So if you think about how intelligence works, imagine for a second that you're in the movie

[07:29:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26971s) Frozen and you're Olaf, the snowman.

[07:29:36](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26976s) If Olaf was an LLM, and there's a scene which is one of my favorites, which is where he rolls

[07:29:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26982s) down a mountain and there's a little stone that pops up and Olaf spreads out, temporarily

[07:29:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26989s) becomes a disassembled snowman and then becomes an assembled snowman.

[07:29:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26992s) Olaf knows how to do that because Olaf can observe as he is acting.

[07:29:58](https://www.youtube.com/watch?v=akEqiXFo3GM&t=26998s) Olaf can, unlike

[07:30:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27000s) LLMs, which autoregressively predict tokens, constantly part of that behavior is that it's constantly observing as it is acting.

[07:30:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27010s) And so this is the big difference between, you know, when you see LLMs try to solve robotics problems or when you look at native architectures like we're working on, it's very, very easy to think of LLMs in the demos to see on Twitter as the thing that will

[07:30:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27024s) Solve everything, there are fundamental limitations where purpose-built models are a much better

[07:30:30](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27030s) fit.

[07:30:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27031s) Today I'm going to talk a little bit about what the other side of that coin looks like.

[07:30:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27038s) For us, it all started with Medal.

[07:30:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27040s) We built the largest video game recorder in the world, and we think of that as the internet.

[07:30:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27047s) People use Medal to record themselves playing tens of thousands of different video games,

[07:30:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27051s) And if you think about it, as a human, we can teleoperate a robot with the exact same

[07:31:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27060s) brain that we played at video game, and in fact, in many cases, we also can use the same

[07:31:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27069s) controller.

[07:31:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27070s) So what this means is that the action space of video games, often because of game controllers,

[07:31:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27074s) and the action space of teleoperations or the real world, in many cases is the same.

[07:31:19](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27079s) And so every single robot, below certain degrees of freedom, ships with the game controller.

[07:31:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27085s) The majority of the supply chains already work this way.

[07:31:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27087s) And so we thought, what if you just take all of the action data that we get through the

[07:31:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27092s) video recorder, which associates the inputs and the actions with the actual frames, we

[07:31:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27098s) label the actions so we understand semantic meaning behind every action, and we use it

[07:31:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27103s) to give it to a vision-based model.

[07:31:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27106s) What you see here on the left is just an intuitive way of how the model works.

[07:31:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27111s) You give the model a set of keys to choose from, essentially a computer use model.

[07:31:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27116s) It gives frames as the input, so you can give it a text prompt as well.

[07:32:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27120s) And you send it through API, and you say somebody waves at you, wave back.

[07:32:05](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27125s) In the input space, the model understands that you have a grid action available, it

[07:32:08](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27128s) will output the action associated with that.

[07:32:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27132s) One way of intuitive understanding how this works, I've obviously simplified here because

[07:32:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27138s) it's predicting the next token, not the next word, but bear with me here.

[07:32:23](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27143s) LLMs took large-scale internet data, then trained it to predict the next text token,

[07:32:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27148s) and now use it for writing software.

[07:32:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27151s) These are all workloads that are really, really well fit for futures that are not simultaneously

[07:32:36](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27156s) unfolding as you are predicting.

[07:32:39](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27159s) Code is turn-based.

[07:32:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27160s) You can say something, the model will do something, you can say something again, the model will

[07:32:44](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27164s) do something again.

[07:32:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27166s) And you can use it for lots of valuable work, but at the end of the day, because it's turn-based,

[07:32:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27172s) it's very, very difficult for it to generalize to environments, especially that include space

[07:32:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27179s) and time.

[07:32:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27179s) So our models, the way that they work is they're continuously receiving frames or receiving

[07:33:07](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27187s) batches of information from the camera of the robot, or it could be a screen.

[07:33:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27192s) And it predicts everything as if it were just a video game.

[07:33:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27198s) So you can think of the way that Anthropic focused on just nailing coding.

[07:33:23](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27203s) So they were never frontier and just base text generation.

[07:33:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27207s) They always focused on being the best at generating code.

[07:33:30](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27210s) And then the bet that they made is that more people would start bringing more problems

[07:33:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27213s) into coding space.

[07:33:34](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27214s) If you think of general intuition, our bet is that you can, as humans, we are generally pixel-to-action converters, plus senses, which in many cases you can convert into pixels as well.

[07:33:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27226s) And you can output actions at the same time.

[07:33:50](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27230s) Those actions are, in this case, either keyboard and mouse or game controller.

[07:33:53](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27233s) So the output tokens of the model are essentially equivalent to that of enhanced computer use model with more actions on the output space to internalize it.

[07:34:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27241s) So you send frames, and you get actions.

[07:34:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27244s) This works, again, because every single thing that you see here can control all of these different embodiments.

[07:34:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27251s) So, as humans, we do not know how to predict motor torque.

[07:34:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27261s) We do not know what value to send to a machine.

[07:34:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27265s) There's this beautiful device that has been made for many years, which actually does that work for us so we can control them.

[07:34:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27272s) Most robotics companies today are trying to solve the problem at the end-to-end level,

[07:34:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27277s) trying to predict exact poses or exact values in the motor,

[07:34:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27282s) instead of just looking and picking up the thing that every robotics company has built

[07:34:50](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27290s) to generalize our very limited abilities to have physics understanding and predict exact values,

[07:34:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27299s) and solve for it.

[07:35:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27300s) And so our approach is, let's just use that.

[07:35:05](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27305s) And so Medal, at the moment, sees about a billion videos uploaded per year.

[07:35:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27312s) This year we're on track for three billion,

[07:35:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27315s) which lead to over a trillion and a half action tokens per month.

[07:35:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27320s) So just for comparison, to think about the bet,

[07:35:23](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27323s) the total amount of data that Frontier Labs use today to train robotics models,

[07:35:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27326s) we collect in, at the moment, half a day.

[07:35:29](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27329s) And so the type of bet, and again, think of the fact that humans can use the exact same

[07:35:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27338s) controllers and their eyes to play video games that they can't help upgrade robots.

[07:35:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27342s) It is the same thing.

[07:35:44](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27344s) We just bet on scaling pixel space the same way that anthropics killed coding space, and

[07:35:50](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27350s) betting that people will just, as we scale data compute into pixel space, that people

[07:35:54](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27354s) will bring more problems into pixel space.

[07:35:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27356s) And so what happens?

[07:36:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27361s) of a sudden you get an incredibly general agent that can go and solve problems on computers

[07:36:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27371s) in the real world.

[07:36:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27374s) in addition, similarly to how Anthropic focuses on doing coding environments, we focus

[07:36:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27380s) on world models, which most of you know is for, the Mirai release world models, to then

[07:36:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27385s) generate additional environments for these models to learn in.

[07:36:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27387s) One way of internalizing a big advantage of this approach against LLMs is if you're an

[07:36:34](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27394s) LLM company and you have to bet on the ability to generate environments for your models to

[07:36:39](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27399s) learn, you have two problems.

[07:36:41](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27401s) One, you actually need to spend the compute to simulate environments, and the more realistic

[07:36:45](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27405s) you try to simulate, the more expensive that gets.

[07:36:48](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27408s) and so imagine you want to learn a new thing

[07:36:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27411s) you want to learn that new thing

[07:36:53](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27413s) in the exact input space of the human

[07:36:57](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27417s) so if we are learning from pixels

[07:37:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27420s) we're learning by looking around

[07:37:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27421s) if I'm trying to communicate something

[07:37:03](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27423s) I want my robot partner to observe that exact same thing

[07:37:07](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27427s) and so what I don't want

[07:37:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27431s) is for that to have to go through an LM

[07:37:13](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27433s) for a digital twin to have to be created

[07:37:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27434s) for the model to have to get deployed

[07:37:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27436s) for it to be a bunch of verifier checks

[07:37:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27438s) whether or not things are equivalent

[07:37:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27440s) and then for the robot to learn much later.

[07:37:22](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27442s) And so one way of thinking about it is

[07:37:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27444s) when you can learn in the native modality of the human,

[07:37:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27447s) you can all of a sudden start solving a lot more problems

[07:37:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27451s) a lot faster.

[07:37:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27452s) What you'll see when these models come out

[07:37:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27453s) is they're significantly faster than any LLM

[07:37:39](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27459s) at doing these things.

[07:37:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27462s) Because games are so action-dense,

[07:37:44](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27464s) So what you see, right, is one thing that's interesting is you saw, for instance, the model predicting multiple actions at the same time.

[07:37:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27472s) We're able to take that action density from gamers and convert them into really complex robotics workloads.

[07:37:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27479s) So, for example, if you are operating a drone and you need to make lots of little adjustments, you really, really, really want to start with games.

[07:38:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27486s) You don't want to start with teleoperations because the human will just take a lot less actions.

[07:38:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27491s) It has a lot less information.

[07:38:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27492s) You want a model that can predict really high APM.

[07:38:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27496s) And so the APM of the data set puts a lot of these problems all of a sudden in reach

[07:38:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27501s) so that it can do a lot of these actions simultaneously.

[07:38:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27504s) And as I mentioned, the model doesn't just understand the key it's predicting.

[07:38:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27508s) It also understands the semantic meaning associated with that key it's predicting, which is really

[07:38:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27511s) important for things like interpretability.

[07:38:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27517s) So we will be releasing the models this year.

[07:38:41](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27521s) we are very happy

[07:38:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27523s) I think the correct word would be

[07:38:45](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27525s) that we're stunned

[07:38:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27527s) by the fact of what these models can do

[07:38:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27529s) it is

[07:38:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27532s) this is a

[07:38:53](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27533s) live stream so I unfortunately couldn't share

[07:38:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27535s) the release videos without

[07:38:57](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27537s) breaking the surprise but if you're in New York

[07:38:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27539s) we are also in New York

[07:39:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27541s) we have started doing a few demos

[07:39:03](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27543s) of the model so if you can

[07:39:05](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27545s) know somebody on the team reach out and we can start

[07:39:07](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27547s) It's showing you some of the capabilities live.

[07:39:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27551s) But we have essentially one model which can play video games at the highest level

[07:39:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27554s) and control over 10 types of robots using the exact same primitives.

[07:39:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27560s) So pixels in, actions out, completely real time.

[07:39:23](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27563s) And we'll be rolling out early access to these models over the next few months.

[07:39:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27568s) We already have some customers that are on them, but it's still private,

[07:39:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27572s) so you're not hearing too much about it.

[07:39:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27573s) But if you scan this QR code, you'll be able to get the models because we go by who signs

[07:39:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27582s) up on the wait list at which times.

[07:39:44](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27584s) So this is a great moment to get in touch.

[07:39:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27587s) As I mentioned, we are in New York.

[07:39:48](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27588s) What you'll find is similar to how Jev and a lot of these decision-making models have

[07:39:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27595s) kind of opened people's eyes on what you can do when you have a new architectural primitive.

[07:39:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27599s) People are ready

[07:40:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27600s) to do away with the unpredictability of an LLM and put something in that is more reliable and faster and you understand how it is going to react as opposed to having to work around it.

[07:40:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27610s) If that resonates, these models are exactly what you're looking for. And all of a sudden you can start exploring the environments that you generate with LLMs.

[07:40:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27621s) All of a sudden, one of the use cases that I'm really, really excited about is, well,

[07:40:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27626s) these models, they can take actions as if a human was on the interface.

[07:40:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27631s) And so if you're doing UX work and you're trying to understand, is this an intuitive

[07:40:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27635s) interface?

[07:40:36](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27636s) Well, this is a very interesting way of solving the problem.

[07:40:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27638s) So all these problems that LLMs can't solve because they can't actually explore like a

[07:40:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27642s) human, you can use these problems and start using these models for.

[07:40:48](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27648s) So the way that they're going to be released is we still expect you to use an orchestrator model.

[07:40:54](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27654s) So you may use something like Gemini or Astra in order to do long-term planning and everything.

[07:40:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27659s) And you'll be able to plug these into your workloads.

[07:41:02](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27662s) And we're actually, the models are so general.

[07:41:05](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27665s) It is genuinely going to, I think, feel like the early LLM days when you have a new toy

[07:41:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27672s) and you just want to figure out what it does and all the things you can use it for.

[07:41:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27676s) my reaction was like that

[07:41:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27678s) when I first started using the checkpoints

[07:41:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27681s) and I think you'll have the same reaction

[07:41:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27684s) and I think a lot of you will

[07:41:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27686s) just like with LLMs

[07:41:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27688s) if you're a bit early and you start understanding these primitives

[07:41:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27691s) I think you'll understand

[07:41:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27693s) that you can even build lots of businesses on top of this

[07:41:36](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27696s) think about for instance

[07:41:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27700s) a science lab

[07:41:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27703s) all of a sudden if you can reliably

[07:41:44](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27704s) orchestrate robots in space and time, and they can take text instructions and do things.

[07:41:50](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27710s) All of a sudden, it's a recursive problem, right?

[07:41:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27715s) Robotics is interesting because one of the big problems of solving robotics is making

[07:41:58](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27718s) more robots.

[07:41:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27719s) And so there are all these businesses that, because it's so bottlenecked by intelligence,

[07:42:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27724s) are going to emerge just from solving this one simple problem, which is pixels to actions.

[07:42:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27730s) And I think if you're early on getting your hands on these, it will have a similar effect

[07:42:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27735s) on getting your hands on early LLMs.

[07:42:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27740s) I am most excited also, one of the things that we're working on which is interesting

[07:42:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27746s) is if you have any friends or family who are in some way need accessibility help for using

[07:42:34](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27754s) their computer, because these models can use computers like a human, all of a sudden, for

[07:42:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27757s) For instance, if you only can use one arm, all of a sudden the model can allow you to

[07:42:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27763s) play video games because you can just control the mouse and the model is predicting the

[07:42:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27769s) actions for your keyboard.

[07:42:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27771s) Because the model also understands which actions you're predicting, because that's the input,

[07:42:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27775s) it's observing as it's acting, it will actually just not predict the things that you're doing

[07:43:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27780s) and it will just predict the keys.

[07:43:02](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27782s) It has all these downstream effects of things that you can build on top.

[07:43:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27786s) I'm an engineer.

[07:43:08](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27788s) If you're interested in building in this space, this is going to be the most interesting toy

[07:43:13](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27793s) that comes out this year.

[07:43:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27796s) I'm fairly certain.

[07:43:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27798s) I also think a lot of the initial usage will sit in simulation.

[07:43:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27801s) If a lot of you are into generating environments with Astra or Gemini, it's perfect.

[07:43:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27805s) Now all of a sudden you can have something that can explore it and you don't have to

[07:43:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27807s) do it yourself.

[07:43:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27808s) itself, and it's perfect timing because of all the environment generation capabilities

[07:43:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27815s) going into these models.

[07:43:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27818s) So really quickly before I wrap up, I want to talk about the team.

[07:43:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27822s) We are based in New York City.

[07:43:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27826s) The Medal team and the general intuition team is here in New York City, so if you're here

[07:43:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27829s) you should reach out to us and we can both show you the models, especially if you want

[07:43:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27835s) to join the team, I think we'll prioritize even higher.

[07:44:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27840s) But we really are looking for people who, A,

[07:44:05](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27845s) want to understand these models as much as we do.

[07:44:08](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27848s) So if you're really, really good at generating

[07:44:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27849s) novel environments, typically if you have some sort

[07:44:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27852s) of research background, but I don't have a research

[07:44:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27855s) background, so that's OK.

[07:44:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27856s) If you come from an engineering background

[07:44:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27858s) and you're very happy to learn, this is fine too.

[07:44:22](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27862s) And ideally what we want is somebody who has experienced shipping end-to-end AI products,

[07:44:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27872s) but it doesn't have to be on the research side.

[07:44:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27873s) So we just want to hire people that understand the entire thing.

[07:44:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27878s) Let's see.

[07:44:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27880s) We have three teams.

[07:44:41](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27881s) So one is role modeling, the other is the policy, so the agent, and the other one is

[07:44:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27886s) working on essentially the data generation and the environment generation.

[07:44:50](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27890s) If you are an engineer, the best one to join is the environment generation team.

[07:44:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27895s) And then you grow into getting access to everyone.

[07:45:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27900s) I'm very, very excited for November.

[07:45:02](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27902s) I really hope that you'll be on the lookout.

[07:45:07](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27907s) As I mentioned, it will be either late November or early December is the current planning.

[07:45:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27911s) One more thing.

[07:45:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27912s) We are going to organize three hackathons.

[07:45:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27915s) The first is in New York City, it will be mid-November.

[07:45:22](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27922s) If you sign up for the partner wait list, we will send out some emails so everybody

[07:45:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27926s) can get access.

[07:45:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27927s) We hope to see everybody at the hackathons.

[07:45:30](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27930s) I think it will be the first time to get your hands on them the way we are playing with

[07:45:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27933s) them internally now.

[07:45:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27935s) And there will be one in New York City and one in London.

[07:45:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27937s) So if you are in either of those cities or want to travel to either of those cities,

[07:45:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27940s) those would be good times to meet the team.

[07:45:45](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27945s) Yeah, let's see.

[07:45:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27947s) I have time for one or two questions,

[07:45:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27949s) and then I think we can wrap.

[07:45:54](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27954s) Let's see.

[07:46:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27960s) How is it different from JEPA?

[07:46:02](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27962s) Yeah, how is it different from JEPA?

[07:46:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27964s) So JEPA doesn't train alongside the action labels,

[07:46:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27966s) so they bet on inferred action labels for the most part,

[07:46:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27969s) which I think, one of the ways of thinking about it

[07:46:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27971s) is JEPA I think in the limit is the more correct approach

[07:46:13](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27973s) but it's just a lot less practical in the short term

[07:46:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27975s) because you need to bet on a separate

[07:46:17](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27977s) scaling law on the action inference itself

[07:46:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27980s) and

[07:46:22](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27982s) we really like

[07:46:23](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27983s) JEPA as a concept, we think it's a

[07:46:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27986s) you know, we use some of the

[07:46:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27987s) things in the models

[07:46:29](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27989s) and

[07:46:30](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27990s) but at the end of the day it's an approach

[07:46:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27992s) not a product and I think we'll

[07:46:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27995s) you know, borrow

[07:46:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=27997s) concepts both ways

[07:46:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28000s) one way of thinking about it is

[07:46:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28003s) because we have so much action data

[07:46:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28006s) let's say you're flying a plane

[07:46:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28007s) and you're moving the rudder in the back

[07:46:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28009s) there's no way of being able to

[07:46:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28012s) tell that from the video frames

[07:46:53](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28013s) it's very very difficult

[07:46:54](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28014s) so our mode on having the action data

[07:46:57](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28017s) really transfers better to

[07:46:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28019s) action sequences than it does

[07:47:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28021s) to world models and to generating

[07:47:03](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28023s) the pixels and that's why

[07:47:05](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28025s) we just went all in on training these general

[07:47:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28026s) foundation models for action predictions

[07:47:08](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28028s) and we use all sorts of techniques to do world models.

[07:47:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28031s) You know, we are own.

[07:47:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28032s) We really like other companies building them as well.

[07:47:17](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28037s) So, great.

[07:47:19](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28039s) One more and then I've got to run, I think.

[07:47:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28041s) Great.

[07:47:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28045s) All right.

[07:47:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28046s) Thank you, everyone.

[07:47:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=28047s) Have a good one.