# Balancing Acceleration and Safety for AI Agents John Feminella · AI Engineer NYC 2026 · October 13 [Watch recording](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3059s) [00:50:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3059s) Hey, [00:51:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3061s) All right, I precariously balanced my water over there. [00:51:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3064s) My name's John Feminella. [00:51:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3066s) I work on AI architecture and engineering at Two Sigma, [00:51:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3069s) which is a hedge fund here in New York. [00:51:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3071s) Before I get too far into things, [00:51:13](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3073s) I have a legal disclaimer slide that I have to show you [00:51:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3076s) that says two things. [00:51:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3078s) One, I promise not to sell or market or offer you any securities [00:51:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3081s) at any point in this talk. [00:51:23](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3083s) and two, everything in this is my own opinion [00:51:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3085s) and not the opinion of Two Sigma. [00:51:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3087s) Thanks for indulging me. [00:51:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3088s) If I don't say those magic words, [00:51:30](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3090s) the 5G microchips implanted in me will explode. [00:51:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3095s) So that slide is a good example, actually, [00:51:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3097s) of how our industry is regulated. [00:51:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3100s) And as a regulated company with lots of controls, [00:51:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3103s) I think one thing that's always interesting [00:51:45](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3105s) is to explore what happens when controls fail [00:51:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3107s) because that gives you a good idea [00:51:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3109s) about why the controls were there in the first place. [00:51:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3111s) And so I'd like to tell you a story about that by showing you another legal slide. [00:51:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3116s) After August 2020, a very interesting contract clause suddenly started showing up in a bunch [00:52:02](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3122s) of commercial loan agreements. [00:52:03](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3123s) And this clause essentially tries to prevent a certain scenario from happening. [00:52:07](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3127s) And it's complicated, but I'll explain it like this. [00:52:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3130s) Let's say that you're a lemonade stand business, and you borrow money from a lender to capitalize [00:52:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3135s) your lemonade stand. [00:52:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3136s) and the bank that's overseeing that transaction, [00:52:19](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3139s) that's in the middle between the borrower and the lender, [00:52:23](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3143s) sends the lender all of the money that you owe in a catastrophic mistake. [00:52:29](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3149s) You'd probably be mad and want the money back, [00:52:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3151s) but the lender might also say, [00:52:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3153s) no thanks, you paid us back and we're all good, [00:52:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3155s) so we'd rather keep the money. [00:52:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3157s) Something like this happened in August 2020. [00:52:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3160s) Revlon, the cosmetics company, had borrowed money under a big 2016 term loan. [00:52:45](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3165s) And that loan was split up into a bunch of pieces across 300-plus lenders, hedge funds, credit funds, CLOs, other banks. [00:52:53](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3173s) And in the middle is Citibank, the administrative agent. [00:52:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3176s) Being the agent in this kind of transaction means that you're performing the plumbing that connects the borrowers with the lenders and managing the lifecycle of the loan. [00:53:05](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3185s) So in this transaction, Revlon pays Citi, and then Citi pays the lenders. [00:53:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3189s) And Citi's not lending its own money. [00:53:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3190s) It's just in the middle, moving people's money around. [00:53:13](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3193s) So here's the plan for August 11th, 2020. [00:53:17](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3197s) Revlon would pay about $8 million of accrued interest to both groups, [00:53:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3201s) and the lender's principal would roll into this new facility. [00:53:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3204s) That's an exchange of paper, not cash. [00:53:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3207s) So the remaining lender's principal would stay put, [00:53:29](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3209s) and the idea here is that only interest should be going out. [00:53:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3213s) Principals should not be moving at all. [00:53:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3215s) And the bank had a piece of internal processing software called FlexCube. [00:53:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3220s) Isn't that a fun enterprise software name? [00:53:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3222s) in FlexCube, the easiest way to do this kind of transaction [00:53:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3226s) is to pretend to record it like Revlon were paying off the entire loan. [00:53:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3231s) And that triggers the correct interest payment calculations to everyone. [00:53:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3235s) You can probably guess what's about to happen next. [00:53:57](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3237s) So you're supposed to send the principal portion [00:54:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3240s) to an internal WASH account that you own as the bank [00:54:03](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3243s) so that it's booked and recorded correctly, [00:54:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3246s) but it never leaves the bank's perimeter. [00:54:08](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3248s) So keep your eye on that WASH account. [00:54:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3250s) What was supposed to happen is that the boxes labeled front and fund and principal should [00:54:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3256s) all include the WASH account number and should all have a check box checked next to them. [00:54:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3261s) So unfortunately it turned out that was what needed to happen, but for various arcane reasons [00:54:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3267s) the boxes were not checked. [00:54:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3268s) And although FlexCube pops up a warning notification that says, uh-oh, money is about to leave the [00:54:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3273s) bank, are you sure you want to do this? [00:54:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3275s) That was correct. [00:54:36](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3276s) Money was supposed to leave the bank because they were making the interest payments. [00:54:39](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3279s) So the three approvers who were supposed to approve that looked at it and said, yep, that's [00:54:44](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3284s) the right thing that's supposed to happen. [00:54:45](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3285s) And although that warning notification popped up, the human operators carrying out the transaction [00:54:50](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3290s) overwrote that. [00:54:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3292s) Remember, Revlon was making that interest payment. [00:54:54](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3294s) So the principal never went to the wash account. [00:54:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3296s) And instead, Flexcube sent the entire outstanding principal of the loan to all of the 300 lenders, [00:55:02](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3302s) about $900 million instead of $8 million. [00:55:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3306s) So every lender got the amount that they would have been owed [00:55:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3309s) if Revlon had paid the loan in full, [00:55:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3311s) but Revlon didn't want to pay off the loan. [00:55:13](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3313s) It wanted to hold on to its money. [00:55:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3314s) That was a great deal for the lenders, [00:55:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3315s) because remember, Revlon was in trouble at the time, [00:55:17](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3317s) so its loans weren't trading at face value, [00:55:19](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3319s) and getting all your money back on a deal [00:55:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3321s) that you thought was only worth 60 cents on the dollar [00:55:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3324s) is a great deal for you. [00:55:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3326s) So they didn't want to give the money back, [00:55:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3328s) even after Citi called them and said, [00:55:30](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3330s) please, that was a mistake, give us the money back, [00:55:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3332s) and their argument was, [00:55:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3332s) we shouldn't have to give the money back [00:55:34](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3334s) because you paid us in full, and that's how loans work. [00:55:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3337s) You eventually pay them back, [00:55:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3338s) and just because you paid it back at a different time [00:55:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3340s) than you were supposed to doesn't mean [00:55:41](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3341s) that we should have to give it back now. [00:55:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3343s) And so to prevent that from happening, [00:55:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3346s) commercial loan agreements everywhere started [00:55:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3347s) including a clause like that one. [00:55:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3349s) So what does a payment error at a bank [00:55:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3351s) have to do with AI agents? [00:55:53](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3353s) Well, the bank had this procedure, it had these controls, [00:55:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3356s) and everybody did what they thought the right thing to do was. [00:55:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3359s) That's the problem I want to talk about for the rest of this talk. [00:56:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3361s) Every day we're giving real agents real autonomy over systems like FlexCube. [00:56:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3366s) And how do we prevent accidents like that one from either from happening without locking everything down or just hoping we'll get lucky that they won't happen? [00:56:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3374s) So here's the crux of the issue. [00:56:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3376s) You can't really know in advance what an agent will do, right? [00:56:19](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3379s) There are these non-deterministic systems. [00:56:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3381s) And that sounds scary until you remember that humans are also non-deterministic systems and they're going to make mistakes too. [00:56:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3387s) And so most firms don't choose to say, well, in order to reduce the risk of humans making a mistake, [00:56:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3392s) we're going to lock them in rooms and prevent them from talking to each other, [00:56:34](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3394s) and they're all going to double-check everybody else's work. [00:56:36](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3396s) We allow them a certain amount of autonomy along with a certain amount of controls. [00:56:41](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3401s) So we give them boundaries, visibility, and we equip them with systems and controls [00:56:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3406s) that let them do their jobs effectively. [00:56:48](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3408s) So in this sort of model, safety and innovation are not opposite ends of a spectrum [00:56:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3412s) that you must trade off on. [00:56:53](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3413s) They're things that you can have at the same time. [00:56:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3416s) So I think you can have your cake and eat it too. [00:56:58](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3418s) And when we were thinking about what controls we wanted to apply to agents, [00:57:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3421s) we also looked at some approaches that didn't work for us. [00:57:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3424s) We don't want to sandbox or lock down everything, [00:57:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3426s) because a sandbox with no real access to systems is safe, [00:57:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3430s) but it's not very interesting or useful. [00:57:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3432s) If you let agents go wild and do whatever they want, [00:57:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3434s) that's maybe powerful and freeing from a cognitive perspective, [00:57:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3438s) but the isolation is very weak there, [00:57:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3441s) and that approach is generally not suitable for highly regulated enterprises. [00:57:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3445s) And then if you just have policy documents that agents look at or that are part of your [00:57:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3451s) specification for the prompt, that's also not very enforceable at runtime. [00:57:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3455s) You can't guarantee that that will be part of the enforcement controls. [00:57:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3458s) So what did work for us was thinking about what mix of controls would get us both safety [00:57:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3463s) and innovation at the same time. [00:57:45](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3465s) So how do we provide boundaries and stop problems but don't hinder folks from operating correctly? [00:57:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3471s) So we have a number of these controls, but today I'll talk about five of them. [00:57:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3475s) When I say controls here, what I mean are the guardrails and shared capabilities that let teams move in ways that don't hinder what they're trying to do. [00:58:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3484s) And we don't want to reach some chasm that we need to cross to let people get their work done and then make everyone wait while we build a bridge to get there. [00:58:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3495s) So these controls are composable. [00:58:17](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3497s) Each one does one job. [00:58:19](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3499s) And then together, they let us run agents with real autonomy at an appropriate level of safety. [00:58:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3506s) So the first problem is that agents were showing up everywhere. [00:58:29](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3509s) We have big teams with lots of agents running. [00:58:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3512s) We have small teams with a few. [00:58:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3513s) We have individual people running stuff in an uncontrolled way. [00:58:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3517s) And these come in every shape. [00:58:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3518s) Could be random chatbots. [00:58:39](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3519s) It could be MCP servers glued together in this Frankenstein hot [00:58:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3523s) podge. [00:58:44](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3524s) It could be cool notebook experiments. [00:58:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3526s) It could be really useful web tools for folks. [00:58:48](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3528s) But it's hard to answer when you look at that a few simple [00:58:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3532s) questions, like who owns this? [00:58:54](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3534s) what tools does it have access to, what should it be doing, what should it not be doing. [00:58:58](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3538s) And if you can't answer those kinds of questions, if you can't govern them, [00:59:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3541s) then you can't really also reuse that. [00:59:03](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3543s) So you're tempted to build multiple solutions to the same problem [00:59:08](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3548s) if you don't know about what exists already. [00:59:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3550s) And then the firm can't also govern those in any kind of legible way. [00:59:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3554s) So one thing the firms sometimes do here is add some kind of gate. [00:59:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3558s) Like don't let anything run until everything is registered and approved [00:59:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3561s) and has gone through this very onerous process. [00:59:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3564s) And that sounds safe, but if you look at what it does to the work, [00:59:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3567s) every idea, even some simple no-pick experiment, [00:59:30](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3570s) is going to wait in line behind something that's really risky. [00:59:34](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3574s) And we don't want to subject everybody who's given the diversity of agents [00:59:39](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3579s) to exactly the same process. [00:59:41](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3581s) So most people are going to give up, [00:59:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3582s) or they'll just get pushed to some shadow IT, [00:59:44](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3584s) less governed path that is less visible, [00:59:48](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3588s) and then we don't get to see anything at all. [00:59:50](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3590s) So that's what motivated the first control is an agent registry. [00:59:53](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3593s) You can't govern it, you can't see, so the agent registry makes agents legible, and it [00:59:57](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3597s) registers, retrieves, and discovers the key [01:00:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3600s) facts about agents that we want to know for any agent. [01:00:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3604s) So those are things like which tools it declares, what endpoints it's going to talk to, which product does it belong to, who owns it. [01:00:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3610s) And humans, other programs, other agents can then use the agent registry to decide what's available at the firm, whether that's useful for them, and to go use that as a source of truth for other questions. [01:00:23](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3623s) So this doesn't require extra effort on your part. [01:00:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3625s) It just happens when you use one of the firm's approved agent building frameworks. [01:00:29](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3629s) It's automatically registered. [01:00:30](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3630s) All that information is recorded there for you. [01:00:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3632s) And if you want to do a manual registration, you can certainly do that. [01:00:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3637s) So the second problem we have is that now that we've got all these agents doing all [01:00:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3642s) these things, we quickly notice that what they're doing is really different. [01:00:45](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3645s) And that means our agents carry really different risks, right? [01:00:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3649s) So a thing that summarizes Jira or linear tickets is really different than a production [01:00:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3655s) deployment agent that can touch a running system. [01:00:57](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3657s) And if we try to use the same review process to cover all of these agents, then every agent [01:01:03](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3663s) is waiting in the same queue for weeks for human review. [01:01:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3666s) And if people are going to route around slow processes, then we wind up in the same situation [01:01:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3671s) that we had before. [01:01:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3672s) So again, you might be tempted to say the answer is something like really detailed, [01:01:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3676s) heavy manual review by all of these different constituencies. Again, that feels safe, but now [01:01:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3681s) we've got a trivial summarizer waiting two months for the same sign-offs as an agent that is touching [01:01:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3687s) production systems. And most of the good agent ideas, most of the ones that have value right away, [01:01:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3693s) start out as small experiments, and you don't want those people to wait in the same line as the stuff [01:01:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3698s) that's touching production right away. So the reviewers are drowning in these low-risk requests, [01:01:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3703s) and the high-risk ones have to share that attention pool. [01:01:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3706s) So we don't want to trade away that speed if we can help it. [01:01:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3709s) So again, because not every agent needs the same review, [01:01:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3712s) and because most agents are purpose-built rather than open-ended, [01:01:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3716s) and when I say open-ended here, [01:01:57](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3717s) you can imagine coding agents like Pi or Claude Code. [01:02:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3720s) When you open them and you start typing, [01:02:03](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3723s) Claude Code or Pi doesn't know what you're about to ask. [01:02:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3726s) It doesn't circumscribe its purpose in any particular way. [01:02:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3729s) But most of the agents that are deployed in enterprises [01:02:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3732s) do have very specific purposes. When you're engaging with that agent, it's to do something [01:02:17](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3737s) specific that's well known and well understood. So they have a very specific reason for existing [01:02:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3741s) and a very circumscribed purpose. And you can tell these agents apart from each other by looking at [01:02:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3746s) what they declare up front, their tools, the endpoints they access, the controls that are [01:02:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3751s) applied to them, etc. So the third problem is that if you know about your agents and what they're [01:02:39](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3759s) doing, you'll quickly notice that many agents don't talk to each other. [01:02:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3763s) Sorry, they don't talk to just one thing or in just one capacity. [01:02:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3766s) The more open-ended the agent is, the more it looks like a Claude Code or Pi, the more [01:02:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3771s) it can do a variety and a diversity of different things. [01:02:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3775s) Call an internal API, run a tool like a web search, that sort of thing. [01:02:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3779s) And each one of those is a different opening, a different door to a different kind of risk. [01:03:03](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3783s) And if you give an agent an entitlement for a role account, then it can do a lot more [01:03:08](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3788s) damaged than something that's just going to read a public web page, for example. [01:03:12](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3792s) So the question is not, can this agent do X, but what's the granularity that we need [01:03:17](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3797s) to solve for for the agent to act successfully? [01:03:21](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3801s) And again, one tempting answer is to kind of be heavy-handed here, or to give every [01:03:25](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3805s) agent the same short allow list of stuff that's safe to do, and then block everything else. [01:03:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3811s) That's safe, but it's also useless. [01:03:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3813s) An agent that can't search, or can't call an API, or can't do any of those things without [01:03:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3818s) asking somebody first can't really do the work that it was intended for. And then [01:03:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3822s) teams either give up on those agents or they find a way around the wall and [01:03:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3826s) both outcomes are costly. So a wide kind of blanket block on everything trades [01:03:51](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3831s) away the innovation that we want to preserve and avoids making a real [01:03:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3835s) decision about risk. And as author John Shedd wrote, a ship and harbor is safe, [01:04:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3840s) but that is not what ships are for. So the fix here is to let agents declare [01:04:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3846s) their purpose upfront and say what tools they're going to use or what tools they [01:04:11](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3851s) need access to. So in the second column here we might say that a research [01:04:17](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3857s) summarizer is going to be allowed to use a web tool to access research, but for a [01:04:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3864s) release deployer agent we're not going to allow them to do that because they [01:04:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3868s) have no reason to use that tool. And so if they try to ask for it they won't get [01:04:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3872s) it by default. So we don't need to make this kind of decision for every tool. [01:04:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3875s) the agents just have to say which tools they want. [01:04:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3878s) And you're, as the agent author, you're in the best position to say, [01:04:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3882s) I need these four things out of the 500 that are available. [01:04:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3887s) So the way we need to enforce that is that when agents run in our managed environments, [01:04:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3892s) they look at the registry that we talked about before [01:04:55](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3895s) to look up what policies apply to it at initialization time. [01:04:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3899s) And that policy gets translated into enforceable statements, [01:05:02](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3902s) like which endpoints are you allowed to access, which hosts are you allowed to access, [01:05:07](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3907s) what other types of resources are available or not available to you. [01:05:10](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3910s) And in this way, nobody's writing a ticket every time they need to change the access. [01:05:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3915s) Nobody's going to talk to IT or some other group outside of your team. [01:05:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3920s) As long as you conform to the risk score that we talked about before [01:05:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3924s) and you're in the agent registry, you're governed, you're approved. [01:05:27](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3927s) So this is how we keep the cost to teams really low for deploying, building, running agents. [01:05:34](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3934s) And the same control architecture works in a variety of settings. [01:05:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3937s) The next thing we noticed was that teams were building a ton of different ways to evaluate agents, [01:05:42](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3942s) and every stack looks different, and some teams don't build anything at all. [01:05:45](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3945s) Sometimes you build this really detailed stack for how your models work. [01:05:49](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3949s) Sometimes there's nothing. [01:05:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3952s) And I think in general we'd be upset if teams were not writing unit tests, right? [01:05:58](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3958s) And we consider evals to be kind of the same thing. [01:06:00](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3960s) If models are changing, that's like a runtime dependency that your application has. [01:06:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3964s) And if your agent is going to change models, it doesn't know without the evals what's going to happen to its outputs and to the quality of those outputs. [01:06:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3974s) So, again, one tempting answer here is let's have some kind of central evaluation methodology [01:06:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3980s) that waits for a full eval suite and a committee sign-off. [01:06:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3984s) But now every prompt tweak, every one-line change is going to sit in that same queue. [01:06:29](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3989s) So we don't want to put more decisions on more reviewers for things like that. [01:06:34](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3994s) We don't want to trade away that iteration. [01:06:36](https://www.youtube.com/watch?v=akEqiXFo3GM&t=3996s) So again, you wouldn't ship software without unit tests, so agents need evaluations for [01:06:40](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4000s) the same reason, and we want to provide the stuff at the level of the firm that's going [01:06:46](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4006s) to be the same for everybody. [01:06:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4007s) So Team A writing evals, Team B writing evals. [01:06:50](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4010s) They're both going to have many of the same features. [01:06:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4012s) So we want to have one shared approach to how we do evals, and have the shared approach [01:06:57](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4017s) be something that's run by the firm and available to everybody for doing their evals. [01:07:01](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4021s) So that covers all the common parts of evals, and all you've got to do is just write what's [01:07:06](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4026s) true about your agent. [01:07:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4029s) And finally, maybe you've had the experience of being asked to approve lots and lots of [01:07:13](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4033s) individual agent actions when there's a human in the loop. [01:07:16](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4036s) So that's fine on a small scale, but once you're operating agents in parallel or in [01:07:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4040s) production, this becomes completely onerous and review requests come in faster than anybody [01:07:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4046s) can be expected to approve them. [01:07:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4048s) After the first few times this is happening, you succumb to review fatigue where you just [01:07:32](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4052s) see yes, yes, yes, yes, yes. [01:07:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4053s) and eventually you click yes on something that's activating a nuclear bomb. [01:07:38](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4058s) So you don't want to click yes by reflex, but that's going to happen inevitably. [01:07:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4063s) One recent example of this is earlier this year, a director at an AI research lab [01:07:47](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4067s) told her open claw agent to suggest emails to delete from her inbox and wait for her approval. [01:07:52](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4072s) But her inbox was so large that eventually the agent compacted its context [01:07:56](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4076s) and lost the part that said, wait for my approval. [01:07:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4079s) And then it deleted hundreds of emails and didn't stop when she told it to stop on her phone. [01:08:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4084s) So again, you might be tempted here to make a human approve every action, but that's not [01:08:07](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4087s) the right way to do it. [01:08:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4089s) A better approach is to let these independent, conservative classifiers make decisions about [01:08:15](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4095s) routine calls, and then send a person only the requests that are most important, the [01:08:20](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4100s) ones that are most likely to make a difference. [01:08:22](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4102s) If you're saying yes, 99.99999% of the time, then that's probably just a rule that the [01:08:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4108s) agent should follow, not a request you need to ask the human for permission for. [01:08:31](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4111s) and the stuff that's unclear or ambiguous or potentially dangerous, yes, surface that to a human, [01:08:36](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4116s) or if it's definitely just dangerous, deny it outright. So this is a way to not have to do all [01:08:43](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4123s) the review work all the time. So we want to take the same approach for both explicit, [01:08:48](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4128s) when explicit rules apply, we want deterministic fast paths. When explicit rules don't apply and [01:08:54](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4134s) there's some ambiguity, we want a classifier to do that work instead. So putting those five pieces [01:08:59](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4139s) back together. You create a registry so that every agent is visible from the day it's created. [01:09:04](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4144s) You score each agent so human review goes in the right spot. You run agents on guardrails so that [01:09:09](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4149s) they only touch what they need. You share evals and metrics so everybody is reaching the same bar [01:09:14](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4154s) without having to reinvent the wheel. And then finally you govern your runtime to make sure that [01:09:18](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4158s) classification happens there and not somewhere else. So each of these controls works together. [01:09:24](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4164s) So I'll leave you with one question. [01:09:26](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4166s) Why do cars have brakes? [01:09:28](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4168s) It's not just so that you can stop, it's so that you can go fast and stop safely. [01:09:33](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4173s) So controls for our agents work the same way. [01:09:35](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4175s) Thank you very much. [01:09:37](https://www.youtube.com/watch?v=akEqiXFo3GM&t=4177s) I'm John Feminella, and I hope you had a good time.