1 00:00:00,661 --> 00:00:01,801 Hey, New York. 2 00:00:02,301 --> 00:00:02,861 Awesome. 3 00:00:03,121 --> 00:00:04,981 Well, welcome to AI Engineer. 4 00:00:06,521 --> 00:00:12,261 I thought given we're at AI Engineer New York, we'd kind of revisit the question of what 5 00:00:12,261 --> 00:00:13,421 does an AI engineer do? 6 00:00:13,721 --> 00:00:20,001 I feel like every year we have an idea of what an AI engineer does, and that definition 7 00:00:20,001 --> 00:00:20,741 kind of changes. 8 00:00:21,561 --> 00:00:25,401 And as I was putting this talk together, it was super timely because Karpathy actually 9 00:00:25,401 --> 00:00:32,341 just tweeted something this weekend where he kind of talks about this growing gap between 10 00:00:33,281 --> 00:00:40,161 basically folks at the frontier who are running thousands of agents and swarms of always-on agents, 11 00:00:40,711 --> 00:00:46,141 and he really kind of drops this funnel where 6 billion-plus users have maybe touched or 12 00:00:46,141 --> 00:00:52,921 interfaced with an LLM, 1 to 2 billion are maybe like casual free-tier users, 20 million are 13 00:00:52,921 --> 00:00:58,281 professionals who have maybe started using it for more, taking some piece of work that 14 00:00:58,281 --> 00:01:00,561 would have taken weeks down to days. 15 00:01:01,201 --> 00:01:07,561 But really, those at the Frontier Labs, which is a super small percentage, it's like 5,000 16 00:01:07,561 --> 00:01:14,761 users, are the ones who are running thousands and thousands of agents at once. 17 00:01:15,241 --> 00:01:19,681 And as I look at this, it really is, I mean, he's kind of dropped a bit of the funnel. 18 00:01:19,681 --> 00:01:22,941 I mean, the first question anyone looks at this is that, where am I on this? 19 00:01:23,321 --> 00:01:34,661 And, you know, he's kind of drawn out a bit of a roadmap of how does everyone get to, in the next couple of years, one of those 5,000 users who are running thousands of agents. 20 00:01:35,281 --> 00:01:41,781 And this has been happening, I'd say, for a couple of decades now, where the abstraction layer has gotten higher and higher. 21 00:01:41,781 --> 00:02:07,601 I mean, definitely faster in the last couple of years, but we've gone from engineers writing higher level abstraction languages to, I mean, just copy pasting code, using local agents to using cloud agents to really the expectation now is how do all of us start running a team of agents or factories of agents internally and build toward that? 22 00:02:08,521 --> 00:02:12,961 And what this has led to is essentially what is the then and now of what an AI engineer 23 00:02:12,961 --> 00:02:13,341 is. 24 00:02:13,541 --> 00:02:21,621 And this is a bit, I think, of a roadmap for some, but I think internally this is what 25 00:02:21,621 --> 00:02:23,681 we've already started to kind of push towards. 26 00:02:24,081 --> 00:02:28,341 As an AI engineer, your job is no longer to just write the feature, write an AI feature, 27 00:02:28,461 --> 00:02:31,601 it's about how do you build the loop to write that feature. 28 00:02:31,601 --> 00:02:37,501 It's no longer about just reading the diff and actually doing code review. 29 00:02:37,661 --> 00:02:40,021 It's actually how do you set up systems and agents to 30 00:02:40,101 --> 00:02:40,881 do the code reviews? 31 00:02:41,061 --> 00:02:47,541 And how do you write evals to actually do the non-deterministic, to review those non-deterministic tasks? 32 00:02:48,281 --> 00:02:58,081 It's actually no longer even about just setting up metrics and software kind of telemetry, but it's actually how do you have agents go through and read the observability data, 33 00:02:58,321 --> 00:03:02,521 and have agents actually consume that telemetry to find patterns? 34 00:03:03,201 --> 00:03:20,041 And so this loop that we've kind of been talking about, which is the build, the eval, ship, you monitor, you improve it, well, the AI engineer is now actually responsible not just for doing each of those steps, but how do you build the loop that actually does each of those steps? 35 00:03:20,661 --> 00:03:30,441 And at Arize, one of the things we've rolled out in our product, and I'm going to talk about a bit of what we've learned in building this, is we built Signal to actually run this loop for AI engineers. 36 00:03:30,441 --> 00:03:36,741 What Signal does is that Signal actually looks at all of the traces, it reads the telemetry, 37 00:03:37,061 --> 00:03:43,461 it identifies patterns in that telemetry, and it can actually put up a fix or a PR if 38 00:03:43,461 --> 00:03:44,821 it has access to the repo. 39 00:03:45,801 --> 00:03:49,281 And Signal is an always-on agent. 40 00:03:49,541 --> 00:03:55,041 So this is an example of how Signal starts, but it actually starts off by just reading 41 00:03:55,041 --> 00:03:55,661 the traces. 42 00:03:56,201 --> 00:03:58,681 Super simple example, but it goes through, reads, 43 00:03:59,501 --> 00:04:01,141 and all of this is configurable, 44 00:04:01,281 --> 00:04:06,281 but a certain volume of traces that came in in some cadence, 45 00:04:06,281 --> 00:04:09,781 it reads it, and then the goal is to actually identify issues 46 00:04:09,781 --> 00:04:11,541 from that telemetry data. 47 00:04:12,141 --> 00:04:15,301 It has to keep track of issues that it's previously seen. 48 00:04:15,441 --> 00:04:18,001 So we have to consider the memory of, 49 00:04:18,781 --> 00:04:21,121 it see something before that it thought was an issue? 50 00:04:21,701 --> 00:04:24,821 Did something seem like an issue, but it wasn't actually captured, 51 00:04:24,821 --> 00:04:29,921 and now as it collects more data, it actually thinks it's much more of a critical issue. 52 00:04:30,281 --> 00:04:34,921 And so Signal actually starts off by reading the traces to identify problems. 53 00:04:35,921 --> 00:04:43,601 so one of the first things that we realized was that the future of who's actually consuming telemetry data 54 00:04:43,601 --> 00:04:44,901 has really changed. 55 00:04:45,081 --> 00:04:48,941 It's no longer about humans consuming telemetry data. 56 00:04:49,341 --> 00:04:53,741 It's not even about humans consuming it and agents kind of taking on the task, 57 00:04:53,741 --> 00:04:56,621 but it really is how do you set up your telemetry data 58 00:04:56,621 --> 00:04:59,741 so that agents are the primary consumers of that data. 59 00:05:00,421 --> 00:05:03,221 And what that does is that that observability data 60 00:05:03,221 --> 00:05:06,201 can become the trigger for actually running 61 00:05:06,201 --> 00:05:09,041 and triggering these always-on agents. 62 00:05:09,301 --> 00:05:12,301 And so the first thing that if you're building a team of, 63 00:05:12,981 --> 00:05:14,881 you know, a factory of really always-on agents 64 00:05:14,881 --> 00:05:18,541 is what triggers these agents to wake up or start. 65 00:05:18,961 --> 00:05:21,341 And there can be a variety of triggers. 66 00:05:21,341 --> 00:05:26,481 For signal, for us, it's configurable to run on an ongoing cadence. 67 00:05:26,761 --> 00:05:34,221 So every three hours, six hours, go pull the last X amount of traces, read them, identify problems. 68 00:05:34,461 --> 00:05:37,681 It can also be triggered by events. 69 00:05:37,901 --> 00:05:44,801 So Slack messages, if some user complains about something, it can be a metric-based trigger. 70 00:05:45,081 --> 00:05:49,781 But these are essentially triggers to kick up the agent actually starting. 71 00:05:50,681 --> 00:05:54,781 Once the agent actually starts, we have to spend some time figuring out, well, what does 72 00:05:54,781 --> 00:05:56,761 an agent run actually look like? 73 00:05:56,921 --> 00:06:04,161 For us, we actually pull in all of the context of the traces, any additional third-party 74 00:06:04,161 --> 00:06:11,141 data, so we allow collecting to APM logs, APM metrics, but all of the third-party telemetry 75 00:06:11,141 --> 00:06:23,041 data that we want to pull in, access to the repo, access to any keys that are needed to 76 00:06:23,041 --> 00:06:24,841 actually run this agent. 77 00:06:24,921 --> 00:06:31,921 But all of this information is put inside of a sandbox, and this is actually what's started 78 00:06:31,921 --> 00:06:34,101 up every time the agent wakes up. 79 00:06:34,341 --> 00:06:36,061 And so the agent's triggered. 80 00:06:36,461 --> 00:06:37,641 We build a sandbox. 81 00:06:37,841 --> 00:06:39,981 We put everything that we need inside of that sandbox. 82 00:06:40,021 --> 00:06:40,761 There's a proxy. 83 00:06:40,761 --> 00:06:44,561 The only way to get information in and out of the sandbox is by the single proxy. 84 00:06:45,261 --> 00:06:49,941 And everything that happens inside of that sandbox is actually already getting traced. 85 00:06:50,201 --> 00:06:56,581 And every output, so the issues that are generated and created are all getting saved to some kind of memory. 86 00:06:57,201 --> 00:07:00,661 And so this happens every single time the agent actually runs. 87 00:07:01,121 --> 00:07:05,581 Every time the agent finishes running, so it's sweeped through all of the traces, 88 00:07:06,821 --> 00:07:09,121 it either found something or it didn't. 89 00:07:09,121 --> 00:07:15,181 there's what actually stays and what actually gets thrown away what stays is 90 00:07:15,181 --> 00:07:19,321 that if it found any issues all of the output is actually saved off in some 91 00:07:19,321 --> 00:07:22,921 kind of memory so that the next time an agent gets spun up it doesn't have to 92 00:07:22,921 --> 00:07:26,861 start from ground zero it has a history of what are the issues that it actually 93 00:07:26,861 --> 00:07:34,241 found the agents are traced so all of the telemetry data of what the agent 94 00:07:34,241 --> 00:07:38,441 called you know if it actually put up a PR all of those things are actually 95 00:07:38,441 --> 00:07:44,581 traced, and any skills that it may have learned while actually doing a run can also get saved 96 00:07:44,581 --> 00:07:45,461 to the actual repo. 97 00:07:45,961 --> 00:07:51,601 But the things that go away is the sandbox gets torn down, any credentials obviously 98 00:07:51,601 --> 00:07:58,621 get removed, the repo access is removed, and so all of the stuff that doesn't need to be 99 00:07:58,621 --> 00:08:01,181 saved kind of gets torn down after the run of Signal. 100 00:08:02,641 --> 00:08:07,461 And we've designed Signal in a way to actually keep this agent durable. 101 00:08:07,461 --> 00:08:18,401 So that means that either if the sandbox fails or if the harness fails, then it shouldn't have to start everything from scratch in the middle of the run. 102 00:08:18,401 --> 00:08:30,221 It should actually just be able to pick up from the step that it failed and all of the state so-called is kind of saved off in kind of a memory outside of the sandbox. 103 00:08:30,221 --> 00:08:38,361 And so we've designed this to actually be a very durable agent so that if it fails, it can spin it back up. 104 00:08:39,201 --> 00:08:43,701 You can launch a new sandbox, launch a new harness to kind of pick back up where it started. 105 00:08:43,961 --> 00:08:50,041 So all of this is essentially the underpinnings of how we built signal to run every single run. 106 00:08:50,301 --> 00:08:54,341 Agent gets triggered, agent gets started, loaded with all the context, the evidence. 107 00:08:54,981 --> 00:08:56,881 It sweeps through the telemetry data. 108 00:08:57,081 --> 00:08:58,781 It either finds something or it doesn't. 109 00:08:58,781 --> 00:09:00,181 All of that gets saved to memory. 110 00:09:00,481 --> 00:09:03,181 If something fails in the middle, the agent can get reset up. 111 00:09:04,141 --> 00:09:09,461 And we built Signal to also be agnostic of the model or the harness. 112 00:09:09,741 --> 00:09:13,401 So today, Signal supports not just Claude Code, which is what we started with, 113 00:09:13,481 --> 00:09:16,181 but it also supports Cursor and Codex. 114 00:09:16,341 --> 00:09:20,581 And the goal is to actually support all of the major, even the open source harnesses out there. 115 00:09:20,581 --> 00:09:27,581 and Signal actually also supports multiple models. 116 00:09:29,561 --> 00:09:31,481 So even if you're using Claude Code, 117 00:09:31,801 --> 00:09:34,741 you can pick any of the models inside the Anthropic family. 118 00:09:35,921 --> 00:09:38,321 And this is really important to us 119 00:09:38,321 --> 00:09:40,881 because one, it's an awesome way for us 120 00:09:40,881 --> 00:09:42,761 to evaluate the quality of the agent 121 00:09:43,881 --> 00:09:45,421 every time we spin it up. 122 00:09:45,541 --> 00:09:48,061 It helps us control costs so we can see 123 00:09:48,061 --> 00:09:54,741 as we use more powerful models, the better quality maybe the PRs are, but can we reduce 124 00:09:54,741 --> 00:10:03,041 the cost and actually still maintain the rigor and maintain the quality of issues that are 125 00:10:03,041 --> 00:10:08,361 actually detected by signal? So this is something we deeply thought about from first principles 126 00:10:08,361 --> 00:10:14,401 when we were designing signal. And this is actually a pretty good segue into what does 127 00:10:14,401 --> 00:10:19,841 an AI engineer need to do? Well, an AI engineer really needs to kind of learn how to write evals. 128 00:10:20,061 --> 00:10:25,301 And so we actually, every time Signal runs, Signal finds an issue, it puts up a PR. Well, 129 00:10:25,401 --> 00:10:32,701 there's evals that are run on top of the agent to test that PR. And an eval can be as simple as 130 00:10:34,001 --> 00:10:38,781 testing something as part of your development workflow. You have a data set. You can be a 131 00:10:38,781 --> 00:10:44,501 simple prompt with outputs that you're testing to make sure that there's no regressions. 132 00:10:45,541 --> 00:10:52,721 But these evals can also, they need to be kind of aligned with human judgment. And I'd argue that 133 00:10:52,721 --> 00:10:58,661 this is actually probably where an AI engineer should be spending a lot of their time as they're 134 00:10:58,661 --> 00:11:03,501 building the system to actually run the loop. They should be actually spending time looking at the 135 00:11:03,501 --> 00:11:09,921 traces with their agents, identifying what the common errors are. And there's a lot of skills. 136 00:11:10,061 --> 00:11:14,561 I think I was hearing Lance yesterday talk about the Claude skill to build evals. We have a skill 137 00:11:14,561 --> 00:11:21,221 with Arize Evaluator to actually help you build and write evals, as well as align those evals 138 00:11:21,221 --> 00:11:28,561 with the human judgment. But this is probably where the shift to defining what is good becomes 139 00:11:28,561 --> 00:11:32,021 really, really important for an AI engineer to actually do. 140 00:11:32,461 --> 00:11:35,961 And for a lot of teams I think we speak with and we work with, 141 00:11:36,141 --> 00:11:38,961 their question becomes, well, how do I trust the evaluator? 142 00:11:39,101 --> 00:11:47,241 And do I really trust that the eval is going to catch? 143 00:11:48,741 --> 00:11:50,901 How do I trust that it's as good as my judgment? 144 00:11:51,161 --> 00:11:54,441 And I'd say what we spend a lot of our own time doing for our own products 145 00:11:54,441 --> 00:11:57,701 is that we actually evaluate the evaluators. 146 00:11:58,561 --> 00:12:05,781 We spend a decent amount of time on actually building these data sets, golden truth data sets. 147 00:12:05,961 --> 00:12:14,881 We run evals, verify them against the human output, and then try to align to make sure that the judge actually agrees with the human 148 00:12:15,881 --> 00:12:20,841 and they're in agreement before we actually ship out a new evaluator. 149 00:12:21,001 --> 00:12:27,541 And these evals can then be used both in an offline testing as well as in an online testing manner. 150 00:12:28,981 --> 00:12:32,061 And this is probably what I would say as an AI engineer, 151 00:12:32,221 --> 00:12:35,401 is more and more of what the job is becoming. 152 00:12:37,361 --> 00:12:40,021 And so what I'm going to leave you guys with is, 153 00:12:40,161 --> 00:13:05,001 as an AI engineer, you've built this agent. You've built this agent to actually run, be always on. You run evals before you actually ship it into production. You can ship it. And then once it goes into production, the goal is to actually trace everything. And tracing everything now means not just tracing the agents, but also tracing the coding agents behind the agents that build 154 00:13:05,021 --> 00:13:12,421 built this. So you can trace Claude Code, you can trace Cursor, you can trace Codex, but 155 00:13:12,421 --> 00:13:17,941 all these coding agents can be traced so you can understand what happened in the factory 156 00:13:17,941 --> 00:13:23,581 of agents that you all are building. And then these agents, all the traces can then also 157 00:13:23,581 --> 00:13:29,881 be evaluated online. So there's an offline evaluation when you're actually testing what 158 00:13:29,881 --> 00:13:31,441 a new change is. 159 00:13:31,581 --> 00:13:36,981 But as it goes into production, these become your online evals that you're running on top 160 00:13:36,981 --> 00:13:38,141 of the telemetry data. 161 00:13:38,401 --> 00:13:45,681 This, for us, ends up becoming the new data that then gets pulled back into the agents 162 00:13:45,681 --> 00:13:52,181 to read the telemetry data, the evals, and surface up problems inside of the traces. 163 00:13:52,181 --> 00:14:06,841 So the system really becomes how do you go from collect data, eval that data, run agents on top of it so that you can find problems and figure out where the gaps are to actually improve your systems. 164 00:14:07,501 --> 00:14:13,621 So if I go back to kind of the question that we started this whole thing with, it's what does an AI engineer do? 165 00:14:13,901 --> 00:14:21,501 I mean, the gap between, I think there's a pretty big gap between what's happening in the labs today and what's happening with a lot of businesses. 166 00:14:21,501 --> 00:14:30,241 And some of it could just be the data inside of businesses are a lot more messy than what is very constrained and testable inside of a lab. 167 00:14:30,461 --> 00:14:34,921 But AI engineers are kind of the role to close that gap. 168 00:14:35,081 --> 00:14:45,861 And I think it really starts with tracing everything, tracing the agents you're actually building, running evals, being able to learn how to test these products that we're really building. 169 00:14:45,861 --> 00:14:50,101 and then how do you build a system so that agents can actually improve 170 00:14:51,041 --> 00:14:53,361 from the telemetry that's actually being collected 171 00:14:54,241 --> 00:14:57,101 so that you're no longer just building individual features 172 00:14:57,101 --> 00:15:02,221 but you're actually building the system to improve your products. 173 00:15:02,681 --> 00:15:04,381 If you want to learn more, come check out. 174 00:15:04,441 --> 00:15:04,961 We have a booth. 175 00:15:05,201 --> 00:15:07,021 You can check out Signal, learn more. 176 00:15:07,361 --> 00:15:09,161 But thank you so much for listening to us.