1 00:00:03,974 --> 00:00:07,154 Hello everyone, welcome. 2 00:00:07,894 --> 00:00:09,674 My name is David. 3 00:00:10,154 --> 00:00:13,354 I lead AI operations at a company called Revolut. 4 00:00:14,514 --> 00:00:17,334 We have 15 minutes today together. 5 00:00:17,894 --> 00:00:24,274 we'll talk about high performance engineering and self-hosting open source model on dedicated GPU. 6 00:00:24,474 --> 00:00:25,034 How does that sound? 7 00:00:27,134 --> 00:00:27,654 Cool. 8 00:00:28,094 --> 00:00:30,054 So it's going to be a technical one, so bear with me. 9 00:00:31,694 --> 00:00:36,394 So maybe to start off, because I know this is the US and you may not know Revolut, 10 00:00:37,274 --> 00:00:38,514 quick words about our company. 11 00:00:39,754 --> 00:00:41,834 We're a digital bank in Europe. 12 00:00:41,834 --> 00:00:43,374 We have 80 million customers. 13 00:00:43,954 --> 00:00:45,994 We're live in 45 markets. 14 00:00:47,334 --> 00:00:50,834 with the most downloaded financial app in Europe for the last year. 15 00:00:51,734 --> 00:00:55,974 And as of July, we're Europe's largest private company. 16 00:00:57,734 --> 00:00:59,254 So we're doing okay. 17 00:01:02,074 --> 00:01:05,314 The emphasis of Revolut is shipping a lot of product, 18 00:01:05,414 --> 00:01:10,114 and so there's a high focus on engineering culture and excellence on that demand. 19 00:01:11,554 --> 00:01:12,814 And so we'll talk about that. 20 00:01:14,074 --> 00:01:16,334 So let's talk about some AI. 21 00:01:18,194 --> 00:01:19,394 We have a chart here. 22 00:01:19,694 --> 00:01:20,794 I'm going to see two lines. 23 00:01:20,974 --> 00:01:25,154 This is a sample of 2,000 engineers' performance over the last year. 24 00:01:26,714 --> 00:01:28,854 So the green line is going to be lead time. 25 00:01:29,094 --> 00:01:32,094 So it's going to be when an engineer picks up a feature 26 00:01:32,094 --> 00:01:35,854 until a feature is merged and ready to be deployed. 27 00:01:36,814 --> 00:01:38,674 The blue line is going to be coding time. 28 00:01:38,814 --> 00:01:39,874 So the green line is in days. 29 00:01:40,034 --> 00:01:41,914 The blue line is in hours. 30 00:01:43,774 --> 00:01:45,914 Just to reset the timeline in case people forget, 31 00:01:45,914 --> 00:01:47,634 because AI moves really fast. 32 00:01:48,654 --> 00:01:52,374 Claude Code came out around February 2025, right? 33 00:01:52,714 --> 00:01:55,134 And that's the Claude Code, Sonnet 3.7, 34 00:01:55,274 --> 00:01:57,834 the one that would delete your prod database, right? 35 00:01:57,954 --> 00:02:00,354 Not the safe cloud and codex that we have today. 36 00:02:00,894 --> 00:02:02,754 So bear in mind when we talk about adoption. 37 00:02:04,654 --> 00:02:07,874 So what you see on this chart is January 25, 38 00:02:08,274 --> 00:02:10,174 first introduced AI in the company, right? 39 00:02:10,694 --> 00:02:11,194 Gemini. 40 00:02:11,614 --> 00:02:15,134 First time people were able to use private data within AI. 41 00:02:16,974 --> 00:02:20,834 Around February, we started playing around with Claude Code, and then January is when 42 00:02:20,834 --> 00:02:23,314 we first launched Claude Code at scale. 43 00:02:24,834 --> 00:02:30,934 What you see is Gemini had 10% improvement in lead time, 20% improvement in coding time, 44 00:02:31,054 --> 00:02:35,034 so that's people no longer need to look at Stack Overflow and reading documentation. 45 00:02:36,294 --> 00:02:39,614 But then Claude Code, not so much impact at the time. 46 00:02:40,534 --> 00:02:45,214 So when we launched it, we were highly skeptical about the cost of AI. 47 00:02:45,214 --> 00:02:48,074 we were seeing engineers spend a thousand a month, 48 00:02:48,214 --> 00:02:49,994 10,000 a month with not so much ROI. 49 00:02:50,594 --> 00:02:51,754 And so when we launched it, 50 00:02:51,874 --> 00:02:54,614 we launched it with strict budget limits. 51 00:02:55,794 --> 00:02:57,314 And so we're looking at the next thing, right? 52 00:02:57,394 --> 00:03:00,434 So how can we provide limited tokens to our engineers 53 00:03:00,434 --> 00:03:03,834 so they can use AI guild-free for a fraction of the cost? 54 00:03:04,034 --> 00:03:05,854 And so we looked into open source. 55 00:03:07,214 --> 00:03:09,654 So around April 26 is when we started 56 00:03:09,654 --> 00:03:11,454 self-hosting open source model. 57 00:03:11,674 --> 00:03:13,234 We're a big fan of GLM for coding. 58 00:03:14,094 --> 00:03:15,534 so around that time what you see 59 00:03:15,534 --> 00:03:17,754 April 26th, GLM 5.1 60 00:03:19,614 --> 00:03:21,174 we see 60% improvement 61 00:03:21,174 --> 00:03:22,954 in coding time 62 00:03:22,954 --> 00:03:25,734 70% improvement in lead time 63 00:03:25,734 --> 00:03:27,834 so traumatic impact 64 00:03:28,814 --> 00:03:29,874 what do we learn from this? 65 00:03:30,094 --> 00:03:31,514 well there's more impact on the lead time 66 00:03:31,514 --> 00:03:32,294 than on the coding time 67 00:03:32,294 --> 00:03:35,434 so what we find is the bottleneck for engineering 68 00:03:35,434 --> 00:03:36,914 is not necessarily coding 69 00:03:37,914 --> 00:03:39,234 you can save a lot of time 70 00:03:39,234 --> 00:03:40,934 not having to dig through your inbox 71 00:03:40,934 --> 00:03:43,054 or messing with Jira and Confluence 72 00:03:43,054 --> 00:03:48,534 and so when you have a limited token you're free to do that. You no longer have to save 73 00:03:48,534 --> 00:03:55,814 your tokens for the end of the quarter or the most difficult task. So where do we stand? 74 00:03:56,674 --> 00:04:02,694 Well open source is now a priority, a revolute, and your economics are favorable, the billing 75 00:04:02,694 --> 00:04:08,214 is flat, we don't have any bill shock any of the month, and the frontier is the exception. 76 00:04:08,214 --> 00:04:10,514 So obviously we keep using codex, code code. 77 00:04:11,294 --> 00:04:12,334 The latest model are fantastic. 78 00:04:13,114 --> 00:04:14,934 Jimmy and I are going very good as well. 79 00:04:15,974 --> 00:04:18,274 But we keep them as exception. 80 00:04:20,254 --> 00:04:21,314 So how do we get there? 81 00:04:23,494 --> 00:04:26,154 Some metrics that matter when you're self-hosting model. 82 00:04:27,274 --> 00:04:31,954 So to serve 2,000 engineers, we roughly need 56 B200. 83 00:04:32,474 --> 00:04:34,614 And that's to host GLM 5.3. 84 00:04:34,794 --> 00:04:36,914 GLM is a big model, right? 85 00:04:36,914 --> 00:04:38,134 8.500 billion parameters. 86 00:04:39,974 --> 00:04:42,174 What you want to aim for is 87 00:04:42,174 --> 00:04:44,734 sub-3 second time to first token. 88 00:04:45,874 --> 00:04:47,394 This is empirical evidence. 89 00:04:47,654 --> 00:04:49,294 As soon as you exceed 5 seconds, 90 00:04:50,514 --> 00:04:51,894 my Slack inbox starts to blow up, 91 00:04:51,954 --> 00:04:52,894 and so we know that's the limit. 92 00:04:54,794 --> 00:04:55,634 In terms of throughput, 93 00:04:55,754 --> 00:04:56,854 it doesn't really matter as much. 94 00:04:57,034 --> 00:04:59,294 We serve, in terms of interactivity, 95 00:04:59,894 --> 00:05:00,854 200 tokens per second. 96 00:05:01,474 --> 00:05:02,214 People are fine. 97 00:05:02,314 --> 00:05:02,914 You don't need to go higher. 98 00:05:04,274 --> 00:05:05,634 The way people work has changed. 99 00:05:05,634 --> 00:05:08,754 which we no longer look at the stream of thinking tokens. 100 00:05:08,974 --> 00:05:10,394 People run tasks in parallel 101 00:05:10,394 --> 00:05:11,614 and check in when the task is done. 102 00:05:13,094 --> 00:05:14,274 But if you're in a high throughput 103 00:05:14,274 --> 00:05:15,614 and you're an engineer at Revolut, 104 00:05:15,894 --> 00:05:17,954 you just connect after 5 p.m. when one is on GPU 105 00:05:18,534 --> 00:05:22,234 you can get 800 TPS and then, you know, it's a dream. 106 00:05:24,314 --> 00:05:25,914 In terms of cost, in terms of impact, 107 00:05:26,294 --> 00:05:28,674 self-hosting, GLM, or blended cost of token 108 00:05:28,674 --> 00:05:30,074 is 30 cents per million. 109 00:05:30,774 --> 00:05:33,094 And so our invoice is one-tenth 110 00:05:33,094 --> 00:05:38,054 of what we were spending on Anthropic and OpenAI. 111 00:05:41,334 --> 00:05:43,054 So there's a lot of tuning 112 00:05:43,054 --> 00:05:45,454 that goes into running open source models. 113 00:05:46,894 --> 00:05:52,434 If you just get GPUs and upload weights on a GPU 114 00:05:52,434 --> 00:05:54,894 and you start doing some inference, 115 00:05:55,134 --> 00:05:56,414 you're going to get terrible performance. 116 00:05:56,514 --> 00:05:58,934 That's because your workload is going to be very dependent 117 00:06:00,494 --> 00:06:01,954 on the parameters that you set. 118 00:06:02,754 --> 00:06:06,834 So if you look all the way to the left of that chart, the blue line is going to be demand, 119 00:06:07,034 --> 00:06:07,894 so that's request per second. 120 00:06:08,414 --> 00:06:11,674 That's going to be what's going to be the bottleneck for your latency for your GPUs. 121 00:06:13,014 --> 00:06:16,054 And the red line is what we're optimizing for, that's the time to first token. 122 00:06:17,694 --> 00:06:23,614 So when we started back in April, you could see request per second around 3, 123 00:06:23,814 --> 00:06:29,214 which is high but not so high, and time to first token over 10 seconds, which is very slow. 124 00:06:30,674 --> 00:06:32,454 All of the data that you see is 125 00:06:32,634 --> 00:06:36,354 the same number of GPU. We did an increase GPU count with LLF56. 126 00:06:37,494 --> 00:06:44,154 And by the end, you see in August, requests per second were over 7, and time to first occur, and we're around 1.6. 127 00:06:44,454 --> 00:06:50,694 So we've had 88% improvement in performance for 76% increased demand. 128 00:06:51,794 --> 00:06:57,254 So the learning from this is there's a lot of parameters that you can improve on, and you really need to resist scaling. 129 00:06:57,454 --> 00:06:59,674 if you're looking to self-host a model. 130 00:07:01,194 --> 00:07:02,474 So let's talk parameters. 131 00:07:04,114 --> 00:07:05,594 There's no secret recipe, 132 00:07:05,694 --> 00:07:07,994 but the things you consider are the following. 133 00:07:09,974 --> 00:07:12,674 First one, long context is a scam. 134 00:07:13,194 --> 00:07:15,214 You really need to be mindful that 135 00:07:15,214 --> 00:07:16,674 even though you can serve one million token, 136 00:07:18,154 --> 00:07:19,794 256 is really what you need. 137 00:07:22,334 --> 00:07:25,094 Quantization, if you're running Blackwell generation GPU, 138 00:07:25,094 --> 00:07:27,774 you don't want to run anything higher than FP4. 139 00:07:28,334 --> 00:07:32,594 And the FP4 is best, roughly going from FP4 to FP8 140 00:07:33,134 --> 00:07:38,594 is a 40% saving in GPU footprint for zero loss in precision. 141 00:07:40,634 --> 00:07:43,054 Other parameters to consider, tensor parallelism, 142 00:07:43,314 --> 00:07:47,434 even though GLM fits on TP4, you don't have enough KV 143 00:07:47,434 --> 00:07:49,134 cache, and so maybe a TP8 works better, 144 00:07:49,294 --> 00:07:50,914 depending on how large your workload is. 145 00:07:52,474 --> 00:07:53,874 and then on the more esoteric 146 00:07:54,654 --> 00:07:55,654 check your cache hit 147 00:07:55,654 --> 00:07:57,914 coding test needs to be above 90% 148 00:07:57,914 --> 00:07:58,774 cache hit 149 00:08:00,374 --> 00:08:01,914 and you can look into optimize 150 00:08:02,554 --> 00:08:03,514 spec decoding 151 00:08:03,514 --> 00:08:05,774 sequential pre-fill, PD disaggregation 152 00:08:05,774 --> 00:08:06,514 all of these 153 00:08:09,314 --> 00:08:09,794 cool 154 00:08:09,794 --> 00:08:12,274 so now let's talk about production setup 155 00:08:13,534 --> 00:08:14,314 GLM obviously 156 00:08:14,314 --> 00:08:15,934 is a text only model 157 00:08:15,934 --> 00:08:17,534 which is a limitation for some people 158 00:08:18,634 --> 00:08:19,954 the trade off I would say 159 00:08:19,954 --> 00:08:23,714 we would make every time is get the smarter model and figure out how to do 160 00:08:23,714 --> 00:08:28,874 image recognition another way. You can easily have a local router, open codex is 161 00:08:28,874 --> 00:08:35,494 one, where you would route any image recognition to Gemini or Luna and keep 162 00:08:35,494 --> 00:08:43,594 GLM for the main task. If you really need some vision model, GLM 5G flash works 163 00:08:43,594 --> 00:08:52,974 really well. We have our UI designer making great use of it. Turnaround for UI product has gone a 164 00:08:52,974 --> 00:09:01,514 really, really long way. And then the last one is going to be about the harness. So all of these 165 00:09:01,514 --> 00:09:07,074 endpoints are going to be OpenAI compatible API with response API. And so you can use them 166 00:09:07,934 --> 00:09:12,214 inside OpenCode Codex, Codex app as well, works very well. 167 00:09:12,574 --> 00:09:14,394 And so that's what we recommend. 168 00:09:19,274 --> 00:09:24,054 And then last message, this one is for the CIO in the room. 169 00:09:24,954 --> 00:09:29,454 There's been a lot of stigma in using Chinese open source model. 170 00:09:30,434 --> 00:09:32,734 So the thing to consider is you're not using 171 00:09:33,674 --> 00:09:36,014 inference, dedicated endpoint that goes to China. 172 00:09:36,014 --> 00:09:39,914 you're just getting the weights and putting them on a US-based service. 173 00:09:41,114 --> 00:09:44,474 So for data sharing side, this is totally fine. 174 00:09:46,434 --> 00:09:50,434 Revolut in particular, because we're an EU-regulated bank, 175 00:09:50,654 --> 00:09:54,314 we fall under the most strict regime in terms of compliance. 176 00:09:55,674 --> 00:10:02,214 so what we find that works is all of these coding tasks are not considered autonomous agents. 177 00:10:02,694 --> 00:10:04,514 They're always with a human in the loop. 178 00:10:04,514 --> 00:10:06,954 And so how do you give reassurance 179 00:10:06,954 --> 00:10:07,874 that you have a human loop? 180 00:10:09,394 --> 00:10:12,674 You start by creating a sandbox that guarantees that. 181 00:10:13,054 --> 00:10:16,174 And the sandbox means now if you've used Codex Claude Code, 182 00:10:16,214 --> 00:10:18,314 it has all of these features of approve for me. 183 00:10:18,974 --> 00:10:21,454 You just have to set the rules for the operations 184 00:10:21,454 --> 00:10:22,374 that you don't want to allow. 185 00:10:22,634 --> 00:10:25,794 So you can allow grep but not gradle, for instance. 186 00:10:26,074 --> 00:10:28,714 And that's gonna give confidence to your regulator 187 00:10:28,714 --> 00:10:31,314 that it's not an autonomous agent 188 00:10:31,314 --> 00:10:32,914 that needs different supervision. 189 00:10:34,874 --> 00:10:37,594 This sandbox also protects you from rogue use. 190 00:10:37,794 --> 00:10:41,494 You may have bad actors inside your company, and you want to shield for that. 191 00:10:42,174 --> 00:10:44,214 You'd be surprised what you find in a large company. 192 00:10:45,874 --> 00:10:47,634 And then the last one is prompt injection. 193 00:10:47,934 --> 00:10:52,714 Not so much a thing now, given how smart models are, but always a thing. 194 00:10:54,334 --> 00:10:56,214 In terms of data privacy, we spoke about it. 195 00:10:56,454 --> 00:11:01,674 As long as the data sits in one of the countries that are in the data equivalency framework, 196 00:11:01,674 --> 00:11:08,934 so US, EU, UK, Israel, you're fine, you can get your GPUs there. All of your inference provider 197 00:11:08,934 --> 00:11:16,054 or GPU provider are going to have zero data retention by default. And then the last thing 198 00:11:16,054 --> 00:11:23,494 you need is a very strong open telemetry so that if you ever have an incident, you need to be able 199 00:11:23,494 --> 00:11:25,754 to go deep and look at the logs. 200 00:11:26,894 --> 00:11:29,234 Also gives you some analytics. 201 00:11:32,914 --> 00:11:34,014 All right, thanks everyone. 202 00:11:34,234 --> 00:11:34,934 That was all for my talk.