1
00:00:00,201 --> 00:00:10,701
So I really love this article from Hannes at DuckDB, where he asks if we lost a decade chasing distributed architectures that presumed compute and storage were slow and expensive.

2
00:00:11,361 --> 00:00:17,221
Capability in every era, though, defines architecture and tooling, and the tooling lags behind consistently.

3
00:00:17,621 --> 00:00:23,441
We first made compute cheaper than storage, and then we gave it to everybody, and then we made it faster.

4
00:00:23,781 --> 00:00:27,341
And every single time, architectures undergo this massive change.

5
00:00:27,881 --> 00:00:33,261
All of our data infrastructure today is built on the idea that transformation is expensive.

6
00:00:33,721 --> 00:00:35,421
What if that wasn't true anymore?

7
00:00:36,781 --> 00:00:42,941
I believe we've solved the final bottleneck in transformation, general intelligence or judgment,

8
00:00:42,941 --> 00:00:50,521
and we've made it accessible on tap all along the Pareto frontier, which puts us right on the verge of a paradigm shift.

9
00:00:50,721 --> 00:00:54,561
So in this talk, we're going to cover why single ontologies always fail,

10
00:00:54,561 --> 00:00:55,021
the

11
00:00:55,141 --> 00:01:07,801
three roles models play in pipelines, intersecting model priors for better results, and Pareto surfing, how today's models can tune a workload for cost, speed, and accuracy all at the same time.

12
00:01:08,461 --> 00:01:09,381
Here's a quick example.

13
00:01:09,761 --> 00:01:20,541
When Jeff launched, we already had an entity resolution pipeline running, working out Ohio's campaign finance records and whether they belong to the same person, company, or committee.

14
00:01:20,701 --> 00:01:23,261
It was projected to cost about $10,000.

15
00:01:23,881 --> 00:01:30,681
Our systems took that new model, worked out where it fit, tested over a dozen approaches, and integrated it.

16
00:01:30,681 --> 00:01:37,801
Ten days after that first call, we had final ground truth for every donor case in Ohio under $800.

17
00:01:38,481 --> 00:01:42,661
That is the power of Pareto surfing and just-in-time transformations.

18
00:01:43,161 --> 00:01:44,921
So, let's get into it.

19
00:01:47,941 --> 00:01:51,641
Ontologies are the enterprise version of mind maps.

20
00:01:51,961 --> 00:01:55,661
It's this tar pit that sucks us all in, including me.

21
00:01:55,661 --> 00:02:01,201
I talked about a single universal schema when I founded Southbridge two years ago.

22
00:02:01,661 --> 00:02:09,621
The popular conception, and it's not wrong, it's just a small part of the picture, is that migrations and ontologies fail because of a lack of buy-in.

23
00:02:09,761 --> 00:02:13,841
And they're abandoned halfway through, that organizations and people never switch over.

24
00:02:14,221 --> 00:02:16,841
All of that is true, 100%.

25
00:02:16,841 --> 00:02:19,461
But buy-in isn't the primary reason.

26
00:02:20,101 --> 00:02:25,781
if you could integrate and migrate all of your information into a single super schema in the blink of an eye,

27
00:02:26,001 --> 00:02:30,521
you still have to contend with the truth that there is no super schema.

28
00:02:30,901 --> 00:02:32,841
There actually are four schemas.

29
00:02:33,121 --> 00:02:37,381
The first one belongs to the source, wherever the information comes from.

30
00:02:38,121 --> 00:02:39,461
Let's take temperature as an example.

31
00:02:39,641 --> 00:02:41,681
Say I ask you to write down the temperature.

32
00:02:42,001 --> 00:02:45,021
The source schema is whatever you have access to.

33
00:02:45,021 --> 00:02:51,181
Your phone gives you your region's temperature, sensor on your desk, which is less accurate, but it's more local, or your watch.

34
00:02:51,461 --> 00:02:56,401
But even your patience in following through on my request is part of that source schema.

35
00:02:56,801 --> 00:02:59,441
The second one is the storage schema.

36
00:03:00,321 --> 00:03:02,921
There's only so much you can write down.

37
00:03:03,101 --> 00:03:05,881
You can't write down the firmware of the device you're using.

38
00:03:06,061 --> 00:03:11,441
You can't even write the time down to the second because you couldn't finish writing all of that inside of a second.

39
00:03:11,761 --> 00:03:14,201
This is the schema imposed on you by storage.

40
00:03:14,781 --> 00:03:17,481
The third one belongs to the decision.

41
00:03:17,941 --> 00:03:20,221
Why do I need you to write the temperature down?

42
00:03:20,441 --> 00:03:22,501
Maybe I'm figuring out what makes you comfortable,

43
00:03:22,501 --> 00:03:25,801
in which case I need temperatures from you spread throughout the day,

44
00:03:25,841 --> 00:03:28,241
and I need to make sure that it's you that's doing the measuring.

45
00:03:28,781 --> 00:03:30,701
Maybe I'm building weather migration maps,

46
00:03:30,761 --> 00:03:32,641
in which case I don't care who does the measuring,

47
00:03:32,761 --> 00:03:35,761
but I care a lot about your exact location,

48
00:03:36,221 --> 00:03:39,121
the time, the device you used, and whether that's changed.

49
00:03:39,661 --> 00:03:44,081
Or maybe I'm deciding whether to install sensitive equipment and where,

50
00:03:44,201 --> 00:03:47,981
and you yourself become this variable that I need to cancel out.

51
00:03:48,641 --> 00:03:51,981
There's a fourth one, though, which we'll get to when we talk about priors.

52
00:03:52,161 --> 00:03:53,241
It's a recent addition.

53
00:03:53,721 --> 00:03:59,261
The true problem is that you will never know all four schemas at the same time,

54
00:03:59,361 --> 00:04:01,761
except at the moment that the decision is being made.

55
00:04:02,081 --> 00:04:04,361
In almost every enterprise and every home,

56
00:04:04,641 --> 00:04:08,361
most of the data being recorded has little idea what it's going to be used for.

57
00:04:08,661 --> 00:04:12,041
Take a customer, the first box in every ontology.

58
00:04:12,041 --> 00:04:19,181
to billing a customer is whoever pays, to support it's whoever calls, to compliance it's whoever

59
00:04:19,181 --> 00:04:24,561
we're on the hook for. And the only time you know is when the decision is known. Should we refund

60
00:04:24,561 --> 00:04:30,701
this order? Can we extend this account more credit? Suddenly, what you need to know, how and to what

61
00:04:30,701 --> 00:04:38,761
degree, becomes extremely clear. This is why I was wrong. There is no universal schema. There is no

62
00:04:38,761 --> 00:04:43,661
concept of a customer that's divorced from the question that you want to answer. And pulling

63
00:04:43,661 --> 00:04:48,321
these schemas together is the process of transformation, of making information actually

64
00:04:48,321 --> 00:04:54,081
useful, taking the context that it was recorded in and merging it with the context of the decision.

65
00:04:54,801 --> 00:05:00,901
And that takes discernment. Doing that merge when the decision space is known instead of ahead of

66
00:05:00,901 --> 00:05:06,841
time is what I mean by a just-in-time ontology. So we'll come back to discernment with a story

67
00:05:06,841 --> 00:05:12,181
from my last job where most of my job was to make sure that certain numbers just did not exist ever.

68
00:05:14,521 --> 00:05:21,941
Before that, a little bit about me and us. We are best described as obsessives. Obsessives who

69
00:05:21,941 --> 00:05:27,841
embed intelligence into pipelines. Pipelines that can scale, that can run over and over again to take

70
00:05:27,841 --> 00:05:33,541
work truly off our hands, and pipelines that can mature in cost, performance, and speed all on

71
00:05:33,541 --> 00:05:40,481
their own. And most of the work we do in data breaks into three regimes. The first is write-ones,

72
00:05:40,781 --> 00:05:46,201
run-ones. If you do any kind of development, think of these as monster-scale immediately

73
00:05:46,201 --> 00:05:51,021
invoked functions. They're built for something single-use, like joining a data set, mining

74
00:05:51,021 --> 00:05:58,021
information, or answering one specific question, and then they get torn down. The second is write-many

75
00:05:58,021 --> 00:06:03,801
run once, where lots of smaller and larger workloads are built for one large common goal.

76
00:06:04,041 --> 00:06:09,261
The closest thing you might know is large migrations. Years of records, moved out of

77
00:06:09,261 --> 00:06:14,921
systems nobody fully remembers, lots of workloads built for one move and verified individually at

78
00:06:14,921 --> 00:06:20,121
source, checked against the others, because all of it only happens once. The third though, and this

79
00:06:20,121 --> 00:06:25,941
is bread and butter, is the most common, is write many, run many. These are repeatable pipelines

80
00:06:25,941 --> 00:06:29,921
that are built and maintained over time as we add more information in.

81
00:06:30,121 --> 00:06:33,821
And at Southbridge, we've moved so much work off the main thread

82
00:06:33,821 --> 00:06:38,041
and into these repeatable systems from benchmark design, tracking, coding,

83
00:06:38,421 --> 00:06:40,661
to making subtitles for movies that we want to watch.

84
00:06:41,461 --> 00:06:45,421
Almost everything we do in the modern world is a transformation of data,

85
00:06:45,541 --> 00:06:48,201
or it's bottlenecked by how hard it is to transform data.

86
00:06:48,481 --> 00:06:51,001
And in all of these cases, our job is simple,

87
00:06:51,161 --> 00:06:53,261
make sure people can rely on the results.

88
00:06:53,761 --> 00:06:59,381
In just two years, we've gone from figuring out how to make early 3.5 turbo level models

89
00:07:00,101 --> 00:07:05,381
reliable on things like healthcare, aging, mining research, to building Hankweave, our

90
00:07:05,381 --> 00:07:09,681
open source harness for data work, and then using that to get models to work together

91
00:07:09,681 --> 00:07:10,661
at terabyte scale.

92
00:07:11,221 --> 00:07:13,041
Today, we have systems that can self-optimize.

93
00:07:13,521 --> 00:07:19,141
You give them the problem, a way to fail safely, a place to work, and they find the shape of

94
00:07:19,141 --> 00:07:19,601
the solution.

95
00:07:19,821 --> 00:07:21,121
They pick the right models.

96
00:07:21,281 --> 00:07:22,241
They run the pilots.

97
00:07:22,401 --> 00:07:23,161
They scale up.

98
00:07:23,261 --> 00:07:26,721
And they Pareto-surfed their way to the best option they can find.

99
00:07:29,901 --> 00:07:37,861
So, story, at my last job, the question we heard most often was, can we make this decision data-led?

100
00:07:38,361 --> 00:07:42,541
And sometimes that was the right question when we had thousands of vessels on the platform.

101
00:07:42,741 --> 00:07:46,641
The problem is that we also asked it when we only had 50 users.

102
00:07:46,981 --> 00:07:52,841
And if you know statistics, you can explain really well why A-B testing 50 users is a bad idea.

103
00:07:52,841 --> 00:07:56,581
Not because you're not going to get a result, but because you will.

104
00:07:57,181 --> 00:08:04,961
So the only real option was to not produce a number at all, because the moment a number exists, it argues for a result.

105
00:08:05,601 --> 00:08:07,401
Today, though, it is worse.

106
00:08:08,061 --> 00:08:10,561
Interactive agents will always give you an answer.

107
00:08:10,761 --> 00:08:11,581
They'll give it to you quickly.

108
00:08:11,841 --> 00:08:15,501
You ask for a transformation, whatever, and you get a contrary and yes, sir.

109
00:08:16,061 --> 00:08:18,421
What this means is mistakes add up quickly.

110
00:08:19,241 --> 00:08:23,341
Moment in chat can be a prod nightmare forever, just waiting to be discovered.

111
00:08:23,961 --> 00:08:29,561
The primary reason that this happens is that intelligence, as it relates to data work today,

112
00:08:30,341 --> 00:08:35,161
occupies three separate roles that coding agents kind of mash into one context window.

113
00:08:35,481 --> 00:08:38,681
Let's pull those apart because it'll help a lot when we get to decision models

114
00:08:38,681 --> 00:08:41,201
because they fit into exactly one of these three.

115
00:08:41,761 --> 00:08:44,661
The first one is building and designing the pipeline,

116
00:08:45,001 --> 00:08:48,161
which is breaking down the problem, writing the early specs.

117
00:08:48,821 --> 00:08:53,361
authorizing spent. The second one, and this is usually confused with the first and given to the

118
00:08:53,361 --> 00:08:58,581
same models, is working inside the pipeline, actually doing the work, reading the PDF,

119
00:08:59,101 --> 00:09:03,521
writing to databases, checking results. Before we get to the third one, I want to make it clear

120
00:09:03,521 --> 00:09:09,281
that these two roles pull in very different directions. The first one wants slower models

121
00:09:09,281 --> 00:09:15,321
with heavier, thicker tokens, models that can think more, walk around a problem and see the angles,

122
00:09:15,801 --> 00:09:19,301
digest documentation, understand how these pipelines get built,

123
00:09:19,461 --> 00:09:23,641
and predict how something will do before it even starts, like opus or up.

124
00:09:24,321 --> 00:09:27,881
The models you want inside pipelines, though, are the opposite.

125
00:09:28,201 --> 00:09:32,141
You want them cheaper, faster, specialized, good at following instructions,

126
00:09:32,701 --> 00:09:34,721
repeatable workers with knowable defects.

127
00:09:35,021 --> 00:09:36,061
You see the same thing in nature.

128
00:09:36,481 --> 00:09:40,581
A leafcutter ant colony, for example, runs on four different sizes of worker,

129
00:09:40,821 --> 00:09:42,801
small, medium, high, X-high.

130
00:09:42,801 --> 00:09:50,581
As you watch any system that you design, review, or hear about, listen for what slots and where in this layer cake.

131
00:09:50,821 --> 00:09:51,561
It makes a big difference.

132
00:09:52,121 --> 00:09:59,281
Because we had this distinction, bringing in JEV in this newly exploding class of decision models was very easy because they're workers.

133
00:10:00,001 --> 00:10:05,301
They're built for quick contact with information, and they're deployed at the bottom end of large workloads.

134
00:10:06,381 --> 00:10:07,841
So third category, though.

135
00:10:08,181 --> 00:10:11,241
This is something we call the hill climbers or surfers.

136
00:10:11,241 --> 00:10:14,701
Models that can take a first one and self-optimize.

137
00:10:14,901 --> 00:10:19,161
Given the right knowledge, instructions, and an incredibly stable surface to stand on,

138
00:10:19,381 --> 00:10:25,501
they can try new things, vary budgets to limit cost exposure, debug quickly, take notes,

139
00:10:25,821 --> 00:10:28,921
surf their way, not just to the right solution, but the optimal one.

140
00:10:29,761 --> 00:10:33,641
We call that Pareto surfing, and we'll cover it at the end.

141
00:10:33,901 --> 00:10:36,321
But first, let's talk about our experience with Jeff.

142
00:10:37,321 --> 00:10:42,761
JEV launched on a Tuesday when we were deep into an entity resolution pipeline.

143
00:10:43,041 --> 00:10:45,841
You were working on donors in Ohio's campaign finance.

144
00:10:45,981 --> 00:10:48,321
And at this point, a lot of people have seen this chart.

145
00:10:48,541 --> 00:10:53,361
We talked about the results before and adapted some of the ongoing work to the data set.

146
00:10:53,441 --> 00:10:55,021
So you can download it for yourself, prompts

147
00:10:55,141 --> 00:10:55,801
and everything.

148
00:10:56,201 --> 00:11:05,841
What we didn't talk about is everything behind that one arrow. And what happened when we ran the same idea on the whole state? Here are all of the architectures we tried.

149
00:11:06,021 --> 00:11:18,381
We started with three designs on a small development set, a cheap reasoning model on its own, Jev with a reasoning model for anything it wasn't sure about, and a reasoning model proposing answers with Jev checking them.

150
00:11:18,381 --> 00:11:26,741
On about 200 held-out families, the cheap reasoning model on its own did as well as Fable for about 2% of the cost.

151
00:11:27,361 --> 00:11:34,681
Jeff first got the core answers right, but it missed nine household legs because it had confidently said no to every one of them.

152
00:11:35,081 --> 00:11:39,021
So we rewrote the questions without touching the models or the thresholds.

153
00:11:39,121 --> 00:11:45,441
The household question now said what counts is enough evidence, and Jeff went from no on all 13 development questions to yes.

154
00:11:46,101 --> 00:11:50,541
And as we scaled it statewide, we ran over a dozen rounds of such qualification.

155
00:11:51,061 --> 00:11:56,481
Each round fixed a definition, what a committee is, what a filer means, when two people live

156
00:11:56,481 --> 00:11:57,661
at one address, what does that mean?

157
00:11:58,241 --> 00:12:06,561
Every one of Ohio's 203,317 cases now has a label that says what the answer rests on

158
00:12:06,561 --> 00:12:07,581
and how far it was checked.

159
00:12:08,161 --> 00:12:11,301
This is the hierarchy we discovered for this specific problem.

160
00:12:12,081 --> 00:12:15,641
And here's everything we learned about putting decision models into pipelines.

161
00:12:15,841 --> 00:12:17,941
I'll leave this up so you can screenshot it.

162
00:12:18,281 --> 00:12:19,641
But here's where the money went.

163
00:12:19,841 --> 00:12:25,981
The whole statewide job came to $795 on an $800 budget that was set ahead of time.

164
00:12:26,881 --> 00:12:29,301
Checking cost about four times as much as doing.

165
00:12:29,541 --> 00:12:36,241
$545 on second opinions, settling disagreements, and the audit against 128 for first answers.

166
00:12:36,821 --> 00:12:40,021
And before Jeff, we expected this work to cost about $10,000.

167
00:12:40,021 --> 00:12:45,781
And now that the pipeline is built and verified, we expect the audit budget for the next state to be close to zero.

168
00:12:46,421 --> 00:13:01,081
The most amazing thing is that our agent, 4A, running on Astra and then later on Opus, found most of these approaches incrementally on its own by building the system, running it, debugging it, and trying new things.

169
00:13:01,681 --> 00:13:04,861
It ran for 10 days, tried over a dozen architectures.

170
00:13:05,321 --> 00:13:13,121
Its hanks ran for more than 4,000 times, alongside 90,000 calls to Jeff, tried practically every model on the Pareto frontier.

171
00:13:13,481 --> 00:13:17,541
And the models inside those runs went through roughly 6 billion tokens.

172
00:13:17,881 --> 00:13:22,921
And once all of that was done, we wrote what we learned back into our library of agentic patterns.

173
00:13:23,361 --> 00:13:28,961
This work leaned on these existing ones, and we're adding new tested ones like the checking letter.

174
00:13:30,181 --> 00:13:36,101
This is, to me, the value of taking work off the main thread for humans and for agents.

175
00:13:36,301 --> 00:13:39,741
Because every single one of these iterations was a version-controlled Hank.

176
00:13:39,961 --> 00:13:43,541
They were easy to revert, understand, debug, and critique.

177
00:13:44,181 --> 00:13:46,721
So can models do this natively now?

178
00:13:47,721 --> 00:13:48,481
Not quite.

179
00:13:48,721 --> 00:13:51,381
They're missing four things, which we'll go over.

180
00:13:51,561 --> 00:13:55,641
And there's something that doesn't go away, though, as models get bigger and more capable,

181
00:13:55,941 --> 00:13:56,541
the prior.

182
00:13:57,601 --> 00:14:03,781
Just like humans, models bring their own priors, their beliefs about what should be in the world, to every task.

183
00:14:04,041 --> 00:14:08,321
This is the fourth schema that I mentioned before, the one the model brings with it.

184
00:14:10,481 --> 00:14:13,941
Bigger models, in a strange way, are worse at this sometimes.

185
00:14:14,301 --> 00:14:18,121
They hold on to their beliefs and argue for them even more vehemently.

186
00:14:18,381 --> 00:14:19,821
Here's a few that we ran into.

187
00:14:20,441 --> 00:14:24,141
One pipeline looked for in-kind spelled with an underscore.

188
00:14:24,681 --> 00:14:27,601
The file spelled it with a capital I and a hyphen.

189
00:14:27,921 --> 00:14:33,261
So the model counted 15,658 in-kind contributions as ordinary money.

190
00:14:33,501 --> 00:14:34,521
Didn't double check.

191
00:14:34,781 --> 00:14:37,521
The loan payment repayments also disappeared the same way.

192
00:14:38,141 --> 00:14:42,141
Another one tried four date formats when the file used to fifth.

193
00:14:42,721 --> 00:14:46,141
99.4% of the transaction dates then came out empty.

194
00:14:46,481 --> 00:14:50,241
And every row count and dollar matched, so the agent moved on.

195
00:14:51,341 --> 00:14:58,721
Several runs noticed that loan amounts were negative, decided that the sign was redundant, and stored the absolute value.

196
00:14:59,221 --> 00:15:07,161
A net of $784,000 became $5.3 million of debt in the end, validated by multiple models.

197
00:15:07,501 --> 00:15:12,861
And this is quickly becoming the largest class of error I see across companies at DataWork as models take over.

198
00:15:13,181 --> 00:15:18,201
Sometimes you can solve this just by not vibing, by checking results and applying human intuition.

199
00:15:19,001 --> 00:15:22,481
what you're actually doing there when you do this is interesting.

200
00:15:22,881 --> 00:15:25,701
You're intersecting your priors with the models.

201
00:15:26,021 --> 00:15:30,861
In that first example, any human would catch missing in-kind contributions,

202
00:15:31,001 --> 00:15:33,121
even you campaign finance or think to check.

203
00:15:33,361 --> 00:15:34,221
It's how we found it.

204
00:15:34,701 --> 00:15:38,081
But humans, speaking as a human on this particular task, are expensive.

205
00:15:38,381 --> 00:15:39,441
They are slow.

206
00:15:39,681 --> 00:15:40,881
They pause timelines.

207
00:15:41,241 --> 00:15:44,681
They're even more non-deterministic than models, and they hate interruptions.

208
00:15:45,361 --> 00:15:51,761
but it turns out you can also get really far with cross model prior intersection on our 200 held

209
00:15:51,761 --> 00:16:00,001
out families jev and luna agreed on 194 and none of those shared answers was wrong every error sat

210
00:16:00,001 --> 00:16:04,961
in the six families where they disagreed models from different families different pedigrees tend

211
00:16:04,961 --> 00:16:09,921
to disagree more and have different priors than models in the same group for example the exception

212
00:16:10,421 --> 00:16:14,741
I'm not going to say who are models that are distilled outside the same company.

213
00:16:15,921 --> 00:16:26,661
So with that kind of work, you can shrink the surface quite a bit and the errors along with it down to a much smaller but persistent class of problems that apply to LLMs as a species.

214
00:16:27,181 --> 00:16:35,261
So in one benchmark, four different model families produce the same total down to this end because they all shared a sign convention that the answer key didn't.

215
00:16:35,261 --> 00:16:41,521
And once you get that far, the only way forward is to list them, look out for them, and apply specific solutions.

216
00:16:43,321 --> 00:16:47,961
So going back, what are the four things we're missing to be able to self-optimize?

217
00:16:48,601 --> 00:16:51,261
Number one is a stable deployment surface.

218
00:16:51,561 --> 00:16:58,161
We use Hankweave for this, a harness that takes the fast-moving surface of agent runtimes like CloudCode and Py,

219
00:16:58,161 --> 00:17:03,641
and that exposes a consistent, well-tested log and API interface with checkpointing,

220
00:17:03,741 --> 00:17:07,081
rollback, budget enforcement, everything built in. Something you can build on.

221
00:17:07,621 --> 00:17:12,181
But other things fit this description. To be fair, for headless data work,

222
00:17:12,341 --> 00:17:15,741
Hankweave, I would say, is still the oldest, most stable, and the most open.

223
00:17:16,021 --> 00:17:18,181
I'll leave the skill up if you'd like to try it.

224
00:17:18,781 --> 00:17:24,901
Number two is some way to bound your experiments. You can't test unless you can control cost,

225
00:17:24,901 --> 00:17:32,101
time, and access as resources. The third one is know-how and methodology. We found that this new

226
00:17:32,101 --> 00:17:36,861
class of models can take everything we've learned from two years of deployments, agentic patterns,

227
00:17:37,061 --> 00:17:42,081
debugging techniques, prompting, and apply it to complex problems. With the right abstraction layer,

228
00:17:42,241 --> 00:17:47,841
you can express it very token efficiently and the models can implement it. The fourth and the most

229
00:17:47,841 --> 00:17:53,381
important one is a failure surface. Archimedes said, give me a place to stand and I will move

230
00:17:53,381 --> 00:17:58,061
the earth. And in our case, we haven't met a problem or a constraint that we couldn't iterate

231
00:17:58,061 --> 00:18:03,821
ourselves to solving, given a stable stack to build on and a failure surface to iterate against.

232
00:18:04,261 --> 00:18:09,701
That last one has been the hardest to put into words and into tooling and instructions that helps

233
00:18:09,701 --> 00:18:15,761
models get there. The current best models are still only barely good at this, sparks of intelligence,

234
00:18:16,041 --> 00:18:21,421
so to speak. The whole process, though, involves going through six stages. First, we spec the job

235
00:18:21,421 --> 00:18:27,781
and we split it. For Jeff, that means splitting some 800,000 rows into 200,000 smaller specialized

236
00:18:27,781 --> 00:18:32,981
cases, and each of those cases into typed questions, what kind of donor, one donor,

237
00:18:33,601 --> 00:18:37,441
you know, different, different questions that needed to get answered. Then we decide the failure

238
00:18:37,441 --> 00:18:42,261
surface. In this case, we ran good models on a small slice of the data, with some of it being

239
00:18:42,261 --> 00:18:48,561
manually checked, to set a baseline that we can fail against. In other cases, we've set automated

240
00:18:48,561 --> 00:18:54,141
failure systems, we've set schema checks, we've also covered behavior as a thing that we can

241
00:18:54,141 --> 00:18:59,241
measure against, but it's very important to have to define a failure surface before you ever start.

242
00:19:00,001 --> 00:19:06,341
But now comes the fun part. We run minimal bounded pilots to test architecture. And once

243
00:19:06,341 --> 00:19:11,461
something starts working, we move those checks into rigs, code that runs between bounded work

244
00:19:11,461 --> 00:19:17,501
segments that we call codons. These codons are then reused, looped independently, validated,

245
00:19:17,501 --> 00:19:24,681
and placed in sequence. Finally, working pipelines get deployed to production where they can slowly

246
00:19:24,681 --> 00:19:30,821
mature. And that middle part is where the surfing happens. We find an architecture that gives a

247
00:19:30,821 --> 00:19:35,621
solid result. We use it as a baseline and we swap models and some other parts for faster,

248
00:19:35,761 --> 00:19:39,521
cheaper options, compare against known intermediates, tighten the requirements for

249
00:19:39,521 --> 00:19:44,781
success, each time moving closer and closer to the ideal. And you just watch the frontier move.

250
00:19:45,441 --> 00:19:53,721
Being able to break a complex problem into pieces means each piece can move to its own best position, as Adam Smith would say.

251
00:19:54,441 --> 00:20:02,081
In a system like this, ontologies exist as intermediates between intelligence and result, which makes them easier to reuse.

252
00:20:02,421 --> 00:20:05,841
That's the other half of what I mean by just-in-time ontologies.

253
00:20:06,061 --> 00:20:12,281
It's built when the job needs it, kept with its receipts, and reused when the next job needs the same thing.

254
00:20:12,281 --> 00:20:18,901
And once you have those, you can make these things cheaper. You can run them in parallel across data sets to scale up.

255
00:20:19,961 --> 00:20:27,641
I close with an apology, right, that the answer is there is no ontology. Not a universal one, at least. There never was one.

256
00:20:28,061 --> 00:20:35,161
The good news is that this makes every ontology and schema valid, as long as you can label and preserve the context around it.

257
00:20:35,161 --> 00:20:46,421
And if we can harness this cheap intelligence that's starting to percolate through the world around us, we're rapidly approaching a future where any information in any format is made as useful as it can be.

258
00:20:46,701 --> 00:20:50,821
And at Southbridge, we're working as hard as we can to make that future a reality.

259
00:20:51,201 --> 00:20:51,621
Thank you.