1
00:00:04,741 --> 00:00:09,321
Hello everyone, this talk is about the danger of your LLMZ formation leaking through shared

2
00:00:09,321 --> 00:00:10,081
GPU hardware.

3
00:00:10,441 --> 00:00:14,721
If you're running a model in the cloud on a GPU you share with other tenants, someone

4
00:00:14,721 --> 00:00:16,341
next to you can learn things about it.

5
00:00:16,541 --> 00:00:20,721
They can tell what model you're running, how fast it's responding, and even what kind

6
00:00:20,721 --> 00:00:25,681
work is doing right now, all without touching your data, just by watching timing. That's what

7
00:00:25,681 --> 00:00:30,581
this talk covers, what I quote tenets on the same hardware can learn about your LLM from timing alone

8
00:00:30,581 --> 00:00:37,681
and what you can do about it. To start, I'm Branislav Brzak, a cyber security researcher employed

9
00:00:37,681 --> 00:00:43,281
at Stealthium. At Stealthium we specialize in securing and providing observability into the environments

10
00:00:43,281 --> 00:00:49,201
AI systems run on. My area of focus is low-level GPU-related security research. That's something

11
00:00:49,201 --> 00:00:56,061
these stock covers. When you share a GPU between tenants, you would expect full isolation, but

12
00:00:56,061 --> 00:01:00,621
the isolation provided may not prove to be sufficient. In the case of this attack, the

13
00:01:00,621 --> 00:01:05,641
attacker doesn't even need any special permissions like root access. It doesn't need any traditional

14
00:01:05,641 --> 00:01:10,981
exploit either. The only thing it does is time the parts of the hardware that are still shared among

15
00:01:10,981 --> 00:01:17,281
tenants. This is a demonstration video of what the victims and the attackers dashboards look like.

16
00:01:17,281 --> 00:01:20,581
On the left, we have Qwen3 model doing inference.

17
00:01:20,961 --> 00:01:25,821
On the right, the attacker uses a side channel that Qwen3 leaks into as it's running.

18
00:01:26,121 --> 00:01:31,101
From that, it determines exactly what model it is and what kind of text it's producing right now.

19
00:01:33,441 --> 00:01:36,481
These are just the headline claims, but not the whole story.

20
00:01:36,681 --> 00:01:41,221
We can tell which model it is, architecture and depth, with the 97% accuracy.

21
00:01:41,681 --> 00:01:45,541
We can tell how fast is the coding, with the 99.3% accuracy.

22
00:01:45,541 --> 00:01:50,421
We can tell when and how much, the busy periods, token counts, and the answer length.

23
00:01:50,901 --> 00:01:57,201
What we can tell what kind of work it's doing, code, pros, JSON, or retrieval.

24
00:01:57,401 --> 00:02:03,081
What we cannot tell though are the actual prompt, answers, or contents, just timing.

25
00:02:03,361 --> 00:02:07,241
And all of this on real H100s serving real LLMs.

26
00:02:08,481 --> 00:02:14,181
If you're using NVIDIA hardware, GPU isolation is achieved using multi-instance GPU or MIG

27
00:02:14,181 --> 00:02:14,721
for short.

28
00:02:14,721 --> 00:02:21,881
Each isolated MIG slice has its own physical compute units and dedicated VRAM that provides

29
00:02:21,881 --> 00:02:23,741
hardware isolation on those parts.

30
00:02:24,121 --> 00:02:29,261
In our example, the LLM we are serving runs on a slice with 3 compute units and 40 GB

31
00:02:29,261 --> 00:02:30,001
of VRAM.

32
00:02:30,221 --> 00:02:34,101
The attack runs on just one compute unit and 10 GB of VRAM.

33
00:02:36,301 --> 00:02:41,581
What's isolated are compute units, memory capacity, bandwidth quotas, and faults and errors.

34
00:02:41,581 --> 00:02:44,621
The L2 cache is partitioned per instance too.

35
00:02:44,861 --> 00:02:46,501
There is no cross-instance eviction.

36
00:02:46,721 --> 00:02:48,801
So it's not shared in that sense.

37
00:02:48,981 --> 00:02:53,261
The actual leaks are memory barriers or members, which are GPU-wide.

38
00:02:53,561 --> 00:02:57,961
One instance's barrier still touches every L2 partition on the card.

39
00:02:58,521 --> 00:03:01,381
Timing is the one thing that isn't isolated.

40
00:03:01,661 --> 00:03:06,241
So when one tenant is busy, the other tenant's own reads get slightly slower.

41
00:03:06,241 --> 00:03:13,941
The key point is that a GPU-wide memory barrier takes effect in every L2 partition on the card.

42
00:03:14,181 --> 00:03:17,721
The next slide builds the whole attack on that one fact.

43
00:03:19,761 --> 00:03:24,141
The vector is that the attacker only ever reads its own memory.

44
00:03:24,501 --> 00:03:27,561
It only times its own reads, it never touches the victim's memory.

45
00:03:27,781 --> 00:03:33,061
There are three steps. Load from its own small buffer, time the load, and repeat every 5 microseconds.

46
00:03:33,061 --> 00:03:37,821
This primitive and the MIG member leak itself isn't our discovery.

47
00:03:38,181 --> 00:03:43,461
That goes to the researchers behind behind bars at using security.

48
00:03:43,781 --> 00:03:45,241
A big thanks for their work.

49
00:03:47,421 --> 00:03:49,761
Now try looking at the timing differences.

50
00:03:50,241 --> 00:03:56,441
With the naked eye comparing your tenant idle versus serving an LLM, you wouldn't even be able to discern them.

51
00:03:56,701 --> 00:03:59,401
The median latency is only 3 microseconds higher.

52
00:03:59,401 --> 00:04:03,181
To the attacker though, those 3 microseconds are all it needs.

53
00:04:04,101 --> 00:04:07,281
enough to gather all the information we already talked about.

54
00:04:09,481 --> 00:04:14,201
Qwen3 with 8 billion parameters is running on the victim's slides throughout this one.

55
00:04:14,481 --> 00:04:16,981
All captured data was taken from a live model.

56
00:04:17,401 --> 00:04:21,081
At the top is GPU activity over time, 1 burst per token.

57
00:04:21,461 --> 00:04:24,881
Below is the frequency spectrum of the attacker's own latency trace.

58
00:04:24,881 --> 00:04:30,701
There is a fundamental 881Hz which is just tokens per second plus harmonics, since each

59
00:04:30,701 --> 00:04:34,621
token is a mix of distinct frequencies, not one clean one.

60
00:04:37,121 --> 00:04:40,501
This is a real trace, one token in the time domain.

61
00:04:40,861 --> 00:04:43,341
The token shows a total of 39 bumps.

62
00:04:44,261 --> 00:04:50,101
bumps with green peaks are the layer invocations, while orange ones are additional between layer

63
00:04:50,101 --> 00:04:50,601
processing.

64
00:04:50,601 --> 00:04:56,841
In a later slide we will demonstrate how we determine the actual layer counts even though

65
00:04:56,841 --> 00:04:58,301
those extra bumps are present.

66
00:04:59,601 --> 00:05:01,321
we look at the decode speed itself.

67
00:05:01,701 --> 00:05:04,901
Just by timing we found the token rate completely bind.

68
00:05:05,201 --> 00:05:09,161
So speed alone is something an attacker can recover accurately.

69
00:05:09,401 --> 00:05:12,061
But speed by itself isn't a clean fingerprint.

70
00:05:12,361 --> 00:05:18,981
Qwen3 8 billion and Llama 3 8 billion decode only 1.6% apart in their token rate.

71
00:05:18,981 --> 00:05:21,821
So from speed alone you'd think they're the same model.

72
00:05:22,021 --> 00:05:24,541
It narrows things down but it doesn't nail it.

73
00:05:24,881 --> 00:05:26,961
We need something sharper, layer counts.

74
00:05:27,201 --> 00:05:29,301
That's what the next few slides get into.

75
00:05:30,801 --> 00:05:34,081
We use the same fold and average tick from before.

76
00:05:34,381 --> 00:05:37,881
This time we run it once per model instead of once per token.

77
00:05:38,141 --> 00:05:42,681
What comes out is a bit count that lands almost exactly on a whole number.

78
00:05:42,681 --> 00:05:48,881
The mean distance from an integer across 22 traces is just 0.028.

79
00:05:49,181 --> 00:05:50,161
That's the tell.

80
00:05:50,561 --> 00:05:56,121
If this were just a fixed hardware artifact divided by different token rates, it'd expect

81
00:05:56,121 --> 00:05:57,581
fractions, not whole numbers.

82
00:05:57,941 --> 00:06:04,321
Each count works out to the model's layer count plus 1 to 3 additional peaks.

83
00:06:06,921 --> 00:06:08,981
model keeps its own signal fingerprint.

84
00:06:09,301 --> 00:06:11,581
A mix of tempo and harmonic structure.

85
00:06:11,581 --> 00:06:14,861
To be clear, this is not a trained machine learning classifier.

86
00:06:15,001 --> 00:06:17,341
There's no training set, no labels, no configuration.

87
00:06:17,821 --> 00:06:19,641
It's under 30 lines of code.

88
00:06:19,861 --> 00:06:23,541
The method checks each candidate's depth from 16 to 48.

89
00:06:23,781 --> 00:06:29,041
It measures how strongly that depth's expected signature shows up in the real trace.

90
00:06:29,501 --> 00:06:31,401
Then it picks which are of its best.

91
00:06:31,761 --> 00:06:37,501
For Qwen3, 8 billion 36 layers wins Cleary by 4.0 times over the next candidate.

92
00:06:37,781 --> 00:06:40,961
Because this is a best fit test, not a lookup table,

93
00:06:40,961 --> 00:06:43,541
it also works on models it never seen before.

94
00:06:43,741 --> 00:06:46,701
That includes proprietary on unreleased ones,

95
00:06:46,961 --> 00:06:52,061
since there's nothing it needs to have seen in the first place.

96
00:06:55,261 --> 00:06:57,381
And this works surprisingly well.

97
00:06:57,881 --> 00:07:01,301
The layer fit classifier also outputs the confidence level.

98
00:07:01,441 --> 00:07:05,281
Across R testing, only two least confident answers

99
00:07:05,281 --> 00:07:06,801
ended up being wrong ones.

100
00:07:09,141 --> 00:07:14,941
This same method also picks up fine tunes and distillations that share the same base

101
00:07:14,941 --> 00:07:15,301
model.

102
00:07:15,681 --> 00:07:19,681
That's because the architecture and layer count carry over unchanged.

103
00:07:20,161 --> 00:07:22,921
Take three related models as an example.

104
00:07:23,341 --> 00:07:25,881
Llama 3 8 billion from meta has 32 layers.

105
00:07:26,461 --> 00:07:32,701
Llama 3.1 8 billion also from meta uses that same 32 layer backbone with minor upgrades.

106
00:07:32,701 --> 00:07:38,521
A R1 distilled Llama 8 billion from DeepSeek is an improvement on top of that.

107
00:07:38,961 --> 00:07:45,881
The per-layer timing comes out to only 1.6 nanosecond median latency shifts of each other.

108
00:07:46,341 --> 00:07:48,861
The rest of the signal looks the same.

109
00:07:49,221 --> 00:07:52,721
Regardless of this, our classifier is still able to discern between them.

110
00:07:54,241 --> 00:07:56,641
Now we ask a deeper question on the same channel.

111
00:07:56,861 --> 00:07:59,581
Not which model, but what is doing right now.

112
00:07:59,581 --> 00:08:01,421
We look at four workflow types.

113
00:08:01,601 --> 00:08:03,901
Prose, code, JSON, and retrieval.

114
00:08:04,101 --> 00:08:08,641
We give a workload on every half second based on the last five seconds of timing.

115
00:08:09,121 --> 00:08:12,201
Out of 40 half-second workloads 31 were right.

116
00:08:12,801 --> 00:08:17,841
And four full responses were correctly identified by majority votes.

117
00:08:18,341 --> 00:08:22,421
This is the same classifier from the video earlier misses included.

118
00:08:23,341 --> 00:08:25,401
One condition is worth stating.

119
00:08:25,801 --> 00:08:28,501
This was measured with speculative decoding turn-on.

120
00:08:28,501 --> 00:08:30,021
That's an opt-in in vLLM,

121
00:08:30,361 --> 00:08:52,881
not the default. Here's the mechanism. How predictable the text is changes how many drafted tokens get accepted at each step. That changes step time and that's the signal we're picking up. We tested 11 configurations, dense and mixture of experts, BF16 and FP8. The same figure page showed up every time, just at very different strengths.

122
00:08:52,881 --> 00:08:59,521
17.2% spread on GPT OSS 20 billion vs 0.9% on GLM-4-9B.

123
00:09:00,061 --> 00:09:04,541
10 out of 11 round pros fastest when the chance alone would only give you 1 in 4.

124
00:09:04,841 --> 00:09:10,621
So say the attacker already knows the model from the earlier slides and specular the recording

125
00:09:10,621 --> 00:09:11,901
happens to be on.

126
00:09:12,281 --> 00:09:15,121
It can also tell what kind of text is coming out.

127
00:09:15,941 --> 00:09:18,801
Now we take the Shadeship out of the picture entirely.

128
00:09:19,241 --> 00:09:22,201
MIG needs the attacker on the same physical die.

129
00:09:22,201 --> 00:09:29,921
This attack doesn't. The setup here is an 8x H100 server connected over an NVSwitch fabric.

130
00:09:30,321 --> 00:09:37,721
The victim runs tensor parallel across GPU 0 and 1. The attacker just needs any other GPU in that

131
00:09:37,721 --> 00:09:44,121
same server, GPU 2 in this case. Peer access over NVSwitch is a real intended feature. It's

132
00:09:44,121 --> 00:09:51,981
how multi-GPU jobs normally work, not a bug being abused. MIG tensor parallelism also can't coexist

133
00:09:51,981 --> 00:09:58,141
on the same device. So this is really a different population of deployments, not just the same

134
00:09:58,141 --> 00:10:06,981
attack repeated. The probe runs on GPU 2. It pointer traces memory that actually is on GPU 0

135
00:10:06,981 --> 00:10:15,721
over NVLink, timing every hop along the way. An idle hop takes about 890 nanoseconds. Once the

136
00:10:15,721 --> 00:10:19,541
victim starts working that climbs up to 300 microseconds.

137
00:10:20,281 --> 00:10:27,801
These are three half-second captures and you can actually count the tokens by eye.

138
00:10:28,121 --> 00:10:33,921
What we lose compared to MIG is the layer bit is below the noise at this

139
00:10:33,921 --> 00:10:37,401
distance. So over-enviling the fingerprints we

140
00:10:37,401 --> 00:10:41,421
were left with is mostly tempo, not layer count.

141
00:10:42,461 --> 00:10:49,641
Here we test the five models on identical serving setup. Only the model itself gives away which one it is.

142
00:10:49,881 --> 00:10:57,081
This method gives 79.2% accuracy. Now it fails exactly where you'd expect the tempo to fail.

143
00:10:57,341 --> 00:11:03,801
LAMA 3.58 billion and R1D Steel are both 8 billion models that they code 3Hz apart.

144
00:11:04,161 --> 00:11:08,981
So they blur together, where MIG's layer counts would have separated them easily.

145
00:11:08,981 --> 00:11:16,101
The takeaway is that the NVLink signal-to-noise ratio is worse than MIGs, since it loses the

146
00:11:16,101 --> 00:11:23,041
layer bit. But worse signal-to-noise ratio isn't the same as safety. 79% against the 20% baseline

147
00:11:23,041 --> 00:11:30,361
is still a real fingerprint, not noise. First, some good news. Our own product at work

148
00:11:30,361 --> 00:11:35,021
caught the attack straight out of the box. Stealthium monitors every single kernel launch

149
00:11:35,021 --> 00:11:40,421
across every GPU, analyzes suspicious workloads and runs them through a full alert pipeline,

150
00:11:40,801 --> 00:11:46,641
which is something not every vendor can do. Here is what made this one stand out. A single

151
00:11:46,641 --> 00:11:53,561
kernel relaunched about 3000 times, hitting the exact same 40 kilobytes of memory over and over

152
00:11:53,561 --> 00:12:00,101
instead of moving to new data. Only one kilowatt of data was sent back to the CPU the entire time

153
00:12:00,101 --> 00:12:03,841
because the timing itself was the payload, not the output.

154
00:12:04,141 --> 00:12:10,641
In real terms, that is under 600 milliseconds start to finish with fewer than 50 tokens over reply.

155
00:12:10,861 --> 00:12:14,421
We caught it before the response was ever half typed.

156
00:12:17,501 --> 00:12:18,941
So, three takeaways.

157
00:12:19,461 --> 00:12:22,461
First, placement is something operators actually control.

158
00:12:22,661 --> 00:12:28,041
So do not put an untrusted tenant's MIG slice right next to sensitive model.

159
00:12:28,041 --> 00:12:30,121
Second, know the trade-off.

160
00:12:30,421 --> 00:12:35,401
MIG rules out peer-to-peer communication, meaning there is no tensor parallelism on

161
00:12:35,401 --> 00:12:35,841
MIG.

162
00:12:36,281 --> 00:12:42,601
Sharding over NVLink instead limits what leaks to timing alone rather than the layer accounts.

163
00:12:43,141 --> 00:12:45,661
Third, watch the GPU itself.

164
00:12:45,941 --> 00:12:52,001
Make sure the host infrastructure is actively verifying that no malicious workloads are running

165
00:12:52,001 --> 00:12:52,681
on the hardware.

166
00:12:52,681 --> 00:12:57,841
That's exactly what Stealthium does by catching these probes straight from the Chrome behavior.

167
00:13:00,221 --> 00:13:05,361
Sharing a GPU means timing alone can give away the model, the speed, and the workload.

168
00:13:05,781 --> 00:13:10,221
That's true even across NVLink, without needing any export at all.

169
00:13:10,681 --> 00:13:15,741
Stealthium sees this happen in real time. Most organizations still don't have that kind of visibility.

170
00:13:16,161 --> 00:13:22,121
Thanks again to the researchers of Behind Bars, this whole talk builds upon their work.

171
00:13:22,121 --> 00:13:23,441
Thank you all for watching.