1
00:00:00,160 --> 00:00:02,880
All right, let's jump right in. 
So what's the big hook here? 

2
00:00:03,120 --> 00:00:06,000
We're talking about a 1,000,000 
token context window. 

3
00:00:06,160 --> 00:00:08,240
Yeah, you heard that right. 
The problem? 

4
00:00:08,600 --> 00:00:11,320
Well, standard attention 
mechanisms have a quadratic cost

5
00:00:11,400 --> 00:00:14,040
that makes that scale just 
completely impossible from a 

6
00:00:14,040 --> 00:00:16,120
budget standpoint. 
And the solution? 

7
00:00:16,360 --> 00:00:19,400
That's DeepSeek V4, a new 
architecture that completely 

8
00:00:19,400 --> 00:00:22,560
rethinks how attention works, 
making these huge contexts not 

9
00:00:22,560 --> 00:00:24,360
just possible, but actually 
efficient. 

10
00:00:24,600 --> 00:00:26,360
We're going to get into all the 
technical goodness. 

11
00:00:26,880 --> 00:00:28,880
I mean, really let that number 
sink in for a second. 

12
00:00:29,200 --> 00:00:31,680
One million tokens. 
That's not just a long book, 

13
00:00:31,800 --> 00:00:33,120
that's practically a whole 
library. 

14
00:00:33,320 --> 00:00:35,800
You can feed an entire code 
base, a massive pile of 

15
00:00:35,800 --> 00:00:38,440
financial reports, or a whole 
corpus of legal documents into 

16
00:00:38,440 --> 00:00:41,040
the model in one go. 
And this isn't just about 

17
00:00:41,040 --> 00:00:43,440
reading more text, right? 
This unlocks the door to these 

18
00:00:43,440 --> 00:00:46,760
long horizon complex agentic 
workflows that have until now 

19
00:00:46,760 --> 00:00:48,040
been pretty much science 
fiction. 

20
00:00:48,720 --> 00:00:51,800
And that of course, leads us to 
the massive engineering 

21
00:00:51,800 --> 00:00:54,200
challenge. 
The classic attention mechanism,

22
00:00:54,280 --> 00:00:56,360
you know, the very heart of the 
Transformer. 

23
00:00:56,560 --> 00:00:59,800
It just doesn't scale. 
Its computational complexity 

24
00:00:59,800 --> 00:01:02,360
grows quadratically with the 
length of the sequence. 

25
00:01:02,600 --> 00:01:05,720
For any of you architects or ML 
UPS folks out there, you know 

26
00:01:05,720 --> 00:01:08,080
this is the killer. 
As you approach a million 

27
00:01:08,080 --> 00:01:12,240
tokens, the cost both for 
compute and for memory just goes

28
00:01:12,240 --> 00:01:14,560
through the roof. 
It becomes astronomical. 

29
00:01:14,960 --> 00:01:16,800
So this is the key insight 
really. 

30
00:01:17,120 --> 00:01:20,120
This isn't a problem you can 
just brute force with more Gu's.

31
00:01:20,480 --> 00:01:22,760
You can't just throw more 
hardware at it and hope for the 

32
00:01:22,760 --> 00:01:24,480
best. 
You have to be smarter. 

33
00:01:24,640 --> 00:01:27,240
You have to innovate at the 
fundamental architectural level.

34
00:01:27,560 --> 00:01:30,960
And that is exactly what Dee Eek
V4 does by introducing this 

35
00:01:30,960 --> 00:01:33,920
really novel hybrid attention 
mechanism that was designed from

36
00:01:33,920 --> 00:01:36,240
the ground U for these extreme 
scale contexts. 

37
00:01:36,600 --> 00:01:39,800
So what's the secret sauce? 
How do they actually manage to 

38
00:01:39,800 --> 00:01:42,120
pull this off? 
Well, the engine that's driving 

39
00:01:42,120 --> 00:01:44,720
all this efficiency is a hybrid 
approach to attention. 

40
00:01:44,800 --> 00:01:47,960
And it's built on two really 
cool new concepts, something 

41
00:01:47,960 --> 00:01:51,600
called compressed sparse 
attention or CSA, and heavily 

42
00:01:51,600 --> 00:01:54,880
compressed attention or HCA. 
OK, so let's start with the 

43
00:01:54,880 --> 00:01:58,080
first piece of the puzzle, 
compressed sparse attention. 

44
00:01:58,480 --> 00:02:00,520
The idea here is incredibly 
clever. 

45
00:02:00,720 --> 00:02:04,000
Think of it this way, instead of
every single query token having 

46
00:02:04,000 --> 00:02:07,200
to look at every single previous
key value token in the cache, 

47
00:02:07,320 --> 00:02:10,680
which is, you know, 
computationally insane, CSA is 

48
00:02:10,680 --> 00:02:13,320
much smarter. 
It first groups tokens into 

49
00:02:13,320 --> 00:02:16,920
blocks, and then compresses the 
KV caches for those blocks into 

50
00:02:16,920 --> 00:02:20,600
a single entry. 
Then, and only then, it uses a 

51
00:02:20,600 --> 00:02:23,760
sparse mechanism to just pick 
out the most relevant of these 

52
00:02:23,760 --> 00:02:26,480
compressed blocks. 
It's like a two stage filter 

53
00:02:26,480 --> 00:02:29,200
that dramatically slashes the 
amount of processing you have to

54
00:02:29,200 --> 00:02:32,160
do. 
So let's break that down step by

55
00:02:32,160 --> 00:02:34,560
step. 
First, it compresses, right? 

56
00:02:34,800 --> 00:02:37,120
Reducing the sequence length 
right off the bat. 

57
00:02:37,520 --> 00:02:40,960
But this is where the real magic
happens in Step 2 with this 

58
00:02:40,960 --> 00:02:43,720
lightning indexer. 
It's a super lightweight 

59
00:02:43,720 --> 00:02:47,320
mechanism that just zips through
and scores how relevant each 

60
00:02:47,320 --> 00:02:50,200
compressed block is. 
It's not doing full attention, 

61
00:02:50,320 --> 00:02:52,080
it's just finding the right 
signals. 

62
00:02:52,480 --> 00:02:56,560
A top case selector then picks 
the winners and only then does 

63
00:02:56,560 --> 00:02:59,680
the model perform its full multi
query attention on this 

64
00:02:59,680 --> 00:03:03,080
drastically, and I mean 
drastically reduced set of KV 

65
00:03:03,080 --> 00:03:05,800
entries. 
All right, so that was CSA. 

66
00:03:06,040 --> 00:03:09,280
Now what's the other part of 
this tag team that's heavily 

67
00:03:09,280 --> 00:03:13,520
compressed attention or HCA? 
Now this one is even more 

68
00:03:13,520 --> 00:03:15,640
aggressive. 
It takes these huge blocks of 

69
00:03:15,640 --> 00:03:18,200
tokens and smashes them down 
into a single entry. 

70
00:03:18,720 --> 00:03:21,600
The trade off is that after this
really extreme compression, it 

71
00:03:21,600 --> 00:03:24,720
then performs standard dense 
attention, but only over the, 

72
00:03:24,800 --> 00:03:27,440
you know, handful of super 
condensed entries that are left.

73
00:03:27,760 --> 00:03:30,720
You can think of it as creating 
a very quick but very global 

74
00:03:30,840 --> 00:03:33,240
bird's eye view summary of the 
entire context. 

75
00:03:33,800 --> 00:03:36,320
And here's the beautiful part. 
It's not an either or a 

76
00:03:36,320 --> 00:03:39,600
situation. 
Deep Sig V4 is a hybrid model. 

77
00:03:39,920 --> 00:03:42,960
It cleverly interleaves both of 
these mechanisms. 

78
00:03:43,200 --> 00:03:45,520
CSA is there to provide the 
balance, right? 

79
00:03:45,720 --> 00:03:49,120
It maintains a good level of 
detail while still being super 

80
00:03:49,120 --> 00:03:52,080
efficient. 
And then HCA comes in to act as 

81
00:03:52,080 --> 00:03:55,240
the global context specialist, 
making sure the model never 

82
00:03:55,240 --> 00:03:58,040
loses sight of the big picture, 
even when it's dealing with 

83
00:03:58,040 --> 00:04:01,840
these crazy long sequences. 
OK, but hacking the attention 

84
00:04:01,840 --> 00:04:03,240
mechanism is only half the 
battle. 

85
00:04:03,400 --> 00:04:06,400
If your model is super deep, you
have another huge problem, just 

86
00:04:06,400 --> 00:04:08,920
getting the signal to flow 
cleanly from layer 1 all the way

87
00:04:08,920 --> 00:04:12,160
to, say, layer 100. 
So for all the researchers in 

88
00:04:12,160 --> 00:04:14,280
our audience, this next part is 
definitely for you. 

89
00:04:14,760 --> 00:04:17,720
DeepSeek V4 introduces something
called Manifold Constrained 

90
00:04:17,720 --> 00:04:20,959
Hyper Connections, or MHC. 
So what is it? 

91
00:04:21,240 --> 00:04:24,240
Well, at its core, MHC is 
basically a version of the 

92
00:04:24,240 --> 00:04:27,240
residual connections we all know
and love, but on steroids. 

93
00:04:27,640 --> 00:04:30,040
They took that information 
highway that lets gradients and 

94
00:04:30,040 --> 00:04:32,640
signal skip layers, and they 
made it wider. 

95
00:04:33,000 --> 00:04:35,720
But crucially, and this is the 
key part, they put a 

96
00:04:35,720 --> 00:04:38,840
mathematical constraint on the 
transformation matrix that maps 

97
00:04:38,840 --> 00:04:40,520
the signal from one layer to the
next. 

98
00:04:40,920 --> 00:04:43,320
And that constraint is the 
secret to preventing the signal 

99
00:04:43,320 --> 00:04:46,160
from either dying out or just 
exploding in these really, 

100
00:04:46,160 --> 00:04:48,840
really deep networks. 
And here's how that works. 

101
00:04:48,840 --> 00:04:50,840
First they expand the residual 
stream. 

102
00:04:51,160 --> 00:04:53,280
Then comes the clever bit, the 
constraint. 

103
00:04:53,720 --> 00:04:56,320
For those of you who love the 
math, they constrain the mapping

104
00:04:56,320 --> 00:04:59,520
matrix to the manifold of doubly
stochastic matrices. 

105
00:04:59,720 --> 00:05:01,400
Now what does that mean in plain
English? 

106
00:05:01,640 --> 00:05:04,920
It means the transformation is 
guaranteed to be non expansive. 

107
00:05:05,080 --> 00:05:08,400
It can't blow up the signal, and
the practical result of that is 

108
00:05:08,400 --> 00:05:11,400
no more exploding gradients 
during back propagation, which 

109
00:05:11,400 --> 00:05:13,880
as you know has been a nightmare
for training very deep models 

110
00:05:13,880 --> 00:05:16,000
for years. 
This is a massive win for 

111
00:05:16,000 --> 00:05:19,160
training stability. 
Now of course an amazing model 

112
00:05:19,160 --> 00:05:22,880
design on AER is one thing, but 
actually training and deploying 

113
00:05:22,880 --> 00:05:25,320
it at scale? 
That is a whole different beast.

114
00:05:25,640 --> 00:05:29,080
So for all you architects and ML
OS engineers listening, this 

115
00:05:29,080 --> 00:05:32,040
next part is for you. 
We're talking about the massive 

116
00:05:32,040 --> 00:05:36,160
engineering effort that actually
makes DeepSeek V4 possible, and 

117
00:05:36,160 --> 00:05:38,840
the list of otimizations here is
just wow. 

118
00:05:39,240 --> 00:05:41,800
They developped customfused 
kernels for their mixture of 

119
00:05:41,800 --> 00:05:44,640
experts layers that cleverly 
overlap communication and 

120
00:05:44,640 --> 00:05:47,400
computation to hide network 
latency. 

121
00:05:47,760 --> 00:05:50,600
They even built their own domain
specific language called tile 

122
00:05:50,600 --> 00:05:53,080
Lang to crank out optimized 
kernels faster. 

123
00:05:53,360 --> 00:05:56,360
They built bitwise deterministic
libraries, absolutely crucial 

124
00:05:56,360 --> 00:05:59,400
for debugging and 
reproducibility, and get this 

125
00:05:59,640 --> 00:06:04,760
quantization aware training down
to FP4A4 bit float for the Moe 

126
00:06:04,760 --> 00:06:07,360
weights and parts of the 
attention mechanism that is 

127
00:06:07,360 --> 00:06:10,400
cutting edge stuff. 
This is really a master class in

128
00:06:10,400 --> 00:06:13,360
systems optimization. 
OK, so we've talked 

129
00:06:13,360 --> 00:06:16,120
architecture, we've talked 
infrastructure, but what does it

130
00:06:16,120 --> 00:06:18,560
all actually add up to? 
What's the real payoff? 

131
00:06:18,840 --> 00:06:21,480
It's time for our neural signal 
check where we cut through the 

132
00:06:21,480 --> 00:06:24,120
noise, and look at why this 
development actually matters 

133
00:06:24,120 --> 00:06:27,800
from a technical standpoint. 
Man, you've got to love a good 

134
00:06:27,800 --> 00:06:30,000
graph. 
And these two, they tell a 

135
00:06:30,000 --> 00:06:33,040
powerful story. 
On the left, compute. 

136
00:06:33,360 --> 00:06:36,840
On the right memory size. 
Now look at that dashed line. 

137
00:06:37,000 --> 00:06:39,720
That's the old way. 
See how it just skyrockets? 

138
00:06:39,960 --> 00:06:42,720
It almost goes vertical. 
That's quadratic complexity in 

139
00:06:42,720 --> 00:06:45,280
action. 
Now look at the solid blue line 

140
00:06:45,280 --> 00:06:48,120
for V4. 
It's a a completely different 

141
00:06:48,120 --> 00:06:50,760
trajectory. 
It's almost flat in comparison. 

142
00:06:51,240 --> 00:06:54,040
That isn't just an improvement, 
that's a broken curve. 

143
00:06:54,360 --> 00:06:57,200
That's the architecture paying 
off in a huge way. 

144
00:06:57,760 --> 00:07:00,240
And if you want the cold hard 
numbers for our neural signal, 

145
00:07:00,240 --> 00:07:03,440
check here they are at a 
1,000,000 token context. 

146
00:07:03,560 --> 00:07:09,000
The V4 Pro model needs only 27% 
of the compute and 10% of the KV

147
00:07:09,000 --> 00:07:11,520
cache memory. 
Compared to its predecessor and 

148
00:07:11,520 --> 00:07:15,080
the even leaner Flash model, it 
drops that to just 10% of the 

149
00:07:15,080 --> 00:07:20,000
compute and 7% of the memory. 
Let me repeat that 110th of the 

150
00:07:20,000 --> 00:07:22,160
compute. 
This isn't just making things a 

151
00:07:22,160 --> 00:07:24,600
little cheaper. 
This is moving million token 

152
00:07:24,600 --> 00:07:28,160
contexts from a maybe in a few 
years research project to a we 

153
00:07:28,160 --> 00:07:31,080
can deploy this now reality. 
And you don't have to take my 

154
00:07:31,080 --> 00:07:33,280
word for it. 
The team at Deep Seeks said it 

155
00:07:33,280 --> 00:07:35,920
themselves. 
This breakthrough ushers in a 

156
00:07:35,920 --> 00:07:39,080
new era of million length 
contexts. 

157
00:07:39,320 --> 00:07:41,760
They're not calling it an 
incremental improvement. 

158
00:07:41,840 --> 00:07:44,160
They're calling it a 
breakthrough that starts a new 

159
00:07:44,240 --> 00:07:45,880
era. 
And honestly, looking at the 

160
00:07:45,880 --> 00:07:48,840
numbers, it's hard to disagree. 
So I want to leave you with this

161
00:07:48,840 --> 00:07:51,760
thought, especially if you're 
ACTO or an architect trying to 

162
00:07:51,760 --> 00:07:54,920
plan your next two years. 
The main barrier to truly long 

163
00:07:54,920 --> 00:07:58,440
context AI, that raw compute and
memory cost, it's just been 

164
00:07:58,440 --> 00:08:01,640
fundamentally broken. 
The question is no longer can we

165
00:08:01,640 --> 00:08:03,640
do it. 
The question now is what are we 

166
00:08:03,640 --> 00:08:06,600
going to build with it? 
What kind of complex multi step 

167
00:08:06,720 --> 00:08:09,240
agentic systems, The ones you 
put on the back burner because 

168
00:08:09,240 --> 00:08:11,400
they just seemed impossible. 
Which of those are you pulling 

169
00:08:11,400 --> 00:08:14,240
onto your road map today? 
That's the real impact here.