1 00:00:00,160 --> 00:00:04,480 Imagine you're trying to read like a 10,000 page encyclopedia 2 00:00:04,480 --> 00:00:07,040 to prepare for this massive exam, right? 3 00:00:07,040 --> 00:00:08,680 That sounds absolutely terrible, right? 4 00:00:08,680 --> 00:00:12,080 It's a nightmare and if you read every single letter sequentially 5 00:00:12,080 --> 00:00:17,440 like CAT space SAT, you would just never finish. 6 00:00:17,480 --> 00:00:19,080 No, you'd completely run out of. Time. 7 00:00:19,080 --> 00:00:21,840 Exactly. But what if you could somehow, I 8 00:00:21,840 --> 00:00:24,480 don't know, ingest an entire paragraph at a glance? 9 00:00:24,480 --> 00:00:26,760 Like you aren't reading the words in strict order, you're 10 00:00:26,760 --> 00:00:31,160 just absorbing this, this localized cloud of meaning. 11 00:00:31,360 --> 00:00:32,880 Kind of like getting the gist of it all at once. 12 00:00:32,880 --> 00:00:37,080 Yeah, exactly. And somehow this totally blurry, 13 00:00:37,080 --> 00:00:41,080 disorganized method of absorbing information allows you to master 14 00:00:41,080 --> 00:00:44,560 the material 2 1/2 times faster. Which is just a staggering 15 00:00:44,560 --> 00:00:45,640 claim. It really is. 16 00:00:45,880 --> 00:00:48,840 Well, welcome to build Wiz AI. We are so thrilled you're 17 00:00:48,840 --> 00:00:50,360 joining us today. Yes, welcome. 18 00:00:50,360 --> 00:00:51,920 Our mission here is pretty simple. 19 00:00:52,360 --> 00:00:55,160 We take a stack of, you know, cutting edge research, we strip 20 00:00:55,160 --> 00:00:58,560 away all the noise, and we extract the most valuable mind 21 00:00:58,560 --> 00:01:02,000 expanding Nuggets so that you stay well informed without that 22 00:01:02,000 --> 00:01:05,040 crushing info overload. Because there is a lot of noise 23 00:01:05,040 --> 00:01:07,760 out there. So much noise and today's deep 24 00:01:07,760 --> 00:01:11,640 dive is tackling a massive, like literally billion dollar 25 00:01:11,640 --> 00:01:14,000 bottleneck in artificial intelligence. 26 00:01:14,080 --> 00:01:17,600 Right, we are talking about the sheer brute force cost of pre 27 00:01:17,600 --> 00:01:20,400 training large language models. It's just so expensive. 28 00:01:20,400 --> 00:01:21,880 It's. Astronomical training these 29 00:01:21,880 --> 00:01:26,520 frontier models requires mountains of compute power. 30 00:01:26,520 --> 00:01:30,040 You're talking 10s of millions of dollars in GPU time and 31 00:01:30,040 --> 00:01:33,840 massive energy consumption just to feed raw text into the neural 32 00:01:33,840 --> 00:01:35,200 network. Just to read the text. 33 00:01:35,320 --> 00:01:38,080 Exactly. It is the single biggest barrier 34 00:01:38,080 --> 00:01:41,080 to entry in the field right now. It completely dictates who gets 35 00:01:41,080 --> 00:01:44,000 to build foundational models. And who just gets totally priced 36 00:01:44,000 --> 00:01:45,880 out, right? Which brings us to the source. 37 00:01:45,880 --> 00:01:48,880 For today's deep dive, we are unpacking this fascinating 38 00:01:48,880 --> 00:01:52,280 preprint paper titled Efficient Pre Training with Tokens 39 00:01:52,440 --> 00:01:54,240 position. Such a cool concept. 40 00:01:54,240 --> 00:01:56,320 It really is. It's authored by Bowen Peng, 41 00:01:56,400 --> 00:01:59,600 Theo Gigant and Jeffrey Klisnel from Noose Research. 42 00:01:59,680 --> 00:02:02,040 A great team over there. Yeah, they do incredible work 43 00:02:02,440 --> 00:02:06,080 and our mission today is to explore a brilliant drop in 44 00:02:06,080 --> 00:02:09,160 method they've developed that speeds up AI pre training by up 45 00:02:09,160 --> 00:02:12,120 to 2.5 X. Without changing the models 46 00:02:12,200 --> 00:02:14,160 actual architecture which is key. 47 00:02:14,280 --> 00:02:17,600 That is the crucial part. But before we look at this new 48 00:02:17,600 --> 00:02:20,200 method, we kind of have to talk about how the industry has been 49 00:02:20,200 --> 00:02:23,280 trying to solve this compute crisis, because it's not like 50 00:02:23,280 --> 00:02:25,160 researchers have just been sitting on their hands. 51 00:02:25,160 --> 00:02:26,960 Oh, definitely not. But reading through the 52 00:02:26,960 --> 00:02:31,200 background context of this paper, it feels like every 53 00:02:31,200 --> 00:02:36,160 previous attempt to speed up training hits this very specific 54 00:02:36,160 --> 00:02:38,080 wall. Yeah, the field has certainly 55 00:02:38,080 --> 00:02:42,040 tried various angles of attack over the last few years. 56 00:02:42,440 --> 00:02:46,320 Initially, a lot of focus went into information maximization. 57 00:02:46,960 --> 00:02:49,080 The thought process there was pretty straightforward, right? 58 00:02:49,280 --> 00:02:51,880 Compute is wildly expensive. Let's just get more learning 59 00:02:51,880 --> 00:02:54,560 signal out of every single training sample we process. 60 00:02:54,560 --> 00:02:56,600 Make every token count for more. Exactly. 61 00:02:56,960 --> 00:03:00,960 That led to innovations like multi token prediction, where 62 00:03:00,960 --> 00:03:04,000 the model is forced to predict multiple future tokens 63 00:03:04,000 --> 00:03:07,440 simultaneously, right? So it creates A richer, denser 64 00:03:07,680 --> 00:03:10,680 training signal per step. But that doesn't inherently 65 00:03:10,680 --> 00:03:14,200 reduce the massive amount of matrix multiplication the model 66 00:03:14,200 --> 00:03:16,880 has to do, right? Just to process the input text 67 00:03:16,880 --> 00:03:18,360 in the first place. No it doesn't. 68 00:03:18,400 --> 00:03:21,400 The compute burden is still totally there, so then the field 69 00:03:21,400 --> 00:03:24,640 sort of pivoted toward trying to shrink the active size of the 70 00:03:24,640 --> 00:03:27,480 model during a forward pass. Right, that was the shift toward 71 00:03:27,480 --> 00:03:33,520 compute sparsity and the mixture of experts or Moe architecture 72 00:03:33,840 --> 00:03:36,200 is the dominant paradigm. There, which we hear about all 73 00:03:36,200 --> 00:03:37,640 the time now. Yeah, it's everywhere. 74 00:03:37,640 --> 00:03:40,840 You keep the total parameter count high, but you route each 75 00:03:40,840 --> 00:03:44,120 piece of text to a specialized sub network and expert. 76 00:03:45,000 --> 00:03:48,040 So for any given input, only a fraction of the neural network 77 00:03:48,040 --> 00:03:51,520 is actually active. You were doing essentially less 78 00:03:51,520 --> 00:03:54,880 math per token. But even then, when that still 79 00:03:54,880 --> 00:03:57,960 wasn't fast enough, we saw attempts at compressive modeling 80 00:03:57,960 --> 00:03:59,800 right where researchers literally really tried to 81 00:03:59,800 --> 00:04:01,640 compress the sequences internally. 82 00:04:01,800 --> 00:04:04,560 Yeah, so fewer tokens flow through the heaviest, most 83 00:04:04,560 --> 00:04:08,280 expensive layers of the network. But here is the massive catch 84 00:04:08,280 --> 00:04:10,560 with almost all of these historical approaches. 85 00:04:10,960 --> 00:04:14,960 And this is a flaw that Peng, Gigant and Connell are 86 00:04:14,960 --> 00:04:17,000 specifically targeting in their paper. 87 00:04:17,120 --> 00:04:20,360 It's a huge issue. Right, when you change how a 88 00:04:20,360 --> 00:04:24,200 model processes data internally to make pre training faster, 89 00:04:24,200 --> 00:04:27,760 like introducing complex internal compression mechanisms, 90 00:04:28,120 --> 00:04:30,720 you are fundamentally altering the models architecture. 91 00:04:30,720 --> 00:04:32,960 You're baking it in. Yeah, and if you change the 92 00:04:32,960 --> 00:04:36,160 architecture, you permanently change the inference dynamics. 93 00:04:36,160 --> 00:04:39,600 That is the exact crux of the problem because inference is 94 00:04:39,600 --> 00:04:42,280 when the pre training is finished and you are deploying 95 00:04:42,280 --> 00:04:45,760 the model for actual use. When you the user are actually 96 00:04:45,760 --> 00:04:46,960 typing a prompt. Right. 97 00:04:46,960 --> 00:04:50,000 If you introduce a bespoke compression module just to save 98 00:04:50,000 --> 00:04:53,640 money during training, that module is stuck in the model 99 00:04:53,640 --> 00:04:56,360 forever. It complicates deployment, it 100 00:04:56,360 --> 00:04:59,480 can introduce serious latency, and it often requires custom 101 00:04:59,480 --> 00:05:02,760 infrastructure just to serve the model to end users. 102 00:05:02,840 --> 00:05:05,240 Which makes the author's approach here so refreshing. 103 00:05:05,240 --> 00:05:07,680 They set this very strict constraint on themselves. 104 00:05:07,960 --> 00:05:10,680 I actually want to quote their core research question verbatim 105 00:05:10,680 --> 00:05:13,480 because it perfectly frames their entire hypothesis. 106 00:05:14,120 --> 00:05:17,720 OK, here's what they asked. Can we improve a language models 107 00:05:17,720 --> 00:05:21,920 pre training efficiency without modifying the final architecture 108 00:05:21,920 --> 00:05:24,360 and its inference dynamics? And their answer to that 109 00:05:24,360 --> 00:05:26,040 question is a resounding yes. It's. 110 00:05:26,400 --> 00:05:28,520 Wild. They do it through a methodology 111 00:05:28,520 --> 00:05:33,200 they call Token Superposition Training, or TST, and the way 112 00:05:33,200 --> 00:05:36,800 they pull this off is by fundamentally blurring the lines 113 00:05:36,800 --> 00:05:39,600 of language during the initial phases of training. 114 00:05:40,360 --> 00:05:42,840 I really want to spend some time visualizing this for you because 115 00:05:42,840 --> 00:05:46,360 the mechanism is genuinely wild. So in standard pre training, an 116 00:05:46,360 --> 00:05:50,200 LLM processes a sequence of tokens strictly left to right. 117 00:05:50,200 --> 00:05:53,640 Right it sees a token, computes its mathematical embedding, and 118 00:05:53,640 --> 00:05:57,000 moves to the next. But what TST does in its first 119 00:05:57,000 --> 00:06:00,360 phase, which they call the superposition phase, is it takes 120 00:06:00,400 --> 00:06:03,480 a contiguous sequence of tokens. Like a chunk of text. 121 00:06:03,640 --> 00:06:08,920 Yeah, let's say a bag of 4 to 8 tokens and it physically merges 122 00:06:08,920 --> 00:06:11,800 them together before the heavy computation even begins. 123 00:06:12,120 --> 00:06:14,440 Right, they take the mathematical embeddings of those 124 00:06:14,440 --> 00:06:17,920 individual tokens and average them into a single combined 125 00:06:17,920 --> 00:06:19,800 representation. Just mash together. 126 00:06:19,840 --> 00:06:22,000 Exactly. They call this an S token or 127 00:06:22,000 --> 00:06:25,040 superpose token. So from the perspective of the 128 00:06:25,040 --> 00:06:28,000 models deepest, most computationally intensive 129 00:06:28,000 --> 00:06:33,200 layers, those 4 distinct tokens now appear as one single dense 130 00:06:33,200 --> 00:06:35,960 block of information. So going back to that speed 131 00:06:35,960 --> 00:06:39,040 reading analogy from the beginning, if the text is the 132 00:06:39,040 --> 00:06:42,480 cat sat on the model isn't computing the relationships for 133 00:06:42,480 --> 00:06:44,640 the then for cat then for SAT? No, it's not. 134 00:06:44,640 --> 00:06:47,600 It is computing the math for a single blurry mathematical 135 00:06:47,600 --> 00:06:50,000 average of that entire localized concept. 136 00:06:50,160 --> 00:06:52,960 It's absorbing the general vibe of that chunk of the sentence. 137 00:06:53,440 --> 00:06:55,400 Vibe, I love that. And by doing that they are 138 00:06:55,400 --> 00:06:59,160 drastically increasing the data throughput per FLOP. 139 00:06:59,200 --> 00:07:01,600 And a FLOP. Just to clarify, a floating 140 00:07:01,600 --> 00:07:04,840 point operation is the basic unit of computing power. 141 00:07:05,160 --> 00:07:07,960 Exactly. If your model's layers are only 142 00:07:07,960 --> 00:07:11,600 processing one superpose token instead of four individual ones, 143 00:07:12,000 --> 00:07:15,880 you're basically pushing four times the amount of text through 144 00:07:15,880 --> 00:07:18,800 the network for the exact same computational budget. 145 00:07:18,960 --> 00:07:21,800 That's incredible, but hold on, I need to wrap my head around 146 00:07:21,800 --> 00:07:24,640 the output side of this. OK yeah, because the whole point 147 00:07:24,640 --> 00:07:28,000 of pre training a language model is next token prediction, right? 148 00:07:28,440 --> 00:07:31,920 You show it text, it guesses the next token, and you use a loss 149 00:07:31,920 --> 00:07:35,560 function to grade its guess. Right, normally you use A1 hot 150 00:07:35,560 --> 00:07:37,720 cross entropy loss. Right, there is one 151 00:07:37,720 --> 00:07:40,640 mathematically correct answer, and if the model guesses wrong, 152 00:07:40,640 --> 00:07:43,800 it gets penalized. But how do you grade a model 153 00:07:43,880 --> 00:07:46,400 when the input is a blurry bag of text? 154 00:07:46,400 --> 00:07:48,560 Well, you have to fundamentally change how you measure the 155 00:07:48,560 --> 00:07:51,040 mistake. Because the input is superposed, 156 00:07:51,280 --> 00:07:53,400 the output target must be superposed too. 157 00:07:53,600 --> 00:07:56,200 Oh, interesting. So they swap out the standard 1 158 00:07:56,200 --> 00:07:58,880 hot loss for a multi hot cross entropy loss. 159 00:07:58,960 --> 00:08:02,040 Multi hot, meaning there are multiple valid targets at the 160 00:08:02,040 --> 00:08:03,480 same time. Exactly. 161 00:08:03,880 --> 00:08:07,080 It asks the model to predict the next bag of words without any 162 00:08:07,080 --> 00:08:09,480 regard for their exact sequence. Wait, really? 163 00:08:09,480 --> 00:08:12,440 Order doesn't matter. Order does not matter at all. 164 00:08:13,160 --> 00:08:15,840 The model doesn't have to know that cat comes before SAT. 165 00:08:16,080 --> 00:08:20,000 It just has to predict that cat and SAT are highly probable 166 00:08:20,000 --> 00:08:23,880 concepts that belong in the ucoming localized chunk of text. 167 00:08:24,320 --> 00:08:26,200 Wow. Yeah, it is predicting the 168 00:08:26,200 --> 00:08:29,040 probability distribution of future concepts rather than a 169 00:08:29,040 --> 00:08:32,840 strict linear sequence. OK, I am seeing a massive like 170 00:08:32,840 --> 00:08:35,240 glaring issue with this methodology. 171 00:08:35,240 --> 00:08:36,159 I think I know what you're going to. 172 00:08:36,360 --> 00:08:41,600 Say if we are training a multi billion parameter model to read 173 00:08:41,600 --> 00:08:45,320 blurry bags of text, and we are rewarding it for predicting 174 00:08:45,320 --> 00:08:47,840 blurry bags of future text, where the sequential order 175 00:08:47,840 --> 00:08:51,080 doesn't matter at all. When you actually deploy that 176 00:08:51,080 --> 00:08:54,120 model and ask it to write a Python script or compose an 177 00:08:54,120 --> 00:08:56,720 e-mail, it is going to spit out absolute gibberish. 178 00:08:56,720 --> 00:08:59,760 Oh yeah, it produces pure unadulterated nonsense. 179 00:09:00,440 --> 00:09:02,200 OK, so I'm not crazy. Not at all. 180 00:09:02,320 --> 00:09:04,360 The authors highlight this fact very clearly. 181 00:09:04,560 --> 00:09:06,880 If you take a model trained exclusively with this phase one 182 00:09:06,880 --> 00:09:10,560 TST method and try to use it for generation, it fails completely. 183 00:09:10,640 --> 00:09:12,400 Totally useless. Pretty much. 184 00:09:12,880 --> 00:09:15,640 It has learned a mixed probability of future tokens, 185 00:09:16,080 --> 00:09:19,640 but it has completely lost the ability to string them together 186 00:09:19,640 --> 00:09:23,000 into syntactically valid human readable sequences. 187 00:09:23,000 --> 00:09:24,960 Which sounds like a spectacular failure. 188 00:09:25,160 --> 00:09:28,240 If the model can't generate A coherent sentence, what is the 189 00:09:28,240 --> 00:09:31,280 point of the compute savings? The point is that language 190 00:09:31,280 --> 00:09:34,480 generation is actually a two-part problem, even if we 191 00:09:34,480 --> 00:09:36,720 historically treat it as one. OK, unpack that. 192 00:09:36,760 --> 00:09:39,720 There is the deep conceptual understanding of how the world 193 00:09:39,720 --> 00:09:42,520 works, right? The relationships between facts, 194 00:09:42,640 --> 00:09:45,920 logic, and semantics. The ideas? 195 00:09:45,920 --> 00:09:49,360 Yes, and then there is the surface level syntax, the rigid 196 00:09:49,360 --> 00:09:52,400 grammar rules required to express those concepts to a 197 00:09:52,400 --> 00:09:55,320 human. TST essentially separates those 198 00:09:55,320 --> 00:09:57,600 two learning processes. And that brings us to the 199 00:09:57,600 --> 00:10:00,000 crucial Phase 2, the recovery phase. 200 00:10:00,000 --> 00:10:02,320 This is the part of the paper that genuinely blew my mind 201 00:10:02,560 --> 00:10:05,320 after running this hyper fast super pose training. 202 00:10:05,320 --> 00:10:08,120 For the vast majority of the training run, they just hit the 203 00:10:08,120 --> 00:10:09,440 brakes. Hard they do. 204 00:10:09,440 --> 00:10:11,040 Completely stop the TST. They don't. 205 00:10:11,040 --> 00:10:12,520 Just change a hyper parameter, right? 206 00:10:12,640 --> 00:10:16,280 No, they physically he stripped the TST code out of the training 207 00:10:16,280 --> 00:10:19,960 loop to ensure there is absolutely 0 contamination. 208 00:10:20,280 --> 00:10:23,520 They revert the model strictly back to standard single token, 209 00:10:23,520 --> 00:10:26,120 left to right, one hot next coken prediction. 210 00:10:26,120 --> 00:10:29,920 And the model just recovers like it just figures it out. 211 00:10:30,040 --> 00:10:33,920 It enters an incredibly rapid recovery period because the 212 00:10:33,920 --> 00:10:37,440 model has already spent billions of compute cycles learning the 213 00:10:37,440 --> 00:10:40,920 deep conceptual relationships between topics during the Super 214 00:10:40,920 --> 00:10:43,360 post phase. It doesn't need to learn what 215 00:10:43,360 --> 00:10:45,640 things mean anymore. It already knows the concepts. 216 00:10:45,640 --> 00:10:47,760 Exactly. It only needs to relearn how to 217 00:10:47,760 --> 00:10:50,480 format those thoughts. Yeah, the syntax and grammar 218 00:10:50,480 --> 00:10:53,240 snap back into place astonishingly fast during this 219 00:10:53,240 --> 00:10:55,920 brief recovery window. That separation of concept from 220 00:10:55,920 --> 00:10:58,840 syntax is just a brilliant mental model for how these 221 00:10:58,840 --> 00:11:00,720 networks actually learn. It really is. 222 00:11:01,080 --> 00:11:03,560 But let's move from theory to the hard evidence, because the 223 00:11:03,560 --> 00:11:07,120 numbers here are staggering. The researchers started small to 224 00:11:07,120 --> 00:11:11,880 prove the concept, running tests on 270 million and 600 million 225 00:11:11,880 --> 00:11:14,240 parameter models. Right, standard proof of concept 226 00:11:14,240 --> 00:11:16,520 stuff. But the real proof is at scale. 227 00:11:17,120 --> 00:11:21,640 They applied this to a 10 billion parameter A1B model, and 228 00:11:21,640 --> 00:11:24,400 we need to clarify that architecture because it's vital 229 00:11:24,400 --> 00:11:28,040 to why this works so well. Yeah, so the 10BA1B designation 230 00:11:28,160 --> 00:11:32,040 means it is a mixture of experts model with 10 billion total 231 00:11:32,040 --> 00:11:35,880 parameters, but only 1 billion parameters are active during any 232 00:11:35,880 --> 00:11:37,440 single forward pass. Right. 233 00:11:37,440 --> 00:11:41,200 So we're back to Maui. And why does TST synergize so 234 00:11:41,200 --> 00:11:43,680 well with a mixture of experts models specifically? 235 00:11:44,120 --> 00:11:47,520 It all comes down to routing. In a Maui model, you have a 236 00:11:47,520 --> 00:11:50,640 router network that decides which expert sub network should 237 00:11:50,640 --> 00:11:53,720 process a given token. When you feed that router a 238 00:11:53,760 --> 00:11:57,680 superposed token, a blurry average of an entire phrase, the 239 00:11:57,680 --> 00:12:00,200 router actually makes better macroscopic decisions 240 00:12:00,200 --> 00:12:02,160 interesting. Instead of rapidly switching 241 00:12:02,160 --> 00:12:05,600 experts word by word, which can be computationally inefficient, 242 00:12:06,080 --> 00:12:09,240 the router sends the entire conceptual bag of text to the 243 00:12:09,240 --> 00:12:11,280 expert best suited for that broader topic. 244 00:12:11,280 --> 00:12:13,600 So it's smoothing out the routing decision exactly, and 245 00:12:13,600 --> 00:12:16,920 the result of doing this at the 10 billion parameter scale up to 246 00:12:16,920 --> 00:12:20,920 a 2.5 X reduction in total pre training time to reach the exact 247 00:12:20,920 --> 00:12:23,040 same loss as a standard baseline model. 248 00:12:23,400 --> 00:12:25,600 Massive. We really need to pause and 249 00:12:25,600 --> 00:12:29,440 contextualize that for you. A 2.5 X speed up in this 250 00:12:29,440 --> 00:12:32,000 industry is a tectonic shift. It really is. 251 00:12:32,160 --> 00:12:35,360 When a single training run for a frontier model costs 10s of 252 00:12:35,360 --> 00:12:38,320 millions of dollars, cutting that time by more than half 253 00:12:38,320 --> 00:12:40,800 completely changes the economics of AI development. 254 00:12:41,440 --> 00:12:44,760 And critically, that massive speed up did not degrade the 255 00:12:44,760 --> 00:12:47,640 final capabilities of the model. That's the crazy part. 256 00:12:47,640 --> 00:12:51,120 Yeah, the author's rigorously evaluated the recovered models 257 00:12:51,120 --> 00:12:53,400 on standard downstream benchmarks. 258 00:12:53,600 --> 00:12:56,360 These are the tests the industry uses to gauge actual 259 00:12:56,360 --> 00:12:59,600 intelligence, right? Like Hellaswag for common sense 260 00:12:59,600 --> 00:13:03,520 reasoning, ARC for science knowledge, Bull Q for reading 261 00:13:03,520 --> 00:13:06,400 comprehension. The TST models consistently 262 00:13:06,400 --> 00:13:09,360 matched or outperformed the baseline models that were 263 00:13:09,360 --> 00:13:12,040 trained with standard methods for the same amount of 264 00:13:12,040 --> 00:13:14,720 computational effort. So we have a method that is 265 00:13:14,720 --> 00:13:17,120 significantly faster, drastically cheaper, and 266 00:13:17,120 --> 00:13:19,840 mathematically just as capable. Tricks all the boxes. 267 00:13:19,840 --> 00:13:22,760 But I want to dig into the ablation studies the authors ran 268 00:13:22,880 --> 00:13:25,360 because understanding why this works reveals some very 269 00:13:25,360 --> 00:13:27,920 counterintuitive properties of neural networks. 270 00:13:28,360 --> 00:13:31,760 Oh, the ablation studies in Figure 6 of the preprint are 271 00:13:31,960 --> 00:13:33,880 perhaps the most fascinating part of the research. 272 00:13:33,880 --> 00:13:36,880 Yeah, let's unpack that. So TST is composed of two 273 00:13:36,880 --> 00:13:40,320 distinct modifications, right? There's the input superposition, 274 00:13:40,320 --> 00:13:43,680 where you blur the incoming words, and there's the output 275 00:13:43,680 --> 00:13:46,840 superposition, where you use the multi hot loss to predict the 276 00:13:46,840 --> 00:13:50,440 bag of future words. The researchers isolated these 277 00:13:50,440 --> 00:13:53,240 variables to see exactly what was driving the performance. 278 00:13:53,240 --> 00:13:57,240 The classic scientific method. Is it the blurry input doing the 279 00:13:57,240 --> 00:13:59,640 heavy lifting or the blurry output target? 280 00:13:59,840 --> 00:14:03,000 And the surprising answer is that both work independently, 281 00:14:03,320 --> 00:14:06,800 but combining them produces A compounded effect with 0 282 00:14:06,800 --> 00:14:09,600 interference. Really. 0 interference. 283 00:14:09,600 --> 00:14:11,800 None. When they tested input only 284 00:14:11,800 --> 00:14:14,040 superposition, the model learned faster. 285 00:14:14,560 --> 00:14:17,280 When they tested output only superposition, it also learned 286 00:14:17,280 --> 00:14:18,240 faster. Wow. 287 00:14:18,360 --> 00:14:21,320 But when they stacked them together into full TST, the 288 00:14:21,320 --> 00:14:23,120 efficiency game stacked perfectly. 289 00:14:23,280 --> 00:14:26,680 Which implies they are solving 2 fundamentally different 290 00:14:26,680 --> 00:14:29,000 optimization problems at the same time. 291 00:14:29,240 --> 00:14:32,840 Exactly, and the authors propose a fantastic hypothesis for this. 292 00:14:33,360 --> 00:14:36,280 They suggest that the input superposition acts as a form of 293 00:14:36,520 --> 00:14:39,520 pre pre training. Free pre training. 294 00:14:39,520 --> 00:14:43,360 Right, by exposing the neural network to a courser, simplified 295 00:14:43,360 --> 00:14:46,840 distribution of language first it learns the broad statistical 296 00:14:46,840 --> 00:14:50,280 structure of topics before it ever has to waste compute cycles 297 00:14:50,280 --> 00:14:53,200 on the rigid rules of grammar. That is so interesting. 298 00:14:53,360 --> 00:14:55,280 And it's a phenomenon we've actually observed in other 299 00:14:55,280 --> 00:14:57,480 domains, like in computer vision. 300 00:14:57,720 --> 00:15:00,800 Models often train faster if you feed them lower resolution 301 00:15:01,160 --> 00:15:04,040 course image patches 1st and gradually increase the 302 00:15:04,040 --> 00:15:05,840 resolution. Makes total sense. 303 00:15:05,960 --> 00:15:08,120 This course to find learning schedule appears to be a 304 00:15:08,160 --> 00:15:11,800 universal property of efficient neural network training, not 305 00:15:11,800 --> 00:15:14,600 just an isolated trick for text. It makes perfect sense if you 306 00:15:14,600 --> 00:15:17,760 think about like a master painter starting a portrait. 307 00:15:17,800 --> 00:15:19,520 Oh, that's a good analogy. They don't pull out a 308 00:15:19,520 --> 00:15:22,040 microscopic brush and start detailing the individual 309 00:15:22,040 --> 00:15:25,240 eyelashes on a blank canvas. No, that would take forever. 310 00:15:25,440 --> 00:15:27,800 Exactly. They use a massive brush to 311 00:15:27,800 --> 00:15:30,560 block out the broad shapes, the lighting, the foundational 312 00:15:30,560 --> 00:15:32,760 colors. Only when the structural 313 00:15:32,760 --> 00:15:35,800 foundation is set do they switch to the tiny brush for the 314 00:15:35,800 --> 00:15:39,200 precise details. And for years we've essentially 315 00:15:39,200 --> 00:15:42,680 forced AI to paint with the tiny brush from day one. 316 00:15:43,080 --> 00:15:46,840 TST finally let's the model block out the canvas first. 317 00:15:46,840 --> 00:15:48,560 That captures the mechanics beautifully. 318 00:15:48,600 --> 00:15:51,280 But you know, we have to put our critical lens on because every 319 00:15:51,280 --> 00:15:53,400 methodological leap comes with a trade off. 320 00:15:53,720 --> 00:15:56,120 We established the massive winning compute time. 321 00:15:56,560 --> 00:16:00,960 Where does TST extract its toll? Well, the toll is paid in data. 322 00:16:01,920 --> 00:16:05,120 To achieve that 2.5 X compute speed up on the 10 billion 323 00:16:05,120 --> 00:16:08,520 parameter model, the TST model had to consume almost twice as 324 00:16:08,520 --> 00:16:11,240 much raw text as the baseline. Almost twice as much. 325 00:16:11,240 --> 00:16:13,040 Yeah. To hit the same performance 326 00:16:13,040 --> 00:16:16,800 target, the standard baseline model process roughly 1.05 327 00:16:16,800 --> 00:16:20,400 trillion tokens. The TST model had to ingest 2 328 00:16:20,400 --> 00:16:22,400 trillion tokens. OK, that is a massive 329 00:16:22,400 --> 00:16:24,320 discrepancy. It is, and it highlights one of 330 00:16:24,320 --> 00:16:27,080 the most intense debates happening in AI research right 331 00:16:27,080 --> 00:16:28,640 now. The tension between being 332 00:16:28,640 --> 00:16:30,360 compute bound versus being data bound. 333 00:16:30,640 --> 00:16:34,360 If your lab has access to endless data but you just can't 334 00:16:34,360 --> 00:16:37,800 afford the electricity or the GPU time to process it, which 335 00:16:37,800 --> 00:16:41,320 describes almost everyone, then TST is the Holy Grail. 336 00:16:41,640 --> 00:16:44,320 You burn through your data faster and arrive at a highly 337 00:16:44,320 --> 00:16:46,360 capable model for a fraction of the cost. 338 00:16:46,880 --> 00:16:50,280 That's the dream scenario. However, if we look at the macro 339 00:16:50,280 --> 00:16:54,680 trends, many researchers believe humanity is rapidly approaching 340 00:16:54,680 --> 00:16:57,320 a data wall, right? We are simply running out of 341 00:16:57,320 --> 00:17:00,960 high quality human generated Internet text to train these 342 00:17:00,960 --> 00:17:02,800 massive networks. On we've read the whole 343 00:17:02,800 --> 00:17:04,240 Internet. Pretty much. 344 00:17:04,720 --> 00:17:08,000 So if the industry becomes data bound, a methodology that 345 00:17:08,000 --> 00:17:11,599 demands twice the token count to achieve its efficiency becomes 346 00:17:11,599 --> 00:17:14,280 much harder to justify. But the authors are completely 347 00:17:14,280 --> 00:17:15,800 transparent about this imitation. 348 00:17:15,920 --> 00:17:19,000 Very transparent, and they note that if data scarcity does 349 00:17:19,000 --> 00:17:22,280 become the primary bottleneck, the output only version of their 350 00:17:22,280 --> 00:17:24,800 method might actually be the most viable path forward. 351 00:17:25,000 --> 00:17:26,560 Because it doesn't need as much data. 352 00:17:26,680 --> 00:17:29,360 Right, because it provides a performance boost without 353 00:17:29,360 --> 00:17:32,400 drastically inflating the data requirements like the full TST 354 00:17:32,400 --> 00:17:34,320 does. We also have to acknowledge the 355 00:17:34,720 --> 00:17:36,960 regarding reproducibility at this scale. 356 00:17:36,960 --> 00:17:38,320 Yeah, that's important to touch on. 357 00:17:38,480 --> 00:17:41,880 Because running a 10 billion parameter model on 2 trillion 358 00:17:41,880 --> 00:17:44,440 tokens is an incredibly expensive endeavour, the 359 00:17:44,440 --> 00:17:47,800 researchers couldn't run a multiple identical large scale 360 00:17:47,800 --> 00:17:50,040 training runs. No, it's just too costly. 361 00:17:50,200 --> 00:17:53,760 In a perfect scientific vacuum, you want to run an experiment 362 00:17:53,840 --> 00:17:57,960 dozens of times to calculate strict statistical significance 363 00:17:58,200 --> 00:18:01,400 and rule out the possibility of a singularly lucky run. 364 00:18:01,760 --> 00:18:04,640 But the sheer cost forces a compromise In the experiment 365 00:18:04,720 --> 00:18:08,720 Mental Design, they demonstrated robust, repeatable results at 366 00:18:08,720 --> 00:18:12,680 the smaller 270 million and 600 million parameter scales. 367 00:18:13,120 --> 00:18:15,960 But at the frontier scale, the industry largely relies on 368 00:18:15,960 --> 00:18:19,200 single, massive training runs is proof of concept. 369 00:18:19,200 --> 00:18:22,240 So taking all of this together, the 2.5 X speed up the 370 00:18:22,240 --> 00:18:25,480 separation of concepts from syntax, the heavy data diet. 371 00:18:25,720 --> 00:18:28,640 Why does this trajectory matter for you, the listener, and for 372 00:18:28,640 --> 00:18:31,120 the ecosystem at large? There are a few big reasons. 373 00:18:31,240 --> 00:18:34,640 The most immediate impact is the democratization of artificial 374 00:18:34,720 --> 00:18:37,880 intelligence. If we drastically lower the 375 00:18:37,880 --> 00:18:42,040 compute barrier, we open the door for smaller research labs, 376 00:18:42,040 --> 00:18:44,760 universities and open source communities to build 377 00:18:44,760 --> 00:18:47,200 foundational model. They no longer need a multi 378 00:18:47,200 --> 00:18:49,120 billion dollar supercomputer to compete. 379 00:18:49,120 --> 00:18:51,320 Exactly. And beyond the democratization 380 00:18:51,320 --> 00:18:54,920 aspect, the reduction in energy consumption is a critical win. 381 00:18:55,880 --> 00:18:59,320 Training these models accounts for a massive growing share of 382 00:18:59,320 --> 00:19:02,960 global data center power usage. It's a huge environmental issue. 383 00:19:02,960 --> 00:19:06,080 Slashing the time those GPU spend running at maximum 384 00:19:06,080 --> 00:19:09,120 capacity translates directly to a dramatically lower carbon 385 00:19:09,120 --> 00:19:11,680 footprint for the industry. It fundamentally changes the 386 00:19:11,680 --> 00:19:13,480 physical footprint of the technology. 387 00:19:14,000 --> 00:19:16,840 But looking ahead, where does this research go next? 388 00:19:17,000 --> 00:19:19,640 Because the paper mentioned some fascinating next steps that they 389 00:19:19,640 --> 00:19:21,880 didn't have the space to fully explore here. 390 00:19:21,880 --> 00:19:25,000 Yeah, the most tantalizing open question is how TST interacts 391 00:19:25,000 --> 00:19:27,480 with long context memory. Oh, this is cool. 392 00:19:27,680 --> 00:19:31,400 Because the methodology folds text into these dense superposed 393 00:19:31,400 --> 00:19:36,480 bags, the model is effectively looking at a much longer horizon 394 00:19:36,480 --> 00:19:39,920 of information at any given moment compared to a standard 395 00:19:39,920 --> 00:19:41,080 sequential model. Right. 396 00:19:41,080 --> 00:19:43,920 So the theoretical implication being that it might naturally 397 00:19:43,920 --> 00:19:47,040 improve a model's ability to process and recall information 398 00:19:47,040 --> 00:19:49,240 across incredibly long documents, right? 399 00:19:49,320 --> 00:19:52,680 Exactly like an entire code base or a full length novel. 400 00:19:52,840 --> 00:19:56,080 They didn't specifically benchmark long context retrieval 401 00:19:56,080 --> 00:20:00,160 in this preprint, but it feels like the logical next frontier 402 00:20:00,160 --> 00:20:01,240 for this architecture. We. 403 00:20:01,240 --> 00:20:02,640 Will definitely be watching for those. 404 00:20:03,440 --> 00:20:05,360 For sure. Well, let's summarize the 405 00:20:05,360 --> 00:20:08,480 journey we've been on today. We started with the grueling 406 00:20:08,480 --> 00:20:10,840 linear cost of standard pre training. 407 00:20:11,480 --> 00:20:14,280 We explored how token superposition training 408 00:20:14,280 --> 00:20:17,840 completely flips that paradigm by physically blurring tokens 409 00:20:17,840 --> 00:20:21,040 into superposed bags. Forcing the model to learn the 410 00:20:21,040 --> 00:20:24,320 broad probability of future concepts using a moulting hot 411 00:20:24,320 --> 00:20:25,480 loss function. Right. 412 00:20:25,760 --> 00:20:28,800 We navigated the reality that this hyperfast concetual 413 00:20:28,800 --> 00:20:32,160 learning phase initially roduces completely nonsensical text 414 00:20:32,560 --> 00:20:35,040 gibberish, but it sets a profound foundation. 415 00:20:35,480 --> 00:20:38,840 And finally we saw how a strict recovery phase snaps the models 416 00:20:38,840 --> 00:20:42,600 grammar back into place, yielding a 2.5 X speed up and at 417 00:20:42,600 --> 00:20:46,680 the 10 billion parameter scale without ever altering the final 418 00:20:46,680 --> 00:20:49,600 inference architecture. It is an elegant, highly 419 00:20:49,600 --> 00:20:52,480 pragmatic solution to a deeply complex problem. 420 00:20:52,480 --> 00:20:55,600 It really is. We want to explicitly credit and 421 00:20:55,600 --> 00:20:59,320 thank the authors of this work, Bowen Peng, Theo Gigant, and 422 00:20:59,320 --> 00:21:02,120 Jeffrey Connell at News Research for their preprint paper, 423 00:21:02,400 --> 00:21:04,720 Efficient Pre Training with Token Superposition. 424 00:21:04,880 --> 00:21:06,600 Incredible. Absolutely. 425 00:21:07,200 --> 00:21:10,280 As we wrap up today's deep dive, I want to leave you with a 426 00:21:10,280 --> 00:21:12,960 thought to Mull over, kind of building on the underlying 427 00:21:12,960 --> 00:21:16,400 philosophy of this paper. If the world's most advanced 428 00:21:16,400 --> 00:21:19,520 artificial neural networks actually learn foundational 429 00:21:19,520 --> 00:21:23,280 structural concepts better and faster when we blur the rigid 430 00:21:23,280 --> 00:21:27,240 order of words, does that mean human language is bogged down by 431 00:21:27,240 --> 00:21:30,400 its own syntax? We spend so much time obsessing 432 00:21:30,400 --> 00:21:33,160 over the perfect sequence of words, the exact grammar, the 433 00:21:33,160 --> 00:21:36,480 precise phrasing. But maybe true understanding 434 00:21:36,480 --> 00:21:38,920 isn't found in the strict linearity of a sentence. 435 00:21:39,160 --> 00:21:41,760 Maybe it's found in the gravity of the concepts they form 436 00:21:41,760 --> 00:21:43,680 together. That is a fascinating way to 437 00:21:43,680 --> 00:21:45,720 look at it. Thank you so much for joining us 438 00:21:45,720 --> 00:21:48,280 on this deep dive on Build Wiz AI. 439 00:21:48,640 --> 00:21:50,760 Until next time, stay curious.