Imagine you're trying to read like a 10,000 page encyclopedia
to prepare for this massive exam, right?
That sounds absolutely terrible, right?
It's a nightmare and if you read every single letter sequentially
like CAT space SAT, you would just never finish.
No, you'd completely run out of. Time.
Exactly. But what if you could somehow, I
don't know, ingest an entire paragraph at a glance?
Like you aren't reading the words in strict order, you're
just absorbing this, this localized cloud of meaning.
Kind of like getting the gist of it all at once.
Yeah, exactly. And somehow this totally blurry,
disorganized method of absorbing information allows you to master
the material 2 1/2 times faster. Which is just a staggering
claim. It really is.
Well, welcome to build Wiz AI. We are so thrilled you're
joining us today. Yes, welcome.
Our mission here is pretty simple.
We take a stack of, you know, cutting edge research, we strip
away all the noise, and we extract the most valuable mind
expanding Nuggets so that you stay well informed without that
crushing info overload. Because there is a lot of noise
out there. So much noise and today's deep
dive is tackling a massive, like literally billion dollar
bottleneck in artificial intelligence.
Right, we are talking about the sheer brute force cost of pre
training large language models. It's just so expensive.
It's. Astronomical training these
frontier models requires mountains of compute power.
You're talking 10s of millions of dollars in GPU time and
massive energy consumption just to feed raw text into the neural
network. Just to read the text.
Exactly. It is the single biggest barrier
to entry in the field right now. It completely dictates who gets
to build foundational models. And who just gets totally priced
out, right? Which brings us to the source.
For today's deep dive, we are unpacking this fascinating
preprint paper titled Efficient Pre Training with Tokens
position. Such a cool concept.
It really is. It's authored by Bowen Peng,
Theo Gigant and Jeffrey Klisnel from Noose Research.
A great team over there. Yeah, they do incredible work
and our mission today is to explore a brilliant drop in
method they've developed that speeds up AI pre training by up
to 2.5 X. Without changing the models
actual architecture which is key.
That is the crucial part. But before we look at this new
method, we kind of have to talk about how the industry has been
trying to solve this compute crisis, because it's not like
researchers have just been sitting on their hands.
Oh, definitely not. But reading through the
background context of this paper, it feels like every
previous attempt to speed up training hits this very specific
wall. Yeah, the field has certainly
tried various angles of attack over the last few years.
Initially, a lot of focus went into information maximization.
The thought process there was pretty straightforward, right?
Compute is wildly expensive. Let's just get more learning
signal out of every single training sample we process.
Make every token count for more. Exactly.
That led to innovations like multi token prediction, where
the model is forced to predict multiple future tokens
simultaneously, right? So it creates A richer, denser
training signal per step. But that doesn't inherently
reduce the massive amount of matrix multiplication the model
has to do, right? Just to process the input text
in the first place. No it doesn't.
The compute burden is still totally there, so then the field
sort of pivoted toward trying to shrink the active size of the
model during a forward pass. Right, that was the shift toward
compute sparsity and the mixture of experts or Moe architecture
is the dominant paradigm. There, which we hear about all
the time now. Yeah, it's everywhere.
You keep the total parameter count high, but you route each
piece of text to a specialized sub network and expert.
So for any given input, only a fraction of the neural network
is actually active. You were doing essentially less
math per token. But even then, when that still
wasn't fast enough, we saw attempts at compressive modeling
right where researchers literally really tried to
compress the sequences internally.
Yeah, so fewer tokens flow through the heaviest, most
expensive layers of the network. But here is the massive catch
with almost all of these historical approaches.
And this is a flaw that Peng, Gigant and Connell are
specifically targeting in their paper.
It's a huge issue. Right, when you change how a
model processes data internally to make pre training faster,
like introducing complex internal compression mechanisms,
you are fundamentally altering the models architecture.
You're baking it in. Yeah, and if you change the
architecture, you permanently change the inference dynamics.
That is the exact crux of the problem because inference is
when the pre training is finished and you are deploying
the model for actual use. When you the user are actually
typing a prompt. Right.
If you introduce a bespoke compression module just to save
money during training, that module is stuck in the model
forever. It complicates deployment, it
can introduce serious latency, and it often requires custom
infrastructure just to serve the model to end users.
Which makes the author's approach here so refreshing.
They set this very strict constraint on themselves.
I actually want to quote their core research question verbatim
because it perfectly frames their entire hypothesis.
OK, here's what they asked. Can we improve a language models
pre training efficiency without modifying the final architecture
and its inference dynamics? And their answer to that
question is a resounding yes. It's.
Wild. They do it through a methodology
they call Token Superposition Training, or TST, and the way
they pull this off is by fundamentally blurring the lines
of language during the initial phases of training.
I really want to spend some time visualizing this for you because
the mechanism is genuinely wild. So in standard pre training, an
LLM processes a sequence of tokens strictly left to right.
Right it sees a token, computes its mathematical embedding, and
moves to the next. But what TST does in its first
phase, which they call the superposition phase, is it takes
a contiguous sequence of tokens. Like a chunk of text.
Yeah, let's say a bag of 4 to 8 tokens and it physically merges
them together before the heavy computation even begins.
Right, they take the mathematical embeddings of those
individual tokens and average them into a single combined
representation. Just mash together.
Exactly. They call this an S token or
superpose token. So from the perspective of the
models deepest, most computationally intensive
layers, those 4 distinct tokens now appear as one single dense
block of information. So going back to that speed
reading analogy from the beginning, if the text is the
cat sat on the model isn't computing the relationships for
the then for cat then for SAT? No, it's not.
It is computing the math for a single blurry mathematical
average of that entire localized concept.
It's absorbing the general vibe of that chunk of the sentence.
Vibe, I love that. And by doing that they are
drastically increasing the data throughput per FLOP.
And a FLOP. Just to clarify, a floating
point operation is the basic unit of computing power.
Exactly. If your model's layers are only
processing one superpose token instead of four individual ones,
you're basically pushing four times the amount of text through
the network for the exact same computational budget.
That's incredible, but hold on, I need to wrap my head around
the output side of this. OK yeah, because the whole point
of pre training a language model is next token prediction, right?
You show it text, it guesses the next token, and you use a loss
function to grade its guess. Right, normally you use A1 hot
cross entropy loss. Right, there is one
mathematically correct answer, and if the model guesses wrong,
it gets penalized. But how do you grade a model
when the input is a blurry bag of text?
Well, you have to fundamentally change how you measure the
mistake. Because the input is superposed,
the output target must be superposed too.
Oh, interesting. So they swap out the standard 1
hot loss for a multi hot cross entropy loss.
Multi hot, meaning there are multiple valid targets at the
same time. Exactly.
It asks the model to predict the next bag of words without any
regard for their exact sequence. Wait, really?
Order doesn't matter. Order does not matter at all.
The model doesn't have to know that cat comes before SAT.
It just has to predict that cat and SAT are highly probable
concepts that belong in the ucoming localized chunk of text.
Wow. Yeah, it is predicting the
probability distribution of future concepts rather than a
strict linear sequence. OK, I am seeing a massive like
glaring issue with this methodology.
I think I know what you're going to.
Say if we are training a multi billion parameter model to read
blurry bags of text, and we are rewarding it for predicting
blurry bags of future text, where the sequential order
doesn't matter at all. When you actually deploy that
model and ask it to write a Python script or compose an
e-mail, it is going to spit out absolute gibberish.
Oh yeah, it produces pure unadulterated nonsense.
OK, so I'm not crazy. Not at all.
The authors highlight this fact very clearly.
If you take a model trained exclusively with this phase one
TST method and try to use it for generation, it fails completely.
Totally useless. Pretty much.
It has learned a mixed probability of future tokens,
but it has completely lost the ability to string them together
into syntactically valid human readable sequences.
Which sounds like a spectacular failure.
If the model can't generate A coherent sentence, what is the
point of the compute savings? The point is that language
generation is actually a two-part problem, even if we
historically treat it as one. OK, unpack that.
There is the deep conceptual understanding of how the world
works, right? The relationships between facts,
logic, and semantics. The ideas?
Yes, and then there is the surface level syntax, the rigid
grammar rules required to express those concepts to a
human. TST essentially separates those
two learning processes. And that brings us to the
crucial Phase 2, the recovery phase.
This is the part of the paper that genuinely blew my mind
after running this hyper fast super pose training.
For the vast majority of the training run, they just hit the
brakes. Hard they do.
Completely stop the TST. They don't.
Just change a hyper parameter, right?
No, they physically he stripped the TST code out of the training
loop to ensure there is absolutely 0 contamination.
They revert the model strictly back to standard single token,
left to right, one hot next coken prediction.
And the model just recovers like it just figures it out.
It enters an incredibly rapid recovery period because the
model has already spent billions of compute cycles learning the
deep conceptual relationships between topics during the Super
post phase. It doesn't need to learn what
things mean anymore. It already knows the concepts.
Exactly. It only needs to relearn how to
format those thoughts. Yeah, the syntax and grammar
snap back into place astonishingly fast during this
brief recovery window. That separation of concept from
syntax is just a brilliant mental model for how these
networks actually learn. It really is.
But let's move from theory to the hard evidence, because the
numbers here are staggering. The researchers started small to
prove the concept, running tests on 270 million and 600 million
parameter models. Right, standard proof of concept
stuff. But the real proof is at scale.
They applied this to a 10 billion parameter A1B model, and
we need to clarify that architecture because it's vital
to why this works so well. Yeah, so the 10BA1B designation
means it is a mixture of experts model with 10 billion total
parameters, but only 1 billion parameters are active during any
single forward pass. Right.
So we're back to Maui. And why does TST synergize so
well with a mixture of experts models specifically?
It all comes down to routing. In a Maui model, you have a
router network that decides which expert sub network should
process a given token. When you feed that router a
superposed token, a blurry average of an entire phrase, the
router actually makes better macroscopic decisions
interesting. Instead of rapidly switching
experts word by word, which can be computationally inefficient,
the router sends the entire conceptual bag of text to the
expert best suited for that broader topic.
So it's smoothing out the routing decision exactly, and
the result of doing this at the 10 billion parameter scale up to
a 2.5 X reduction in total pre training time to reach the exact
same loss as a standard baseline model.
Massive. We really need to pause and
contextualize that for you. A 2.5 X speed up in this
industry is a tectonic shift. It really is.
When a single training run for a frontier model costs 10s of
millions of dollars, cutting that time by more than half
completely changes the economics of AI development.
And critically, that massive speed up did not degrade the
final capabilities of the model. That's the crazy part.
Yeah, the author's rigorously evaluated the recovered models
on standard downstream benchmarks.
These are the tests the industry uses to gauge actual
intelligence, right? Like Hellaswag for common sense
reasoning, ARC for science knowledge, Bull Q for reading
comprehension. The TST models consistently
matched or outperformed the baseline models that were
trained with standard methods for the same amount of
computational effort. So we have a method that is
significantly faster, drastically cheaper, and
mathematically just as capable. Tricks all the boxes.
But I want to dig into the ablation studies the authors ran
because understanding why this works reveals some very
counterintuitive properties of neural networks.
Oh, the ablation studies in Figure 6 of the preprint are
perhaps the most fascinating part of the research.
Yeah, let's unpack that. So TST is composed of two
distinct modifications, right? There's the input superposition,
where you blur the incoming words, and there's the output
superposition, where you use the multi hot loss to predict the
bag of future words. The researchers isolated these
variables to see exactly what was driving the performance.
The classic scientific method. Is it the blurry input doing the
heavy lifting or the blurry output target?
And the surprising answer is that both work independently,
but combining them produces A compounded effect with 0
interference. Really. 0 interference.
None. When they tested input only
superposition, the model learned faster.
When they tested output only superposition, it also learned
faster. Wow.
But when they stacked them together into full TST, the
efficiency game stacked perfectly.
Which implies they are solving 2 fundamentally different
optimization problems at the same time.
Exactly, and the authors propose a fantastic hypothesis for this.
They suggest that the input superposition acts as a form of
pre pre training. Free pre training.
Right, by exposing the neural network to a courser, simplified
distribution of language first it learns the broad statistical
structure of topics before it ever has to waste compute cycles
on the rigid rules of grammar. That is so interesting.
And it's a phenomenon we've actually observed in other
domains, like in computer vision.
Models often train faster if you feed them lower resolution
course image patches 1st and gradually increase the
resolution. Makes total sense.
This course to find learning schedule appears to be a
universal property of efficient neural network training, not
just an isolated trick for text. It makes perfect sense if you
think about like a master painter starting a portrait.
Oh, that's a good analogy. They don't pull out a
microscopic brush and start detailing the individual
eyelashes on a blank canvas. No, that would take forever.
Exactly. They use a massive brush to
block out the broad shapes, the lighting, the foundational
colors. Only when the structural
foundation is set do they switch to the tiny brush for the
precise details. And for years we've essentially
forced AI to paint with the tiny brush from day one.
TST finally let's the model block out the canvas first.
That captures the mechanics beautifully.
But you know, we have to put our critical lens on because every
methodological leap comes with a trade off.
We established the massive winning compute time.
Where does TST extract its toll? Well, the toll is paid in data.
To achieve that 2.5 X compute speed up on the 10 billion
parameter model, the TST model had to consume almost twice as
much raw text as the baseline. Almost twice as much.
Yeah. To hit the same performance
target, the standard baseline model process roughly 1.05
trillion tokens. The TST model had to ingest 2
trillion tokens. OK, that is a massive
discrepancy. It is, and it highlights one of
the most intense debates happening in AI research right
now. The tension between being
compute bound versus being data bound.
If your lab has access to endless data but you just can't
afford the electricity or the GPU time to process it, which
describes almost everyone, then TST is the Holy Grail.
You burn through your data faster and arrive at a highly
capable model for a fraction of the cost.
That's the dream scenario. However, if we look at the macro
trends, many researchers believe humanity is rapidly approaching
a data wall, right? We are simply running out of
high quality human generated Internet text to train these
massive networks. On we've read the whole
Internet. Pretty much.
So if the industry becomes data bound, a methodology that
demands twice the token count to achieve its efficiency becomes
much harder to justify. But the authors are completely
transparent about this imitation.
Very transparent, and they note that if data scarcity does
become the primary bottleneck, the output only version of their
method might actually be the most viable path forward.
Because it doesn't need as much data.
Right, because it provides a performance boost without
drastically inflating the data requirements like the full TST
does. We also have to acknowledge the
regarding reproducibility at this scale.
Yeah, that's important to touch on.
Because running a 10 billion parameter model on 2 trillion
tokens is an incredibly expensive endeavour, the
researchers couldn't run a multiple identical large scale
training runs. No, it's just too costly.
In a perfect scientific vacuum, you want to run an experiment
dozens of times to calculate strict statistical significance
and rule out the possibility of a singularly lucky run.
But the sheer cost forces a compromise In the experiment
Mental Design, they demonstrated robust, repeatable results at
the smaller 270 million and 600 million parameter scales.
But at the frontier scale, the industry largely relies on
single, massive training runs is proof of concept.
So taking all of this together, the 2.5 X speed up the
separation of concepts from syntax, the heavy data diet.
Why does this trajectory matter for you, the listener, and for
the ecosystem at large? There are a few big reasons.
The most immediate impact is the democratization of artificial
intelligence. If we drastically lower the
compute barrier, we open the door for smaller research labs,
universities and open source communities to build
foundational model. They no longer need a multi
billion dollar supercomputer to compete.
Exactly. And beyond the democratization
aspect, the reduction in energy consumption is a critical win.
Training these models accounts for a massive growing share of
global data center power usage. It's a huge environmental issue.
Slashing the time those GPU spend running at maximum
capacity translates directly to a dramatically lower carbon
footprint for the industry. It fundamentally changes the
physical footprint of the technology.
But looking ahead, where does this research go next?
Because the paper mentioned some fascinating next steps that they
didn't have the space to fully explore here.
Yeah, the most tantalizing open question is how TST interacts
with long context memory. Oh, this is cool.
Because the methodology folds text into these dense superposed
bags, the model is effectively looking at a much longer horizon
of information at any given moment compared to a standard
sequential model. Right.
So the theoretical implication being that it might naturally
improve a model's ability to process and recall information
across incredibly long documents, right?
Exactly like an entire code base or a full length novel.
They didn't specifically benchmark long context retrieval
in this preprint, but it feels like the logical next frontier
for this architecture. We.
Will definitely be watching for those.
For sure. Well, let's summarize the
journey we've been on today. We started with the grueling
linear cost of standard pre training.
We explored how token superposition training
completely flips that paradigm by physically blurring tokens
into superposed bags. Forcing the model to learn the
broad probability of future concepts using a moulting hot
loss function. Right.
We navigated the reality that this hyperfast concetual
learning phase initially roduces completely nonsensical text
gibberish, but it sets a profound foundation.
And finally we saw how a strict recovery phase snaps the models
grammar back into place, yielding a 2.5 X speed up and at
the 10 billion parameter scale without ever altering the final
inference architecture. It is an elegant, highly
pragmatic solution to a deeply complex problem.
It really is. We want to explicitly credit and
thank the authors of this work, Bowen Peng, Theo Gigant, and
Jeffrey Connell at News Research for their preprint paper,
Efficient Pre Training with Token Superposition.
Incredible. Absolutely.
As we wrap up today's deep dive, I want to leave you with a
thought to Mull over, kind of building on the underlying
philosophy of this paper. If the world's most advanced
artificial neural networks actually learn foundational
structural concepts better and faster when we blur the rigid
order of words, does that mean human language is bogged down by
its own syntax? We spend so much time obsessing
over the perfect sequence of words, the exact grammar, the
precise phrasing. But maybe true understanding
isn't found in the strict linearity of a sentence.
Maybe it's found in the gravity of the concepts they form
together. That is a fascinating way to
look at it. Thank you so much for joining us
on this deep dive on Build Wiz AI.
Until next time, stay curious.