Welcome back to the Deep Dive. So we're shifting gears a little
bit today. Yeah, a new format.
A new format. Usually we look at, you know,
the results of technology, how it changes society, the market
impact. Right, the big picture stuff.
But today we are taking a strictly engineering
perspective. We're launching what we're
calling the Practical AI Digest. I love this.
So we're going to take one specific architecture, just one
strip away all the marketing hype and explain exactly how it
works so you can actually understand the mechanism under
the hood. Exactly.
And the topic for today is something that I think a lot of
people feel they understand on a surface level, but the actual
mechanics are quite surprising. Yeah, we are looking at the
engine behind the massive, massive explosion in AI art
we've seen over the last couple of years.
We're talking about latent diffusion models, LDMS. This is
the thing that powers Stable Diffusion is basically the
reason you can now generate these incredible high fidelity
images on a home computer. Instead of needing a giant
server farm. And that is the key hook here.
I mean everyone has seen the output you know mid journey
deli, stable diffusion, you type a cyberpunk city in the style of
Van Gogh and boom 5 seconds later you have a masterpiece.
But very few people ask the next question, which is what is the
engine driving this? Yeah.
And specifically, how did we get from models that needed, like
you said, a super powder to something that can run on a
gaming laptop? And that's the core conflict.
You know the problem we need to address before this paper?
The high resolution image synthesis with late diffusion
models paper. This whole field was incredibly
expensive. It was a witch man's game,
totally. Let's put a number on expensive
because I think people just assume AI lives in the cloud and
it's, you know, basically free. Oh, it's so far from free.
Yeah. I mean, if you look at the
source material, they highlight that training one of these
powerful pixel based diffusion models.
Yeah, it took hundreds of GPU days.
Hundreds. Yeah, they cite examples from
like 150 up to 1000 V 100 days. OK, let's just clarify that for
you listening. AV 100 is a top tier enterprise
grade graphics card. So 1000 V 100 days means if you
had one of these very expensive cards, it would take you almost
three years to train the model once, just once.
Exactly. And that's just the training
part, the inference. So actually making an image
after you type a prompt, that was also painfully slow.
Why? Because the neural network had
to sequentially evaluate a massive, massive amount of data.
So the mission of this deep dive is to unpack the architecture
that solved this bottleneck. We're not just making images.
We are, I guess, making them efficiently.
We're going to break down the latent part of latent diffusion
models. It's what democratize high
resolution image generation and the whole trick.
The core idea was separating the compression phase from the
generative phase. OK, so let's get into that
Section 1, The intuition and the problem with pixels.
So before we can understand why latent diffusion is so special,
we need to understand what standard diffusion even is.
Right, the baseline. If I'm a developer looking at
this for the first time, what's the basic mechanism?
The basic idea is something called iterative denoising.
It's. You can think of it as a model
that's designed to learn a data distribution.
Bye. By gradually removing noise.
It's a two way St. the forward. Process and the reverse process.
Right, so the forward process is fixed.
It's easy. You take a clean image, let's
say a photo of the cat, and you just gradually add Gaussian
noise to it. A little bit at a time.
Step by step, so you know, step one, it's a little grainy.
Step 50 it's very grainy. By step 1000 is just Ure static,
the image is completely gone. OK, that seems simle enough.
Ruining a iPhoto is easy. It is the magic is the reverse
process. This is where you train a neural
network to look at that Ure static and predict the noise
that was added at the previous step.
Ah, so it's learning to go backwards.
Exactly. Because if you can predict the
noise, you can subtract it. Yeah.
And if you do that over and over, step by step backwards
from total randomness, you eventually reveal a clear image.
So the model is kind of like a professional art restorer.
It's looking at this messy noisy signal and saying, OK, I know
what part of this is just static and what part is actual
structure. And it just wipes the static
away. That's a perfect way to put it.
And ideally you end up with a beautiful high fidelity image.
You do, but here's the problem, and it's a huge problem with
standard models like Dally 2 or the original guided diffusion
models. They do all of this in pixel
space. Meaning it's literally
calculating every single pixel. Every single pic.
Think about it, you have a 10/24 by 1024 image.
That's over 1,000,000 pixels and each.
One has three color channels. Red, green and blue.
So it's more like 3,000,000 data points, and the model has to
process that entire massive grid for every single step of the
denoise. Every step which can be hundreds
or thousands of steps. OK, I see it now.
And this is where we hit what the paper calls the pixel space
bottleneck, right? And this right here is probably
the most critical conceptual take away from this whole thing.
Digital images contain a massive amount of information, right?
But not all of that information is actually important.
Semantically important. What do you mean by semantically
important? OK, well imagine a photo of a
brick wall. The semantic information is this
is a brick wall. It's red.
The mortar is Gray. The concept.
The concept exactly, but the actual digital image also
contains all this high frequency detail.
High frequency meaning like the microscopic grain on the brick.
Yes, imperceptible noise. Tiny, tiny shifts in the
gradient of a shadow on one single brick.
The exact texture of the mortar if you zoomed in 1000 times.
All these little details take up a huge amount of data to encode.
But do they matter to me as the person looking at the picture?
Not really. I mean, if I showed you a
picture of that wall and then I showed you another one where I
just slightly blurred the texture or shifted the noise
pattern, you still just say that's a brick wall.
Right, the wallness of it hasn't changed.
Exactly. But the standard diffusion
models, they don't know that they.
Can't tell the difference between the concept and the tiny
details. No, because they're what are
called likelihood based models. Mathematically, they're
obligated to model everything, so they spend a completely
disproportionate amount of their compute power modeling these
imperceptible high frequency details.
So it's like trying to write a book summary by, I don't know,
analyzing the microscopic fibers of the paper on every single
page. Instead of just reading the
plot. Instead of just reading the plot
points, yeah. That is the perfect analogy.
You are burning these incredibly expensive GPU cycles just to
figure figure out the texture of the paper, but all you really
want is the story. That's the difference between
perceptual compression and semantic compression.
So the goal of the latent diffusion model is to stop
looking at the paper fibers. Yes, the goal is to find a
representation of the image, a different space that is
computationally way way cheaper, but is perceptually equivalent
to the original. We want to find a space where we
can do the heavy lifting of generation without being bogged
down by millions of useless pixels.
Precisely. OK, this brings us to Section 2,
the system architecture. The paper splits this whole
system into two very distinct phases, and phase one you said
is strictly about compression. Right, first we need to get out
of pixel space and to do that they introduce a component
called an auto encoder. OK, for the engineers listening,
let's breakdown the specs of this auto encoder.
So an auto encoder has two parts.
They have an encoder, which we can call egoder, which we'll
call encoder and encoder. The encoder takes your high
dimensional image 6 LR. That's your massive grid of RGB
pixels, and it crunches it down to something called a latent
representation. We call that D getter.
So $6 is the big original image and Zeidol is the tiny
compressed version, correct? And Zelda lives in what we call
latent space. It's a much, much lower
dimensional space. The paper talks about a specific
parameter for this, a down sampling factor, which they call
Philidol. And what is that control?
It just controls the ratio of compression.
So if Philidol is equal to 4, the latent representations has a
height and width that's 1/4 of the original image.
And if $5 is 8, it's 18. So we're shrinking the map.
But I mean, that brings me back to my earlier concern.
When you shrink an image, you lose quality.
If I take a big photo in Photoshop and I shrink it down
to 10% and then try to blow it back up, it looks awful.
It's all pixelated and blurry. How does the LDM avoid that?
This is such a critical design choice in the paper.
They don't just use a standard loss function like an L1 or L2
loss. Mean squared error things like.
That right, those standard loss functions tend to produce really
blurry results because the model is trying to average out all the
uncertainty. Right, the mean of all possible
sharp edges ends up being just a blurry gradient.
Exactly. So to keep things sharp, they do
something clever. They use a perceptual loss
combined with a patch based adversarial objective.
Adversary. OK, that sounds like we're in
Jan territory. It's borrowed directly from Jans
Generative Adversarial Networks. They essentially have a second
network, a discriminator, that looks at the reconstructed image
that comes out of the decoder. And it critiques it.
It critiques it. It basically asks does this
little patch of the image look like a real texture?
Is this sharp or is this blurry mush?
So it forces the auto encoder to be honest, it can't just get
away with a blurry approximation.
No, it has to maintain what they call local realism.
Local realism I like. That, and that's what ensures
that even though Siller is highly compressed, when you run
it back through the decoder dollar, the output image looks
crisp and detailed. Now, there's a technical nuance
here about regularization. The paper mentions two ways to
keep this latent space well behaved.
Why do we care if the latent space is well behaved?
Well, imagine if you just mapped a bunch of images to a latent
space and the points were just scattered completely randomly.
So a cat is here, and a millimeter away is a toaster.
In in between it's just garbage static.
That would be a high variance chaotic space.
You can't navigate that. You can't, and you certainly
can't train a diffusion model on it.
Diffusion models need a smooth continuous space that they can
walk through, so regularization is about forcing the latent
vectors to follow a standard predictable distribution.
And what are the two options they talk about?
Option A is called KL regularization.
This adds A slight penalty that kind of nudges the latent
distribution towards a standard normal distribution is very
similar to how variational auto encoders or VA ES work.
OK. And option B.
Option B is VQ regularization, which stands for Vector
quantization. This uses a kind of code book of
learned vectors. It quantizes the latent space
into discrete separate steps. This is more like what you see
in VQGAN. So which one did they end up
going with? They experimented with both, but
the kale regularization version is what became the standard for
Stable Diffusion. It just offers a really nice
balance of quality and stability.
OK, so let's just recap phase one.
We take an image, we use an encoder to crunch it down into a
tiny efficient latent vector 7. We then use a decoder to prove
that we can turn Zeiger's back into a sharp looking image, but
we haven't generated anything new yet.
Not a thing. We're just compressing and
decompressing. It's just building the stage.
Exactly. Phase 2 is where the performance
actually happens. This is latent diffusion.
This is the core shift. We are no longer diffusing
pixels, we are diffusing the latent vectors.
Right. We take that compressed
representation 70 hours, we add noise to that, and then we train
a model to remove the noise from Ziller.
So we are essentially hallucinating the compressed
concept of an image rather than the image itself.
That's a great way to put it. And because ZZS is so, so much
smaller than the original Image 60's, the computational cost
just drops off a Cliff. OK, let's talk about the
backbone. What's the neural network that's
actually doing this work? Is it a transformer?
It feels like everything is a transformer these days.
Right, that's the default assumption.
Daily one used a transformer. It did, but no.
And this is a really specific and I think very smart
engineering decision. They use a time conditional U
net AU. Net.
Why AU? Net.
So the paper argues that Transformers are naturally
sequence based. They treat data like a stream of
words in a sentence. One after another.
But images, even latent images, they aren't sequences, they're
2D spatial grids. They have structure.
The top left corner is related to the top right corner in a way
that the first word of a sentence isn't necessarily
related to the last. Exactly.
A unit is built from convolutional layers, and
convolutions are designed specifically to process spatial
grids. They have what's called an
inductive bias towards images. They naturally understand things
like locality and, you know, translation and variance, so.
By using a unit, they're working with the structure of the data
and not trying to force it into a different shape, and I assume
this is faster. Much faster.
Remember our down sampling factor $5?
If $5 is 8, the latent space has 64 times fewer data points than
the original pixel space, so the UNET is processing a tiny, tiny
fraction of the data that a standard diffusion model would
have to. And that explains how we can run
this on a single consumer GPU. That's the whole ball game.
This design allows for training on limited resources and,
crucially, very efficient inference.
Let's look at the math for just a second.
We don't need to get bogged down, but for the engineers
trying to implement this, what's the objective function?
What is the model actually trying to minimize?
It's a simplified version of the standard diffusion loss.
The formula is the LLDM equals the expectation of the squared
difference between epsilon and epsilon Theta.
OK, let's translate that into plain English for sure, yeah.
You have the actual noise that's epsilon which you just sampled
from a normal distribution. OK, And you have the predicted
noise that's epsilon Theta, which comes from your neural
network, EU net, right? EU net looks at the noisy latent
vector Z dollar at a specific time step T dollars, and it
makes a prediction. The objective is simply to make
the predicted noise as close as possible to the actual noise.
So the model is just playing a guessing game.
It's saying I see this messy static.
I think the noise you added looks like this specific
pattern. And we grade it on how close
it's guess was. If it guesses the noise
correctly, we can abstract it and take one step closer to a
clean image. That's it.
So now we have a system that can generate random images from
noise, and it can do it really efficiently.
But we don't want random images. No, we want specific images.
We want a cat eating pizza. Right, and this brings us to
Section 4, conditioning. This is the mechanism that
enables text to image generation.
How do we tell the model what to generate?
I mean EU net speaks this spatial image language, but our
prompts are in language sequence language.
We need a bridge between the two.
We do and the solution they implemented is brilliant.
It's called cross attention. This is a mechanism that's
borrowed from the transformer world, isn't?
It it is, yeah. They effectively graphed these
cross attention layers onto the unit backbone, but they make the
really smart architectural move here.
They separate the text processing from the image
generation. They introduce what they call a
domain specific encoder, which they label as taffeta.
Taffeta. It's a totally separate model.
It could be bird, it could be a colleague model that takes your
input, say your text prompt right, and IT projects it into
an intermediate representation. So the text goes into its own
dedicated processor first, before it ever touches the image
generator. Yes, and only then is that
process text representation fed into the unit through the
attention mechanism. If you think of the attention
formula, attention to QKV. Query key and value.
Right, let's map those terms to this system.
OK, who's the query? The query comes from the unit.
It's coming from the intermediate layers of the image
in progress. You can think of it as the image
asking questions. And the keys and values the.
Keys and values come from our conditioning input, so the text
prompt after it's been processed by that tough head encoder.
So visually, can you walk me through what's happening here?
OK, imagine the unit is trying to denoise a little patch of the
latent image. It generates a query that is
effectively asking what is supposed to be in this top left
corner right now. OK, the crossattention layer
then takes that query and compares it to the keys from the
text prompt. If the prompt is a red cat on a
blue table, the keys for red and cat might light up for that
area. They become.
Relevant. They become relevant and then
the values provide the actual information.
This area should have red furry texture.
That's a fantastic explanation that So the image generation
process is literally querying the text prompt for guidance at
every single step of the denoising.
And this design is so incredibly flexible because the unit only
interacts with the conditioning through this clean cross
attention interface. The conditioning input can be
anything. Doesn't have to be text.
Nope. As long as you have a domain
specific encoder atop of data that can turn your input into
the right shape for the key and value pairs, you can.
Plug anything in. You can condition on anything.
You can use semantic maps, so like a color-coded layout.
You can condition on class labels or even other images.
And this explains why the whole Stable Diffusion ecosystem is so
modular. People are constantly swapping
out these conditioning encoders to build things like control net
or IP adapter. Without having to retrain the
core unit from scratch. It's a plug and play
architecture for control signals.
Exactly. OK, so we've covered the auto
encoder, the latent diffusion process itself, and the
conditioning. Now let's get practical Section
5 trade-offs in tuning. If I'm an engineer trying to
implement this, or even just a power user trying to optimize my
workflow, what are the levers I need to know about?
The paper spends a lot of time on that down sampling factor,
$5. Yes, this is the sweet spot
analysis. They tested a whole range of
factors and five five of 124816 all the way up to 32.
And just to remind everyone, $5.00 is how much we compress
the image, right? So let's look at the extremes.
What happens at five $302.00? That's a massive amount of
compression. It is at feet three, 302 hours.
You're squashing a 256 by 256 image all the way down to an 8
by 8 grid. It's tiny.
And the benefit is speed I assume.
The benefit is the diffusion model is lightning fast.
It has almost no data to process.
But the downside is the auto encoder just fails.
It abstracts away way too much information.
And that becomes too simple to be useful.
Exactly. The perceptual quality just
stagnates. You lose all the texture, you
lose text legibility, the reconstruction looks.
I don't know, sketchy because the latent space just doesn't
have the capacity to store the fine details anymore.
OK. And what about the other
extreme? And four of one or two?
Well, 4111 is just pixel space. You're right back to the
original problem. So it's slow high memory usage.
You're not gaining any of the efficiency benefits that make
the LDM architecture so great. So what's the verdict?
Where's the Goldilocks zone? So the paper concludes that the
audible balance is somewhere between in five of four and in
five of eight. This is where they achieved the
best FID scores. And FI.
First yet inception distance. It's basically the standard
metric for measuring how real generated images look compared
to a data set of real images. Lower is better.
OK. And the models they call LDM 4
and LDM 8 down sampling by 4:00 and 8:00, they achieved
state-of-the-art results on day sets like Celebi HQ and
Imagenet. And they were faster.
They significantly outperformed the pixel based models in sample
throughput, which is speed, while also getting lower FID
scores which is better quality. It is so rare in engineering to
get faster and better at the same time.
Usually you have to pick one. It's very rare, and that's
really why this paper was such a breakthrough.
But there's another practical feature that I think is even
cooler. It's called convolutional
sampling. And what does that?
Enable. It enables resolution solution
independence. Most transformer based models
are totally stuck generating images at the exact resolution
they were trained on. Right.
If you trained on 256 by 256, you get 256 by 256.
That's it. But LDMS are different.
Because the model is fully convolutional.
It learns features rather than fixed positions.
It learns what a tree looks like or what sky looks like.
Which means you can take a model that was trained on 256 by 266
images and just apply it to a 10/24 by the 424 canvas.
Yeah, it doesn't just break. It doesn't break, it just
generates more stuff. If you ask for a landscape,
it'll just generate A wider landscape with more trees and
more mountains. The paper shows these amazing
examples of generating huge, consistent panoramas that are
way larger than any of the training images.
That is. Huge for creative applications.
It implies the model understands textures and relationships on a
local level, so it can just tile them indefinitely.
It also enables these really powerful editing workflows like
in painting. So filling in a missing part of
an image. Exactly.
You can mask out a part of an image and just ask the model to
fill it in. The LDM handles this completely
naturally. It just treats the unmasked part
as known latent codes and then diffuses the mask part to match
the context. How does it stack up up against
specialized tools built just for that?
So the paper compares it against specialized in painting
architectures like a model called Llama, and the LDM
achieves competitive performance.
Which is pretty remarkable for a general purpose.
Model. It's amazing.
It wasn't built just for in painting, but it happens to be
great at repair work too. And what about super resolution
upscaling? Same story, you can use LDMS to
upscale low res images and because they're likelihood based
they avoid something called mode collapse.
That's a classic jam problem right?
Where the model just learns to output the same generic good
looking face over and over. Again, exactly.
LDM's preserve diversity, and unlike the big autoaggressive
models, they don't need billions and billions of parameters to do
it. They're just efficient, diverse,
and sharp. So if we step back and look at
the big picture here, the LDM architecture really is a kind of
best of all worlds approach. I think so.
They took the perceptual quality of Jans through that auto
encoders discriminator, they took the stability and diversity
of diffusion models, they took the control of Transformers
using cross attention, and they combined all of it into a system
that's efficient enough to run on consumer hardware.
That is the summary. They identified that the pixel
was the wrong unit of currency for generation.
The latent vector is the right unit.
Let's wrap up with the aha moment.
For me, the genius of latent diffusion is that separation of
concerns. You don't need to paint every
single pixel from scratch. You can dream in this compressed
abstract space, the latent space, and then you have a
separate dedicated translator, the decoder, to turn that dream
into a high fidelity reality. I like to think of it like a
chef versus a chemist. The diffusion model is the chef,
It's the artist. It decides what ingredients go
together to make a delicious meal.
The auto encoder is the chemist. It worries about the precise
molecular structure of the food. But the chef doesn't need to
know chemistry to cook, right? They just need to know what
flavors work together. And by separating those roles.
The chef can work much much faster.
And the impact of the separation has been huge.
It completely reduced the barrier to entry from tech giant
supercomputer down to single GPU.
It effectively democratized high resolution image synthesis.
Yeah, and that's where we're seeing this incredible explosion
of creativity and tools in the open source community right now.
So here's a final provocative thought for you to Mull over.
We've just spent this whole time talking about how compressing
images into latent space let's us generate visuals efficiently,
right? We're already seeing similar
logic being applied to audio generating music by diffusing
spectrograms in a latent space. Yeah.
What happens when we start applying this latent diffusion
logic to video day or to entire 3D environments?
We are already standing on that precipice.
I mean, if we can dream up a static snapshot on a home
computer today, are we moving toward a future where we can
generate entire realities? You know, movies, video games,
immersive worlds, all of it all optimized to run on the hardware
you already on? We're moving from generating
snapshots to generating realities.
And the engine under the hood is exactly what we talked about
today. It's going to be a wild.
It certainly is. I hope this deep dive gave you a
solid mental model of how the magic trick actually works.
Thanks for listening and we'll see you in the next deep dive.