Welcome back to the Deep Dive. So I was looking at a photo of a
modern beta center the other day and it just, it struck me.
It didn't look anything like the server rooms I used to see in,
you know, corporate offices. The clean white rooms with all
the blinking blue lights. Read The Sterile Environment.
Yeah, this this looks like the engine room of a nuclear
submarine. It basically is an engine room.
It really felt like it just rows and rows of these black monolith
cabinets in this massive plumbing and structure
everywhere for liquid cooling, and you could almost feel this
low vibrating hum of incredible energy just from the picture.
It felt heavy. It is heavy physically and in
terms of power draw. And that's really what triggered
today's deep dive. We spend so much time analyzing
the mind of AI, the algorithms, prompt engineering, the ethics.
Chatbots that write poetry. Exactly.
But we almost never stopped to talk about the brain, the actual
physical matter that makes any of that thinking possible.
Which is, you know, kind of ironic.
Yeah, because without the specific breakthroughs in
silicon and physics we've seen over the last decade, none of
the software we talked about would even exist.
Not at all. Jet GPT would still just be a
theoretical paper on a dusty shelf somewhere.
So that is our mission today. We are stripping away the hype
to look at the metal. We're digging into the A
Sixteens AI hardware explained guide and also the latest info Q
Trends report to really understand the physical engine
of artificial intelligence, right?
We are not talking about prompts today.
We were talking about the silicon that makes them think.
The Foundation. And I'll be honest, I'm kind of
relieved we're doing this because I want to finally
understand why NVIDIA is so important beyond just, you know,
the stock price and it. Really is the bottleneck of the
future. I mean, software is infinite,
you can always write more code. But physics?
Physics is finite. And the hook here, and this is
the thing that really surprised me when I was reading through
the sources, is that this entire revolution is, well, essentially
an accident. It's completely accidental.
That's the amazing part of the story.
That part just blew my mind. The hardware that's driving the
most advanced technology in human history.
It wasn't built for scientists. No, it wasn't built for AI
researchers at all. It was built for teenagers.
Exactly. It was built for gamers trying
to get, you know, ridiculously high frame rates on Call of
Duty. And then later for the crypto
miners? And later for the crypto guys
mining Bitcoin in their basements.
Yeah, the gaming KC and the Bitcoin mining rig, those were
the prototypes for the AI supercomputers we see today.
OK, so help me bridge that gap. How do we go from shooting
zombies and 4K resolution to building large language models?
What is the architectural link between a video game and a
neural network? It all comes down to a really
fundamental difference in how computers will not think, but
how they process instructions. To understand AI hardware, you
have to understand the difference between the CPU, the
central processing unit that's probably in your laptop right
now, and the GPU, the graphics processing unit.
Let's start with the CPU then. This is the Intel or AMD chip,
the brain of personal computing for what, 40 years now?
Right. And you can think of the CPU as
a genius mathematician, or maybe a really sharp high-powered
executive. OK, I like that analogy.
It's designed to be incredibly versatile.
You can run your operating system, open a huge spreadsheet,
play a song, check your e-mail, manage your WI.
Fi all at the same time, basically.
Yeah, it switches between all these different tasks
constantly. It's the ultimate multitasker.
So it's a generalist. It is, but structurally it's a
sequential thinker. A classic CPU is designed to
take a complex logic problem and work through it step by step.
One instruction at a time. One instruction at a time.
Historically now, even modern CPUs with multiple cores, say 12
or 16 cores, are still just a small team of those genius
mathematicians. They're optimized for latency.
Latency, meaning speed on one task.
Exactly. They want to finish one hard
task as fast as possible so they can get to the next one.
So if you give a CPU a calculus problem or a complex logic tree
like if this then that, but it's not, then go over here.
It solves it instantly brilliantly.
Yeah, now let's look at the GPU, the graphics processing unit.
The other side of the coin. Completely.
If the CPU is that genius mathematician, the GPU is an
army of 10,000 laborers. None of them are particularly
smart on their own. They can't run an operating
system. They're terrible at complex
logic branches or jumping between different tasks.
But there are just a ton of. Them massive amounts.
While ACPU might do say 10s of operations per cycle, a modern
AI accelerator or GPU can execute over 100,000
instructions per cycle. 100,000 that's that's a completely
different scale. It's a different universe.
They are built for throughput. They're designed to do 1 simple
thing, but do it to a massive amount of data all at once.
And this is where the gaming connection comes back in.
Because rendering a video game, that's not a logic puzzle,
right? Think about your screen.
A 4K screen has roughly 8 million pixels to render a
single frame of video game, and you want sixty of those every
second. You have to calculate the color,
the lighting, the texture for every single one of those 8
million pixels that the the exact same time.
You can't do it 1 by 1. You can't.
You don't want to calculate pixel one, then pixel 2, then
pixel three. It would take forever.
You need to do the whole screen at once.
So it's a parallel problem. It is the definition of a
parallel problem. It's a massive, relatively dumb
math problem where you perform the exact same calculation
millions of times simultaneously, and it turns out
that is exactly mathematically what a neural network is.
OK, I want to double click on that.
Why is a neural network like a video game screen?
That feels like a leap. It's because deep learning, at
its very core isn't a complex logic tree like old software.
It's not a bunch of isn't statements, right?
It represents information as massive stacks of matrix
multiplications. It's just rows and columns of
numbers. We call them matrices being
multiplied against each other over and over again to find
patterns. So when I asked Strat GPTA
question, it's not thinking in a straight line like ACPU
processing a script. Not at all.
It's activating billions of parameters all at once to
predict the next word. Correct.
It's taking a massive grid of numbers that's your input and
multiplying it through layers of other massive grids of number.
Those are the weights of the model.
You don't need the genius mathematician for that.
You need the army of laborers. You need the army of laborers
doing massive simultaneous arithmetic.
You know, this reminds reminds me of that famous Marc
Andreessen quote from what 2011 Software is Eating the World?
But looking at these reports and seeing how the demands of neural
networks are just completely reshaping the entire
semiconductor industry, it feels like the dynamic has shifted.
It feels like hardware is now eating software.
I'd say it's a symbiotic relationship, but you're right,
the hardware is now dictating the capability.
And if you're a software engineer listening to this, the
big take away here is that you have to stop thinking in loops,
which is that sequential CPU style of.
Thinking and start thinking in tensors.
And start thinking in tensors absolutely.
OK, let's define that word, because we hear it everywhere.
Tensor flow, tensor cores, TPU. It sounds incredibly technical,
but what actually is a tensor? It really sounds like sci-fi
jargon, doesn't it? But it's actually just geometry.
I mean, you know what vector is. Yeah, a line of numbers like a
list in an array, right? A1D array and a matrix.
That's a grid of numbers, like a spreadsheet, rows and columns
SO2DA. Tensor is just the next step up.
It's a multi dimensional array. It could be a cube of numbers
that would be 3D. Yeah.
Or a structure with even more dimensions.
It's just a generic term for a container of numbers.
So it's just a bucket for numbers.
Essentially, yes. And the reason we talk about
them so much is that companies literally started building chips
specifically to move these containers around.
You mentioned Google's TPU, the Tensor Processing Unit.
The name is literal. The name is completely literal.
It is a processor physically designed to manipulate these
specific data containers and multiply them against each other
as fast as humanly possible. So we have the architecture down
the T-21 because it's a parallel math engine.
OK, now let's talk about the landscape of players.
We can't have a conversation about AI hardware without
talking about the king of the hill, NVIDIA.
Their data center revenue is just astronomical.
But here's something I've always found a little puzzling.
If the hardware is just about parallel math, why hasn't Intel
or AMD or Qualcomm just caught up?
That's a great question. I mean why is NVIDIA a trillion
dollar company and everyone else seems to be fighting for the
scraps? Is their silicon really that
much better? And this is probably the most
misunderstood part of the entire NVIDIA story.
Everyone looks at the H-100 or the new Blackwell chips and they
think the magic is in the silicon.
And it's not. Well, the silicon is amazing,
don't get me wrong. It's a marvel of engineering,
but that's not the Moat. So what is?
The Moat is software. You're talking about CUD.
CUDA, Exactly. It stands for Compute Unified
Device Architecture. It's the software layer, the
programming model that allows developers to actually talk to
the GPU. And they've been working on this
for a while. NVIDIA released this almost 20
years ago, way way before the AI boom.
They spent decades building out libraries, tools, and
optimization protocols that let programmers easily send
instructions to the graphics card for general purpose
computing. So it's the ecosystem, it's not
just the chip. It is 100% the ecosystem.
If I'm a researcher at a university and I download an
open source model from GitHub today, the chances are what, 99%
that the code is written to run on CDA.
Which means it runs out-of-the-box on NVIDIA
hardware. Instantly out-of-the-box, no
hassle. And if I try to run that same
code on say an AMD chip. It breaks, or it one's terribly
slowly. You have to use these clunky
translation layers or you have to manually rewrite the code for
AM DS competing platform ROCM. Which nobody wants to do that.
Friction is massive. It's vendor locking at the
deepest, most fundamental level. The hardware is great, sure, but
the software ecosystem is what keeps the competition at Bay.
Engineers want things to just work.
But the competition is coming. The sources list a few key
players that are trying to chip away at this lead.
We mentioned Google's TPU, which they mostly use internally.
Who else is making serious moves?
Well, AWS, Amazon Web Services is a fascinating one, right?
Amazon isn't really trying to sell chips to the public, but
they are building custom silicon for their own massive cloud
servers, and they've split the problem into two distinct parts.
They have Tranium in Inferentia. And that distinction is really
important for listeners to understand, right the whole life
cycle of an AI model, what's the difference between training and
inference? Think of training as the models
university education. That's when it's learning
everything it knows. Reading the whole Internet.
Literally, it's reading the entire Internet, finding
patterns, adjusting billions of tiny weights in its network.
This process takes weeks, sometimes months.
It requires enormous data sets, and it consumes just insane
amounts of power. That's what training was
designed for. Brute force learning, high
bandwidth, massive clusters of chips working together.
Inference is when the graduate finally gets a job.
It's when you actually use the model.
You ask it a question that gives you an answer that has to happen
in milliseconds. It requires less power per
transaction, but it needs to be incredibly fast and responsive,
so low latency. That's what infringe is for.
So AWS is trying to specialize the hardware for the specific
stage of the AI life cycle instead of using one general
purpose GPU for everything. Exactly.
It's all about efficiency and cost optimization at scale.
And then you have the startups that are just pushing the
boundaries of physics itself. The one that always, always
blows my mind is Cerebrous. All right, the way for scale
engine, I saw a picture of this thing, it looks, it looks
ridiculous. Looks like a prop from a sci-fi
movie. So usually to make a chip you
start with a big silicon wafer. Looks like a big round dinner
plate made of silicon and you etch hundreds of little square
chips onto. It then you cut them up, then
you cut it. Up Intel does this, NVIDIA does
this, everyone does this. Cerebras looked at that process
and said, what if we just didn't cut it?
They use the whole plate as one chip.
The entire wafer is the processor.
It's the size of a keyboard. It has 2.6 trillion transistors
on a single piece of silicon. 2.6 trillion.
That number is hard to even conceptualize.
Why on earth would you do? That to beat the speed of light.
OK, explain that. One of the biggest slowdowns in
AI at scale is moving data from one chip to another across a
physical wire. Wires have resistance.
Signals take time to travel, even at lightspeed.
If the whole computer is 1 giant chip, the data never has to
leave the silicon. It's all local.
It moves at effectively the speed of light across the wafer.
It's an attempt to completely eliminate the communication
bottleneck between chips. So we have these massive chips.
We've got NVIDIA, Google, Cerebras, all pushing the
physical limits. But there's a practical side to
this, right? We can't just keep building
bigger and bigger brains if we can't fit them in our pockets or
if they cost $10 million to run for an afternoon.
Exactly. Which brings us to optimization.
Right. And this is where the rubber
really meets the road for engineers.
If you're building an AI product today, you are constantly
fighting against memory limits. You have this massive, beautiful
model, and you have to figure out how to fit it onto a GPU
that has a fixed amount of VRAM. You can't just, you know,
download the Internet into your laptop.
And one of the key techniques mentioned in the Info queue
report is quantization. It sounds like something from
Star Trek, but let's break it down.
What are we actually doing when we quantize a model?
At its simplest, we are reducing its precision to save space.
In a computer, numbers are typically stored as floating
point numbers, and the standard for scientific computing has
always been 32 bit or FP-32. That means every single weight,
every parameter in the neural network is represented by a
string of 32 zeros and ones. Which gives you incredibly high
precision. You can represent tiny, tiny
fractions out to dozens of decimal places if you need to.
Exactly. But engineers started asking a
really important question. Do we actually need that?
Think about it this way. If I ask you to drive a car
through a garage door, do you need to know the width of that
doorway down to the micrometer? No, not at all.
I just need to know if it's, you know, roughly wider than my car.
Precisely, neural networks are probabilistic.
They are fuzzy systems. They don't usually need 32 bit
precision to get the right answer.
So engineers started chopping the bits.
How far down? They went to 16 bit which is
half the size. Then they went even further to 8
bit integers or in date. So you're literally cutting the
data size in half, and then in half again you're down to 1/4 of
the original. Size and the benefits are
massive. First, memory.
An 8 bit model takes up 1/4 of the space of a 32 bit model.
Suddenly a model that required 4 expensive GPU's can run on just
one. And the second benefit?
Speed. Computers can multiply 8 bit
numbers way, way faster than they can multiply 32 bit
numbers. It's just less math for the
silicon to do. But there's no free lunch in
physics or engineering. You're throwing away information
when you do this. What's the trade off?
What do you lose? You lose accuracy and you
introduce noise, but the biggest and most dangerous risk is
something called overrun or underrun.
Explain overrun. I want to visualize this OK.
Think about the odometer on a really old car, the kind with
the physical spinning numbers. It only goes to say 99,999
miles, right? If you drive one more mile, what
happens it? Rolls over to all zeros 0000000.
Right, that's an overrun. You tried to store a number that
was too big for the container you had.
Now in computing, an 8 bit integer is a container that can
only hold a number between 9:00 is 128 and plus 127.
That's it. That's the hard limit.
That is a tiny, tiny range. It is.
So imagine you're doing a matrix multiplication in your neural
net and the correct mathematical result is 200.
The 8 bit container can't hold it.
The computer doesn't know what to do.
It might wrap around and call it a negative number.
Which would completely trash the logic of the model.
Instant garbage output. It's like your model confusing
very hot with very cold just because the thermometer broke at
the top. So the art of quantization is
actually something called normalization.
Normalization, yes. Before you run the math, you
have to look at all of your data and mathematically squash it.
You scale everything down so that the biggest and smallest
numbers fit perfectly inside that -128 to plus 127 range.
It's like trying to pack a suitcase.
You have to fold everything just right.
If you just shove it all in, you'll break the zipper.
That is a perfect analogy. Quantization is the art of
folding the data so it fits in the hardware suitcase.
If you fold it too much, you wrinkle the close, you lose too
much accuracy. If you don't fold it enough, the
suitcase doesn't close, you get an overrun.
That makes perfect sense. O we're otimizing the software
because the hardware has these very real limits.
And I want to talk more about those limits.
We always hear about Moore's law that chips get twice as good
every two years. Is that?
Is that actually still true? This is a huge debate in the
hardware community, according to the sources.
Strictly speaking, Moore's Law is alive.
It is. We are still somehow doubling
the number of transistors we can pack into a square millimeter.
Yeah, Apple's M1 chip had 16 billion transistors a few years
ago. The newer chips have way more.
We are getting denser. But I sense a but coming.
There's a huge But while Moore's Law is alive and kicking,
another less famous law called Dennard scaling is dead, and its
death is causing major headaches for everyone from NVIDIA all the
way up to the power grid. I have not heard of Dennard
scaling. What is it?
Dennard scaling was a beautiful observation from back in the
1970s. It basically said that as
transistors got smaller and more dense, they also used less
power. So for a long, long time, we
could pack twice as many chips into the same space and the
total power usage stayed pretty much flat.
It was magical. We got free performance.
And that stopped working. It crashed and burned about 15
years ago. The physics just stopped
cooperating. We hit a quantum leakage limit.
So now when we pack more transistors onto a chip, the
power usage doesn't stay flat. It goes.
Up it skyrockets and power equals heat.
And we're right back to the nuclear engine room analogy we
started with. We are.
A modern NVIDIA H-100 GPU can draw 700 watts of power.
By itself, just one chip. Just one chip.
Now put eight of them in a server rack, which is a standard
configuration, and you have thousands and thousands of watts
of heat being generated in a box the size of a mini fridge.
That's like running a couple of space heaters inside your
computer case. It's insane.
It's enough to melt silicon. This is why you don't really see
fans in these high end AI servers anymore.
Fans aren't enough. We are literally pumping fluid
special dielectric coolant directly through the servers to
carry the heat away. And this is the hard physical
limit we are now hitting. It is the limit.
We can't make the chips run faster in terms of clock speed,
you know, gigahertz because they would literally catch fire.
Have you noticed that computers haven't actually gotten that
much faster in terms of gigahertz in the last decade?
We've been stuck around 3 to 5 gigahertz for a long time.
That's. Wild, I hadn't really thought
about that, but you're right, my processor speed is about the
same as it was on my computer from 2015.
Exactly. The only way we get more
performance now is by going wider.
More cores, more parallelism, bigger chips.
But bigger means more power and more heat.
It's a vicious cycle. This physical reality, the heat,
the power, the sheer cost, it seems to be forcing a change in
the entire industry's direction. The Info Q report really
highlights a move towards small language models or SLMS. Yes,
this is a direct logical response to the hardware
constraints for the last few years.
The trend was just bigger is better.
GPT 3, GPT 4 are llama 3. Exactly.
Trillions of parameters, but those models require massive
clusters of GPU's that cost hundreds of millions of dollars
and sucked down megawatts of power from the grid.
Which is just unsustainable for most companies.
If I'm a mid sized business, I can't afford my own H-100
cluster You. Can't even rank one easily.
So now we are seeing a correction.
Companies are leasing models like Microsoft's FI 3 or Mistral
or Tiny Llama. These are models that are
specifically designed to be small and efficient.
And the whole pitch for these is that they are good enough for
most tasks. Exactly.
They might not be able to write a Symphony or code an entire
application from scratch like GBT 4 can, but if you just need
to summarize an e-mail or organize a spreadsheet or answer
a common customer support query, they're perfect.
And, crucially, they can run on the edge.
The edge, which just means locally right on my own device.
Right. Instead of sending your data to
some massive server farm in Virginia to be processed by a
700 Watt GPU, you process it right there on your laptop or
even your phone. I've noticed this with all the
new hardware releases, the Apple M4 chip, the new NVIDIA RTX
cards for laptops, the whole AIPC push from Microsoft.
They are all marketing their NPU or AI performance now.
They're optimizing that consumer grade hardware to run these
small language models efficiently.
And this isn't just about saving money on cloud compute bills.
This is a huge, huge deal for privacy and security, which the
report highlights as a major enterprise trend.
Right. Because if I'm a bank or
hospital or a law firm. You do not want to upload your
patient data or your client's sensitive legal strategy to open
AI's cloud. You just don't.
The risk is too high. But if you can, run a small
language model entirely on your own secure laptop completely
disconnected from the Internet. That solves the problem.
It solves a massive problem. So the hardware constraint, the
fact that these big ships are scarce, hot and expensive, is
actually driving better privacy and more efficient software
design. Scarcity breeds innovation.
It has to. The sources mentioned that the
demand for AI hardware outstrips the supply by a factor of 10.
Right now. You literally cannot buy your
way out of the problem. You have to engineer your way
out of it. So you have to use quantization.
You have to use quantization. You have to use SLMS. You have
to be clever. OK, so let's synthesize all of
this for the listener. We covered a lot of ground.
We started with the the happy accident of the gaming PC, how
graphics cards turned out to be the perfect engine for parallel
processing because of how they render pixels.
Which all runs on matrix math, or more generally, tensors.
That's the native language of the modern AI chip.
Right, but because that hardware is physically constrained by
heat and the death of Dennard scaling, we can't just brute
force it forever by making things bigger.
We have to be clever. We have to be clever.
We use optimization tricks like quantization to shrink the data
from 32 bit down to 8 bit while carefully managing that risk of
overrun. And the whole industry is now
moving towards small language models to run things efficiently
on the edge, which solves for both power and privacy.
It's a full stack view. You can't really understand the
AI software revolution without understanding the silicon it
lives on. The hardware dictates the
architecture, and the architecture is now dictating
the future of the software. And I want to leave the listener
with a final thought about that future because everything we've
discussed today, the liquid cooling, the 700 Watt chips, the
massive data centers, it all sounds incredibly energy
intensive. It is.
It's arguably the single biggest bottleneck we face as an
industry. And it raises a really profound
question. Are we approaching a wall?
A physical wall. A hard physical wall.
If we can't make chips run faster because they'll melt, and
we can't just make them infinitely bigger because they
use too much power, what happens next?
What's the next paradigm? I think the future might not be
bigger AI, the future might have to be denser AI, and for that
you have to look at biology. The human brain.
It's the most advanced neural
network in the known universe. It has 86 billion neurons,
trillions of synapses and do you have any idea how much power it
runs on? I'm going to assume it's not 700
watts. It runs on about 20 watts. 20
watts. It's like a dim light bulb.
Exactly. A dim light bulb.
Meanwhile, to simulate anything even remotely close to that
level of connectivity, a cluster of H1 hundreds needs its own
dedicated power plant. The gap between biology and
silicon, that massive efficiency gap, That's the next frontier.
We have to figure out how to get that level of intelligence
without boiling the oceans to do it. 20 watts versus a MW.
That really puts the whole thing in perspective.
We are still essentially using brute force.
We are in the steam engine era of AI.
It works, it's powerful, but it's loud, hot, and wildly
inefficient. The electric motor era hasn't
even started yet. So next time you look at your
laptop or your gaming console, don't just see a screen and a
keyboard. Look at it as a parallel math
engine. It's a baby supercomputer, and
we're only just now figuring out how to really turn it on.
And if you're a developer out there, stop writing loops.
Start thinking in tensors, because that's where the entire
world is going. Thanks for diving deep into the
engine room with us today. It's absolutely fascinating to
see what's really under the hood of all this.
Always a pleasure. We'll catch you on the next deep
dive. Keep learning.