I want to start today with a mental image.
I want you to picture the absolute cutting edge of
artificial intelligence. Right now, most people, when I
ask them to do that, they picture a massive data center.
You know, rows and rows of servers blinking lights.
Cooling fans roaring like jet engines.
Exactly. Consuming enough electricity to
power a small city. They're picturing NVIDIA H1
hundreds stacked right up to the ceiling.
And they wouldn't be wrong. I mean, that is certainly where
the large language models live. That's sort of the big AI
paradigm, right? But today we are going to go in
the, well, the complete opposite direction.
I want to shrink our world down. Way down.
We're leaving the cloud, we're leaving the server farm.
We were even leaving the smartphone.
It's probably in your pocket right now.
We're headed to the edge. Yeah, the the extreme edge.
We are talking about microcontrollers.
We're talking chips that cost less than $5, chips that can run
on a coin cell battery. And the mission today for this
deep dive is to figure out how you can take that massive, you
know, brain melting AI capability and shove it into a
piece of silicon smaller than your fingernail.
It's a fascinating area. It's often called tiny ML and
honestly I think it's a harder problem than the big stuff.
Really harder. Oh for sure.
When you have basically infinite computing power, you can get a
little easy. When you have $5 and a few
kilobytes of memory, you have to be a genius.
And to really set the stage for why this is so hard, I want to
take us back in time a bit to the early 1980s, Carnegie Mellon
University. Right, the birth place of so
much computer science history. It really is.
So picture a group of grad students.
It's the 80s, so you can imagine the big hair, the chunky
glasses. They're working in their office
and they have a problem. A very, very serious problem.
Let me guess, Yeah, they needed coffee.
Close. They wanted a Coca-Cola.
The fuel of all night coding sessions.
I get it. But the vending machine is all
the way down the hall, maybe even in another building, and
there was just nothing worse than trekking all the way there,
putting your quarters in and getting a warm.
Coat oh the worst. Or finding out it's empty.
Right. A classic engineer's dilemma.
Laziness really is the mother of invention.
It truly is. So they hacked the machine, they
installed these little micro switches inside the columns of
the Coke machine to track how many bottles were left.
OK. And this is the key part.
How long they had been there so they could estimate if they were
cold? That's brilliant.
Then they wired it all up to the departmental server.
Which meant they could just sit at their desk terminals and
check the stock. They invented the first Internet
of Things device because they didn't want to walk down the
hall for a warm soda. It's a great story, but it also
highlights a very specific kind of computing right?
That Coke machine was doing something, well, very simple.
It was just logic. If then.
Exactly. If the bottle count is greater
than 0, return true. If the time in the machine is
greater than 3 hours, return cold.
It's a lookup table. It's basic and Fast forward to
today, we have something like 250 billion microcontrollers
MCU's out there in the world. Billion with AB.
They're in your car's braking system, your microwave, your TV
remote, soil sensors in farm fields.
Everywhere. And for the most part, they're
still kind of stuck in that Coke machine era.
They're reactive. They just follow simple
preprogrammed rules. But we're greedy now.
We don't want them to just follow rules anymore.
No, we want them to see. We want them to hear.
We want to understand context. We want deep learning on them.
And that right there is where we hit the wall.
The memory wall. Let's unpack that term because I
think people hear chip and they just assume it's like a mini
computer. I mean, my phone is essentially
one big chip and it has gigabytes of RAM.
And that is such a dangerous assumption.
We need to be very, very precise about the hardware we're talking
about today. The source material for this
dive, it focuses on a specific, really popular commercial
microcontroller, the STM 32F746. What's?
Your name. Oh, engineers love their
alphanumeric codes, but here are the specs that matter.
A typical smartphone, like you said, has maybe 8 gigabytes of
RAM, right? This microcontroller has 320
kilobytes of RAM. Wait, wait.
Kilobytes. Kilobytes.
That is what, roughly 25,000 times less working memory than
your phone. I have single e-mail attachments
that are larger. Than that definitely do.
And then there's the storage, the flash memory.
This is where you keep the program itself.
Your phone might have 128 gigs, 256.
This chip has one MB. One MB.
So we are trying to fit a neural network which can take up
hundreds of megabytes just to store its weights into one MB of
program space. And then run it in a working
memory space smaller than a floppy disk from 1995.
That sounds. It is if you try to take a
standard small deep learning model, something like Mobile Net
V2, which is widely used on phones, and you try to load it
onto this chip. Yeah, it's not that it runs
slowly. It physically cannot exist
there. It just doesn't.
Fit the Arameters alone need 68 megabytes.
You have one MB of total space. It's like trying to park a
Greyhound bus inside a coat closet.
OK, so it was considered impossible.
It pretty much was, until the research we're looking at today.
We're doing a deep dive into a system called MCU Net, which was
developed by researchers at MIT, specifically A-Team led by Ji
Lin and Song Han. And looking at the results they
sent over, they didn't just get a model to run, they achieved
over 70% accuracy on Imagenet. Which is the absolute gold
standard benchmark we're talking about recognizing 1000 different
categories of objects. Golden retrievers, Fire trucks,
Espresso machines. All on a $5 chip.
With 70% accuracy, that is a massive deal.
The previous best attempts were stuck down around, what, 54
percent or they discouraged? So how did they do it?
I mean, did they just find a better way to compress the file?
A better zip algorithm? No.
And that's the old way of thinking about this problem.
To really get why MCU net is so revolutionary, we have to look
at how this kind of AI is usually built.
It's typically a two stage process with, well, with two
different tribes of people. The tribes.
OK, I like this analogy. So you have tribe one, the model
designers. These are the architects.
They work in Python using libraries like Pytorch or
Tensorflow. Their whole world is designing
the neural network structure. They care about accuracy.
And they assume hardware is just this infinite resource pool
somewhere in the cloud. Exactly.
They're drawing the blueprints for a beautiful massive
cathedral. Then they hand those blueprints
over to Tribe 2, the system designers.
These are the folks right in the compilers, the drivers, the low
level code. Their job is to take that
blueprint and actually make it run on the real silicon.
They're the construction crew. Precisely.
But in the world of microcontrollers, this hand off
is a complete disaster. The architect hands over a
blueprint for a cathedral. The builder looks at the
physical site and says, but we only have a 10 foot by 10 foot
lot and our total budget is $5. So the builder just yells back
at the architect. Make it smaller.
Right, and the architect who doesn't understand construction
just starts chopping things off randomly.
OK, let's remove the Spire. Get rid of the lobby.
They make the model Dumber and Dumber and Dumber until it
finally fits. But there's a catch, I'm
guessing. Here's the kicker.
The builder is also using really inefficient tools.
Standard libraries like Tensorflow, light, micro, they
have a lot of overhead. They waste space by their very
nature. So the builder is wasting space
on the tiny lot, which forces the architect to make the
building even smaller and Dumber than it actually needs to be.
It's a vicious cycle of inefficiency.
A vicious cycle. OK, and this is where MCNET
proposes a new way system algorithm Co design.
Which means what exactly? It means the architect and the
builder are the same person, or at the very least they are in
the same room looking at the same constraints and designing
the engine and the car simultaneously.
OK, that makes intuitive sense. So let's break this down.
The paper splits MCU net into two main components.
There's tiny Nas on the algorithm side.
And tiny engine on the system side.
Let's start with tiny Nas. Nas stands for Neural
Architecture Search. Correct.
I've heard of this. This is basically the concept of
AI designing AI, right? That's a great way to put it.
Instead of a human engineer manually picking all the layers,
I'll put a convolution here, a pooling layer there.
You write an algorithm that automatically searches through
millions or even billions of possible architectures to find
the best one for your task. But wait a second.
NASA is notoriously computationally expensive.
I've read about Google running these searches on thousands of
TP us for weeks just to find one good model.
A great point. If our whole goal here is to
save resources for a tiny chip, isn't running a massive power
hungry search totally counter intuitive?
It would be if he used a standard Nas.
Standard Nas is expensive, and it's usually just optimizing for
one thing, accuracy. But tiny Nas has to solve a
much, much harder problem, which is it has to find a model that
is not only accurate, but also strictly obeys the hard physical
constraints of the microcontroller.
It can't use one single byte more than 320 kilobytes of RAM,
not one. So it's constrained optimization
on steroids. And there's another wrinkle.
Microcontrollers are incredibly diverse.
There are thousands of different chips out there with slightly
different memory sizes, different features.
So you can't just find one perfect model.
Exactly. If you have to rerun a massive
expensive search for every single specific chip you might
want to use, well, it's just not scalable.
It's not practical. So how does Tiny Nest solve
that? Problem.
It starts by being very, very clever about where it looks for
a solution. This is the first big
innovation, the concept of automated search space
optimization. OK, the outline mentions
something here that sounds pretty technical.
The CDF of FLOPS. I'll be honest, that sounds like
a tax form. Can you help us visualize what
this actually means? I can try.
Let's stick with an analogy. Imagine you're trying to pack a
suitcase for a trip. OK, I'm with you.
You have a very small suitcase that's your 320 KB memory limit
and you want to pack as much utility into that suitcase as
you possibly can't close gadgets, shoes right now.
FLOPS stands for floating point operations.
In our little analogy, FLOPS are like the utility or the
smartness of the items you're packing.
Generally an AI, the more math, the more FLOPSA model performs,
the smarter it is. So.
I want to pack as many FLOPS, as much smartness into my little
suitcase as I can. Exactly.
Now imagine you have different wardrobes you can choose your
items from before you start packing.
Some wardrobes are full of bulky, inefficient clothes.
You know, a giant wool coat that takes up half the suitcase but
isn't actually that warm. That's a bad search space.
Other wardrobes are full of high
tech thin thermal gear. You can fit a lot more warmth, a
lot more FLOPS into the very same suitcase.
So before Tiny Nest even starts picking the specific items, the
specific model it first analyzes the wardrobes.
That is exactly what it does. That's the CDF part, the
cumulative distribution function.
It's a statistical method. It basically samples the
wardrobes. It looks at the probability
distribution of what's inside. It asks if I pick random items
from wardrobe A, how many FLOPS can I usually fit in my
suitcase? Now what about wardrobe B?
And it picks the wardrobe that offers the highest density of
math per byte, the most efficient set of options.
Spot on, and the results were staggering.
They found that by simply picking the right search space,
the right wardrobe, before they even started the fine grained
search, they could boost the final models accuracy by over
4%. 4% is huge in this field. It's.
Massive. And that's before they've even
chosen a single layer. That's fascinating.
So it's not just about the final model, it's about the potential
of the model family you start with.
Right. So once it's picked the best
search space then he uses something called one shot Nas to
pick the specific final architecture.
Why one shot? What does that mean?
Because training a neural network from scratch is the slow
part. Training millions of them to
test each one would take 100 years.
So instead they train a super network.
A super network. They give it like a giant
network that contains every possible variation of the
smaller networks inside it all at once.
So like a Swiss army knife that is every possible tool already
attached to it. That's a perfect analogy.
They train that big Swiss army knife knife just one time.
Then they use a clever fast evolutionary algorithm to select
the specific subset of tools, the specific subnetwork that
works best for a specific chip. So it can tailor the model.
It custom tailors the suit to the exact measurements of the
hardware. It asks OK for this STM 32 chip
with 320 kilobytes of RAM, is it better to have a high resolution
image with fewer channels or a lower resolution image with more
channels and more layers? And for a different chip with
say 512 kilobe, the answer might be different.
The answer will almost certainly be different.
It automatically finds that perfect trade off to squeeze
every last drop of performance out of that specific memory
budget. OK, so we have our custom design
model. The architect has done their job
perfectly, but now we have to actually build it and run it.
And that brings us to the other half.
Tiny engine. This is the system side and for
the engineers listening, this is where you should really lean in
because this part directly challenges the industry standard
way of doing things. And the standard way would be
using libraries like Tensorflow, Light micro or CMSIS and and
yes. Most of these existing
frameworks use what's called an interpreter based approach.
Can you explain that? Why is an interpreter a bad
thing for a microcontroller? Let's go back to the kitchen.
An interpreter is like a cook who has to read the recipe line
by line while they are actively cooking.
So they read the first line, chop one onion, they stop, they
put the book down, they find the knife, they find the cutting
board, they chop the onion, and they pick the book back up, find
their place, and read the next line.
Saute the onion in a pan. They put the book down again.
There's a lot of switching back and forth, a lot of overhead.
Tons of overhead, yeah. But even more importantly, you
need space on your counter for the cookbook itself.
And that cookbook here is the model graph, the metadata that
tells the system what layer comes next, what its parameters
are. I see, on a server with 32
gigabytes of RAM, you just don't care if the cookbook takes up
100 kilobytes. But on our little chip that
metadata, that cookbook can consume 20% or even 30% of your
total precious memory. Just to know what to do next,
that feels incredibly wasteful. It is so tiny.
Engine takes a completely different approach.
It is a compiler. So instead of the cook with the
open book. It's like a robot arm in a
factory that has been hardwired to perform only that one
specific recipe. There's no book Tiny Engine
looks at the model that tiny ass designed and it generates
specific custom binary code for just that model says I see you
need a three by three depth wise convolution here.
I will write the exact machine code for that.
And I see you don't use any pooling layers anywhere in this
model. OK, I will just delete all the
pooling code. So there's no cookbook at
runtime, no interpreter. None.
The logic is baked directly into the executable file, and this
simple change slashes the final binary size by a factor of 5.
Compared to Tensorflow light, it frees up all that flash sorts.
But what about the SRAM, the counter space?
You said earlier that was usually the hardest constraint.
And this is where a tiny engine gets really, really clever with
something they call model Adaptive Memory scheduling.
I saw a note about this, they compared it to Tetris.
It is exactly like Tetris in a standard library.
Memory allocation is often lazy and inefficient.
The system looks at layer one and says, OK, I need 50
kilobytes for this, it grabs a chunk of memory.
Then layer 2 needs space, it grabs another chunk.
Often they just allocate a huge buffer for the worst case layer
and let it sit there unused for most of the time.
So you end up with all these gaps, wasted space like a bad
game of Tetris. Exactly.
But tiny engine, because it's a compiler, knows the future.
You can see the entire life cycle of the network before it
ever runs. It knows that tensor A is
created at Step 5 and is never ever used again after step 6.
So it's step 6.001. That memory is free to be used
for something else immediately. And tiny engine scheduler slides
tensor V right into that same physical memory address.
It creates a perfect memory schedule where the buffers
overlap in time with no wasted gaps.
It's like extreme hot dusting for data.
As soon as one number leaves the seat, another one sits right
down. That's a great way to put it,
and because it knows the whole topology ahead of time, it can
make smarter decisions about things like tiling.
If a layer is too big to fit in memory all at once, you have to
chop it into smaller tiles. Tiny Engine calculates the
perfect tile size based on what else is on the desk at that
specific moment in time. That seems incredibly efficient,
but there's one more trick in tiny engine that the paper
highlighted and I really want to understand it.
The in place depth wise convolution.
Yes, this is a very low level optimization, but its impact is
huge. It's brilliant.
OK, walk us through it. So in a standard convolution,
you're taking an input image or a feature map from a previous
layer, and you're applying a filter to it to create an output
map, right? To do this you almost always
need 2 separate memory buffers, one to hold the input so you can
read from it, and a second separate one to write the output
into. So if the image data is size N,
you need at least 2 N of memory at that moment.
Yes, your peak memory usage is 2 N You can't just overwrite the
input while you're working, because you might need those
same input pixels for the next calculation.
However. Many modern efficient networks
use a special type of layer called depth wise convolution.
The key detail here is that the filter only looks at 1 channel
at a time, so the output for channel 1 depends only on the
input from channel 1, doesn't need to look at Channel 2 or
channel 3. OK, so the channels are
independent. How does that help with memory?
Because the researchers have this key insight, as soon as you
have calculated the output for pixel 000 in channel 1, you will
never ever need the input for pixel 00 in channel one ever
again. So you can just overrate it.
You can write the answer directly on top of the question.
That's like destructive editing. Exactly.
It's like painting over a canvas.
You don't need a second canvas to copy your work to.
You just transform the image right there in place.
And what does that do to the memory requirement?
It drops from 2 N. To north plus a tiny tiny
buffer. So basically it drops to north,
it cuts the peak memory usage for those specific layers almost
perfectly in half and. When you are fighting for every
single KB, cutting your peak usage in half is.
That's miraculous. It really is.
And this brings us right back to the Co design principle.
The Architect Tiny Nass knows that the Builder Tiny Engine has
this special in place trick up its sleeve.
So it changes what the architect designs.
Of course Tiny Nass is now more likely to design networks that
use a lot of these depth wise convolutions because it they are
super cheap on memory when tiny engine runs.
They feed off each other. The capabilities of the system
software directly influence the design of the AI model itself.
Precisely. So we've built the car, we've
custom tuned the engine, we've stripped out all the unnecessary
way. Let's take it to the track.
What are the actual performance results?
The headline number as we mentioned at the top is 70.7%
top one accuracy on Imagedet. Can you contextualize that for
me? Again, is 70% considered good in
the grand scheme of things? 70% is widely considered the
threshold of real world utility. You know 50% is basically a coin
toss. It's useless. 60% is OK for a
cool demo, but 70% is where you start to see things that could
be commercial products. It means the system is reliable
enough to actually be useful. And how did this compare to the
previous state-of-the-art for this kind of chip?
O the previous best solution they benchmarked against which
was using a quantized mobile net V2 model.
This standard CMSISNN library it only got 54% accuracy.
That is a massive gap, 16 points.
That's the difference between a prototype and a real product.
And that's not even the whole story.
In many of their tests, the mobile Net V2 model simply
wouldn't run at all. It would just crash with an out
of memory error. So MC Net doesn't just make it
better, it actually makes it possible in the first place.
OK, what about speed? Usually when you compress things
this much or get this clever, you have to pay a penalty and
latency. Not here because of that
compiler smaller approach, no interpreter overhead, Tiny
engine was consistently 1.7 X to 3.3 X faster than running a
model on tensorflow light micro. Faster, smaller and more
accurate. It's the Holy Trinity of
embedded systems engineering, and they proved it on other
tasks too. They benchmarked it on Visual
Wake words. That's like Hey Siri or OK
Google but for video, right? Exactly.
The task is just detecting if a person is present in the frame.
It's used for things like smart doorbells or security cameras to
wake them up. MC Net ran 2.4 times faster than
the existing solutions. And was it more accurate?
Yes, and this is where the trade-offs get interesting.
The previous winner of the Visual Wake Words challenge had
optimized really hard for low memory, but it was very slow.
MC Net managed to optimize for both small memory and low
latency. They also mentioned they tried
it on object detection. They did using a tiny version of
a Yolo model which stands for You Only Look Once.
This is a much harder task than just classification because you
have to draw a bounding box around the object you find
right, And on that task they achieved a 20% improvement in
mean average precision while staying under a strict 512
kilobytes RAM limit. A 20% improvement on a chip this
constrained is huge. It's the difference between a
Smart car seeing a pedestrian and not seeing a pedestrian.
It's a game changer. So this works.
The technology is real. We have successfully shrunk the
AI brain. I want to spend our last few
minutes talking about the So what?
Why should you, the listener, care?
What does this unlock for the world?
Well, I think there are three major implications that jump out
immediately. The first one is privacy.
You have that great line in the prep material.
What happens on the toaster stays on the toaster.
Exactly. Right now, to have a truly smart
home, you basically have to accept a certain level of
surveillance. Your smart speaker, your
doorbell camera, they are constantly streaming raw audio
and video data to the cloud to be processed.
Which a lot of people are, you know, increasingly uncomfortable
with, and for good reason. For very good reason.
With a system like MCU Net, we can move that processing to the
extreme edge. The audio analysis, the person
detection, the face recognition, it all happened on that $5 chip
inside the device itself. The raw data never leaves your
house. So the device just sends a
simple signal, a flag, door opened or person detected, not
the actual video of you opening the door.
Right. And that is a massive selling
point for consumers who value their privacy.
OK, what's the second big implication?
The second one is democratization.
Just lowering the cost of entry. Drastically if you need $1000
GPU or an $800 smartphone to run useful AI, that automatically
excludes a huge part of the world and a huge number of
potential use cases. But if you can, run it on a $5
chip. You can put it in literally
everything. You can put crop disease
detection sensors in the fields of developing nations and they
can run on a small solar cell. You can put smart monitoring
into basic low cost medical devices.
You can put it in toys, appliances, infrastructure.
It makes AI truly ubiquitous. And the third one, I think the
third one is green AI. We talked a lot about the
massive carbon footprint of training these huge models like
PPG 4, but running the models inference consumes a ton of
energy too. Every single time I ask my smart
assistant a question, a GPU has to spin up somewhere in a data
center. That's a good point.
Running a model locally on a microcontroller consumes a tiny,
tiny fraction of the energy of sending data to the cloud,
processing it there, and sending the result back.
Yeah, it enables always on intelligence without constantly
draining the battery or the power grid.
This brings me to a final, maybe slightly provocative thought to
leave our listeners with. For the past 20 years, we've
gotten used to the Internet being cloud centric.
The brains are all in the server farms, the endpoints, our
phones, our laptops are basically just fancy screens.
Right, they're thin clients. But if these 250 billion devices
suddenly wake up, if every light bulb, every toaster, every shoe
has a tiny but effective neural network inside it that actually
works, are we moving from that cloud centric world to an edge
heavy world? That is the big question, isn't
it? If the edge becomes truly
intelligent, the entire architecture of the Internet
could change. We might stop moving raw data
around. We would start moving knowledge
insights. We don't send the video of the
living room, we just send the insight.
The cat is on the couch. The bandwidth requirements
plummet, but the distributed computing power of the whole
network skyrockets. We might be accidentally
building a global distributed supercomputer, a digital nervous
system for the physical world. And MCU net feels like it could
be the synapse that makes it all connect.
It's certainly a key enabling technology, a huge step in that
direction. Too.
So for the developers and the engineers listening out there,
maybe take a look at your own stack.
If you're just grabbing a standard library because it's
easy, you might be leaving a huge amount of performance on
the table. Don't just buy the suit off the
rack. It might be time to tailor it,
or better yet, learn how to codesign it from the fabric U.
Tailor it, I like that. We will leave it there.
Thank you for helping us shrink the world and get the synthesis
today. It was a pleasure.
And thank you for listening to the deep dive.
We'll see you next time.