Here is something that will sound completely
counterintuitive if you have been following the AI hardware
race. The biggest bottleneck in
running large language models right now is not compute.
It is not the number of GPU's you have, it is memory.
Specifically, it is a data structure called the KV cache.
And if you are running any kind of production LLM system, it is
almost certainly the thing that is quietly eating your budget,
throttling your throughput, and limiting how many users you can
actually serve. I am MO and this is the
practical AI digest. Today.
We are talking about KV cache compression, the memory wall
that nobody in the industry wants to have an honest
conversation about, and two recent breakthroughs that might
actually change the game. Let me set the stage with a
number that should make you uncomfortable.
A single instance of a large language model processing
100,000 tokens of context can consume 40 gigabytes of GPU
memory just for the KV cache alone.
That is roughly half the high bandwidth memory on an
enterprise grade GPU like the A-100.
Not the model weights, not the activations, just the cache of
keys and values that the attention mechanism needs to
look back at previous tokens. And that is for one user, one
request. Now multiply that by the number
of concurrent users you need to serve, and you start to
understand why inference costs are where they are.
The reason this happens goes back to how Transformers work at
a fundamental level. Every time a model generates a
new token, it needs to attend to all the previous tokens in the
sequence. The naive approach would be to
recompute the attention for every single previous token on
every generation step, which would be catastrophically
expensive. So instead we cache the key and
value projections from each layer for each token and reuse
them. That is the KV cache.
It is a brilliant optimization that makes autoregressive
generation tractable, but it comes with a cost that scales
linearly with sequence length, linearly with the number of
layers, and linearly with the hidden dimension.
For a 70 billion parameter model with 80 layers, you are looking
at about a MB per token per layer.
At 100,000 tokens, that is 8 gigabytes per layer times 80
layers. The math gets ugly fast.
This is what infrastructure people call the memory wall, and
it is not a theoretical concern. WEK published an analysis in
April 2026 framing this as the defining infrastructure
challenge of the current AI generation.
Their argument is sharp and I think correct.
Organizations have been throwing GP US at the inference problem,
but GP US do not solve a memory bandwidth bottleneck.
You can have all the floating point operations per second in
the world, and if your attention mechanism is waiting on memory
reads from a bloated KV cache, you are memory bound.
The GPU cores are sitting idle waiting for data, and you are
paying for that idle time at GPU prices.
The practical impact hits in three ways.
Throughput goes down because you can fit fewer concurrent
requests in memory. Latency goes up because memory
access patterns get less cache friendly as the KV cache grows,
and cost goes up because you need more GPUs not for compute,
but for memory. This is why a company serving a
chatbot to 1,000,000 users is not spending most of its
inference budget on matrix multiplications, it is spending
it on storing and fetching the conversational context for each
user. And there is a compounding
dynamic that makes this worse. Over time, as models get larger
and context windows get longer, the KV cache grows in both
dimensions simultaneously. A model with twice the layers
and twice the context window has four times the KV cache.
Meanwhile, GPU memory has been growing linearly, roughly
doubling every two to three years.
The KV cache is growing quadratically while the hardware
is growing linearly. That is the definition of a
wall. There is also a subtlety here
that gets lost in the benchmarks.
The memory wall does not just affect peak memory, it affects
memory bandwidth utilization. Modern GPU's like the H-100 have
enormous compute throughput over a petaflop of FP8, but their
memory bandwidth tops out around 3 terabytes per second.
When your attention mechanism has to read a 40 GB KV cache for
every single generated token, you are doing 40 gigabytes times
the number of attention heads of memory reads per token.
At 3 terabytes per second, that is over 13 milliseconds per
token just for memory access. For comparison, the actual
matrix multiplications take a fraction of that.
The model is literally waiting for data, not crunching numbers.
This is why you see companies like Rock building custom chips
with massive on chips RAM, why Cerebras has their wafer scale
approach, and why even Nvidia's Blackwell architecture
prioritizes memory bandwidth over raw flops.
The industry has figured out that compute is abundant, memory
is scarce, and the KV cache is where that scarcity hurts the
most. So what do you actually do about
it? The field has been working on
this hard, and 2025 and early 2026 have produced some
genuinely promising approaches. I want to walk through two of
them in detail because they represent fundamentally
different strategies, and understanding why they work
tells you something important about where inference
optimization is heading. The first is KVTC, which stands
for KV Cache Transform Coding. This was published at ICLR 2026,
and it takes an approach borrowed from signal processing,
specifically from how we compress images and video.
The insight is elegant. The raw key and value vectors in
the KV cache have significant statistical redundancy.
Adjacent tokens tend to produce similar key vectors because the
underlying representations change gradually.
Different attention heads within the same layer often encode
partially overlapping information, and the
distribution of values across the hidden dimension is highly
non uniform, with some dimensions carrying much more
information than others. KVTC exploits all three of these
redundancies in a pipeline. First, it applies PCA based
feature decorrelation to remove the cross dimension
correlations. This is essentially rotating the
coordinate system so that each dimension is as independent as
possible, which is exactly what transform coding does in JPEG
and video codecs. Then it applies adaptive
quantization, allocating more bits to high variance dimensions
and fewer bits to low variance ones.
And finally it runs entropy coding on the quantized values
to squeeze out the remaining statistical redundancy.
The results are striking. On standard reasoning
benchmarks, KVTC achieves up to 20 times compression with
negligible accuracy loss. On some specific workloads it
hits 40 times or higher. And critically, it is training
free. You do not need to fine tune the
model. You do not need calibration
data. You apply it at inference time
and it works. The compression and
decompression are fast enough that the time spent compressing
is more than offset by the memory bandwidth you save on
cache reads. 20 times compression on a 40 GB KV cache
means you are down to 2 gigabytes.
That changes the economics of inference entirely.
You could serve 10 times as many concurrent users on the same
hardware, or you could extend your context window by an order
of magnitude without adding GPU's.
There is a nuance in the KVTC approach that I think is
underappreciated. The PCA rotation step is not a
one-size-fits-all transformation.
The authors compute the principal components per layer
and per attention head because different layers encode
fundamentally different kinds of information.
Early layers tend to capture syntactic and positional
patterns where the key vectors cluster tightly.
Middle layers handle semantic composition, where the variance
is distributed more evenly. Late layers handle task specific
reasoning, where a few dimensions dominate.
By adapting the transform per layer, KVTC avoids the common
trap of compression methods that work well on average but blow up
on the layers that matter most for accuracy.
The entropy coding step is also worth a closer look.
After quantization, the distribution of quantized values
is highly non uniform. Some quantization bins are hit
far more often than others, and that non uniformity is free
compression if you exploit it. KVTC uses arithmetic coding, the
same family of algorithms that powers modern video codecs like
H265, to squeeze out this remaining redundancy.
The overhead of the coding itself is negligible because the
decoder runs on the GPU's tensor cores during the attention
computation. You are effectively getting free
compression from statistical structure that was always there,
but nobody was bothering to exploit.
One more thing about KVTC that surprised me.
The authors show that their compression actually improves
cache locality. In some cases, the compressed
cache entries are smaller, which means more of them fit in the
GPU's L2 cache during the attention pass.
This creates A virtuous cycle where compression reduces memory
footprint, which improves cache hit rates, which reduces memory
bandwidth pressure, which speeds up the attention computation
beyond what the raw compression ratio would predict.
The measured speed up is sometimes higher than the
compression ratio alone would explain, and this locality
effect is why the second breakthrough is Turboquant from
Google Research, also at ICLR 2026.
Turboquant takes a different approach.
Instead of the full transform coding pipeline, it focuses
specifically on quantization, but does it in a mathematically
novel way that eliminates the accuracy penalties that previous
quantization methods suffered from.
The key innovation is a technique called polar quant.
Traditional quantization methods need to store per block
normalization constants, scaling factors that map the quantized
integers back to the original floating point range.
These constants take up space and introduce error at the
boundaries between blocks. Polar quant eliminates them
entirely by representing values in polar coordinates where the
magnitude and angle can be quantized independently with
tighter bounds. It sounds like a small change,
but it removes an entire class of quantization artifacts.
On top of Polar Quant, Turboquant adds something called
Quantize Johnson Lindenstrauss, or QJL.
This is a dimensionality reduction technique that
projects the key vectors into a lower dimensional space while
preserving the relative distances between them, which is
what the attention mechanism actually cares about.
The Johnson Lindenstrauss lemma guarantees that random
projections preserve distances with high probability, and
Turboquan shows that you can quantize the projected vectors
to very low bit widths without violating this guarantee.
The combined result is 3 bits per element in the KV cache.
That is a six times memory reduction on NVIDIA H-100 GPU's.
The attention computation itself gets up to 8 times faster
because the compressed cache fits better in the GPU's on chip
memory hierarchy. And here's the part that made me
sit up. 0 measurable accuracy degradation, not small
degradation, not acceptable degradation. 0 on long bench, on
needle in haystack tests, on standard reasoning benchmarks.
No calibration data required, no fine tuning.
You apply it at inference time and it just works.
Now you might be wondering how GULO validated the 0 degradation
claim, because that sounds too good to be true and honestly
when I first read it I was skeptical too.
But the key is in how attention actually works.
The attention mechanism computes dot products between query
vectors and key vectors, then uses those scores to weight the
value vectors. What matters for the output is
not the exact values of the keys, but the relative ordering
of the dot product scores. If your compression preserves
the ranking, the ARD Max, the softmax distribution, then the
output is functionally identical.
Polar quant and QJL are designed specifically to preserve these
relative distances, and the empirical results confirm that
the preserved ordering translates to preserved output
quality across a wide range of tasks.
There is also a practical consideration that the paper
addresses what happens at the boundaries when you compress mid
sequence. Do you get artifacts at the
transition between compressed and uncompressed cache entries?
Turboquan handles this by compressing the entire cache
uniformly so there are no boundary effects.
Every entry gets the same treatment, which means the
compression is invisible to the model.
I want to be clear about why this matters beyond the obvious
memory savings. The memory wall is not just a
cost problem, it is a capability ceiling.
Today the practical limit on context length for most
production deployments is not the models advertised context
window, it is how much KV cache memory you can afford.
A model that advertises 1,000,000 token context window
but requires a TB of KV cache memory at that length is not
really a million token model for anyone running real workloads.
KV cache compression directly translates into longer usable
context at the same hardware cost.
It also changes the calculus on model architecture.
One of the reasons mixture of experts models are popular is
that they reduce the per token compute cost, but they do not
reduce the KV cache size because every token still needs full key
and value projections across all layers.
With aggressive KV cache compression, dense models become
more competitive again because their main disadvantage, higher
memory usage per token gets compressed away.
I also want to put this in an economic frame because the
dollar amounts clarify the urgency in a way that technical
metrics sometimes do not. A single H-100 GPU costs roughly
$30,000, or about $4.00 per hour on the major cloud providers.
If your KV cache is consuming half the GPU memory, then half
of that $4.00 per hour is going to store in context, not
generating tokens. For a company running 100 GPUs
for inference, that is $200 per hour, roughly $175,000 per month
just on KV cache memory. A six times compression from
Turboquant cuts that to under 30,000.
A 20 times compression from KVTC cuts it to under 9000.
These are not theoretical savings.
They are the difference between an inference deployment that is
economically viable and one that is hemorrhaging money.
And it compounds. When you free up GPU memory from
KV cache, you can fit more concurrent requests for GPU,
which means you need fewer GPU's for the same throughput.
Fewer GPUs means less data center power, less cooling,
fewer network switches, less operational overhead.
The KV cache is the first domino.
Compress it and everything downstream gets cheaper.
So what does the practitioner playbook look like?
If you are running inference in production today, let me give
you 5 concrete things. Benchmark your KV cache memory
usage per request at your actual workload lengths.
Not the models maximum context. Your actual 50 and P99 context
lengths. Most teams I have talked to do
not know this number and it is the single most important metric
for capacity planning. Evaluate KVTC or Turboquant on
your specific model and workload.
Both are training free and can be tested without modifying your
model. KBTC gives higher compression
ratios but with more computational overhead.
Turboquant gives lower compression but with faster
decompression and 0 accuracy risk.
The right choice depends on whether you are memory bound or
compute bound. If you are not ready for a
compression library, start with the low hanging fruit sliding
window attention on your KV cache where you only keep the
most recent end tokens plus a few anchor tokens from the
beginning. This is not as principled as the
compression approaches, but it is trivial to implement and gets
you meaningful memory savings for conversational workloads.
Set up memory observability. You should be tracking KV cache
size per request, peak memory usage for GPU, and the ratio of
KV cache memory to total GPU memory utilization.
If that ratio is above 50%, you are leaving throughput on the
table. Plan for the convergence,
Google's official turboquant implementation is expected in
the second quarter of 2026, and the major inference frameworks
VLLM, Tensor, RTLLM, and the rest will integrate these
techniques within a release cycle or two.
The teams that understand the memory wall now will be ready to
deploy compression the day it ships in their stack.
The teams that do not will be 6 months behind paying six times
more for memory than they need to and do not sleep on the 2nd
order effects. Once KV cache compression is
standard, the entire model serving landscape shifts.
Longer context windows become practical on smaller hardware.
Edge deployment of large models becomes feasible.
The cost per query drops enough that use cases that were
previously uneconomical. Real time document analysis
always on coding assistance with full repository context.
Persistent multi turn agents that remember everything.
All of those become commercially viable.
The memory wall is not just a technical constraint, it is a
capability gate and we are about to open it.
There is one more thing I want to address because I know some
of you are thinking it. What about multi query attention
and grouped query attention? These are architectural changes
that share key and value heads across multiple query heads and
they do reduce KV cache size. GQA with eight groups instead of
full multi head attention cuts the KV cache by the number of
query heads divided by the number of key value groups.
Lama 3, Gemma 2 and most recent models use some form of this.
But here is the thing. GQA is an architectural decision
made at training time. You cannot retrofit it onto an
existing model. And even with GQA the KV cache
still scales linearly with sequence length.
At long enough context you hit the wall again.
The compression techniques I have been talking about, KVTC
and Turboquant, work on top of whatever attention architecture
the model uses. They are complementary, not
competing a model with GQA plus turboquant gets the benefits of
both. The memory wall is real, it is
expensive, and it is solvable. The research is ahead of the
production tooling right now, but that gap is closing fast.
Most important thing you can do today is stop thinking about
inference optimization as a compute problem and start
thinking about it as a memory problem, because the GPU is not
your bottleneck, The KV cache is.
That's the one for this episode in MO.
This is the practical AI digest. If you got something out of
this, send it to one person who'd actually use it.
No subscribe campaign, no rating farm, just one person.
See you on the next one.