Over the last few weeks, the New York Times, Fortune, TechCrunch,
and many more have all covered the same new trend.
It's called token maxing, a new trend that's spreading literally
everywhere. Yet this month a consultancy
called Jellyfish just released a study of 12,000 developers
across 200 companies. The engineers at the bottom of
this graph that they created ship code changes for about 28 P
each. The engineers at the top of that
graph ship them for $89.00 each. The heavy users produce twice as
much output as 319 times the cost.
This is what token maxing is. Today I'm going to discuss many
of the silly stories behind this new trend with engineers gaming
leaderboards to encourage them to spend lots of money in tokens
to explain what's really happening, what token maxing is
and why the same dynamic is coming to every industry and
job, not just software. And importantly, why it's such a
big problem that you must catch at work and what you should be
doing instead. This is a loop of Jack Horton.
I hope you enjoy the show. So let's start at the top and
discuss what the new trends of token maxing actually is and and
to be honest, why it's a bit of a big problem.
So for context, in in this month, say April 2026, the CTO
of Uber just got on stage at a big industry event and admitted
that his entire 2026 AI budget was completely spent.
Now four months earlier, he'd rolled out Claude code to about
5000 engineers and and by March, 95% of them were using AI every
single day, with 70% of the code they were submitting being AI
written. But only 11% of that was
actually running in Uber systems.
So 95% of them were using AI. 70% of code that was being
submitted were created by AI, but only 11% made it to
production. These are very important numbers
and they tell a really good story about what's happening
right now. And this is going to be
something that will be mirrored across many different jobs and
industries. As I've said a few times,
programming is just ahead of the game and knowledge workers
generally are going to be going through many of the same
challenges, ideas, new ways of working that developers have
gone through over the last 18 months.
So for context, a token is roughly 3/4 of a word.
So when an LLM gives you 3/4 of a word, you span the token.
And to give you a sense of scale, the entire text of War
and Peace, the book is about 1,000,000 tokens.
And very funnily, there's there's VCs and people blogging
online and releasing articles all the way through 2026 about
burning about 250 million tokens in a single day by running a
swarm of agents all operating in parallel.
So 250 million tokens by the way, is is about roughly 250
copies of War and Peace consumed in a single day by a single
person. I think Jensen Huang of NVIDIA
said on the All In podcast recently that if a half
$1,000,000 engineer, so they pay him half $1,000,000 a year, did
not, and I quote, consume at least $250,000 worth of tokens a
year. He said he's going to be deeply
alarmed. And by March, the New York Times
had called this a basically a new status game all over Silicon
Valley. And the measure is quite simple.
Right now, it's just consume, consume more tokens, spend more
and prove that you're doing lots of stuff.
Now over the last few weeks, this story has taken an
interesting term because engineers at Meta and Microsoft
and Sales Force and a bunch of other big companies have been
found to be deliberately burning tokens.
So running lots of agents trying to have massive contacts.
Windows, which costs lots of money, going over and over on on
prompts, just looping prompts because they worked out that
their managers were using AI consumption as some form of
performance metric. And of course they gained it.
And those engineers that were at the top percentage in terms of
usage and token spend were producing about twice the
output, but at 10 times the cost.
And Meta's internal tool called Chordonomics is essentially
something that went public recently.
So they built this leaderboard ranking system and it ranked
people by the number of tokens they'd consumed.
So about 60 trillion tokens were used or spent in a 30 day
window. To put that 60 trillion tokens
into context, if every employee at Meta read a full novel every
single day for 30 years, they would collectively consume about
0.1% of that total number. And Meta by the way, has 78,000
employees. And that leaderboard handed out
badges like Token Legend, Cachet, Wizard, session
Immortal. And Meta CTO was, you know,
shouting on the rooftops, basically saying that, you know,
they're, they're top ranked engineers on these leader boards
have become 5X to 10X more productive, saying that this is
easy, this is like, you know, printing money basically.
And when it went public, of course, they then pulled the
leaderboard because it's quite embarrassing.
And I'll go into why now this is very developer focused, but
we're going to see the same in every single function in some
form. People would so desperately want
their teams to be using AI that they'll just want them to
consume tokens. You know, if token consumption
is going up, surely outputs going up.
Now the question for me, and I think you can probably tell my
opinion here, is this actually making people more productive?
So if we discuss that point, you know why I think tokens don't
mean more output. So Jellyfish that that
consultancy that released that study that I mentioned at the
start of this discussion. So they measured pull requests
shipped per quarter across 12,000 developers over 200 big
companies. Now a pull request for for
anyone outside of engineering is a standard unit of developer
work, a chunk of code that's been submitted for review, it's
been approved and then added to the product.
So shipping poor request has often been, other than merge
requests, I guess the closest thing to let's say maybe a
universal output. Now the engineers that we use in
the least AI burned around $3 of tokens over 1/4, so over three
months and shipped 11 poor requests each.
Now the engineers that we're using the most burn $1820.00 of
tokens 1/4 and ship 2313 more PRS pull requests.
So that's twice the output but about 600 times the cost.
And the word output I guess is a is a loaded word because a lot
of what the heavy users produce doesn't necessarily ship
everywhere. Remember Uber's numbers, 70% of
that committed code was written by AI, but only 11% of that
actually was running in production.
Now, it's quite easy to diagnose.
You know, just because you're producing lots doesn't mean it's
good. This is a new area.
It's an area that everybody's trying to innovate from a
process perspective, a systems perspective, trainings and skill
perspective. But also there's just natural
bottlenecks in a process. If you think about the theorem
of constraints, process will always move as fast as the
slowest moving part. And right now there's certain
areas that are just bottlenecks. You know, review is a
bottleneck. A senior engineer might be able
to review, say, 2 to 400 lines of code in a day, let's say,
yeah, AI can generate that in 90 seconds.
Now, people are doing automations on top of this and
automated testing. But still, that doesn't mean
that all of this is quality enough to pass QA.
So that is quality assurance and testing to actually get inside
the product. And again, just because you're
burning lots of tokens, are you doing it effectively?
Are you being efficient with your token spend, you know,
putting 1000 words PDF into a context window and then asking
to do, you know, lots of questions on that is going to
use loads of tokens? Are you actually achieving
anything there? Now none of this is new as a
problem. There's a principle called good
hearts law. So this is when a measure
becomes a target, it ceases to be a good measure.
Now a good example is a paper by someone called Khrushchev, and
it's about a Soviet candle factory.
And the way they evaluated progress was by weight.
Now, if the chandeliers grew heavier and heavier, so all the
workers were producing heavier and heavier chandeliers until
they started pulling ceilings down, they fulfilled the plan.
Now the question is, is does anyone really need that plan?
Do they actually solve any problem?
No, it doesn't. You're just creating heavy
chandeliers and measuring value based on that.
Bill Gates once said the same thing.
It's like measuring programmers by amount of code written is
like measuring the progress of us building a, let's say a plane
by weight has literally no value.
Wells Fargo famously measured stuff by cross sells to selling
more products to the same customers and ended up with
about 3 1/2 million fake customers.
All of this started as a reasonable proxy and KPI to try
and get people to do the right things that the organization
wanted, but obviously stopped working the moment everybody
realised that that's how they were being assessed.
And it doesn't actually deliver any value.
If that measure does not deliver value and everyone gains that
system, you're getting more of a bad thing.
And token maxing is just the newest member of that family.
It's the new trend. Now there are some people that
push back on this. You know, you could argue that
if your engineers spend 1000 a month on say LLMS and ship 10%
more, that's arguably cheap. It's not too expensive, you
know, Shopify's VP of engineering has been making the
argument very publicly. Shopify CEO makes a very similar
point that only through massive usage of AI will you get the
right skills to eventually become productive.
And and the reality is both are right.
If the output is real, then token usage is also great
because you're getting good output and clearly lots of it.
But the jellyfish numbers that study that Jellyfish released
and the example by Uber suggests that often this just still isn't
the case. And I'll come on to some of the
reasons I think this is the case, by the way.
And one of them, which is one of the reasons I got so interested
in this episode, is what mindset is actually solving for
internally and as a result has been solving for for other
customers and partners. But let's talk about, I guess,
the reason that things are getting so expensive and things
aren't necessarily improving in terms of productive output.
And as a result, what I think people should be focusing on
instead. So as I said, I, I think the
case for heavy consumption is partly right.
It's going to help people gain skills.
It's going to force people to actually adopt this technology
and get over themselves. But this still doesn't explain
why things are so expensive and things still aren't very good in
terms of quality output. And I think there's about 3 or 4
reasons. And, and the first reason I
think is, is pricing. So that previous example of
measuring people by a proxy, like lines of code written or
the weight of a chandelier, well, when those came about,
that didn't cost as much. So when a programmer in say 1982
row, let's say 500 lines of code, those lines sat on a disk
for nothing, didn't cost people money.
Whereas AI coding tools aren't sold by the seat, they're sold
by the token. So Anthropic, for example, move
their big enterprise contracts towards the late end of 2025
from a flat roughly $200 per user model with tokens included
to a more metered compute model. So you get $20.00 for a base
seat and then you get usage charge.
So when legacy contracts all over the world have started
expiring around, you know, March, April, May 2026, and all
these companies have really been drinking from the kool-aid of
token maxing, many customers bills will be tripling overnight
for Anthropic. Let's say a typical big company
runs maybe 150 to $250 a month per person, and then heavy users
run 5 to 10 times that. You can see how the numbers
start to get really expensive when you've got 78000
developers. And then when you have some
agents that are just running continuously, doing huge tasks
and using that entire token limit in just a matter of days.
Well, imagine if you have 7000 people with those agents.
Again, if you remember the episode I discussed recently,
the Open Core episode and why it's such an important software
invention and why Jensen Fuang is in love with it, is because
these things are going to consume so many tokens, which
Dr. LLMS business models which ultimately benefits NVIDIA
anyway. I think the second reason is
culture. Right now we've got this weird
culture like Matter and many of the big companies where you've
got this leaderboard of consuming tokens, you're Jensen
Wang and wanting to people to spend, you know, quarter of
$1,000,000 in tokens per year. So really this token consumption
has become quite performative in a way that you know, let's say a
company selling $20.00 per month per seat per user just didn't
see and didn't experience in the same way.
And when that metric is really publicly visible and the boss
says no limit and you get rewarded a pat on your head,
obviously people will start to play it and gain it.
I think the third reason here is a hidden quality cost.
You know, 43% of AI generated code requires lots of manual
debugging after it's gone through QA.
So testing, and there's been some studies to say that some
developers are spending about 38 to 40% of their week on
debugging and verifying whether what the agent has actually
built is the thing they meant to build.
So that heavy token usage by an agent.
So generating lots of words, although it seems really
attractive, it's only real if a human clears it up or you have
some form of really trustworthy testing software and suite that
automates that. And these things aren't always
easy to automate reliably. It's not easy.
And that clean up time isn't showing up on the token
dashboards. It shows up when you try to
deploy. And this is going to be the same
across all spaces and industries for those in every single, let's
say a law firm or a marketing team, a sales team, wherever you
are. The role of managers is also
quality assurance and quality checking.
You need to make sure what we're producing as an organization is
up to scratch. And again, maybe you can get AI
and go, yeah, AI, go and check this, but maybe you also need
people. So you have explosions of output
from some people and then lots of people having to verify that.
I think this comes to the other problem, which is contacts and
orientation of agents and then being able to reliably produce
things that you want it to produce.
Now some people are solving this with skills, with MD files to
give agents context and rules and processes.
And I've done episodes on that recently that have been really,
really popular. But I think still that it's just
a trend of the day. It will go away over the next 18
months because ultimately this system still relies on people
updating those documents, updating those skills, still
keeping them so fresh that they don't go stale.
And an agent reliably using that at scale across large
organisations, which although it's infinitely better where we
are now than 18 months ago, I still don't think it's perfect.
I think we're a way off from what will come as the next
evolution. And mindset is seeing this
because we're quite far ahead of most people from a process
perspective internally. And we've already really faced
that problem. We actually built our own tool
to solve that as an internal tool, which the team are
absolutely living. And I'll give you a good
example. So when you ask an agent, say,
to make a change in a code base, it might not have all the
context of your code base itself because it can't just, you know,
scan all of your code base because the context window blows
up. So a lot of people create
markdown files, so essentially documents which specify specific
ways things work, how my APIs work, how this works, how that
works. They also have processes they
enshrined. So how to deploy, how to push
tickets to Jira or Gitlabs, your product project management
tools. You then get people to pass
information. So a product team would go and
write up what you're trying to build in the spec, you pass it
to engineers and that might go into an MD file.
And then MD file gets given to agents and people create like
we've been doing quite complex setups, like really powerful
setups, clever setups of skills and MD files that essentially
enable an agent to operate and do good coding.
And this is the same that's happening now with called Co
work and all organisations as I've covered in other episodes.
So you create this fantastic system.
However, that doesn't mean the agent actually producing great
work. A lot of the time it produces
random work. That doesn't mean that all of
your engineers are able to suddenly use these things really
well. It doesn't mean that you can
easily track all of the key important decisions that are
made. Because in a project management
tool, you have tickets or you have tasks, but an agent doesn't
think in that way, doesn't think in a small, like tiny task, add
a headline, write subcopy, review.
It thinks in larger chunks of work, but those larger chunks of
work, therefore, when they're executed, have to rely on the
agent accessing that MD file structure, the skills that
you've created to actually do the work.
And then you have to test whether that's been done.
Because if it writes 10,000 words of or lines of code, so
let's say doesn't mean that all of that is actually what was
intended to be built. You often find things that
accidentally are snook in, you know, why did you do that?
Oh, I thought this was a good idea.
And again, I'm going to keep coming to this.
This is the same thing that's going to be happening at scale
with knowledge workers. And you know, three or four
months in or five months in from having on D files and skills,
you will start to find that suddenly some of these go stale.
People stop updating them. They don't use them properly
because naturally people just drift, especially when it's
nebulous. You know, it's, it's documents
like these things of skills that I, you know, I, I've been
promoting them and they're really great.
But they are just a document specifying how to do something.
And that information must be continually updated 24/7 for
every decision. And the more decisions,
strategies, projects you do, the more skills and MD files you
need. And it just becomes very bloated
and chaotic. Trust me, very quickly.
So this problem around token consumption relies on everyone
updating. This essentially document sets
that live somewhere. But then you need to track who's
allowed to update that document. Has this person used that skill?
Can I track if they've used that skill?
Wait, someone over here updated that document which changed our
process? OK, so we need a review process
to allow only some people to update an asset like an MD file
or a skill, but not others. But then again, things go stale.
Work isn't actually a high quality anymore.
So this is kind of what happens in a big system when you start
to really scale it and scale AI first processes.
So yeah, this is it's it's a real problem.
So you get drift from agents delivering the wrong things, you
get documents that go outdated, people not keeping everything up
to date, and essentially the entire system starts to slowly
decay and performance goes down, yet token consumption goes way
up. That's honestly one of the
reasons we've actually solved this and been solving this
internally for ourselves. We call it Memex.
It's essentially a system whereby you specify strategies
at the top and instead of using project management tasks, the
document which has access to your code base, which is
indexed, has access to specific files and other function and all
the skills turns that into tasks which are managed in a different
way. So for anyone that's really
interested in that, do reach out because we're actually having
lots of conversations about this internally to share how to use
something like this and how to implement better AI first
processes at the moment. Now I think that's a good
opportunity to now go and what I think people should be measuring
instead, how they should go about this.
Companies that are thinking about this really, really
effectively over them from a process perspective and how
we're solving that with internal tools like I described.
I think also the way you measure output is important.
So for example Salesforce have been trying to measure something
called agentic work units. So AW use as a complete counter
to token maxing instead of counting tokens consumed.
An AWU is essentially a completed unit of business
relevant work. So let's say Singapore Airlines
use AW use to track customer resolution time, another company
use them to track the quality of product recommendations.
And about 2.4 billion AWUS were generated across the system and
ecosystem in Q42025 alone. Now if you're running a team,
the job is therefore to define what your AWU is.
The unit has to represent a complete piece of work you'd
otherwise have to pay the human to do, and it has to track
whether that work actually stuck.
So for engineering, it's not necessarily pull request
generated or pull requests merged into a Co base.
It's pull requests that made it into production and stayed
there, or bugs per shipped PR. So per shipped piece of work,
how many bugs were generated? How many incidents were traced
back to AI assisted code? Or track the humour review hours
your team is burning on AI code for sales.
It could be qualified opportunities that progressed a
sales stage that was AI assisted versus not illegal.
Maybe it's contract sign, not drafts produced of content
actually published that drove traffic for marketing, not the
content generated. Again.
I keep coming back to that chandelier factory example.
What a stupid stupid. Proxy and token matching I
believe is the exact same and it will end.
And it is a big mistake to get obsessed with that.
It's maybe a good starting place to get your team using it, but
it's not going to produce value. So yeah, those organisations
that, yes, try and get people to adopt these tools but care about
how you organise contextual information that goes beyond
just MD files and skills that don't try and get people to gain
the system to spend tokens and think about what they're really
trying to measure, are going to succeed here and produce actual
value, not just trash. Anyway, that's it for this week.
Thank you for listening and I'll see you next time.