Anthropic just released a major labour market report.
It contains a chart showing the gap between what AI could
theoretically do across every occupation and what it's
actually doing. And that gap is, well, pretty
big. I think the chart is a very
honest picture of where things stand with AI today.
Studies have found that AI could theoretically handle 94% of
tasks in computing and math is doing 36%.
In legal work apparently could handle 90%.
He's doing 20. In most categories, the gap is
even wider. Today, I'm going to cover what
is really slowing this down and why this ceiling may be just too
high to start with, and what technology leaps we must make
for AI agents to truly impact 93 and 94% of tasks across
different jobs. This is in the Loop with Jack
Horton. I hope you enjoy the show.
start from the top. And there's a couple stories
from this really interesting study, but most importantly the
chart. So Anthropic published a report
called Labor Market Impacts of AI, a new measure and early
evidence. And it introduced something they
called observed exposure, which is basically a metric that puts
theoretical AI capability next to real world usage data to see
which jobs are actually being impacted to try and get get away
a little bit from the hype and the media cycles.
And the report is summarized well with the radar chart.
So if you can picture those almost a circular chart and it
looks like there's a spider's web in the middle of it.
They used to tell the user at school to rate how good I was at
different skills around cooking in cooking class.
And essentially what you see in this image is a large blue
spider diagram that stretches out towards the edges for a
bunch of different occupations of management, business,
computer, architecture, life sciences, social services,
legal. And that's a theoretical impact
of AI on these occupations. And I can tell you it's a much,
much bigger spider web than the red one, which is the actual
current observed usage. And this is measured by
Anthropic's own data. And the numbers are really
interesting between how much AI could theoretically transform
these occupations versus observed usage like computer and
maths. 94% theoretical but only 33% observed.
Legal, meaning 90% theoretical but barely passed. 20% observed.
Almost across every domain. It's the same pattern apart from
a bunch of domains that basically hasn't been impacted
at all. You've got, you know, grounds
maintenance, food serving, agriculture, construction,
installation and repair. None of those have been touched
by AI really. Obviously Anthropic's
interpretation is that this is a major growth story.
They said, and I quote, as capabilities advance, adoption
spreads and deployment deepens, the red area will grow to cover
the blue area. And in their framing, the red is
the blue's younger self. Give it time and convergence is
obviously inevitable. However, you could read it
differently. That gap could also be telling
us where the boundaries are, not where it's heading.
And Anthropic is looking at that distance and seeing potential.
Whereas, as we know, some people will see it as a big chasm that
may or may not ever be crossed. So yeah, let's discuss that.
So from one perspective, you could argue that the ceiling is
actually lower than it looks. So that blue shape, that
theoretical ceiling comes from a 2023 Open AI paper called GPTS
Are GPTS. It's almost three years old now.
The authors went through every task in the ONET database, which
is a primary source or US source of occupational information,
covering about 900 jobs. And in the study, Open AI scored
each 11 if an LLM alone could double the speed, 0.5 if it
could with additional tools, 0 if it was out of reach.
Anthropic adopted this as the upper bound of their chart.
The issue is how those scores were generated though.
In the study, they were asking whether a task could be speed up
by a model with, say, a 2000 word input limit and no access
to current information on the Internet.
That's not measuring what happens when the AI has to be
integrated into a complex procurement workflow at bank.
It's not measuring what happens when the output has to pass a
legal review, or when the data lives in 3-4 different systems
that just don't talk to each other, or when the person using
it doesn't trust it. It's measuring what's possible
in a very clean, you know, very controlled hypothetical
environment, and then trying to project that across the entire
economy of the occupation. Anthropic does flag this to be
fair to them, but their argument is that the models just have
improved, so the real ceiling is probably much higher than the
bully suggests. But you can run that logic in
the opposite direction too, you know.
The original scoring was probably way, way, way generous
because it was measuring what a model could do under perfect
conditions with no friction, no context, no complexity, and also
during a period when hype around LLMS was at its absolute peak.
So if anything, you could argue the blue is too big, not too
small. See, I, I, I think studies like
this are really helpful to understand, but not obviously
where I get very interested. We should come on to, I guess in
the secondary part of this discussion, because I've talked
a lot about this. Benchmarks these days are, well,
just as much about the importance of vibes.
So the vibes of this new LLM and how it feels and B, the
complexity of real world deployment that that's what
these benchmarks rarely capture. So the question I'm actually
interested in is, how do we make this translate into a situation
where conditions aren't perfect? Because these benchmarks are all
in perfect conditions. The chart suggests that the
answer is not nearly as simple as we'd like to think, because
that red spider's web is much, much smaller than the blue
theoretical one. Let's explore that.
Why isn't AI impacting more tasks?
I want to say something quite clearly before we get into this
discussion because it could sound like I feel somewhat
negative about the situation where it's not actually where I
land. I would describe myself as a
realist argument about the situation because I believe AI
is going to be transformative. I work in the industry, we
actively build systems and products to deploy it into big
companies. So yeah, it's more of a
realistic argument about how long transformation actually
takes before technology becomes truly upscale.
Anyway, there are a few things to explain this gap.
I think in theoretical expectation and reality 1 is
about the technology, which something we as in mindset are
actively working on. So it's really interesting and
can really give a kind of insider's perspective on.
And the others are human problems.
And I think really the human and organizational problems are the
hardest to fix. Let's start with the people
problem. The first problem is that the
adoption curve is more of a Cliff rather than a slope right
now in AI open AI S own numbers tell us a really good story.
ChatGPT has about 900 million active users or weekly active
users less now after the quick GBT movement.
And about 50 million of those are actually paying power users.
So the top 5% of those paid subscribers use reasoning
capabilities about 7 times more than a median paying customer.
So if we do the maths on that, 5% of 50 million is 2.5 million,
that's 0.25% of the total user base actually using AI in a way
that could truly reach that theoretical expectation of how
AI will impact an occupation. Most people therefore we can
assume are using it for very basic things or in a way that
isn't going to transform their occupation if this were a normal
adoption curve. So in technology, when a new
technology appears, we have we see something called an S curve.
So it's a standard model for how new technologies get adopted.
It looks like a flattened S on its side, which means a slow
uptake at the start. Early adopters are small
numbers, then a steep acceleration in the middle as
mainstream users pile in, which is called the tipping point.
And then it kind of flattens out at the top as you hit saturation
and only really the laggards are left to start using it.
Well, if we're experiencing that the middle should be filling by
now. And you could argue there is and
that when you're in the middle of it you are very impatient.
But 3 1/2 years after ChatGPT launch and it isn't.
What we're looking at is close to a Cliff edge right now.
A tiny spike of heavy users right at the top and then the
near vertical drop to everyone else.
So a handful of people have obviously figured it out and a
vast majority of people are just bounced off or haven't found the
use case or skill set to make it stick.
And I'll say this because the data seemingly points to it.
I think probably have a skill issue.
Now, it could be me being impatient here.
And therefore that Cliff will become that S curve over time
and it's just going to take a little bit longer.
We'll have to see. But I don't think most people
have developed the ability to use these tools in ways that
produce seriously meaningful results.
And nobody is teaching them how to do this at scale.
I bet it's not being taught at school.
The second problem is that the economy moves far slower than I
guess the AI industry wants to admit.
So large companies don't RIP out enterprise systems they're
running on because something better came along.
They're locked into 1218, two year contracts with platforms
that, you know, meet their needs quite well.
Even if those platforms aren't AI native and aren't that
impressive, people will be forced into waiting.
Now that that will last for long, you know, by the end of
that contract, they'll be at risk of losing or they'll be
forced to start integrating other companies into that
solution, which will slowly eat and eat and eat away at, I guess
the perceived value of the big company.
So the cycle time for these enterprise software deployments
are measured in years. You know, some companies take
1218 months to deploy your technology.
So therefore there will be a slow adoption enterprise level,
even if they've all adopted ChatGPT, there's still 10s of
thousands of other SAS of technology platforms who are
solving problems and who are not AI native.
And you know, only 3% of Microsoft 36 fives, 450 million
users have even adopted or opted into Copilot yet.
So again, it will take time. The third is that the technology
stuck is actually harder to integrate into apps people use
daily. And so I see it as also a supply
problem. So what I mean here is that you
see the supply side every single day.
So people might use AI if it was in front of them in a technology
solution they already use everyday, but it's not.
So they therefore would have to learn to be an expert on other
AI native tools. So the claws, the chat, GBTS,
the many, many others, because we work with technology
companies, SAS companies, typically large B to B SAS
companies deploying agents into their products.
And the complexity is really, you know, quite staggering when
you go from, say, a prototype or a quick demo, You need an entire
stack to make all of this work. You got huge amounts of
complexity to embed and integrate these into solutions.
You need to identify how it's all going to work and even the
use cases that you want to deploy for.
First, you know, you need an agent framework.
You need reasoning and logic. You need to wire this into all
your systems. You need to understand how
agents understand data and APIs. You need to build APIs for that.
You need to create MCPS. You need to call it, create a
tool calling layer. You need to then have a testing
layer. You need to be able to surface
the right information and test that that's the right
information because big enterprise companies really care
about compliance. You need a purpose built
interface for every type of interaction, like widgets
appearing in charts and charts. All this has to then work at
production scale, not just for, you know, 2000 people or 100
people. And I'd say that about 75% of
the companies we talked to are either just getting started or a
little bit into a build. And these platforms aren't small
though, you know, companies with 10/15/20 thousand enterprise
customers worldwide. And so without that supply, IE
the big companies that already have millions of users
integrating this into their products in a really effective
way, it's really hard for most people to use AI.
So the real question therefore is will it grow?
And I'm confident that it will. And I think there's a really
core component to what will make it grow and that's making the
technology itself reliable. So this comes to the fourth
reason I think that there's still a gap between theoretical
and real world usage. And this is most interesting to
me because it's what we're obsessed, we're trying to solve
right now. It's, you know, something that
we actively are always in at mindset.
So essentially there was a really good paper that
highlights this point and discussion here.
And it's called Towards a Science of AI Agent Reliability.
And it essentially puts rigor behind something.
You know, I've been saying on this podcast for a long time
that the industry is often measuring the wrong thing.
You know, you draw a line between capability, So can a
model do this task and reliability?
Will it do it consistently, safely and predictably in
conditions that are rarely perfect?
And we don't just mean that they do the right thing most of the
time. We mean something much more.
And this is what the paper breaks down really
interestingly. A colleague and therefore an AI
agent must be able to #1 be consistent.
This means they get it right consistently, and that means not
right today and therefore wrong tomorrow on the same task #2
Robustness. They can't just fall apart soon
as conditions aren't perfect for them #3 Calibration.
They're able to tell you when the unsure rather than
confidently guessing and #4 is safety.
So when they do make a mistake, their mistakes are fixable, and
they're not just catastrophic failures.
Again, what you'd expect from a colleague.
And to assess agent reliability right now, the authors broke
down all of this into 12 metrics across 4 dimensions and tested
14 different models from Open AI, Google, Anthropic, spanning
a year and a half of product releases.
They run each task five times with different sets of
paraphrase instructions. And they also deliberately broke
things. So they tried to cause tool
failures and API calling failures.
They tried to simulate environmental faults to see how
agents handled imperfect conditions.
They asked the models to report their own confidence to test
whether they could tell when they're wrong.
Overall they had 500 total benchmark runs and the headline
finding was that 18 months of LLM capability gains produced
only modest improvements in reliability.
All three major providers custard quite closely together,
suggesting that this is an industry wide challenge, not a
company specific one. And for me, I don't think this
is solved at the model layer, it's sold at the application
layer. So for example, on consistency,
whether the model gave the same answer when asked the same thing
twice. Scores range from 30 to 75% on
robustness. So whether it can deal with
imperfect conditions. Models handled infrastructure
failures like big server crashes quite well, but apart from that
didn't really do that well on predictability, whether the
model knew it's wrong. Performance was also very mixed
and this speaks to a challenge within implementing AI right
now. I think that, you know, mindset
sees all the time it's in that we're actively working through.
And as I said, it doesn't get solved at the model layer.
It gets solved at the application layer.
And again, models are trying to bring in more application
technologies. So they're trying to turn their
model into an agent itself in the way it works.
And this also is why SAS companies have such a big
opportunity really to create perfect agents for their
specific niche because a general purpose, Claude, although is
absolutely incredible and can do many things, if you make a hyper
specific agent for a specific area that you're an expert in
can be really, really powerful and more consistent, more
robust. And the reason that we have to
deal with this is essentially what you're asking is do we want
an agent to have deterministic controls or have high inference?
So essentially, can an agent decide and reason what to do
high inference, or do we, as say the agent creator, give it
specific instructions that it's not really allowed to stray
from? So if you give that agent a step
by step plan to follow, that's all it can ever do.
So if a person comes in and wants to go in any other
direction, it's not going to be able to do it.
Or if you let it reason itself and decide, it might go off in
different direction to solve the exact same problem.
Because ultimately an agent, all it's doing is observing,
planning, acting, checking, adjusting.
So it makes a plan, it sees how that plan goes, sees what data
and information it gets from that plan, checks whether that's
the right thing, and if it isn't, adjust.
I'll check in with the humor so that that's what enables agents
to do really incredible things that blow your mind.
And it will deal with imperfect conditions really well.
It can tell you when it needs help.
So it can act more like a person, but it doesn't mean it's
reliable. There's loads of different
things that can go wrong when it's reasoning.
So this is where that trade off between consistency and
flexibility becomes really important to rigid a set of
instructions. It can't help people in any ways
other than that step by step plan.
Too much inference means that it's likely to be inconsistent.
And so there's lots of ways to try and solve this mindset,
solving this in really interesting ways.
We're we're creating something called blueprints, which is more
like giving an agent a set of DNA, which is a set of
instructions that can last over many months.
So rather than just, you know, a quick process, you can give it
the ability to solve, say 50 problems with what we call
tools. Now a tool might be, you know, a
integration to a separate system to help a user do something.
It could be a small set of instructions that the agent can
follow, but a, the blueprint is essentially its DNA over many,
many, many months, like a long term objective.
It might be the HR recruitment, an entire employee life cycle of
how it expects to help people, or a product discovery journey
that might take six months. So it describes a set of things
an agent must take people through over a long period of
time. So yeah, this is a really big
area of investment by all these big companies.
It's really, really important to solve and all of it does take
time and therefore it takes time for adoption.
And we've seen this before. In 1987, there was a writer that
said, see the computer age everywhere but in the
productivity statistics. And in February, Apollo's chief
economist said the exact same thing about AI.
So yeah, 15 years into the computer revolution,
productivity actually slowed, especially during the IT build
out. It dropped from 2.9% annual
growth in productivity to 1.1%. But then between 1995 and 2005,
it surged. The technology hadn't
necessarily improve that much. The change was the organizations
finally rebuilt how they worked with it.
The technology was super consistent, it was accessible,
and it took a decade and a half to do.
Let's conclude. The dominant narrative I think
says that there's a massive market disruption that's
imminent. And I think we're starting to
see that. You know, Dario Amadei, the CEO
Anthropic, has said this over and over again.
But clearly the story, I think this data support is a little
bit more complicated. AI's economic impact is
definitely real, but held back by reliability limits that the
industry hasn't yet fully solved and organizational human
barriers that are just not easy to solve because theoretical
baselines are always more aspirational than measurement.
So yeah, Will AI transform work? Yes, Computers did reshape the
economy, but on their own schedule, not on what the
industry wanted it to be. And because everything
surrounded, the technology had to mature with it.
And I think the same thing is happening right now.
And this type of reliability research is going to force us
all in the industry to measure what matters.
So anyway, I hope you enjoyed the episode.
I hope you learned something. Thank you for listening and I'll
see you next week.