So on Wednesday, the US Treasury Secretary and the chair of the
Federal Reserve called an emergency meeting with CE OS of
America's biggest banks. The subject wasn't interest
rates or inflation or, you know, pending World War Three.
It was an AI model. Anthropic has now built
something that could find and exploit security flaws in
virtually any software, it was pointed out, and the financial
system wanted to know how exposed it really was.
Anthropic have decided not to sell this.
Instead they've called all the biggest companies from Apple,
Amazon, Google, banks, governments to form a coalition
called Project Glass Way. And they've kept the model
behind closed doors, only giving access to those companies.
Today I want to walk through what this model actually does
and what it means for us in practice, Why the sceptics have
some really interesting points here, and why the way this was
all handled tells us more about but where AI governance is truly
at and what risks we really face.
This is In the Loop with Jack Horton.
Hope you enjoy the show. So let's start with what Mythos
actually did. So Mythos is Anthropic's newest
AI model. And what makes it quite unusual
is that it turned out to be extremely good at, you know, one
thing, you know, instead of being the best in the world at
writing, and maybe it was really good at writing.
It turned out it was really good at finding security flaws in
software, not small business, not small theoretical problems
that real flaws in real applications that are out there
right now and used by basically everyone in the world.
And these bugs have been missed by human experts for years.
A little bit of context here. Every bit of software we use,
you know, our browsers, our phones, the servers that handle
online banking, there's just loads of bugs in it.
And most of these are absolutely harmless, but some are what
security researcher would describe as zero days.
So flaws that nobody knew about but could let an attacker take
control of an entire local machine.
So you take control of your laptop just because you clicked
on a website. If you heard about a company or
a hospital or a government being hacked, it's usually because
someone found and exploited one of these zero day problems.
And you know, only a small group of elite researchers typically
take on the job of finding these, whereas Mythos has just
found them in minutes, you know, across many different systems
all at once, and then without any human guidance or prompting,
worked out exactly how to exploit them.
And there are a few examples to talk through because, you know,
it really tells us what this would mean in practice at scale.
You know, it found a bug in Free BSD.
So it's an operating system that basically runs a lot of Internet
infrastructure and servers, networking equipment, basically
the really boring backbone stuff that most people just never see.
And that bug has been in the code since 2009.
Ultimate Testing Tools had run through it over 5,000,000 times
and it never caught this bug, whereas Mythos found it and
actually built a multi step attack across different network
requests that were given an attacker full control of
everything. So in practical terms, that
means anyone that could reach their server could read every
file that exists on it, install whatever they wanted, and use it
as basically a launchpad to attack any of the machines on
that same network. He also found a 27 year old
networking bugging Open BSD, which is another operating
system known for being one of the most security countries in
the world, and the the bug had been there since 1998.
It found vulnerabilities in every major web browser and it
found a chain of four separate bugs and flaws that if you
string it together and attack, meant that if you visited a
particular website and then Tucker could literally take
control of the browser, the security, it can break out of
the operating security and take control of your laptop.
You wouldn't need to download anything or click anything.
It would literally take control of your PC.
Mythos has found thousands of these zero day vulnerabilities
and few of them, 1% have actually been patched so far.
And Anthropic have published cryptographic proof that these
discoveries exist by basically tamper proof time stamps showing
that these things really do exist.
And they found these flaws before anyone else did.
And you know, none of this was the goal.
You know, Anthropic actually set out to build the best coding
model in the world. And it turns out that also makes
it really good at attacking software.
Now it's worth having a conversation about the skeptics
because there are a lot of skeptics out there and write
this out. For example, the headline number
that Anthropic have given out is thousands of 0 day flaws and
books and Anthropic as Human reviewers check 190 of these
reports and reviewers agreed with the models assessment about
89% of the time, but that means there's still thousands that are
unchecked. So there's a lot of
extrapolation there to say that there are thousands of 0 day
bugs and flaws. Another example is the security
research. Different labs for example, have
tested some of the vulnerabilities on cheaper
smaller models and they ran 8 experiments and found eight of
the problems. So it's 100% matching Mythos.
Now there's there's, again, lots of nuance and extrapolation that
that was only 8 problems. And it was also focused on the
things that Mythos had already found.
And on top of that, what those models definitely didn't do is
then build an entire attack plan by itself.
And another thing that I think is most compelling for me is
that there's just this timing aspect.
So Anthropic is about to go towards a stock market listing
and it'll be valued at about $380 billion, apparently, you
know, in a model that's too dangerous to release is
brilliant PR in the world for everyone getting excited and
putting lots of money in them. And there are many arguments out
there saying that Anthropic are essentially withholding the best
models so they can charge big enterprise huge amounts of money
and essentially keep everyone else a generation behind.
And, and there's good reasoning for that, you know, those
listening, most likely me, our team, we're power users and
power users cost model providers money.
You know, we might be paying. You know, I've got a couple of
accounts at $200 per account. We definitely use more than $200
worth of credits every month. Again, intelligence isn't cheap,
but you know, the long tail of you, let's say hundreds of
millions of users out there that use it, fraction of the amount
that maybe I do, That's where they make their money.
So then suddenly finding a really good model, charging
absolute fortunes to big enterprise, who wants a
competitive advantage like that and will it be able to pay those
huge sums of money is a very clever strategy.
And there's also an aspect which is that we have been here
before. You know, in 2019, Open AI
announced a model called GPT 2, and they refused to release it.
You know, they said it was too dangerous.
It could write so convincingly that it's going to fuel
disinformation and propaganda. And so they staged it out over
nine months, releasing progressively larger versions.
And, you know, by the time the model had come out, they had big
test groups and nobody found any risk in the end.
But obviously, it was brilliant PR for them to raise more money
and attract more exciting people to come work for them, come
invest in them. Though what I would add on to
that is a little bit of a nuance, which is, you know, a
mythos is a bit different. You know, Anthropic isn't
withholding anything. They're in fact literally
publishing all the evidence, all the reports, and allowing
anybody to test them themselves. And arguably you can't really
have a staged release process when the model is able to just
attack software at will and take down the entire Internet.
It's a bit too dangerous to go and risk it.
So, yeah, is this real or is this marketing?
And it's a good question. I think it's both and I think
it's OK to have both. You know, two things can be true
at once. It probably is quite dangerous
or they're worried that it's dangerous and it's a wonderful
PR opportunity. But personally, I think the
bigger and more important question to ask of instead of is
this PR marketing? Because, you know, of all the
companies, I think Anthropic is probably the most ethical.
You know, we covered the episode where they fought back against
the US government and the Department of War.
But the bigger question really is who decides that something is
or is not safe? At what point does this become
safe enough to release now? So like Anthropic is a good
example, looked at what they built, concludes that gosh, this
was too dangerous for the public and decided which specific
companies could get to use this. And they called it Project Glass
Wing. You know, Amazon, Apple, Google,
Microsoft, 40 other organizations get access.
And yes, Anthropica committed $100 million in computer credits
to fine and patch any vulnerabilities.
But essentially they are still a private company making a private
assessment from the private coalition, arranged a private
solution with no, say, treaty inspectors, independent
oversight, no democratic input. Yet when it comes to nuclear
weapons or industries, you have international oversight,
independent governance bodies who are deciding who might get
access, whether they're actually breaking the rules, whether
they're being safe. Now, I'm not trying to be all
too European on this in terms of trying to create huge regulation
that makes it really difficult for these organizations to
operate because that's definitely not me.
However, I am often quite balanced when it comes to
wanting to pair risk and reward. So maybe they are the right
organization to take the right steps and make the right
choices, but equally at all organizations, the right
organization. I don't trust open AI as far as
I could throw them. See, I think this tells us where
we are in the maturity of AI and governance generally, that we
still rely on private companies and we still haven't got the
systems in place which Anthropic of openly acknowledged to keep
society safe as these things are rolled out.
Anyway, what I think to conclude, what sticks for me
isn't where the mythos is, you know, super powerful and, and I
actually really hope it is because it's quite exciting.
It's going to help me at work, it's going to help me at life.
And to be honest, it probably is, or at least close to it.
But what sticks with me is that this is the first time the AI
industry as a whole has agreed, really, that this capabilities
are too dangerous to release based on real results rather
than, you know, theoretical risk.
And the best way we could manage this risk was obviously a group
of private companies deciding amongst themselves what to do
about it. So yeah, I think the question on
my mind right now is who decides what happens next or when the
next one appears, or on whose authority and accountability do
these things get released? Anyway, thanks for listening and
I'll see you next week.