Welcome to Crying Out Cloud. The podcast will make you laugh,
cry and reconsider all of your cloud security fears.
I'm Eden. I'm here with my amazing Co host
Amitai. Howdy.
We're back. It's been a while.
We didn't do a news episode, but we're going to fill you in on
everything you need to know. What are we going to tell them
about Amitai? So today, First off, we have a
guest, Romney McCarthy from the Wiz Research team, who's going
to come and talk about his research into VS Code extension,
supply chain attacks and the work he did there, which is
really cool. Next, we'll be talking about a
Redis vulnerability, also discovered by Wiz Research at
POM 2. On that we called a Redis shell.
Then we're going to talk about a security incident that happened
to F5 and try to figure out what that's about.
We're going to talk a bit about how attackers are sort of
incorporating AI into their malware and using AICLIS that
they find on target machines and why they might be doing that and
what it might mean. And then we'll do a new section
that we're calling. Cricket.
Like cricket? Take note that Amita has
wonderful activities for his children.
Play DoH. Chalkboard.
How do you know these are for my children?
Maybe I like to play with Play DoH in my spare time.
Yeah, we should write out the the volumes on the chalkboard
like it's. Oh, I can do that.
I can do that next time. I'll do that next time, OK?
For our next topic, we have the VS Code extension supply chain
attack. Rami is on the pod today to talk
about it. Rami is the head of Risk
Research at Wiz. Welcome back to the pod.
Thrilled to be a frequent fire. Let's just start with what did
you find? Tell us.
Yes, this is something we shared out from the Wiz Research blog
recently. It was actually a relatively
long research process for what I would say is an issue with a
really simple root cause. Fundamentally what we found was
that VS Code extension marketplaces, so VS Code being
most popular developer IDE that has extensions that add
functionality. These are distributed through
both the official VS Code marketplace run by Microsoft and
the open VSX marketplace that is tied to AI enabled ID ES like
for example cursor. Those extensions were frequently
leaking secrets from their publishers.
So publishers were shipping out these bundles of code.
And with any bundle of code that gets shipped out to the
Internet, it tends to leak secrets.
And that was the initial insight.
We actually started by looking for malware, and then we
stumbled on this problem of secrets and decided to take a
deeper look there. And secrets on their own are
problematic, but we have a lot of cases of secrets leaking
these days. I mean, Wiz has published
research on how AI has contributed to increased secrets
leakage. We've seen secrets in MPM and
Pypy on GitHub. Of course, the bigger issue came
when we made another connection, which was a lot of the secrets
that were leaked were actually publishing tokens for these
marketplaces. So what you'd have is you'd have
a very popular extension that was distributed that carried
with it ability to update that extension across the entire user
base. So extensions in BS code are
auto updating. If you ship a new version, it
gets pushed out to everyone's laptop.
And in the end, we figured out that we could push malware
directly to somewhere around 185,000 developer laptops who
were using vulnerable extensions across both marketplaces.
So imagine if you found it, who's nice and lovely and
awesome, But if an attacker found these tokens, like, what's
the scope of what they could have done with this?
Yeah. So there are two references I'd
make here when we talk about the scope of impact.
So the first is that we recently had a worm bull attack on NPM
happen. This was, you know, a recent
incident where NPM packages were compromised and once
compromised, poisoned additional NPM packages.
That incident, you know, made made the news, was far reaching,
had a pretty severe impact across the landscape and was
eventually centrally stopped by NPM and by GitHub.
So we could worm this malware, right?
If we're able to distribute to 185,000 desktops, we could
immediately perpetuate that malware on to other supply
chains. But that's NPM packages where
that's extensions that are published by victims of the
initial attack. So that's just like one recent
incident that leveraged a similar capability on a much
smaller scale. The NPM attack, the worm was
seated with three or four packages, I think.
And so here we have well over 100 different extensions, a
really diverse user base that gives you no more legs to it.
The other issues that shortly after I published this research,
actually someone attempted something very similar
maliciously in the Open VSX marketplace.
This reported by a couple firms, but there was a attempted worm
through Open VSX. It doesn't seem to be
particularly well crafted or particularly successful.
And in that case they used one leaked token to publish at least
one, maybe 3 or 4 to publish a malicious the Open VSX
extension. And so that may have impacted,
you know, a handful of real world users.
The possible scope of install base was relatively low.
It was in the few thousands. And so that didn't give enough
leverage, enough attack surface for a worm to get off the
ground. That's when we're looking at
some of these worms. There is a a tipping point where
if you can get broad enough distribution initially, it
becomes really hard for the industry to react and outpace
it. It tends to spread
exponentially. And so the larger the starting
population, the larger the eventual impact.
Is this affecting Enterprise Oregon?
Is it affecting Just like anyone can install this, these these
extensions, right? So it.
Impacts everyone, enterprise or not.
There are a few ways it's specifically impactful to
enterprises and I think a lot of enterprises or who cares about
these issues, individual developers obviously have have a
less rigor around their security than enterprise would.
So a few things we've seen recently in the Shihalud and
Singularity attacks, we've seen developer accounts leaking
enterprise secrets, and that similar could have happened
here. If you have a developer who has
enterprise credentials mingled with a personal IDE instance, it
would have been very easy for those lines to blur.
The second thing is that there are actually many impacted
extensions here that are internal to companies but
distributed publicly. So you know, for example, Acmico
has a internal developer efficiency extension.
And the easiest way to get it out to people is to just put it
on the marketplace, right? Download.
It will owe off you as soon as you download the extension and
log in. So, you know, we're not risking
anyone else using this extension but internal and some of those
extensions had leaked secrets as well.
So those were companies where you could immediately attack
their developers in a really targeted way, including one we
make reference to anonymized of course, is a $30 billion Chinese
mega core had a internal developer extension with
hundreds of users using it. So that's like a company that's
sort of really at risk where this to have been discovered and
weaponized by an attacker. It's crazy.
First of all, what has been the response?
Like obviously you have a mega company and they have a
different response than maybe like just the average person on
the Internet. But what has been the response
to the finding from the community?
And then I'd also love to hear from Microsoft and working with
them. Yeah.
So we, we were very careful when disclosing and eventually when
publishing this finding to be as responsible as possible to give
the most time for the industry to respond, for vulnerable folks
to respond. I will say we've seen a similar
pattern with secrets leakage, where in any form of mass
disclosure, it can be hard to get disclosure acknowledged and
remediated by the victims themselves, the publishers,
right? Some of these publishers put up
a popular extension five years ago and don't have access to the
e-mail anymore. However, we also spent six
months working with Microsoft centrally to address this and
that took a few forms. So one is Microsoft is much
better quick than me personally to do a notification campaign to
VS Code Marketplace publishers. And so they did a mass
notification campaign multiple rounds.
They also removed extensions where there was no response.
So if you were exposing a secret and they couldn't get in touch
with you, they would proactively protect you by taking down that
extension in some cases. And finally they added server
side prevention here for leaking future secrets.
So Microsoft owns GitHub, right? They have a long history of
secrets detection and awareness. Like the GitHub Secret scanning
program. They've now applied that to the
BS Code Marketplace. So when you go to publish an
extension, you will get stopped and warned if there's a secret,
and in fact they block it. There's no opting into leaking
secrets in your BS code extensions anymore.
So what should people using BS code extensions and orgs do to
protect themselves? I think if you're an enterprise,
you have two ways of doing security here. 1 is you can
limit what extensions are run centrally, manage it.
Or two, you can limit the impact of a single developer
workstation being packed. Those are both hard problems.
I think some companies have done a really strong push on the
latter, right? They're they're moving to
Chromebooks for many roles. They're removing credentials
from machines. They're using virtual
development environments, right? There's hardening you can do
that makes this less of an issue when it happens, but maybe a
more targeted approach would be extension.
Allow listing and management centrally and and this applies
to all extensions. The importance of the supply
chain attack here is that it doesn't matter how complicated,
risky, interesting the extension is.
We found a lot of cases where we found the leaked tokens to
themes, and themes are just like configuration bundles for colors
in your VS Code instance. They don't even have codes.
So as a security team, you're inclined to allow that sort of
thing, right? Like it's not as scary as
something with code, but there's nothing stopping someone taking
a theme and hijacking the developer account and pushing
out malware. There's nothing actually
limiting themes on the code execution front.
So another example of really where you have to be relatively
restrictive and also consider whether auto updates are are the
best fit for your organization or not.
And what in terms of like what like just individuals can do,
are there any like open source tools that they can use to sort
of scan their extensions or just in terms of best practices like
what that what should they be avoiding?
If the features aren't in the platform, individuals are gonna
really struggle always. That's my belief.
Right. Like if a platform has security
hardening, you can always talk about like the Apple App Store,
right? Apple is responsible for review.
They are ensuring that only safe things go through.
I think Step 1 is gauge your trust in the marketplace.
Like if you're using open VSX, there's a set of security
controls. If you're using the official
VSXVS code marketplace, there's a higher bar right now for
security. And so considering where you
source your extensions is part of your risk math.
I think also we, we talked about how to measure risk in 3rd party
packages, extensions in generic way, popularity history, recency
right there. All these metrics you can use as
sort of a, a thumb in the wind. And So what I would say is the
active malware we've seen, barring the supply chain
takeovers, which are impossible for a user to defend themselves
against, frankly, right? Like there's don't use any
extension because if it gets taken over, like you can't do
anything, right. Instead what I would say is
focus on very popular extensions.
Look at the publishing date, look at how many downloads it
has. Look at how many reviews people
will pump fake downloads to make sure it's not just a brand new
version. People do typo squatting.
Go directly to the marketplace. Don't follow Google search
results. We know people do SEO poisoning
as part of attacks. They'll put fake ads in.
Don't go and follow some sort of AI hallucinated slop squatted
extension name and really just consider that these are part of
your tax service. I think it's, we've gotten
really comfortable, same thing in NPM with JavaScript packages,
just like Willy nilly expanding our tax service by adding on
other people's code, other people's exposure.
And every VS Code extension you bring means you're not just
trusting the code as it currently is, but if you're
having auto updates, you're trusting the publisher and their
personal security and and just keeping that in mind.
When you found this and you had your big moment of glory, what
did you do to celebrate and what'd you have for dinner that
night? I did not.
Here's the fun part about research like this is you don't
get to celebrate until you publish, because you know you're
in for a long road of responsible disclosure.
And so in fact, I I flew to the US the night after finally
publishing this blog and really turned off my laptop and closed
myself down for comments. Attended a beautiful wedding, so
we'll call that the celebration but and.
It's closing your laptop is a celebration.
Anytime my laptop is closed, it's a good day.
Amazing. OK, Thank you for coming.
Thank you for sharing. Come back soon.
We appreciate all of your work and all of your insight.
Thank you. Give me nine months to find
something else interesting. I'll be right back.
All right, reddest vulnerability, a significant new
vulnerability has been discovered.
So what do we know about this so far, Amitai?
Well, I mean, we know a lot because I mean, it was
discovered by Wiz. I will say that some of the
details aren't public yet. Reason being that we're trying
to give people some time to patch before we reveal to the
world how this vulnerability actually allows.
RC Benny, a researcher from Wizard Research who discovered
it recently, gave a talk at Hexicon about this.
So if you were at Hexicon, you probably know the answer.
And if not, then you'll have to wait for our blog post or ask
someone who was a hexagon. We have to give him a tease on
the podcast. A little tease.
Well, I can say that it definitely allows RC, which is
not obvious because it was very difficult to do so from what I
hear. Do we know how many
organizations were potentially affected by this?
So this affected all past versions because it was part of
the Lua engine which is which has has always been part of
Redis as far as I know. And this probably existed for
around 13 years. And Redis is used, according to
our data, in around 75% of cloud environments.
And this affected all of them. And it's really, really
widespread. A lot of people install Redis as
part of their products. As a lot of people have Redis in
their environment, then they don't even know about it just
because it comes packaged alongside other software that
uses Redis. So Redis is is very prevalent
which makes this vulnerability very important to batch.
OK so if I am listening and I want to know what to do, what do
I do? So First off, you need to find
where this vulnerability is likely to be exploited.
Which means that you need to search for cases where it it's
publicly exposed, where Redis is actually exposed to the Internet
and running in a way that allows you to anyone to authenticate.
Which means either have no authentication or have like a a
weak or default password. Now interestingly, Redis does
actually have in its default configuration.
It doesn't allow anyone to authenticate anonymously.
It doesn't even allow you to expose it to the intranet by
default. However, there are many flavours
of Redis and many different ways to install Redis, and in some of
them, some of them which are really common, authentication is
actually set to allow anonymous access by by default.
So depending on how you install Redis, you are either affected
by this or not. So our suggestion is again,
focus on publicly exposed instances.
They're misconfigured to allow anonymous authentication.
OK, next topic is the F5 security incident.
So there's a network device manufacturing company F5.
They had a breach by a Chinese threat actor.
F5 makes like products like big IP.
They serve things like as a firewall or a router.
What's crazy about this, and we'll get into a little bit more
details in a minute, but the TLDR is the threat actor was in
their network for two whole years and F5 isn't exactly clear
on what they did in the environment, but they did access
their source code. Yeah, I, I guess this, this
could be classified as infosec drama in some ways, although the
drama is, is sort of between F5 and and China.
I guess it's also sort of between F5 and like the legal
system and their customers and the rest of the community
because F5 took a while to disclose this.
Many times it takes a while to disclose these things just
because it takes a while to determine if there's actually
been any material impact and material impact, at least in the
states, kind of determines whether you, whether you
actually need to disclose this publicly.
So there were two delays here. 1 is it took them a while to even
realize that they had been breached.
And once they realized this, it took them a while to reach the
point where they felt comfortable sharing this with
the world, telling people what happened.
One thing that they did, which was I, I, I think it made a lot
of sense before they went public, was they enlisted the
help of a few companies to basically check their source
code and figure out if it was likely that the actor inserted
anything malicious into their source code during the time that
they were there. The simplest explanation is they
just wanted the source code to have it.
Maybe they were trying to steal other things, like personal
information about their customers.
Maybe they were trying to steal like their documentation.
We know that they managed to gain access to information about
vulnerabilities that hadn't been published yet, which in theory
could have been like really cool for the threat actor.
But in actuality, these vulnerabilities, when you look
at them, they don't really seem to be very interesting and of
themselves. But if they were there for for a
while, then any vulnerability published over the past few
years, in theory they might have had advanced notice about.
However, just stealing source code and just feeling
documentation doesn't seem like it would be worth the effort.
And it it kind of makes sense that they'd want to do a bit
something more. So I think everyone's sort of
wondering what they were doing there.
But what's interesting is or interesting depends who you are,
but I find fascinating, we find fascinating is the Chinese like
operations aspect of this. Like we've seen a trend of
Chinese operations as of late. Where does this tie into it?
What does it tell us about their larger maybe motives or what we
should be looking for for other things they might get themselves
into? Saying that Chinese productors
or China in general has an interest in edge devices like a
5 big IP, that that makes a lot of sense.
Like it makes sense that they'd want information or the means of
conducting a supply chain attack or stealing vulnerabilities
about relevant to edge devices like big IP.
The idea is that this lets them get into get initial access into
networks without requiring any interaction with victims.
Like you don't need to do social engineering, so the attacker can
kind of just control the situation.
It's the equivalent of like a zero click vulnerability on on
an iPhone. Like you decide when you get
into the organization and, and nobody can stop you because you
have a zero day vulnerability and these things are sort of
meant to be exposed. You can't not expose them.
And also edge devices are very comfortable for productors
because there's not a lot of visibility on them.
Usually can't install any sort of EDR or, or, or sensor or
anything like that. So it becomes a bit of a blind
spot. So you could deploy malware on
there and sort of conduct your operation from there without
anyone being the wiser. Other than that, China has been
a very aggressive in their pursuit of vulnerabilities.
They don't just research like they steal vulnerabilities, they
steal information from vulnerability researchers.
They steal information from companies that have information
like F5 has about big IP. And there are also rumors
circulating that that China has also stolen, basically Chinese
offensive organizations have stolen or abused Chinese
defensive and organizations access to early information
about vulnerabilities in order to weaponize them and use them
against their their own targets, which is one of the reasons that
a few Chinese companies have have been blocked from from
these early access feeds. OK, well if we learn more we'll
tell you more. Next topic, emerging use of
malware invoking AI, termed living off the LLM, which is a
great theme. I'll start there.
We get a lot of questions about how AI is changing the game in
cyber attacks, reasonably so. Very trendy topic.
And while some people may argue that AI attacks are over hyped,
there's really a clear trend over the past year where we see
actors that are leveraging AI tooling found on workstations.
Some examples? Lame hug Amazon Q developer
extension compromise singularity dot dot dot amitai.
What is living off the LLM? I also really like that name.
I think we took it from a article from Oak Ridge National
Laboratory, I think. I don't know if they coined
this, but I think that's where I sort of saw this this recently.
And yeah, the idea is that you have like living off the land,
which is a term that's been around for a while where instead
of fed actors sort of bringing their own tooling into the
environment which it which has a higher likelihood of being
detected, they basically live off the land in the sense that
they use things that are already there.
So on all of our computers and in servers in in our company
networks, there is already tooling that they can use or
abuse for their own means for their own ends.
So the idea is that they have to be smart about how they do this.
And but it's things that are like already running things that
are already generating a lot of noise that they can hide in.
And now with the advent of people installing CL is for the
various AI services like like Claude and, and, and open AI and
Gemini. So with those things being
installed by people on their laptops, attackers are now
starting to take advantage of that.
And it's, it's really interesting because like like in
the examples you mentioned, basically attackers are are
building in functionality into the malware that they're
deploying on Victor machines to take advantage of those CL is if
they are present on the machine. In this case, the prompt becomes
the indicator of compromise, right?
So what kind of monitoring needs to be put in place to capture?
Be aware of that this is happening to detect it.
So what you're going to see in these cases is you're going to
see basically, let's say a script that the malware is using
when the attacker puts it on the machine once they gain access.
And that script is going to be invoking the LLM through the
AICLI and it's going to be prompting things like help me
find all of the sensitive data on this laptop.
Those prompts in and of themselves can be IOCSI want to
reference the project called Nova, developed by Thomas.
I hope I'm saying his name correctly, Thomas Rosia for
Microsoft. And he built this project that's
sort of a a way to sort of make these sort of IO CS actionable
by sort of scanning the logs of a ICLIS, for example.
And you can look for prompts that are known to be associated
with specific malware because the prompt itself might be like
a specific string. It might always be the exact
same sentence. Often times before doing that
sort of thing like helping find all of the sensitive thousands
machine, they first need to jailbreak the LLM.
And those jailbreak prompts are also sort of recognizable
because they often repeat themselves and they're often
phrased the exact same way. So you can use like similar to
scanning for hashes or scanning for certain strings that they
might indicate. So in the malware you can also
scan for specific prompts. We can ask you a naive question
that can I tell if something happened to my personal
workstation or just as an organization, can I do
monitoring? So that that's a good question.
I think if you are using NAICLII think you have logs in some
cases. However, in many cases those
logs will cost a bit more money or a lot more money depending on
how much you use them. So for this to be done at scale,
an organization does need to have that logging enabled in
their in their cloud account, for example.
However, you can do these scans locally like you can scan your
machine for the malware for the prompt like in the malware
binary, for example. So like there are two different
places you can do this scam. For the attackers that are
pursuing this, like why are they using this mechanism?
What does it give on? I think our leading theory right
now is that they're doing this for for stealth, for operational
security, just because this means that when you scan the
malware itself, it doesn't appear necessarily malicious
because again, it just contains a prompt, which is just works,
right? It contains a sentence and that
isn't something that Ed Rs or malware scanners are are looking
for yet at scale. But again, if we start doing
things like what Nova sort of allows you to do at scale, then
that might make this technique a bit more a bit more difficult to
implement. Interesting.
If you want to learn more about this, there's a wonderful blog
post by Scott Piper on malware invoking AI and highly
recommend. We will link it, check it out.
OK, to close off, we have a new section, Quick Hits.
It is to cover things you cannot do anything about, but we've
decided you should know about #1 the Crimson Collective threat
actor. This is great.
This is by, well, great, depending on who you are.
Just like the the F5 incident is fun.
It's it's great in the same sense that it was fun.
It is a group that appears to be related to Lopsys and Scattered
Spider, IE the Calm Teenagers, and they seem to target
companies only with red logos like Red Hat and Nintendo.
Now it's unclear if they only target red logo companies or
when they successfully hit a red logo company they claim it by
the Crimson Group. Do you think red is their
favorite color? How do you think they got to
this? I'm sure this is some like in
this is some inside joke. I mean it, it has to be.
I mean that I don't, I don't, I don't see what else it could be.
Maybe this is opposed to the end of the section, but what could
you do about this? Yeah.
So I think the only thing you could do is potentially change
your logo. I mean, if they're going after
red, red logo companies, then if you go to red logo, you know
what you need to do. Get creative.
So our second quick hit was a crazy bug in Inter ID discovered
by Dirk Chan and Malema, who's a great Azure researcher.
And this vulnerability was a cross tender vulnerability that
could have allowed any attacker with relatively minimal effort
to gain admin privileges on any Azure customer.
And Microsoft has fixed it. So you don't need to do anything
about it. I think he did write up a few
ideas on how you could potentially verify for yourself
whether you were exploited by this.
So do check out his blog post which we will link to.
And then tell us in the comments if you liked quick hits.
We could keep it running or we could kill it.
So let us know. And on that note, if you enjoyed
the show, be sure to subscribe and share a link to the podcast,
but not your cloud keys. And as always, if your cloud
security strategy is making you cry, don't worry.
Just cry out. Cloud security.
Security.