Welcome to Crying Out Cloud, the podcast that will make you
laugh, cry, and reconsider all of your cloud security fears.
I'm Eden and I'm here with my amazing Co host Amitai.
Hello. Amitai, what are we going to
talk about today? So today we're going to talk
about React to Shell, the vulnerability that lit the
Internet on fire, Shai Hulud, 2 Point O and a zero day in gogs.
And finally, we'll be talking about is a vulnerability
affecting Apache Tikka. OK, for the first one, RAC to
shell it is a vulnerability in React server components.
It's a software library for front end apps, but it
essentially runs React on the back end.
What was interesting about this is it went down as an embargo.
What happened exactly? So what happens with sort of big
bugs like this where the people who discover it or the vendor
whose products are affected, if they know that this is going to
cause a splash and if they know that this is something that's
going to be heavily exploited, or if they have a feeling it's
going to be, then they'll basically place an embargo on
the information and share it with a few select other
companies or researchers. Part of the goal there is to
make sure that vendors that use the software heavily have chance
to patch before the vulnerability becomes public.
Most big companies have have a pretty good way of doing going
about this and making sure that things stay under wraps.
Sometimes this stuff leaks ahead of time that has happened before
and then the embargo is essentially broken and they need
to to publish the vulnerability ahead of time, which isn't
great. The reason they do this in the
1st place is just because once you publish A vulnerability once
everybody knows about it, It's a race against time until it gets
exploited. So you want to give defenders
advantage? Yeah.
And this one specifically is big, right?
Like in this case, the embargo was important because it had
like a lot of impact, a lot of potentially or actually impacted
environments. So React itself is incredibly
popular. Many people use it.
React Server components is a bit rarer, but specifically it is
being used in something called Next JS, which is a very, very
popular library used by tons of websites.
Our data shows that a very large percentage of cloud environments
use Nix JS and for this vulnerability specifically,
around 50% of environments had this vulnerability somewhere in
their environment, even on whether it was internal or or
external resources. And around 10% actually had it
exposed from the outside, like actually validated as
exploitable, which is a very large number.
Yeah, huge. And who took advantage of this?
So basically everyone, a bunch of a bunch of vendors, including
ourselves, started putting out information about this
vulnerability saying, OK, we're starting to see exploitation,
We're starting to see PO CS popping up.
And then within a few days, I think Google and a few other
companies were already claiming that Chinese productors were
taking advantage of this and, and exploiting this immediately.
We saw usage of this for crypto mining.
We saw usage of this for all sorts of purposes, both cyber
criminal and and nation state. That's what happens when you
have a really good and really, really widespread vulnerability.
You're you're going to use it. It was funny because while the
real exploits were dropping, there were also like a lot of
fake proof of concept exploits going around, which creates a
bit like a fog of foggy situation.
But some of the AI ones, we're doing something.
I don't think it's funny. I think it's funny they were
leaving like helpful comments in the code which no actual
attacker would do. So it was clearly an AI POC.
Yeah, that was fun to see. Like in the beginning they were
they were completely fake, like they were just someone made a
best effort using CHA GPT, decided to publish it, decided
to scan the Internet with it. It didn't really work.
And within a few days, things started to look a bit more a bit
more interesting and a bit more more impactful.
But the ones that were impactful was interesting.
Is that a bunch of more file list?
Like they didn't leave a trace on the disk, which made them
hard to detect if you didn't have an agent.
Yeah, that was really interesting.
Generally, because of the way that node, that next JS is
working, this basically caused the situation where a lot of the
activity didn't really leave any trace on the disk, so you needed
to have some sort of runtime protection, some sort of EDR in
order to to be aware of what was going on.
There wasn't much logging, at least not by default.
We actually looked into what logs our customers should enable
to make it a bit easier to to track this stuff, but none of
the stuff is enabled by default. So what should people do in a
situation like this? Well, at this point it's very
easy to test if you are. If you are exposed to this, our
recommendation, as always, is to patch and focus on a publicly
exposed applications. Do it Our second story,
Shyhalood 2 Point O. We're already seeing a sequel
very fast. What is the first Shyhalood?
So Shyhalood was a worm their their creator named them after
named them after the worm from Dune.
Which is unfortunate because Dune is great and there are a
bunch of other things named after Dune.
So this causes confusion. We didn't choose the name.
Anyway, it was a worm that sort of burst its way through the NPM
get ecosystem, where basically you had a few initial NPM
packages that were compromised by a malicious actor, and then
they used that access in order to publish basically fake
malicious versions of those legitimate packages.
People downloaded them, and then that spread to their machines,
that spread to their CICD systems.
And from there the malware automatically searched the
machine for any credentials they could find for GitHub or NPM.
And then they used those credentials to basically recurse
through the ecosystem and take advantage of whatever access
those victims had to NPM in order to publish more versions
of the malicious library and so on and so forth.
And the first one was was horrible, and a lot of companies
were impacted by this and a lot of credentials were stolen this
way. And the second one was it was
also not great. Not great for sure.
I do think what's interesting about this, it's like a low
level of sophistication, but it's incredibly persistent.
So it was inevitable the second one would happen.
But it is interesting because from the first time around, the
attackers learned they got smarter, they made, they didn't
make the same mistakes twice, particularly around when they
couldn't find credentials on a machine.
The new version works in waves where it searches get up for any
credentials that it can pivot to.
Meaning like if they couldn't find on my machine credentials,
but they could find on AMI ties, they would just use AMI ties.
Doubled my thing. So this had a big scale, but
it's interesting how it got to a big scale.
Can you kind of talk about like how they were able to use the
right packages to get far? It seems like the recipe for a
successful Shankillude style campaign, and we don't actually
know if this was even the same attacker.
It might have been a completely different one, there might be
many, many, many attackers working, but it seems like the
the sort of the recipe here, like the the key to success is
you need to choose your patient zeros very well.
In this case, their patient zeros were super popular
packages that exists across tons and tons and tons of
environments that are downloaded like millions of times a week.
And the way they gained initial access to those to the
irrelevant maintainers was they took advantage of what we called
a phone request, which is a vulnerability that affects
GitHub actions that are basically misconfigured.
And it's relatively prevalent. And, and the the main thing is
that it's very easy to find. There's a lot of open source
tooling out there, like offensive security tooling that
basically anyone can run across any repository in the world.
And, you know, they just scam GitHub all the time.
The second they find that a type of on the bill like that, they
exploit it, they steal the keys. And then, you know, they they
take advantage of the of the fact that they now have the
ability to publish a version of a super popular package to just
trojanize that package and insert malware.
And in this case, there were around 10 packages owned by, I
think like two or three vendors that got popped in this case.
And that was enough to 'cause this to be to propagate through
the entire ecosystem really, really, really, really quickly.
The key here is the attacker needs some way to hijack super
prevalent packages and from there they can just go to town.
How can you protect yourself? So that's a good question.
If there's a sort of a shared responsibility model here that
we need to consider, there are things that NPM, GitHub, GitLab,
pipee, etcetera can do here to make this a lot more difficult
to happen in the 1st place. Like making sure that you can't
publish a version of your package to NPM without MFA, for
example. And that is something that that
NPM is starting to work on. And you could also, in terms of
the GitHub action, you could make the Pone request a thing of
the past by just making a lot more difficult to exploit.
That's also something that GitHub is is improving,
improving out a lot in terms of what you can do as an
organization to prevent yourself from being exposed to this sort
of thing. First of all, if you don't want
to become patient zero, you need to make sure that you are have a
super hardened release system where you never publish anything
unless you intend to, where you have review and control over
every, every time you you publish a a version to a version
10 PM, you want to make sure that you don't have any
credentials for your maintainers lying around anywhere.
There's a lot of really basic things you can do to make sure
that you are not the next patient zero to this in terms of
preventing yourself from being a victim here, from being like a
carrier of the worm. The easiest thing you can do is
just not download the latest version of any package like
always wait a few days or a week.
That's turned out to be super effective at preventing the
spread. It's basically like doing like
quarantining people during, during COVID, I guess, But just
instead of quarantining in space, you're quarantining in
time. Never meet anyone new unless
they've been around for more than a few days and you know
that they're not sick, I guess. And you know, you could use a
proxy within your organization to enforce this.
You can enforce it at the level of the tooling that you use to
pull packages from, from MPM and PIPI.
Lots of things you can do here. OK, story #3 Goggs. 0 day Goggs.
If you don't know, it's an open source alternative to GitHub.
And Amitai's team actually found, or maybe more precisely
stumbled into this zero day vulnerability.
What did they find? So we weren't looking for
vulnerabilities in the traditional senses, weren't like
we were. It wasn't like we were reviewing
the source code and trying to find vulnerabilities or anything
like that. The first thing that Gili and
Yala from my team saw when they were looking into this was
basically post exploitation activity on a machine that had
Gogs installed. And the machine seemed to be not
at risk, like there were no vulnerabilities being detected
on it or anything like that, just the malware running there.
And they started looking into this and tried to figure out
what the mower was doing. And then the next question,
which is an important question, is how did they get there?
And the answer is almost always either a vulnerability or a
misconfiguration. And in this case, they couldn't
find anything. So that's when they started to
look in a bit more deeper into the code itself, trying to
figure out what might have happened here.
And that is how they actually found that there is a
vulnerability affecting Gogs in its latest version, and that the
attacker had probably realized this a while ago and began
exploiting this. How widely was this exploited?
Like how big is the territory of compromise?
So once they realize that there was a vulnerability here, they
started looking for other cases where this might have happened.
We we looked at scan data to see where else in the world GOGS is
exposed. It seems to mostly be used in
Asia. And once we start looking into
this, we actually found that like 70 or 80% of all instances
of this technology that that are exposed in the world were
targeted by this attacker, like this attacker really went to
town. Who is the attacker?
Do we know? We don't know.
I mean, they're just an opportunistic cybercriminal
focusing on using compute to mine cryptocurrency.
That's pretty much all we know. And is it fixed?
Like do the maintainers take care of it?
So unfortunately not. We reported this a few months
ago. They're taking their time to fix
this. That's sort of why we decided to
go public with this. We waited a long time and then
we eventually decided, OK, enough's enough.
So TLDR, don't expose your bogs to the Internet.
Yeah, probably best practice anyway, but at the moment that's
really the only option you have because you can't patch this.
Our last story, Apache Tikka. Apache Tikka, because you can't
really tell from the name what it is, is APDF parsing library.
It is used under the hood by many apps.
People want to parse PDFs. So to explain how this works,
imagine you have a cool startup that lets people upload PDFs and
you do some sort of AI analysis for them.
Like you read their tax returns or something, but they have to
upload APDF. That's the the core thing.
If they use tika tika tika, an attacker can upload a malicious
PDF that would trigger an RC on the server where it's parsed and
then all hell breaks loose. I mean Ty and I were talking
about this before and I was talking about how it's really
hard to triage. Can you talk about why?
So any vulnerability that's really prevalent is
automatically hard to handle because you just have a lot of a
lot of cases that you need to prioritize between.
In this case, it's not like you can just say like I'm going to
prioritize publicly exposed machines that are running Tikka
or publish cases where I can, where I can see Tikka running
from the outside when using an authenticated scan or something
like that. Because Tikka is running behind
the scenes. And there's also not much you
can do here. And it's not like it's dependent
on how you're using Tikka. It's just like if you're
uploading the PDF, then then you're then you're at risk.
Like if you accept PDF from untrusted sources, you're going
to be at risk. And you know, you might have
Tikka on some like internal machine that's getting things
from a front end. So it makes it a bit a bit
difficult to to prioritize. Some of the stuff that we've
been recommending customers do is a, there are cases where
Tikka is literally exposing the server to the Internet.
It doesn't do that by default, but that might exist in your
organization. So that's definitely what you
should handle first, because that's the most at risk.
Second, you can, if you're able to do that, search for cases
where you can find Tikka installed in a machine that also
has a bunch of PDFs like. That might indicate that it's
being used for PDF parsing and actually in use.
And other than that, basically focus on cases where you can
validate in runtime that Tikka is actually running.
Because again, it comes under the hood in a lot of cases.
Like it's bundled along with a bunch of other software, but it
isn't necessarily always in use. So you should focus on cases
where you have at least some signal that it's actually being
used. Sounds like a solid plan.
OK what are the pot aways? Take a pods from this episode.
So react to shell patch. Not really a lot of options
here. You can use WAFF rules but
they've been proven as not infallible, so best to patch
Shivalu 2 point O make sure that if you are an organization that
is publishing to MPM, make sure that you're using MFA to do so.
And if you are consuming packages from MPM, you might
want to consider. Not consuming latest.
You might want to consider having some sort of delay on how
fast you're consuming packages because that will give you a
window in which other people might get infected and then
they'll warn you about it. GOG 0 day you can't patch
because there's no patch available.
So just don't expose gogs to the Internet.
Always best practice and Apache Tikka patch, but since it's
incredibly prevalent, you should probably focus on cases where
it's literally exposing the server to the Internet, or if
it's being used by an app that is being exposed to the
Internet. Or if you can find it on servers
in your environment that have a bunch of PDFs, because that
might indicate that they are actually being parsed by
Bachitika, which means that it might be accepting PDFs from
untrusted sources. Maybe not the ABS, but that
gives you some sort of signal to work with.
All right, well, if you enjoyed the show, be sure to subscribe
and share a link to the podcast, but not your cloud keys.
And as always, if your cloud security strategy is making you
cry, don't worry. Just cry out cloud.