Welcome to Crying Out Cloud, the podcast that will make you
laugh, cry, and reconsider all of your cloud security fears.
It's Enan and Amitai yo, and today we have you've all on to
break the news about a super cool vulnerability found.
Yep, Yep, definitely. Cool.
Yuval is a researcher on our vulnerability research team.
Previously he was a principal security researcher at Palo
Alto. He is a Microsoft MVR Top 10 as
a researcher. And yeah, he's our colleague and
he's super cool. So we're super happy to have him
on. Happy to win.
OK, you've all discovered a flaw in Amazon's own code build
projects. I'm not going to give too much
more details on, I'm going to let him tell us more himself.
But let's start off with how did you get the idea, the
inspiration to work on this research?
So the idea actually came from a threat actor right as I was
joining Wiz. This was a blog about an
attacker that managed to take over an AWS Gita repository.
So we saw that and we thought it was really peculiar.
I don't know, I mean, time, maybe you remember this attack.
It was kind of a weird one because the entire motivation of
the attacker was to just cause destruction.
He, he apparently didn't like the service or something like
that. And he just tried to, you know,
'cause he didn't try to like Krypto Mine or something like
that, just to cause destruction. And he, although he had a typo
in his payload which caused it to ultimately fail, he was
actually able to get his payload to run on like end user
computers. So it was a really interesting
attack and that really sparked our interest into AWS code build
the service because the entire attack was built by abusing that
service. So how did you find this?
Like what were you looking for? So when we looked at the
original attacks, we just try to understand how is this even
possible? How can you exploit code build
to take over the underlying GitHub repository?
And what we found out is that code build.
The way it works, when someone submits a pull request to the
GitHub repository, it will trigger a build based on the
pull request branch right? And once that happens that the
code of the build runs in an environment that has a
credentials to GitHub, like a GitHub token, because code bill
needs to do stuff like update the commit status if the build
failed or succeeded and turn It turns out that more often than
not, those bills are those. This token is quite privileged.
And we saw that in a lot of cases, it will actually let you
take over the entire repository. So we immediately, when we saw
this with, we went to look for how does AWS recommend to, you
know, not allow this kind of attack?
Because it's kind of crazy. And we saw that the main
mechanism back there was using something that's called an
actual ID filter, which basically is just a, you know, a
list of user IDs whose pull requests are allowed to trigger
builds. So we, we looked at it and while
we, it seemed like a, you know, a valid solution, we thought it
was like not the best user experience.
Like for each time someone joins your team, you need to figure
out his GitHub user ID and then update this, you know, list on
AWS. So we were kind of suspicious
and we wanted to check whether, you know, people are actually
implementing this correctly, right?
Like as a misconfiguration. Yep, definitely the if people
are actually configuring their code build correctly.
So we wrote like a script that looks at GitHub and tries to
find something that's called public code build projects.
So in code build you can actually set something that
makes your code build settings public just on a web page.
So that was extremely helpful for us because we could search
for those and actually see their configuration, whether they use
an actor ID filter. We ran, we found a bunch of AWS
repositories that use the actor ID filter.
So we're like OK, good for them. Like nice, they'll be actually
doing what their documentation says you need to do.
And we kind of left it at that and continue to other stuff
until like 1 evening. I just, I don't know why it
popped into my head and something about the syntax of
the list seemed a bit off Seemed weird.
You would expect expect it to be like a list of numbers separated
by like a comma or something like that, just a list.
But it actually was the separation was a pipe character
like I don't know if this is a good indication, just a pipe
character and and then it clicked to ask that this is
actually a regex. It's not the simple list and
it's it's well documented. We just didn't like who didn't
saw that it's actually a regex filter.
And the reason is because the actor ID filter is not the only
type of filter you could have like file system path filters
and other filters where regex makes sense and that's why it's
the underlying engine. But for a list of IDs it's it's
not really the best solution. And immediately what we thought
about is whether this list was encode.
What does that mean? That you put 2 characters at the
end and beginning of a list to ensure that what you are
matching is exactly the match. Otherwise regex would match even
if the string contains it. So let's say you're searching
for ABC with regex. If you don't put the encores
also 123ABC567 will also match right?
So immediately we thought, wow, if this is possible, we might be
able to bypass this filter. So what does that look like?
I mean, cuz it sounds like you'd need to have at least a partial
match, right? In order to make this work,
yeah. So how do you exploit that?
I I would say that Neil didn't thought it was exploitable.
Neil is our head of research. He didn't think it would work.
I, I immediately called him and I was like, wow, we we might be
able to bypass and he was like, no way this is going to work.
I was like, we're putting this on the record that Neil did not
believe this could work. I put this on record, but after
he saw that he was supportive at the end anyway.
So, so like you say, we want, we wanted to be able to understand
if we could even get like a GitHub user whose ID is a super
string of like an approved maintainer ID, right?
So that got us to look into, you know, how even GitHub IDs are
distributed and it turns out that they are just a sequential
like list. So the first GitHub user has ID
one, then like in 2015 IDs have like 4 digits, then in 220 they
put balloon to like 7 digits and today they are already at the 9
digits. What did you do next?
You figured this out and then. And then we had a lot of luck
because AWS actually employs really I would say early GitHub
users. So their ideas were kind of
relatively short. They had like 6 or 7 seven digit
ideas. So that meant that right now,
because they are already 9 digit ideas, we could actually find
that ID that is like a super string like contains the idea of
an approved the maintainer. Oh, interesting.
Yeah, yeah. So how did you action this?
What did you do? So we we needed to claim an ID
that actually contains the the smaller ID like the approved ID.
We know that those ideas are out there.
The thing is that GitHub creates like 200,000 users every day,
right? So it's kind of how to get a
specific ID because a lot of users are being created all the
time, right? So we needed a way to like
automatically create a lot of users in an like in a second so
we can catch the idea that we want, right?
We had the target idea that we want to catch and we wanted to,
when we see that this is the idea is about to come up, it's
about to be created. We wanted to just create a flood
of new users and we found a way to do that using a GitHub apps.
It turns out that when you register an app, it also creates
like a bot user. So when we did that, we're
actually able to create a lot of GitHub apps very quickly, and
that allowed us to create a flood of new users and actually
claim an ID that was a super string of an approved maintainer
ID. Could you use this to like for
other reasons? Like let's say you really want a
specific ID number like a vanity license plate.
Like could that could the same technique be about?
I love Eden, yeah. You could, you could definitely
do something like that. I don't know if Gitab will patch
it soon, but as of right now you can do that, yeah.
You should buy them while they're hot, while they're
still, while it's still possible.
Yeah. OK, So tell us about, let's call
it D-Day, the day that you you ran the script to actually to do
this in the flesh. What happened?
What did you have for breakfast? How'd you run the script?
How long did it take? Yeah, so we actually, there were
two D days. We were like preparing
everything and then it was already like 8 in the evening
and we near we looked each other in the eyes and we're like, OK,
we're not let's not do an all nighter.
Let's just go to sleep and do it fresh in the morning.
So we came fresh in the morning and we we chose a target.
The target was the AWSSDK for JavaScript.
It was one of the, you know, affected the repositories that
had this vulnerable actor ID filter.
So we we created a demo environment first, but when we
actually did it, it was definitely nerve racking and it
actually took much more time than we expected because they
use like a huge machine for their build with like, I don't
know, 70 CPUs and huge memory. So everything was so slow, which
really added to the to the pressure, I would say.
But after we submitted the PR from the user that we created
that had the correct ID that could bypass the filter, it
worked. We got the GitHub token from the
build environment. We checked its permission.
It was quite a powerful user. It was basically, it was
basically an admin of the repository.
And that was, you know, we, we expected it to happen, but it
was still definitely quite shocking.
So what could an attacker have done with this?
Like if, if, if the evil Yuval somewhere in Russia, let's say,
would have would have brought this to his boss, the evil near
and decided to pursue this. What?
What could they have done with? This a lot actually because this
specifically this repository. We chose it because we thought
it really demonstrated the impact because it is present in
our based on our analysis in like 2/3 of cloud environments,
it is heavily used this SDK and one of its user is actually the
AWS console. So this code in this repository
runs inside database console, which is basically the brain of
like the entire AWS cloud. Every user uses it.
So if you manage to use this admin privileges of the
repository to push, you know, malicious code that could
potentially end up in like 1,000,000 of AWS consoles, which
is quite scary. And it's not, you know, we
obviously decided not to do that, not to do like a POC of
that because we were like, we didn't want to touch something
that. Because you're the good you are.
You're not. You're not the evil you are.
Yeah, definitely. But it's not like only a
theoretical attack because just a month before we saw like the
the attack that we discussed at the beginning that someone
actually did that and managed to get his payload to end users.
So we know this is actually something that, you know, it
could work. So we're really happy that we we
found it in time and we were able to get it this close to
AWS. But let's say right now, first
of all, it's super cool, but let's say right now you're an
AWS customer, which likely you are.
Do you need to be concerned about this?
Like AWS is using their SDK also internally.
So what's the impact on any given AWS user?
It could have been massive, but thankfully AWS patched it.
It's no longer an issue so right now you shouldn't be concerned.
I I'd say you could like if you are using AWS code build, there
are definitely lessons here for you as well because AWS series
basically a user of AWS code build with, you know a big
misconfiguration or flaw. So the misconfiguration is still
possible, like someone else could make the same mistake
again tomorrow. Yep, Yep, definitely.
But it's AWS released actually some features now that make it
easier to like make sure only approve maintenance are able to
create the builds. So there are actually new
improvements that AWS introduced as a response to that original
attack that allow you to not be able.
You don't need to maintain a list anymore, right?
You just you just need to enable it.
And AWS behind the scenes checks the permission of the user that
created the PR and if he has an appropriate permission like, you
can see that he's the owner of the repository.
Only then a build will be triggered.
But The thing is that this is only for a new code build
projects. This is only enabled by default
for new code build projects. And so if you have an existing
1, you should really like. It's also the biggest
recommendation in the blog post. You should definitely enable
this feature, it's super important.
What would be the equivalent here in in other platforms?
Similar to code build? Like?
Is it likely that their customers have of other of other
competing platforms might need to be worried about this sort of
thing as well? I think it's relevant in
anywhere that you have like an external CI system to where your
source code is. So because a lot of time that CI
system needs to be able to interact back to the source
depository. So normally GitHub, but it could
be GitLab would be bucket or any of those platforms and it needs
to like report it. It needs a token to report back
what happens in the build, and it's the job of the underlying
CI system to make sure that that token, if possible, isn't
exposed to build and that by default it doesn't have a lot of
permission. A few months ago there was
another issue in a cold rabbit, I think that was pretty severe
that had something to do similar to that where the credentials
were in the build. So it's definitely important to
make sure the permit, the token, the privileges that you give
your CI system over the source code are limited.
Yeah, we. Worked closely with Amazon
around the disclosure, obviously.
What was that like? Oh, they were great.
They actually fixed it I think in less than.
I don't know if it was 24 or 48 hours, but they immediately
disabled the projects and then introduced the fixes.
And also they made this, you know, amazing feature, which is
a much, much easier to use. So everyone can actually benefit
from a better, you know, better configuration for COVID.
OK. Anything else you want to impart
to people about this vulnerability?
Anything else they should know? I think the really the lesson is
that you should never have built with privileged permissions that
can be triggered from external pull request.
This is like the first. There are a lot of improvements
that you can make to your CICD system defenses, but this is
like #1 you shouldn't allow external users to be able to
trigger bills in your environment.
That's like the most important thing and the most important
lesson. We should print that with our
title. Yuval, Congrats.
This is super cool and thank you for explaining it to people.
Even I understood it, which means I think everyone can
understand it. Yeah, have a good one.
If you enjoy the show, be sure to subscribe and share a link to
the podcast, but not your cloud keys.
And as always, if your cloud security strategy is making you
cry, don't worry. Just go out.