Welcome to the Decipher Podcast. I'm Dennis Fisher.
I'm here with Lindsay O'Donnell Welch, and we're going to talk a
little bit about this kind of crazy F5 story that emerged this
week. Linds, you know, we've both been
doing this a long time and we've seen a lot of these kind of
incidents, you know, intrusions at tech companies have become
more and more common, especially from, you know, well resourced,
state backed actors who are looking for sensitive
information, you know, product information, whatever they can
get their hands on. But this one seems to have a
whole bunch of different layers and different kind of
storylines. And it's only been about 24
hours as we're recording this since the news came out.
But as I was just reading the initial reports from F5
yesterday in the middle of the day, looking at it, we've both
read enough of these statements and public reports to kind of
read between the lines and you can see what they're saying and
what they're not saying. You know the the public
statements are that an unnamed government backed actor had long
term is the phrase they used access to F5 networks.
Yep, long term persistent access.
Right, that's the, that's the absolute basics.
They all, they said that the attackers were able to get some
product information. They also had access to some
information about undisclosed vulnerabilities and some limited
customer information and financial information, I
believe. So, you know, kind of a of grab
bag of really sensitive information, but there's a lot
that's unsaid in the the statement as well.
They don't say how long the attackers were in there.
They don't attribute this attack to any, they don't even
attribute it to a country or anything like that.
It's just sort of the, the bare bones of it.
And then, you know, it kind of goes into the the customer
impact. But this is one of the more
thorny and potentially really damaging ones that I think we've
seen in a while. Yeah.
And you know, just taking a step back, looking at F5 big IP, as
soon as I saw that in the news, I was kind of like, OK, this is.
And I was just telling you, this is a product that has come up
like, you know, a handful of times every year because it's
such a popular type of appliance that threat actors have been
targeting. And if you look at the, the
platform, the product itself, it makes sense, right?
I mean, if there's, it's on the edge of the network, it, there's
kind of, if you get a foothold into that, it's easy access to
then launch out lateral movement or some of these other vectors.
But then also looking at F5 like the customer base I'd imagine is
pretty valuable as well. For threat.
As if they want to target that. So it's, you know, just the
reasons why, you know, the threat actors, especially nation
state actors might be interested in this.
I think is important to highlight here.
The other thing too is, you know, you mentioned this kind of
high level bare bones that was released yesterday.
Since then, you know, there's been some news reports that have
come out, I think from Bloomberg.
It was basically giving a little bit more information into, you
know, what the scope and scale of this was, which, you know, it
seems as though reports have linked the attack to China and
then also that threat actors have actually been inside F5
network for 12 months, about a year.
So just. That's a long time.
That's a long time, yeah. If you really think about it, a
year is of of access into your network.
Just think of all the things that happened and all the
potentials. It's just hard to even know the
true extent of this incident, you know?
It really is. I, I can't really recall any
publicly known incident where an attacker was inside of an A
victim network for a year, 12 months.
I mean, even going back to like the Aurora attacks that hit
Google and a whole bunch of other companies back in like
2010 or 11, whatever year that was, they had long term access,
but it wasn't anywhere close to that long.
You know, this is the kind of this sounds like, you know,
whether these whether this was opportunistic.
You know, the attackers had a bug that they were using against
a bunch of potential targets and happened to get lucky and hit F5
with this. You know, with this, found a
vulnerable instance of whatever and got in there and then, you
know, may have sold that access to somebody or used it
themselves, who knows. Or whether this was a targeted
attack and they were specifically looking for, you
know, looking to get into F5's networks and then see what they
could get from there. Nobody knows.
But but the state back state backed actor targeted seems more
likely. And there were a couple of sort
of weird things or odd phrasing in the F5 statement.
I'm just going to read like 1 portion of it from the What We
Know section. And this came out on October
15th, which was the day everything became public.
So it says we have confirmed that the threat actor
exfiltrated files from our Big IP product development
environment in engineering knowledge management platforms.
These files contain some of our Big IP source code and
information about undisclosed vulnerabilities we were working
on in Big IP. OK, next sentence.
We have no knowledge of undisclosed critical or remote
code vulnerabilities and we are not aware of active exploitation
of any undisclosed F5 vulnerabilities.
So those two statements seem to contradict each other.
I'm not. The only way that I can sort of
make them agree is if the attackers had access to this
vulnerability database or whatever it was for X number of
months, and now all of those vulnerabilities have become
public. So everything that the attackers
had access to is now public through other means.
You know, F5 has disclosed them and patched them or they've
become public in other ways, but that that's not really the way
that it's worded. So it's a little difficult to
kind of parse all of that, but that that's the only way that I
can parse it that makes any real sense to me.
Yeah, that wording was weird. I think it's probably like
legalese, you know, some sort of lawyer speak that probably got
their their hands on that. To your point, though, I know
they did also, they pushed out a security update and I guess it
was like pretty big and it included some high severity
flaws and then also some denial of service bugs that could be
remotely exploited without authentication.
But like what, what worries me too about this is, you know,
reading between the lines again. I mean, if threat actors got
access to their source code, you know, looking at actual 0 days
in the sense of, you know, the fact that threat actors now can
find their own bugs and vulnerabilities that even F5
doesn't know about. And F5 may have disclosed all
this. But it just makes me nervous
that they can, they'll be able to study, you know, these things
and really be able to get to kind of those root issues and
those root weaknesses. And I don't think that's
something that, I mean, they mentioned pen testing or
whatever in the advisory, but like, I don't know if that was
something that F5 has, you know, given me enough confidence in
how they have talked about. They've addressed that.
The other thing too that I, that kind of came to mind was like,
yes, they push out patches, but like if threat actors are able
to have this level of access and able to really study, you know,
this, you know, to this level like the source code and
everything. Like, you know, I'd imagine if
there was any sort of like incomplete patch or like, you
know, something like just the ability for them to really skirt
around that and, you know, have that level of knowledge, it just
makes me nervous about what's going on behind the scenes.
Yeah, there's a, there's a bunch of different really concerning
things there. If they, they got the source
codes, there's all those problems that you mentioned.
They did publish a couple of, they said that they brought in
IO Active and NCC group to do, you know, kind of not the
incident response work that you know, I guess they also brought
in Mandiant and Crowdstrike. That's the kind of thing that
they were doing. I think NCC and IO Active
specifically were looking at the code repositories and things
along those lines to see if there were any modifications.
I read the IO Active letter that said, you know, in the scope of
the work that they looked at, they could, they didn't find any
modifications to 2F5's code base.
But that doesn't, as you said, Lance, it doesn't take into
account the fact that the attackers can now just look at
whatever they took. We don't know how much source
code they took or what it was, but yeah, they can just kind of
pour over that and get the lay of the land, see how see how
everything works. You know, it's, it's yeah, it's
really tough. Yeah, yeah, yeah.
The, and the other thing too I saw that was interesting was
the, there were some reports that were linking this intrusion
to a separate recent report by Google, Google Mandiant, which
basically has some threat Intel parallels and talked about kind
of Chinese espionage groups that were targeting, you know, tech
companies to grab their source code data and analyze it for 0
days. And you know, I'm sure you've
read that one, but it was the one that involved the malware,
the Brickstorm malware. So that was a, you know, another
kind of interesting piece of this as well.
Just kind of. Looking at the parallels there.
I think, you know, if you, if you had to bet your mortgage on
who the attackers were, I think you, I think we'd both bet on a
China backed group. You know, I don't think that
Google has a name for them in that report that you mentioned.
It's one of the UNC groups, I believe one of the groups, not
University of North Carolina, but yeah.
But the, the link to the Brickstorm malware is
interesting too. They don't, there wasn't any
mention in the Mandy or excuse me, the F5 reports from this
week about any specific malware that they found in their
environment or anything like that.
Just judging by the way that these things have gone in the
past, I would expect some sort of public post mortem report
from either Mandy and or crowd strike on this.
You know, it'll, it'll be limited because of the way these
things work and Ndas and all of that.
But if there's some new malware strain or some use of unknown
malware strain like Brickstorm, those are the kind of things
that usually get publicly reported in these cases.
And we haven't mentioned there's also SISA put out an emergency
directive on Thursday or excuse me on Wednesday as well.
You know, with the the usual language of that, you know,
these things apply to all the federal civilian executive
branch agencies. So basically if you've got an
EF5-GO, look at it, see if it's vulnerable, if it is, you know,
pull it off the network, all of that kind of stuff.
And there's, there's usually a 48 hour, 72 hour deadline for
these things. I didn't, I don't recall exactly
what it is for this one, but that gives you a good idea of,
you know, obviously SISA knew about this long before it was
public. That's the way that these things
are coordinated. So I assume that they've been
scrambling around on any other government networks as well, you
know, the DoD and other networks trying to get those in shape.
But at the same time, SISA is being systematically gutted.
So I don't really know how much manpower they have right now to
to get to do this, to respond to something like this.
You know, it's one thing to put out these EDS and, you know,
public advisories, but it's, it's another thing like one of
their functions is to go and help agencies respond to things
like this. I don't know who's left to do
that, you know? Yeah.
And I don't even know who at other agencies themselves would
be leading the charge on this. Great point.
If they're if they're there as well, I know you know, that's,
that's definitely a good big picture thing to be thinking
about right now. I I thought they saw that they
had given them till like Halloween, which I thought was.
Oh, OK, Long, yeah. Now that you're talking about
this, it actually makes sense because they're probably trying
to like, just like with the limited resources and manpower
that they have in place now, it's like.
Yeah, there's probably like 4 people that have to go around
and do all this for every civilian agency.
Yeah, I mean, that's a whole other story.
But it's, it's connected in that, you know, it's a limited
pool of resources to do this stuff, you know?
Right. But well, the you know, that
brings to mind though the timeline here is interesting
too. And the reason I bring that up
is because I, I don't, I'd imagine there's been some behind
the scenes like preparation from a government perspective, if the
government is known about this. And it has because as you
mentioned, they became, they F5 said they learned of this in
August, August 8th or something. And but they only, you know, had
the public SEC filing disclosed this past week.
But part of that is because the that you were mentioning the DOJ
gave them an extension. So behind the scenes, you know,
there has been some coordination that's not leaving federal
agencies scrambling during a government shutdown.
Yeah, it which if you go and read the SEC filing from F5,
which is linked to in our story, you can see that DOJ, as you
mentioned, I think it's September 12th is the date if I
recall correctly, that DOJ said, yes, you can delay this
basically out of the in the interest of national security
because F5's products are everywhere, you know, including
government agencies, as we mentioned, but they're in a ton
of enterprises and cloud providers and everywhere else.
So it's one of those situations where it you know, there the
longest running argument in the security community about when to
disclose bugs and how to disclose them and all that kind
of shit. But in these kind of cases, you
look at it completely legitimate request on F fives part and to
my mind completely legitimate answer from DOJ saying yes, in
this case we should delay this so that everybody who can has a
chance to go in, you know, make whatever changes they can in the
interim. Right.
Yeah, that's that's a good point.
And the other thing about timeline that stuck out to me,
at least when I saw that report of that, you know, the threat
actors had had access to the environment for a year
essentially is, you know what, what was the tip off in August
that eventually led to them discovering this?
I mean, was it an external company or like did they figure
it out themselves? And I don't know, like I feel
like a year like there's granted it's Chinese threat actors, like
they're pretty stealthy, I would say.
Professional had a lot. With like, you know, the malware
and IO CS that Mandy and has disclosed over time.
But like there's kind of, you know, there's there's always
bread crumbs, like there's always clues that point to these
types of intrusions. Like, yeah, yeah, it's 12
months. Like that just always is.
That's just crazy to me. It is and it's, you know, my
sort of educated guess, but not, you know, specifically to this
is in a lot of cases what happens is there's a separate
incident that seems completely unconnected.
And after the details of that become public, either an
internal team or an external team goes and looks for
indications of something similar, whether it was the
exploitation of a specific bug or the use of specific malware
like like Brickstorm. You know, somebody might have
picked up that hunting guide and gone and looked and, you know,
whether it was an internal team at F5 or somebody external was
like, oh shit, this looks like we might have an issue here.
And then they sort of work back from there and you know, see
where the attackers had access in it.
But, you know, that's, that's one scenario that could be a
bunch of others. But in a lot of cases, that's
what you see with especially any kind of widespread
vulnerability. You know, if it's some kind of
like really widespread Windows bug or, you know, anything like
that. And once one incident or
intrusion becomes public, then everybody else starts looking
around for that. They're like, oh God, yeah, you
know, so I don't know. But that that's the most likely
scenario I can think of. Yeah, I think you're right.
That's how a lot of those types of like discoveries are made.
Yeah, I think. And the other thing too is like,
you know, there's a couple of things that we still don't know
at this point. Initial access is one of them.
And so, you know, a lot of there's a couple of other pieces
as well. But it it'll be, you know,
interesting to see like how moving forward, if we see more
of these, you know, facts or, you know.
Findings come out through, you know, a Mandiant disclosure or
something like that or how you know how F5 is going to kind of
move forward in terms of revealing information about
this. Yeah, I would assume they're
going to continue to update what they can, you know in legally
and and you know with the other constraints that they're going
to have. But you know, in terms of the,
the customer impacted, as you said, Lance, they, they released
all these patches yesterday and you know, essentially said go
update, which is, you know, all the guidance there kind of is
right now. And they're, oh, we should
mention we, we didn't, it's my fault we didn't mention this.
But another thing that F5 disclosed yesterday is that they
rotated a bunch of their code signing certificates and keys as
a result of this. They didn't say specifically
what, but they did say that they listed a few big IP and other
products and said if you haven't updated, you know, as a result
of this key and certificate rotation, some of these updates
might not take because, you know, if you haven't gotten
there yet. So that sounds like another sort
of artifact of this whole thing, you know, but that's, you know,
it's just one other piece of it that's there's nothing customers
can do about that, but just more kind of fallout.
Yeah, more fallout, yeah. I know, yeah.
Like we need more of that. Yeah, no, we don't.
But yeah, it it does seem like this is going to be one of those
that we're going to be hearing bits as bits and pieces about
for the next few days and weeks. I would imagine, you know,
something else will come along and grab everybody's attention
and then, you know, four weeks from now we'll see an update and
be like, Oh yeah, yeah, I remember that.
That was like 3 incidents ago. Looking at you like Avanti.
Sonicwall somebody. Yeah, I I saw her friend Matt
Johansson yesterday, You know, somewhere, I don't know if it
was Twitter or wherever was saying, you know, everybody has
breached fatigue at this point. And it's just kind of one of
those it professional hazards in this industry where you're like,
OK, another thing, another thing, another thing.
And it's hard to sometimes stop and take a look and be like,
well, hold on. One of these things is not like
the other. This is a big, this isn't just
an attacker got in and stole some, you know, financial
records from a manufacturer. This is a big thing.
So it we do sometimes, you know, your eyes glaze over at some
point, you're like, OK, here we go again.
But yeah, this is, I think this is a a little different than
most of the other ones we usually see.
Yeah, I think that's that is a really interesting topic.
We should do another podcast on sometime though.
It's just like the level of breach fatigue and like the
issues with communication and the security industry about
like, you know, how high, how much of A priority one thing is
versus another. Because it is like, and you
know, I think the the news honestly has has a big role and
responsibility in this is there's just so many headlines
over and over about, you know, there was this actively
exploited vulnerability, There was this zero day.
But like what actually separates something like this type of
incident from something from, you know, that we saw even
earlier this week, we saw the Oracle extortion attacks, like
continual fallout from that. So like looking at two different
incidents, like what the impact is for customers, how long term
is this going to be in terms of the effect?
You know, all these questions, I think, you know, it's there's
just a lot of a lot of different pieces that go into it.
Yeah. I mean, we, we both know people
that do incident response and all of that kind of stuff.
And those people do often have that kind of like 1000 yard
stare where they've seen so many things.
There's, you know, they're immune to those the same way
that, you know, like EMTs have a certain look where they're like
they've seen everything at some point.
But if you're a customer or, you know, an internal security team,
you have to treat every one of these things as its own, you
know, discreet animal and be like, OK, let's go and look at
this on its own. And you can't, you can't afford
to let that fatigue really pile up on you, because then you
can't really do the job that you're supposed to be doing, you
know? Yeah, that's true.
Makes like really difficult it. Does yeah.
And also even looking at the the advisory to like just the the F5
advisory, for instance, like, you know, using these same types
of languages like long term persistent access, like I know
this is never going to happen. I know this is just me being
optimistic, but like if you're telling your customers that it
was 12 months, just come out and say, say it was 12 months.
Like we're going to find that anyway.
Yeah. Yeah.
It's it's just like being able to better communicate so that we
can really fully understand the impact of some of these things.
Yeah, there are those coded phrases.
It's, you know, long term persistent is something that you
would look at. OK, That's, you know, people
generally won't put long term in a statement unless they
absolutely have to. You know, they'll usually say
limited or, you know, whatever it is, they'll they'll use some
parameters to make it sound squishier than it than it really
is. And, you know, obviously this
doesn't sound squishy at all. It sounds terrible.
So when I first saw that, I was like, yeah, this is, you know,
that's one of the phrases you kind of stop on.
So, yeah, it's again, I, I think we're, we have certainly not
seen the the end of the disclosures for this so.
No, definitely not. Yeah.
All right. Well, we'll be definitely
updating the piece that we wrote on this, and I have a feeling
we'll be mentioning this on future podcasts as well.
So yeah, stay tuned. Yeah.
All right. All right.
Thanks, Lynns. Good to see you.
Thanks. Bye.