Hello, and welcome to the Decipher Podcast.
I'm Dennis Fisher and I have two guests with me today.
Really happy to have two of the three founders of Empirical
Security with me, Jay Jacobs and Michael Reitman.
Guys, thanks so much for being here.
Thanks for having us. Yeah.
Thank you. Yeah, absolutely sad that we
didn't have Ed with us, but I wanted to, you know, he's
probably just sad about the Cubs anyway, so that's fine too.
Much empirical for anyone podcast.
It's yeah, too many big brains. Yeah.
So I mentioned, you know, I want
to talk a little bit about I've known both of you guys for a
little while and I think Jay, I've known you for longer, but
you both have this kind of background in like the kind of
the data-driven side of security, which you know, is a
really fascinating part of it to me, even though I have
absolutely no mind for math whatsoever, much to my parents
chagrin. But tell me where the idea for
Empirical itself came from and and what you guys were were
hoping to accomplish and like how you were hoping to help
enterprises when you started? So Jay and I have been working
together around this problem for almost 15 years now.
I think we first worked together in a data-driven capacity and a
chapter in the DBIR based on kind of security data when I was
Chief Data Scientist in kind of security and Jay was at the
DBIR. We second worked on this when
Santia did an engagement with kind of security to do the
prioritization of prediction reports.
They did nine of them reviewing our internal data in the
aggregate to try to make some statements about vulnerability
management as a whole. And that I think has a life of
its own. I think some of those findings
are still relevant today. And then along the way, we
started working on the exploit prediction scoring system in
2019. We presented kind of the
scaffolding for it in version one at Blackhat and Jay has
taken that to new heights. At first, the idea for Empirical
essentially came from working with vulnerability management
data interfacing with a lot of large organizations.
Kind of security was acquired by Cisco in 2021 and I was exposed
to a whole bunch of those. And every single time we talked
to a sufficiently large or complex or mature organization,
the first pushback would be like, this model is great in the
non targeted attack sense, but I am a special stuff.
Like I have special data, I have a special environment, my
threats are different. This doesn't capture my reality.
And you know, with that hat on, my response was this is the best
we've got. This is the model that is
data-driven and others really aren't at the time.
I think in the back of my head, and I started talking to Jay
about this probably two years ago, was what if we did build
individual models per customer? What if we did incorporate that
data about that customer and build a model specific to them?
What does that world look like? And Empirical has really been us
trying to realize that journey along the way.
We got Ed Bellis to take a look at it and we're going to have
him invest in the company. And he was like, no, that's,
that's a bit too good of an idea.
Why don't I join you guys? As if he didn't have enough to
do. Yeah.
I mean, it's, it's hard to remember now, at least for me,
but there was a time where kind of this data-driven approach to
security like almost didn't exist.
Like it was just, you know, and it wasn't, I mean, it's long ago
in like security community terms, but it's not that long
ago in absolute time terms. You know, like even just 15
years ago, like this was seen as like this sort of like out there
idea of like, what are you going to do with data?
Who knows? Like how are you going to apply
data to like bad guys, you know, trying to get into my network?
What is that going to tell you? You know, we, we still run into
that. Yeah.
Oh, I'm sure. Mortality is absolutely still
out there. Yeah, yeah.
I believe it like do, but do you run into that at like what level
of the organization is, is, where does that resistance?
I, I, I don't think it's a level.
I think it's just something about some people have a very
skeptical bones in their body about data-driven security
stuff. And so I've seen it at every
single level and I've been surprised some people at every
single level are like, this is the greatest thing ever and
really want to get into the math and stuff.
And so it's, it depends. I don't think it's a specific
role or anything. It's just a mentality for some
people. Yeah, I think that's right
because, you know, especially people I think of maybe our
generation J, like people got into security 'cause they like
tech stuff, you know, they like breaking things and they liked
that sort of like, you know, back and forth type thing.
Not because they like graphs and math, you know, certainly not
what they were in for. And you would still see that,
see that sometimes it like at conferences, you know, is it
something like Black Hat that you mentioned, Michael?
It, you know, it was all the guys like trying to break stuff.
And then there'd be like a group of like 12 of you guys over in
the corner, like, you know, in like the data data world talking
about like actual results and like, look, we can prove that
like this stuff works, you know? Yeah.
I think both are really necessary parts of the security
ecosystem, but security's been growing.
I think that part's missing from a lot of discussions about this
that 15 years ago security, some parts of security were a human
scale, let's investigate type of problem.
I think for about 5-10 years, it hasn't been that the volume has
been such that you need to do statistical analysis and try to
be more efficient with what you're working on.
But once you figure out what's efficient, it's still a pretty
manual effort of investigation and figuring out what to do and
actually doing the remediations or mitigations.
I like to think about just how firefighters rarely take a step
back and think about like the statistical incidents of fires
across the city because they're busy responding to the
day-to-day, and I think security has been in that mindset for
some time. Taking that step back is
actually quite a luxury. Yeah, that's a, that's a good
way of putting it. Yeah, I like that.
And it's so you mentioned the EPSS model, which I think is one
of the coolest things that's kind of developed over the the
last few years, which is the exploit prediction scoring
system. OK, Yeah.
So essentially like I'll let you guys describe like how it all
works, but it's the idea is to like it what it says on the
label, like tell you how likely this bug is to be exploited
taking into account a whole bunch of factors and parameters.
But Jay, where did that? And I know it's like from the
first group, the incidents incident response group.
And yeah, so I want to go back to the beginning and, and
Michael sort of glossed over a little bit.
But when we started, when I, I started Santi with Wade Baker
and we were contracting with Kenna and a bunch of other
companies and Kenna had just wonderful data about the
vulnerabilities in existence, the remediation, a bunch of
extra data about vulnerabilities that you just don't find
typically. And then we had another
customer, we're working with Fortinet and we're producing
Fortinet's threat landscape report.
And of course, Fortinet has, you know, I think over 100,000
devices deployed globally and they're all reporting the, the
activity that they're seeing centrally.
And so Fortinet had this wonderful amount of information
about all this activity going on, all these CV ES being
attempted to be exploited, blocked, things like that.
And I was looking at the exploit activity and looking at
Kenneth's data and kind of had some exploit activity, but we're
looking at Fortinet. Like if I bring these together,
we could we could model this like we've got outcome data.
The things that we want to stop is this exploit and then all the
stuff about the vulnerabilities from Canada.
And so bringing those two together was basically the birth
of VPSS. So OK.
That's no small thing, because if you really think about it,
security is two types of data. In general, it's, I looked at a
system and here's the state of that system, which is what a
vulnerability is. It's like this exists in maybe
production, maybe not production on a particular device or
machine or database. And then there's the telemetry
data, which is in real time. These things are happening.
Historically, they've been to sometimes even different groups
in an organization. And I think what Jay did is
essentially cross that divide of data engineering more than
anything to ask the question, well, what what if we looked at
all security data, what could we answer about an organization's
posture in real time? So if you're looking at a given
vulnerability, how does the EPSS like?
What is it taking into account to come up with a a
exploitability prediction score for that bug it?
Takes in as much as I can everything and anything we can
get our hands on. But for the most part, I think a
lot of the strong signals right now we've got about 2800
different variables feeding into the model.
And I think probably 900 to 1000 are actually feeding and helping
the the scores and stuff like that.
So the extra is just sitting there.
But anyway, a lot of it is like one of the key things we'll put
in the vendor of the product and that vendor holds so much
information. Like how prevalent is that
vendor? You know, if you think of, you
see a vulnerability and you're like, and then you see how it's
a Microsoft on, you're like, oh, wait a minute, what was that
vulnerability? You know, like you get these
little tip offs and the same thing with the model.
And so we also do a ton of scraping for exploit code being
published. Those are usually pretty good
indicators. Yeah.
I mean, just everything and anything different chatter.
We try to pull texts of people talking about these
vulnerabilities and then mine that text for keywords and
phrases and all sorts of things that people are talking about.
We try to get at the characteristics of the
vulnerability. So things like CVSSI guess, but
we try to get a little bit more granular than that.
So we try to get more detail about, you know, if you know,
like as a sequel injection and what kind and all the details
that we can get. And it's all just fed into this,
you know, meat grinder of a model that learns all the
patterns. And so when EPSS is rating
something high, it actually EPSS, when we're scoring stuff
daily, it doesn't know about recent exploit activity.
So it's not actually factored into the model.
And so when the model scores something high, it says, hey,
looking at all the attributes to this vulnerability, it looks
like it shares a lot of the attributes with things that
we've seen exploited. And so it's saying like this has
the the markings of things that should be exploited.
And that's what the model is producing that that's sort of,
it looks like exploited stuff. You didn't use the key operative
word, which is prediction, right?
I think that's the difference between EPS and every other
scoring system that came before it.
It's that it is not describing the current state of the
vulnerability, although of course that data is factored in.
It is making a prediction about is it likely to be exploited or
not. Yeah.
And so if if I'm an organization, let's say I'm, you
know, an insurance company and I'm looking at this, you know,
to try and grade vulnerabilities, is it taking
into account or do I have to sort of make these calculations
myself? Like the industry that I'm in,
you know, like how likely it is that, you know, insurance
companies will be attacked with this type of bug, that kind of
thing is, is that kind of context there?
It's, it's not there that that kind of breakdown because all we
do like when we publish EPSS, it's literally just ACV and a
score like that is that is the richness of our data there.
And we've been doing that since we've been publishing since
2021. And so everyday just CV and a
score. So we don't break anything out
by industry or anything like that.
We try to keep it global, just, you know, that initial sort of
global indicator. Yeah, and and the the output is
an actual probability and we actually calibrate that
probability. So when we say, you know, like
2% seems like a very low probability, but it's a 2%
chance that it will be exploited to one out of 50
vulnerabilities. At 2% we expect to see
exploitation activity, you know, and so that's that's a key
thing. If we talk about thresholds, I'm
going to bring back probability into that conversation.
But yeah, we output a probability.
OK, and I, I think I mentioned that earlier this, this Linux
bug that came out this week that, you know, everybody's kind
of freaking out about it's a local privilege escalation bug,
which, you know, not not terrible, but it's got a name
and so people freaked out about it.
It I happen to notice today that the EPSS score for it was like
.1%. Like I don't maybe you guys can
explain that to me. I don't totally know like why
that would be the case, but is it just because it's a local?
I'm not sure. That could be part of it and the
fact that it's on Linux could also play into it.
I mean, there's, there's several different things that can play
into it. Maybe there's no, I don't know
if there is or not, but XY code being published or haven't seen
anything like that. So, so all of those have an
earmark of something that is not likely to be exploited.
Not saying it won't be, but it's just has a very low probability
of being exploited. And the thing that we have to
keep up on though, is to keep watching this.
So something like this, if it's garnering a lot, it's giving a
lot of headlines. That means that exploit code
probably isn't far behind. We're probably going to see some
really detailed write ups of how to exploit it, you know, some
things like that. And that's all going to start to
shift that probability as we see that activity.
And so that's why we're doing daily scores because that's
going to shift. It's going to change as we go
through time. OK, so this is this is a really
interesting point that's super easy to explain in layman's
terms for me, because I do this myself every day.
If there's a weather forecast for a day that I really want to
go to the park, I check it like 3 days before and that
probability could be like 20% chance of rain.
But then the next thing I do is I check it the day afterwards to
see if that probability has changed based on new information
that's rolling in about the environment.
And then right before I go outside to the park and think
that like for the next three hours I'll be outside, I again
check the weather forecast, which is the same exact
technology as EPSS. It's essentially saying given
the information we have historically about this system,
dynamic system, weather system or vulnerability ecosystem, what
is the probability that in the next 30 days is what EPS
measures? The weather forecast is usually
measuring something in the next couple hours.
Is it going to rain or not? Now that is a that's not a
static thing and it's not a static thing because it's not a
static thing in the world is weaponized code comes out as
people embed this and something that is published and somebody
else can pick up as they pair it with another vulnerability that
can take advantage of the local escalation, that probability
will change. So one of the downsides of the
model is that it's making a forecast probability guess.
One of the upsides is that you just can keep running it time
and time again. Like internally we run them
hourly instead of daily to see how that probability is
changing. Do you guys ever run them
against, like, historical vulnerabilities, you know, like
Microsoft stuff that came out like 10 years ago, like that is
is sitting there? Yeah.
We're scoring every single vulnerability that's been
published into the CVE data. Data OK.
And so yeah, we're scoring 1999 vulnerabilities still and there
are things from I think early 2000s, maybe there's a 99 one
too that is scored pretty high and we're still seeing some
activity on really old stuff so. Yeah.
That part is amazing to me. I think the peak actually is
like 10 years, 9 to 11 years, mainly driven by Microsoft.
But like the peak activity are things that have been published
9 to 11 years ago that we're seeing for the activity, which
is crazy. It is.
I mean, there was, I don't know if it was last week or the week
before, but SIS added something to the Kev that was from like
2009. It was, it was, yeah, some kind
of Windows Excel bug, I think. And it's a good.
Segue into some of the work we're doing for the DBIR that's
going to get published any minute now.
So we analyzed re exploitation probability.
Given that something has had exploitation in the past or was
on a catalyst, what's the chance that it's going to get exploited
again in the next 30 days? It's actually a really
interesting question because some things on this, it's a
catalyst to your point are 10 years old and some got on there
three years ago, but we don't know what's happened in the past
three years of that vulnerability.
Yeah, I'll, I'll leave it to Jay to explain re exploitation, but
that that result is pretty interesting.
Yeah, it's, it gets really complex really quick.
But like the question I had is if you see something on like
Scissor Kev's list and all you know is that it's on the list,
maybe it was put on yesterday or three years ago, but all you
know is it's on there. You don't know when it was last
exploited. What is the probability that it
will be exploited in the next 30 days or something?
And I think when I looked and I was using our data and I did it
over like the last six months, but it was something like
roughly about a 60% chance of exploitation activity for any of
them on the Kev. And then when I started looking
at it very closely though, and I looked at our data and I said
when was the last exploitation activity that we saw?
It was this beautiful little curve that just decayed from
like 100%. So it was exploited yesterday.
You got like a 99.8% chance it'll be exploited today.
And then if it didn't get exploited then you've, you know,
then it's like got a 97% chance and then like a 94% chance and,
and it just drops down and sort of flattens out.
And after about a year it, it's about the same.
If we haven't seen activity in a year, you've got about the same
chance of never been exploited. And so just knowing that recent
exploitation activity was a really key bit of information.
If you know something has been exploited, it's really helpful
to know when was the last time it was observed with activity
being exploited. Yeah, that makes a lot of sense
to me because there are, you know, 10s of thousands of known
vulnerabilities in, you know, I don't know how many are in the
Sisi cab, but there's enough. And, you know, attackers don't
need brand new vulnerabilities all the time.
You know, there's plenty of RC bugs in every firewall sitting
on the Internet and all that kind of stuff.
And the interesting thing to me, and I mean, I don't, there's no
real way to sort of quantify this, but you'll see things get
added to Sis's Cav or other Cavs where, you know, say it was an
iOS bug or a Chrome bug that was used in like one of those
targeted attacks against a journalist in, you know,
Morocco. It was chained with like 9 other
nine other bugs and they add it which is cool.
But like, the chances of that ever being exploited again are
probably pretty low because it took somebody eight months to
build that exploit in the 1st place and Apple just patched it
and you know, that kind of thing.
So there, there's all that, that sort of like variance in there
too. But yeah.
Yeah. I mean, yeah.
And we had. To admit to ourselves at some
point that the technology of the models we use for non targeted
attacks, like the Oracle Weblogic 2019 CV that's
constantly being exploited is not the same model and
technology that you would use to deal with targeted attacks.
Right. And I think that that was not
obvious in 20/15/2019. But to take it back to the
beginning of those organizations that are like, I'm a special
snowflake, my threat vector is different.
Some of them are right. They are special and they have
targeted attacks that might get triggered against them.
That's not going to come up in the data sets that we used to
triage the generalized vulnerabilities that are used
against everybody. Yeah.
And I mean, nor should it really.
I mean, there's kind of outliers or I mean, they're super cool to
look at those exploit chains and all that kind of stuff.
And like, think about the amount of effort that went into that
and that sort of thing and how they're, you know, sort of
driving innovation on the defensive side too, how Apple
and Google and Microsoft and everybody else are working to
break those exploit classes or bug classes.
But yeah, I mean, one crazy snowflake iOS bug shouldn't,
shouldn't, you know, drive a bunch of, I mean, they drive
headlines. That's, you know, we know that.
So the the initial kind of idea behind this this podcast for the
three of us was you guys reached out about the EPSS being
mentioned in anthropics documentation.
I guess documentation is not the right word, but blog post about
the Mythos release and I kind of missed it on first glance
because right above that was a piece of advice that said, I
believe patch everything that's in the Kev, right, Right.
Which is like, have you tried not having vulnerabilities
essentially? Like, yeah, your doctor telling
you, have you tried not being sick?
Sure, yeah, that's excellent advice.
But right below that was essentially pick a threshold,
like determine your threshold for tolerance in EPSS and patch
everything above that. Like, you know, so how did that
come? Were you guys aware that that
was going to be in there? Or was it just sort of like, oh
shit, look at this? Yeah, and it's amazing that they
use the language of threshold because one, it's the right way
to think about any model, but 2, they would know they build
models, but two, that question of what is the right threshold
for me has come up on the EPS CIG no less than 100 times over
the past five years. Oh, I'm sure.
Yeah, and I, I keep trying to punt it because of course I have
no idea. You know, it's all about your
risk tolerance, the security controls everything.
You know, like there's so many factors that go into that
threshold setting for for tolerance and, and early on
people were like, hey, you're producing the score.
Where's critical, high, medium and low.
We want to, we want to know where critical is and, and high.
And, and I kept on saying it's not like we're producing the
score, figure it out like, you know, and it's just the other
thing, like if you think about risk, you know, the, the
pressure, the, the exploitation activity coming in and then your
ability to resist it and then the outcome on the other side.
We're only measuring one of those three things, you know, it
is not the whole risk picture. And so for us to put a
threshold, like we're very concerned.
I'm very concerned that if we say here's critical, then people
are like, great, we're going to do this threshold and above,
then we're done. I'm like, that's not how it
works, you know, because I said like 2% seems low, but you're
still going to see some exploitation activity on the low
end, you know, so I can't tell people don't, you know, don't
patch under this or something. So I've always avoided that.
I want to grandstand about criticals and highs in general a
little bit. So this is a philosophy.
We have an empirical where we don't want to tell people what
higher critical is. We want to give them the tools
to determine that for themselves.
But what Jay saying is absolutely correct.
There's for the typical large organization that's using 10
different security ventures. They have 10 different
definitions of what high and critical is, some of which are
correct and maybe that thing is critical, some of which are not
applicable in that particular instance, some of which have
mitigating controls already in place.
So it was a critical a priori. But isn't that organization?
And so I have met many a person who spends their day-to-day
explaining to a regulator or an IT person why something that is
labeled as read high or critical is not read high or critical for
them. And vice versa, something that
is read labeled low is actually super important for that
organization. Adding another layer of
criticality in the exploit prediction scoring system.
We'll kind of do away with the entire concept of probability.
We want to give people the tools to measure likelihood.
That's EPSS. You know, a little bit of an
empirical plug. We've got models that do take
into account mitigating controls and we can build specific ones
for customers whether more of the risk is measured.
But even in those instances, it would be disingenuous for us to
say we understand your organization's risk tolerance
and posture and capacity. And so we're going to tell you
to patch everything above this. Now we want you to select and
have a discussion at maybe even the board level about what is
our risk tolerance? How much risk do we want to
remove from the organization? How much capacity do we have to
do that? Can we really patch 5% of our
vulnerabilities in any given month?
That is a, that is a measurable thing, right?
Some organizations can that have insane budgets, others can't.
Also 5% is going to change dramatically over the next two
years, but that means the raw numbers.
So just taking a step back from the the entire security industry
has been kind of saying, I found 100,000 things and 30% of them
are high. That determination is actually
something that enterprises and users have to be making rather
than vendors. But I I completely understand
the impetus for it, right? Like, you do need to filter the
signal from the noise in some way.
It's just that maybe the filter's in the wrong place.
Yeah, that's a great point. I, I do think, you know, going
back to your point about a lot of organizations thinking
they're special snowflakes, like in some way every organization
kind of is like they, you know, they have their own risk
tolerance, they have their own budget constraints, they have
their own, you know, resource constraints.
And in terms of personnel and like time to get this stuff done
even without, you know, frontier models dumping a giant pile of
bugs on them in the near future. Just, you know, Patch Tuesday,
Oracle, you know, Chrome, all that kind of stuff, not to
mention whatever other, you know, software they're using.
Even well resourced security teams have a tough time keeping
up with that stuff, you know, and so they do use, you know, we
got to get everything that's critical or high slash important
that that's what we're doing this month.
Like we got to get those done by May 15th because that's what you
know, is in my KPIs. Like, you know, like there's
real world shit that determines that stuff.
Right. Oh yeah, the SLA question is an
entirely different one too. If I say I fix all criticals
within seven days, that seven day period is also a risk and
capacity statement that I'm making about those criticals.
But the definition of the critical doesn't include that
seven day period. So you have two knobs to turn
right? One is how quickly do I do
things in this bucket? The second is what what is
critical? What is that bucket?
How small is it? Because if it's really just the
50 CVS that you know could really 'cause serious
exploitation and are being exploited elsewhere in your
environment, maybe you can't get those done in three days.
But if it's 5000, then you can't.
And that three day SLA is actually just you nicely lying
to yourself and others. Yeah, well, we all have those
little lies we tell ourselves. Yeah.
Some are about bugs, some are about the.
Yeah, our sleep schedules and our diets.
But yeah. So what are you guys
anticipating with, like Mythos and these other Frontier models?
I mean, there's been a lot of like there was a media backlash,
you know, Mythos specifically, then there was kind of backlash
to that of like, no, this is really cool.
There's a lot of like interesting stuff coming out of
this. You know, I, I saw a blog post
from the folks at Mozilla talking about like the, you
know, I think it was 200 and something bugs that it was found
in, in Firefox. Maybe I'm overstating it, but it
was quite a lot. And how, you know, kind of game
changing that was for them. I don't really know where this
is going, but it seems like if we have this conversation, like
even three months from now, we might be looking at a completely
different landscape, you know? Yeah.
And for us like I, I've thought a lot about it and actually it
doesn't matter what I think for for the modeling because the
modeling is is self learning. You know, it like EPSS right
now, I don't think I've looked at the data in like a month.
You know, the underlying data, it is going out and grabbing and
collecting all the stuff and producing the scores and it's
reacting to what's going on and I'm not there doing anything
with it. And so I think what the only
thing that I can say that absolutely will happen is I
think we're dealing with it going to be dealing with a
different system. You know, like our system right
now is, was the one that we've been working with for a long
time is that the attackers have finite resources.
The, you know, their attention is by far the most valuable
resource. And that that balance is going
to be shifting and it already is shifting.
And so the, the rate of exploit, the things that are targeted, I
think all of that is going to be shifting.
And I think the, the probably the hardest thing is for people
who've been in the industry for like 20 years or something going
to be like, no, I know what how vulnerabilities are picked.
This is a bad one. And that's a good one.
You know, like that's going to be really hard for them to shift
their perspective, but for the model, it's going to be like,
oh, I'm seeing this new activity or we're seeing it over here now
we're seeing it over here, You know, so we're going to have to
retrain more often. We're going to have to keep up
on the model, but largely that model is going to react.
So, so the retraining part is where I was going with this.
When we trained DPSS Version 4, which we released in March of
last year, it had been two years since the public UPS had been
retrained. So that's the gap between 3:00
and 4:00, just about two years, OK.
We started measuring model performance decay.
Well, as the first thing that we did before we started trading
version four, we had a lot more resources for Version 4.
We'd raise the seed round of funding.
We had some data infrastructure there.
So we're like, OK, how much worse did it get over 2 years?
And the answer was on median, something like 15%.
Jay, correct me if I'm making numbers up something like that
over 2 year decay, that's actually pretty good for any
deterministic model. I was gonna say like what it
what would you have expected? I honestly so, so a couple of
data points is that 48,000 vulnerabilities were released
this year. In 2021 when EPSL started there
I think were 18,000 vulnerabilities were released.
Yeah. So the growth is already like
exponential, right? Just the exponent is not the
same as it will be with these Frontier models.
Yeah, yeah. So I would have expected
actually more of a decay because the system had changed and the
volumes have changed, but OK. What's interesting is that the
vulnerability and by the way, way more exploits too.
Exploits used to be published and exploit DB and Metasploit.
Now like overwhelmingly they're in GitHub randomly all over the
place. So all of those things are
growing exponentially. But the vulnerabilities that are
being exploited, I think last year we tracked like 900 some
and maybe the maybe 1100 or had active exhibition that we're not
new. Yeah, yeah.
Overall, there's like 18,000 that have been and are being
exploited. But of those 48,000 that were
released that year, only about 1000 had exploitation activity.
OK, yeah, yeah. So I think what we're gonna have
to do and we are currently in the process of training at EPS
version 5, which will account for some of this shift change.
We'll make an announcement about it soon and then release it
shortly thereafter after after sig, sig testing.
But we're going to have to start retraining the model more often
because the decay will probably accelerate until we get to
another steady state where maybe there's some limit to the the
frontier models performance. Yeah.
And before the decay was basically just, you know, the
the market is shifting and it shifts slowly and that's what
the decay is. But now I think that the
underlying system is going to be shifting quicker.
And so that's why that decay I think is going to be a lot more
prominent. So retraining off it was going
to be our key to dealing with the next generation of threats.
Yeah, that next generation, I mean, it's, I don't know, the
next like year to 18 months are going to be really interesting
to watch. I mean, you mentioned that, you
know, we used to think like, OK, attacker attention or time was,
you know, a bounded resource that or bounding resource that
they couldn't you can't really make more time.
You can't make more of yourself. But literally they can now they
can sort of, you know, multiply themselves using AI.
And on the on the defensive side, the scarcity of elite
vulnerability research talent was a real problem too.
You know, that there was a a very finite pool of folks during
doing that top level research, you know, here in the US and
around the world. And now it's just like, I don't
know, do you have enough money to feed into the AI machine?
How many bugs do you want to find?
Like so, money is a finite resource.
Sometimes. Not always.
Sometimes. Yeah.
So I think this brings us to the point of why did Anthropic
recommend a deterministic predictive model that's purpose
built for one task and only does one thing when they literally
build a model. They can do just about anything
in a general purpose model. And the reason they recommended
EPSS is it is a model that can measure that difference in
attacker defender performance. And it is asymmetrically cheaper
for the defender to run a predictive purpose build model
for this task than it is for them to query a Jedi model every
30 minutes to figure out which vulnerability they should focus
on next. So as defenders, we have some of
these advantages where we can have sensor networks that give
us real time telemetry about what attackers are doing and
then use them feed small models that are super efficient and
super cost effective. So they might have infinite
attention, but they don't have infinite money.
And creating that kind of financial asymmetry is also
useful. Yeah, that's a great point.
I mean, yeah, the idea of making attacks more expensive in terms
of money and time for attackers is always good.
But we don't, I guess we have to assume that the apex adversaries
that we're worried about all the time in terms of like state
actors from China, North Korea and Russia are doing exactly
the, you know, have their own labs doing this kind of work as
well. Yeah.
You know, the bug finding and exploit development and all that
kind of stuff. So yeah, yeah, I don't.
Know, and the another thing I'll say like they're really high
profile targets that are typically at the top of the list
for state sponsored activity should be very afraid.
And I'm sure that they are. But I think for those not at the
top of the list for that group, I do want to point out that like
the the number of targets at those places generally was not
the limiting factor, right? I mean, like they had
vulnerabilities. They have vulnerabilities, they
will have vulnerabilities tomorrow.
You know, that that is not the limiting factor for most of the
attacks. And so moving forward, there's
going to be even more, you know, it's not going to change a ton.
So they're they're going to probably have to update and
react a little quicker, but hopefully we can, you know, get
the prioritization better so that they can react smarter.
So if Ed was on this podcast, he'd talk about the Mendoza
Line. Yeah.
The Mendoza Line for security is essentially you want to be
better than an amateur and an amateur used to be.
Don't have anything that Metasploit can reach across your
networks, across that data set of organizations that we worked
with for the prioritization and prediction reports, I don't
think there were any that didn't have at least one Metasploit
exploitable vulnerabilities somewhere across their networks.
This was 2021-2022. The thing that's really
different is that the Mendoza line is shifting.
It's no longer have nothing that's exploitable by
Metasploit. It's now have nothing that is
easily exploitable by an amateur wielding a cheap LLM.
So that line is shifting, but the the problem set and like the
the constraints that organization has to deal with it
really aren't because they were never above that Mendoza line
that would existed in the past anyways.
They had to manage to a different line, but the
management process is still very much the same.
Right. Yeah, In the past you were
worried about, Yeah, somebody that got ahold of Metasploit
and, you know, had some idea how to use it.
One of the other things that kind of worries me now is these
LLMS finding bugs in, you know, cloud platforms or SAS platforms
that are used by everybody and their brother.
And, you know, some universal bug like that that you know, was
found by an adversary and they get an exploit for it.
Like there was a stupid C panel bug this this week that like
effects 10s of millions of like shared hosting instance is and
like the average enterprise that is affected and can't fix that,
they need their hosting provider to fix it.
So, you know, things like that, you know, those happen every
once in a while, but I worry that those could be happening
more and more often with, you know, mythos or whatever other
frontier model decides to go and look.
Yeah, I agree and. Those are the things that
enterprise, you can't really plan for those, you know.
There's a whole class of those, right?
Like it used to be the CV. It was something that the
impetus was on the user of the software to remediate and patch
the system. I'm writing Adobe Reader 91.
I got to apply the patch on my endpoint.
Increasingly, and this is anecdotal from just me reading
CVS day in and day out, there are CVS in Salesforce or some
cloud hosting provider that are there almost as like a record,
but I can't do anything about it.
Nobody is using that software. The ownership of the remediation
is on the provider of the software and that is not the
same CVE as a CVE on a device where I have access and I'm
responsible for that. Right.
Yeah, those are those are the ones that I always, they catch
my attention. You're like, those could be
really bad. You know, when you sort of go
and look at how many people have that, you know, that Salesforce,
you know, they're using that cloud service or whatever it
happens to be. Yeah, but.
At the same time, for the average enterprise, it's like,
would I rather have my security team of three people fixing this
or Amazon security fixing? Yeah, no, that's, I mean, that's
one of the reasons you offload that responsibility to those
folks, right? Like, yeah, Amazon and Microsoft
have all the security talent in the world and all the money in
the world. So hopefully they're doing, you
know, they're spending our money wisely.
Vulnerability tax. Yeah, yes, it comes for us all.
Yeah. All right, Guy.
Anything we forgot to hit, I think we we got a lot of we got
a lot of topics in there. Yeah, everything, I think.
I I I really like anthropics recommendation.
I think it is directionally correct in the sense that
remediate the things you know have active exploitation that
are relevant to you. You can't remediate the entire
capitalist because you don't have every vulnerability on that
capitalist. I think the caveat that Jay's
research with the RE exploitation shows and that
folks can read on the DBIR in the coming weeks is that just
being on the capitalist is not right.
It is an indicator of risk, but it's not the only one, right?
So recency of that exploitation really matters.
I think we're trying to challenge the industry to think
about the acronym Kev as not a binary but a continuous
variable, right? Was it exploited yesterday?
Is it still being exploited? What is it patchy?
Is it bursty? And then the second
recommendation is use a cheap, small purpose built
probabilistic model to forecast what's next.
And that recommendation coming from an LLM lab is really
exciting to me. Like, you know, I don't ask
Anthropic what the weather is. I still go to weather.com.
There's a reason for that. I, I had a really funny thought
and I was wondering like, did they folks at Anthropic like use
Claude to generate that blog and Claude just like shoved it in
there and they never reviewed it and they're like, I whatever
publish, you know, that seems right.
It does seem right. Yeah, it'll be.
Poetic. I think the chances of that are
quite high, yeah. I don't remember if there were
actual names on that blog, but it doesn't matter anyway.
Doesn't matter. Yeah, not at all.
All right, right. Well, guys, thanks so much for
for coming on and explaining the world of math to me.
I really appreciate it. Yeah, great talking to you,
Dennis. Yeah, great to see you guys.
All right, take care all. Right.
Bye bye.