1 00:00:12,480 --> 00:00:14,920 Hello, and welcome to the Decipher Podcast. 2 00:00:14,920 --> 00:00:18,320 I'm Dennis Fisher and I have two guests with me today. 3 00:00:18,320 --> 00:00:22,000 Really happy to have two of the three founders of Empirical 4 00:00:22,000 --> 00:00:24,720 Security with me, Jay Jacobs and Michael Reitman. 5 00:00:25,080 --> 00:00:26,320 Guys, thanks so much for being here. 6 00:00:27,000 --> 00:00:28,280 Thanks for having us. Yeah. 7 00:00:28,320 --> 00:00:32,000 Thank you. Yeah, absolutely sad that we 8 00:00:32,000 --> 00:00:34,640 didn't have Ed with us, but I wanted to, you know, he's 9 00:00:34,640 --> 00:00:37,560 probably just sad about the Cubs anyway, so that's fine too. 10 00:00:38,800 --> 00:00:40,440 Much empirical for anyone podcast. 11 00:00:41,280 --> 00:00:44,320 It's yeah, too many big brains. Yeah. 12 00:00:46,160 --> 00:00:48,040 Yeah. So I mentioned, you know, I want 13 00:00:48,040 --> 00:00:50,120 to talk a little bit about I've known both of you guys for a 14 00:00:50,120 --> 00:00:53,040 little while and I think Jay, I've known you for longer, but 15 00:00:54,040 --> 00:00:56,880 you both have this kind of background in like the kind of 16 00:00:56,880 --> 00:01:01,960 the data-driven side of security, which you know, is a 17 00:01:01,960 --> 00:01:04,120 really fascinating part of it to me, even though I have 18 00:01:04,120 --> 00:01:07,920 absolutely no mind for math whatsoever, much to my parents 19 00:01:07,920 --> 00:01:12,120 chagrin. But tell me where the idea for 20 00:01:12,120 --> 00:01:16,120 Empirical itself came from and and what you guys were were 21 00:01:16,120 --> 00:01:19,800 hoping to accomplish and like how you were hoping to help 22 00:01:19,800 --> 00:01:25,920 enterprises when you started? So Jay and I have been working 23 00:01:25,920 --> 00:01:30,760 together around this problem for almost 15 years now. 24 00:01:32,720 --> 00:01:37,440 I think we first worked together in a data-driven capacity and a 25 00:01:37,440 --> 00:01:42,360 chapter in the DBIR based on kind of security data when I was 26 00:01:42,360 --> 00:01:44,640 Chief Data Scientist in kind of security and Jay was at the 27 00:01:44,640 --> 00:01:48,920 DBIR. We second worked on this when 28 00:01:48,920 --> 00:01:51,400 Santia did an engagement with kind of security to do the 29 00:01:51,400 --> 00:01:53,040 prioritization of prediction reports. 30 00:01:54,040 --> 00:01:58,400 They did nine of them reviewing our internal data in the 31 00:01:58,400 --> 00:02:01,720 aggregate to try to make some statements about vulnerability 32 00:02:01,720 --> 00:02:05,840 management as a whole. And that I think has a life of 33 00:02:05,840 --> 00:02:07,160 its own. I think some of those findings 34 00:02:07,160 --> 00:02:11,680 are still relevant today. And then along the way, we 35 00:02:11,680 --> 00:02:15,080 started working on the exploit prediction scoring system in 36 00:02:15,080 --> 00:02:17,800 2019. We presented kind of the 37 00:02:17,800 --> 00:02:23,040 scaffolding for it in version one at Blackhat and Jay has 38 00:02:23,040 --> 00:02:28,080 taken that to new heights. At first, the idea for Empirical 39 00:02:28,080 --> 00:02:33,240 essentially came from working with vulnerability management 40 00:02:33,240 --> 00:02:36,200 data interfacing with a lot of large organizations. 41 00:02:36,200 --> 00:02:40,200 Kind of security was acquired by Cisco in 2021 and I was exposed 42 00:02:40,200 --> 00:02:43,080 to a whole bunch of those. And every single time we talked 43 00:02:43,080 --> 00:02:47,040 to a sufficiently large or complex or mature organization, 44 00:02:47,640 --> 00:02:52,560 the first pushback would be like, this model is great in the 45 00:02:52,560 --> 00:02:55,760 non targeted attack sense, but I am a special stuff. 46 00:02:55,760 --> 00:02:59,120 Like I have special data, I have a special environment, my 47 00:02:59,120 --> 00:03:02,280 threats are different. This doesn't capture my reality. 48 00:03:02,880 --> 00:03:07,680 And you know, with that hat on, my response was this is the best 49 00:03:07,680 --> 00:03:10,200 we've got. This is the model that is 50 00:03:10,200 --> 00:03:12,760 data-driven and others really aren't at the time. 51 00:03:13,960 --> 00:03:17,360 I think in the back of my head, and I started talking to Jay 52 00:03:17,360 --> 00:03:22,040 about this probably two years ago, was what if we did build 53 00:03:22,040 --> 00:03:26,960 individual models per customer? What if we did incorporate that 54 00:03:26,960 --> 00:03:30,360 data about that customer and build a model specific to them? 55 00:03:30,360 --> 00:03:34,160 What does that world look like? And Empirical has really been us 56 00:03:34,160 --> 00:03:35,720 trying to realize that journey along the way. 57 00:03:36,120 --> 00:03:39,160 We got Ed Bellis to take a look at it and we're going to have 58 00:03:39,160 --> 00:03:41,160 him invest in the company. And he was like, no, that's, 59 00:03:41,240 --> 00:03:43,000 that's a bit too good of an idea. 60 00:03:43,000 --> 00:03:47,760 Why don't I join you guys? As if he didn't have enough to 61 00:03:47,760 --> 00:03:48,920 do. Yeah. 62 00:03:49,240 --> 00:03:53,920 I mean, it's, it's hard to remember now, at least for me, 63 00:03:54,360 --> 00:03:58,720 but there was a time where kind of this data-driven approach to 64 00:03:58,720 --> 00:04:00,520 security like almost didn't exist. 65 00:04:00,880 --> 00:04:05,120 Like it was just, you know, and it wasn't, I mean, it's long ago 66 00:04:05,120 --> 00:04:08,960 in like security community terms, but it's not that long 67 00:04:08,960 --> 00:04:12,480 ago in absolute time terms. You know, like even just 15 68 00:04:12,480 --> 00:04:16,880 years ago, like this was seen as like this sort of like out there 69 00:04:16,880 --> 00:04:18,680 idea of like, what are you going to do with data? 70 00:04:19,079 --> 00:04:21,200 Who knows? Like how are you going to apply 71 00:04:21,200 --> 00:04:25,280 data to like bad guys, you know, trying to get into my network? 72 00:04:25,280 --> 00:04:28,120 What is that going to tell you? You know, we, we still run into 73 00:04:28,120 --> 00:04:29,040 that. Yeah. 74 00:04:29,080 --> 00:04:30,880 Oh, I'm sure. Mortality is absolutely still 75 00:04:30,880 --> 00:04:31,960 out there. Yeah, yeah. 76 00:04:32,440 --> 00:04:36,880 I believe it like do, but do you run into that at like what level 77 00:04:36,880 --> 00:04:40,080 of the organization is, is, where does that resistance? 78 00:04:41,320 --> 00:04:43,240 I, I, I don't think it's a level. 79 00:04:43,240 --> 00:04:46,960 I think it's just something about some people have a very 80 00:04:46,960 --> 00:04:51,680 skeptical bones in their body about data-driven security 81 00:04:51,680 --> 00:04:54,160 stuff. And so I've seen it at every 82 00:04:54,160 --> 00:04:57,240 single level and I've been surprised some people at every 83 00:04:57,240 --> 00:04:59,480 single level are like, this is the greatest thing ever and 84 00:05:00,000 --> 00:05:01,520 really want to get into the math and stuff. 85 00:05:01,520 --> 00:05:04,640 And so it's, it depends. I don't think it's a specific 86 00:05:04,640 --> 00:05:06,640 role or anything. It's just a mentality for some 87 00:05:06,640 --> 00:05:09,040 people. Yeah, I think that's right 88 00:05:09,040 --> 00:05:12,160 because, you know, especially people I think of maybe our 89 00:05:12,160 --> 00:05:14,880 generation J, like people got into security 'cause they like 90 00:05:15,320 --> 00:05:17,920 tech stuff, you know, they like breaking things and they liked 91 00:05:18,440 --> 00:05:22,480 that sort of like, you know, back and forth type thing. 92 00:05:22,480 --> 00:05:27,440 Not because they like graphs and math, you know, certainly not 93 00:05:27,440 --> 00:05:30,760 what they were in for. And you would still see that, 94 00:05:30,760 --> 00:05:35,360 see that sometimes it like at conferences, you know, is it 95 00:05:35,360 --> 00:05:37,560 something like Black Hat that you mentioned, Michael? 96 00:05:38,480 --> 00:05:41,040 It, you know, it was all the guys like trying to break stuff. 97 00:05:41,040 --> 00:05:45,240 And then there'd be like a group of like 12 of you guys over in 98 00:05:45,240 --> 00:05:49,600 the corner, like, you know, in like the data data world talking 99 00:05:49,600 --> 00:05:53,560 about like actual results and like, look, we can prove that 100 00:05:53,720 --> 00:05:56,200 like this stuff works, you know? Yeah. 101 00:05:57,320 --> 00:06:01,320 I think both are really necessary parts of the security 102 00:06:01,320 --> 00:06:04,680 ecosystem, but security's been growing. 103 00:06:05,040 --> 00:06:07,880 I think that part's missing from a lot of discussions about this 104 00:06:07,880 --> 00:06:12,880 that 15 years ago security, some parts of security were a human 105 00:06:12,880 --> 00:06:15,440 scale, let's investigate type of problem. 106 00:06:17,240 --> 00:06:20,960 I think for about 5-10 years, it hasn't been that the volume has 107 00:06:20,960 --> 00:06:24,240 been such that you need to do statistical analysis and try to 108 00:06:24,240 --> 00:06:25,960 be more efficient with what you're working on. 109 00:06:27,320 --> 00:06:30,640 But once you figure out what's efficient, it's still a pretty 110 00:06:30,640 --> 00:06:33,840 manual effort of investigation and figuring out what to do and 111 00:06:34,280 --> 00:06:36,560 actually doing the remediations or mitigations. 112 00:06:38,200 --> 00:06:42,160 I like to think about just how firefighters rarely take a step 113 00:06:42,160 --> 00:06:44,800 back and think about like the statistical incidents of fires 114 00:06:44,800 --> 00:06:46,960 across the city because they're busy responding to the 115 00:06:46,960 --> 00:06:50,480 day-to-day, and I think security has been in that mindset for 116 00:06:50,480 --> 00:06:52,200 some time. Taking that step back is 117 00:06:52,200 --> 00:06:56,200 actually quite a luxury. Yeah, that's a, that's a good 118 00:06:56,200 --> 00:06:58,440 way of putting it. Yeah, I like that. 119 00:06:58,560 --> 00:07:03,040 And it's so you mentioned the EPSS model, which I think is one 120 00:07:03,040 --> 00:07:06,040 of the coolest things that's kind of developed over the the 121 00:07:06,040 --> 00:07:12,520 last few years, which is the exploit prediction scoring 122 00:07:12,520 --> 00:07:13,840 system. OK, Yeah. 123 00:07:14,320 --> 00:07:19,800 So essentially like I'll let you guys describe like how it all 124 00:07:19,800 --> 00:07:23,400 works, but it's the idea is to like it what it says on the 125 00:07:23,400 --> 00:07:27,800 label, like tell you how likely this bug is to be exploited 126 00:07:27,800 --> 00:07:32,160 taking into account a whole bunch of factors and parameters. 127 00:07:32,160 --> 00:07:35,120 But Jay, where did that? And I know it's like from the 128 00:07:35,120 --> 00:07:39,320 first group, the incidents incident response group. 129 00:07:39,880 --> 00:07:42,480 And yeah, so I want to go back to the beginning and, and 130 00:07:42,480 --> 00:07:44,040 Michael sort of glossed over a little bit. 131 00:07:44,040 --> 00:07:48,680 But when we started, when I, I started Santi with Wade Baker 132 00:07:48,680 --> 00:07:51,840 and we were contracting with Kenna and a bunch of other 133 00:07:51,840 --> 00:07:56,320 companies and Kenna had just wonderful data about the 134 00:07:56,320 --> 00:07:59,320 vulnerabilities in existence, the remediation, a bunch of 135 00:08:00,800 --> 00:08:04,160 extra data about vulnerabilities that you just don't find 136 00:08:04,160 --> 00:08:06,240 typically. And then we had another 137 00:08:06,240 --> 00:08:08,600 customer, we're working with Fortinet and we're producing 138 00:08:08,600 --> 00:08:10,400 Fortinet's threat landscape report. 139 00:08:10,920 --> 00:08:14,320 And of course, Fortinet has, you know, I think over 100,000 140 00:08:14,320 --> 00:08:18,840 devices deployed globally and they're all reporting the, the 141 00:08:18,840 --> 00:08:20,680 activity that they're seeing centrally. 142 00:08:21,200 --> 00:08:24,240 And so Fortinet had this wonderful amount of information 143 00:08:24,240 --> 00:08:27,240 about all this activity going on, all these CV ES being 144 00:08:27,480 --> 00:08:29,680 attempted to be exploited, blocked, things like that. 145 00:08:29,680 --> 00:08:32,400 And I was looking at the exploit activity and looking at 146 00:08:32,400 --> 00:08:35,880 Kenneth's data and kind of had some exploit activity, but we're 147 00:08:35,880 --> 00:08:38,360 looking at Fortinet. Like if I bring these together, 148 00:08:38,840 --> 00:08:41,480 we could we could model this like we've got outcome data. 149 00:08:41,559 --> 00:08:45,200 The things that we want to stop is this exploit and then all the 150 00:08:45,200 --> 00:08:46,840 stuff about the vulnerabilities from Canada. 151 00:08:46,840 --> 00:08:49,760 And so bringing those two together was basically the birth 152 00:08:49,760 --> 00:08:52,760 of VPSS. So OK. 153 00:08:53,200 --> 00:08:55,840 That's no small thing, because if you really think about it, 154 00:08:55,840 --> 00:09:00,160 security is two types of data. In general, it's, I looked at a 155 00:09:00,160 --> 00:09:02,440 system and here's the state of that system, which is what a 156 00:09:02,440 --> 00:09:05,960 vulnerability is. It's like this exists in maybe 157 00:09:05,960 --> 00:09:08,840 production, maybe not production on a particular device or 158 00:09:08,840 --> 00:09:12,280 machine or database. And then there's the telemetry 159 00:09:12,280 --> 00:09:14,760 data, which is in real time. These things are happening. 160 00:09:15,320 --> 00:09:18,120 Historically, they've been to sometimes even different groups 161 00:09:18,120 --> 00:09:21,480 in an organization. And I think what Jay did is 162 00:09:21,480 --> 00:09:25,520 essentially cross that divide of data engineering more than 163 00:09:25,520 --> 00:09:28,720 anything to ask the question, well, what what if we looked at 164 00:09:28,720 --> 00:09:31,960 all security data, what could we answer about an organization's 165 00:09:31,960 --> 00:09:36,120 posture in real time? So if you're looking at a given 166 00:09:36,120 --> 00:09:40,200 vulnerability, how does the EPSS like? 167 00:09:40,200 --> 00:09:43,520 What is it taking into account to come up with a a 168 00:09:44,520 --> 00:09:47,120 exploitability prediction score for that bug it? 169 00:09:47,840 --> 00:09:51,360 Takes in as much as I can everything and anything we can 170 00:09:51,360 --> 00:09:55,240 get our hands on. But for the most part, I think a 171 00:09:55,240 --> 00:09:59,080 lot of the strong signals right now we've got about 2800 172 00:09:59,080 --> 00:10:01,640 different variables feeding into the model. 173 00:10:02,440 --> 00:10:07,160 And I think probably 900 to 1000 are actually feeding and helping 174 00:10:07,160 --> 00:10:08,720 the the scores and stuff like that. 175 00:10:08,720 --> 00:10:10,680 So the extra is just sitting there. 176 00:10:10,680 --> 00:10:14,880 But anyway, a lot of it is like one of the key things we'll put 177 00:10:14,880 --> 00:10:18,400 in the vendor of the product and that vendor holds so much 178 00:10:18,400 --> 00:10:20,120 information. Like how prevalent is that 179 00:10:20,120 --> 00:10:22,280 vendor? You know, if you think of, you 180 00:10:22,280 --> 00:10:24,640 see a vulnerability and you're like, and then you see how it's 181 00:10:24,640 --> 00:10:26,560 a Microsoft on, you're like, oh, wait a minute, what was that 182 00:10:26,560 --> 00:10:28,440 vulnerability? You know, like you get these 183 00:10:28,440 --> 00:10:30,480 little tip offs and the same thing with the model. 184 00:10:30,480 --> 00:10:34,600 And so we also do a ton of scraping for exploit code being 185 00:10:34,600 --> 00:10:36,240 published. Those are usually pretty good 186 00:10:36,240 --> 00:10:38,920 indicators. Yeah. 187 00:10:38,920 --> 00:10:41,160 I mean, just everything and anything different chatter. 188 00:10:42,120 --> 00:10:45,360 We try to pull texts of people talking about these 189 00:10:45,360 --> 00:10:48,520 vulnerabilities and then mine that text for keywords and 190 00:10:48,520 --> 00:10:51,200 phrases and all sorts of things that people are talking about. 191 00:10:51,840 --> 00:10:53,600 We try to get at the characteristics of the 192 00:10:53,600 --> 00:10:57,680 vulnerability. So things like CVSSI guess, but 193 00:10:58,240 --> 00:11:00,200 we try to get a little bit more granular than that. 194 00:11:01,000 --> 00:11:04,040 So we try to get more detail about, you know, if you know, 195 00:11:04,040 --> 00:11:08,120 like as a sequel injection and what kind and all the details 196 00:11:08,120 --> 00:11:10,440 that we can get. And it's all just fed into this, 197 00:11:10,440 --> 00:11:13,240 you know, meat grinder of a model that learns all the 198 00:11:13,240 --> 00:11:16,400 patterns. And so when EPSS is rating 199 00:11:16,400 --> 00:11:20,800 something high, it actually EPSS, when we're scoring stuff 200 00:11:20,800 --> 00:11:24,400 daily, it doesn't know about recent exploit activity. 201 00:11:24,560 --> 00:11:26,680 So it's not actually factored into the model. 202 00:11:27,680 --> 00:11:30,320 And so when the model scores something high, it says, hey, 203 00:11:30,600 --> 00:11:33,840 looking at all the attributes to this vulnerability, it looks 204 00:11:33,840 --> 00:11:35,920 like it shares a lot of the attributes with things that 205 00:11:35,920 --> 00:11:39,720 we've seen exploited. And so it's saying like this has 206 00:11:39,720 --> 00:11:42,320 the the markings of things that should be exploited. 207 00:11:42,720 --> 00:11:45,080 And that's what the model is producing that that's sort of, 208 00:11:45,280 --> 00:11:49,560 it looks like exploited stuff. You didn't use the key operative 209 00:11:49,560 --> 00:11:51,200 word, which is prediction, right? 210 00:11:51,200 --> 00:11:55,040 I think that's the difference between EPS and every other 211 00:11:55,040 --> 00:11:56,240 scoring system that came before it. 212 00:11:56,240 --> 00:11:59,560 It's that it is not describing the current state of the 213 00:11:59,560 --> 00:12:01,840 vulnerability, although of course that data is factored in. 214 00:12:02,160 --> 00:12:05,520 It is making a prediction about is it likely to be exploited or 215 00:12:05,520 --> 00:12:06,920 not. Yeah. 216 00:12:07,640 --> 00:12:11,920 And so if if I'm an organization, let's say I'm, you 217 00:12:11,920 --> 00:12:15,720 know, an insurance company and I'm looking at this, you know, 218 00:12:15,720 --> 00:12:18,440 to try and grade vulnerabilities, is it taking 219 00:12:18,440 --> 00:12:21,920 into account or do I have to sort of make these calculations 220 00:12:21,920 --> 00:12:25,400 myself? Like the industry that I'm in, 221 00:12:25,720 --> 00:12:30,800 you know, like how likely it is that, you know, insurance 222 00:12:30,800 --> 00:12:34,360 companies will be attacked with this type of bug, that kind of 223 00:12:34,360 --> 00:12:36,240 thing is, is that kind of context there? 224 00:12:37,080 --> 00:12:40,840 It's, it's not there that that kind of breakdown because all we 225 00:12:40,840 --> 00:12:44,120 do like when we publish EPSS, it's literally just ACV and a 226 00:12:44,120 --> 00:12:47,520 score like that is that is the richness of our data there. 227 00:12:47,520 --> 00:12:50,200 And we've been doing that since we've been publishing since 228 00:12:50,200 --> 00:12:54,240 2021. And so everyday just CV and a 229 00:12:54,240 --> 00:12:55,760 score. So we don't break anything out 230 00:12:55,760 --> 00:12:57,920 by industry or anything like that. 231 00:12:57,920 --> 00:13:01,720 We try to keep it global, just, you know, that initial sort of 232 00:13:01,720 --> 00:13:06,360 global indicator. Yeah, and and the the output is 233 00:13:06,360 --> 00:13:09,360 an actual probability and we actually calibrate that 234 00:13:09,360 --> 00:13:11,640 probability. So when we say, you know, like 235 00:13:11,640 --> 00:13:16,200 2% seems like a very low probability, but it's a 2% 236 00:13:16,200 --> 00:13:18,480 chance that it will be exploited to one out of 50 237 00:13:18,480 --> 00:13:21,800 vulnerabilities. At 2% we expect to see 238 00:13:21,800 --> 00:13:24,920 exploitation activity, you know, and so that's that's a key 239 00:13:24,920 --> 00:13:26,520 thing. If we talk about thresholds, I'm 240 00:13:26,880 --> 00:13:29,600 going to bring back probability into that conversation. 241 00:13:29,880 --> 00:13:32,120 But yeah, we output a probability. 242 00:13:32,920 --> 00:13:37,520 OK, and I, I think I mentioned that earlier this, this Linux 243 00:13:37,520 --> 00:13:40,680 bug that came out this week that, you know, everybody's kind 244 00:13:40,680 --> 00:13:44,360 of freaking out about it's a local privilege escalation bug, 245 00:13:44,360 --> 00:13:48,040 which, you know, not not terrible, but it's got a name 246 00:13:48,040 --> 00:13:49,400 and so people freaked out about it. 247 00:13:49,720 --> 00:13:54,200 It I happen to notice today that the EPSS score for it was like 248 00:13:54,640 --> 00:14:00,360 .1%. Like I don't maybe you guys can 249 00:14:00,360 --> 00:14:02,720 explain that to me. I don't totally know like why 250 00:14:02,720 --> 00:14:06,640 that would be the case, but is it just because it's a local? 251 00:14:06,680 --> 00:14:11,320 I'm not sure. That could be part of it and the 252 00:14:11,320 --> 00:14:13,960 fact that it's on Linux could also play into it. 253 00:14:15,080 --> 00:14:17,120 I mean, there's, there's several different things that can play 254 00:14:17,120 --> 00:14:19,200 into it. Maybe there's no, I don't know 255 00:14:19,200 --> 00:14:22,360 if there is or not, but XY code being published or haven't seen 256 00:14:22,360 --> 00:14:25,760 anything like that. So, so all of those have an 257 00:14:25,760 --> 00:14:28,400 earmark of something that is not likely to be exploited. 258 00:14:29,480 --> 00:14:32,400 Not saying it won't be, but it's just has a very low probability 259 00:14:32,400 --> 00:14:36,000 of being exploited. And the thing that we have to 260 00:14:36,000 --> 00:14:38,120 keep up on though, is to keep watching this. 261 00:14:38,120 --> 00:14:41,160 So something like this, if it's garnering a lot, it's giving a 262 00:14:41,160 --> 00:14:43,640 lot of headlines. That means that exploit code 263 00:14:43,640 --> 00:14:45,920 probably isn't far behind. We're probably going to see some 264 00:14:45,920 --> 00:14:48,800 really detailed write ups of how to exploit it, you know, some 265 00:14:48,800 --> 00:14:50,560 things like that. And that's all going to start to 266 00:14:50,560 --> 00:14:52,920 shift that probability as we see that activity. 267 00:14:52,920 --> 00:14:55,640 And so that's why we're doing daily scores because that's 268 00:14:55,640 --> 00:14:57,400 going to shift. It's going to change as we go 269 00:14:57,400 --> 00:15:00,400 through time. OK, so this is this is a really 270 00:15:00,880 --> 00:15:03,440 interesting point that's super easy to explain in layman's 271 00:15:03,440 --> 00:15:05,840 terms for me, because I do this myself every day. 272 00:15:06,760 --> 00:15:09,240 If there's a weather forecast for a day that I really want to 273 00:15:09,240 --> 00:15:12,960 go to the park, I check it like 3 days before and that 274 00:15:12,960 --> 00:15:15,400 probability could be like 20% chance of rain. 275 00:15:16,000 --> 00:15:18,600 But then the next thing I do is I check it the day afterwards to 276 00:15:18,600 --> 00:15:21,400 see if that probability has changed based on new information 277 00:15:21,400 --> 00:15:22,920 that's rolling in about the environment. 278 00:15:23,280 --> 00:15:26,280 And then right before I go outside to the park and think 279 00:15:26,280 --> 00:15:29,280 that like for the next three hours I'll be outside, I again 280 00:15:29,280 --> 00:15:32,080 check the weather forecast, which is the same exact 281 00:15:32,080 --> 00:15:34,960 technology as EPSS. It's essentially saying given 282 00:15:34,960 --> 00:15:38,680 the information we have historically about this system, 283 00:15:39,080 --> 00:15:43,120 dynamic system, weather system or vulnerability ecosystem, what 284 00:15:43,120 --> 00:15:46,520 is the probability that in the next 30 days is what EPS 285 00:15:46,520 --> 00:15:47,960 measures? The weather forecast is usually 286 00:15:47,960 --> 00:15:49,720 measuring something in the next couple hours. 287 00:15:50,240 --> 00:15:53,280 Is it going to rain or not? Now that is a that's not a 288 00:15:53,280 --> 00:15:56,440 static thing and it's not a static thing because it's not a 289 00:15:56,440 --> 00:15:59,480 static thing in the world is weaponized code comes out as 290 00:15:59,480 --> 00:16:02,480 people embed this and something that is published and somebody 291 00:16:02,480 --> 00:16:05,360 else can pick up as they pair it with another vulnerability that 292 00:16:05,360 --> 00:16:08,840 can take advantage of the local escalation, that probability 293 00:16:08,840 --> 00:16:11,760 will change. So one of the downsides of the 294 00:16:11,760 --> 00:16:15,720 model is that it's making a forecast probability guess. 295 00:16:16,200 --> 00:16:18,520 One of the upsides is that you just can keep running it time 296 00:16:18,520 --> 00:16:20,120 and time again. Like internally we run them 297 00:16:20,120 --> 00:16:22,760 hourly instead of daily to see how that probability is 298 00:16:22,760 --> 00:16:26,520 changing. Do you guys ever run them 299 00:16:26,520 --> 00:16:29,840 against, like, historical vulnerabilities, you know, like 300 00:16:30,560 --> 00:16:34,200 Microsoft stuff that came out like 10 years ago, like that is 301 00:16:34,400 --> 00:16:35,760 is sitting there? Yeah. 302 00:16:36,360 --> 00:16:38,400 We're scoring every single vulnerability that's been 303 00:16:38,400 --> 00:16:40,960 published into the CVE data. Data OK. 304 00:16:41,920 --> 00:16:47,000 And so yeah, we're scoring 1999 vulnerabilities still and there 305 00:16:47,000 --> 00:16:50,800 are things from I think early 2000s, maybe there's a 99 one 306 00:16:50,800 --> 00:16:54,520 too that is scored pretty high and we're still seeing some 307 00:16:54,520 --> 00:16:57,640 activity on really old stuff so. Yeah. 308 00:16:58,240 --> 00:17:00,880 That part is amazing to me. I think the peak actually is 309 00:17:00,880 --> 00:17:06,599 like 10 years, 9 to 11 years, mainly driven by Microsoft. 310 00:17:06,599 --> 00:17:09,640 But like the peak activity are things that have been published 311 00:17:09,640 --> 00:17:13,680 9 to 11 years ago that we're seeing for the activity, which 312 00:17:13,680 --> 00:17:15,079 is crazy. It is. 313 00:17:16,040 --> 00:17:18,680 I mean, there was, I don't know if it was last week or the week 314 00:17:18,680 --> 00:17:23,160 before, but SIS added something to the Kev that was from like 315 00:17:23,160 --> 00:17:25,720 2009. It was, it was, yeah, some kind 316 00:17:25,720 --> 00:17:29,040 of Windows Excel bug, I think. And it's a good. 317 00:17:29,400 --> 00:17:33,480 Segue into some of the work we're doing for the DBIR that's 318 00:17:33,480 --> 00:17:34,840 going to get published any minute now. 319 00:17:36,040 --> 00:17:39,600 So we analyzed re exploitation probability. 320 00:17:39,960 --> 00:17:42,560 Given that something has had exploitation in the past or was 321 00:17:42,560 --> 00:17:45,520 on a catalyst, what's the chance that it's going to get exploited 322 00:17:45,520 --> 00:17:48,360 again in the next 30 days? It's actually a really 323 00:17:48,360 --> 00:17:50,560 interesting question because some things on this, it's a 324 00:17:50,560 --> 00:17:53,920 catalyst to your point are 10 years old and some got on there 325 00:17:53,920 --> 00:17:57,040 three years ago, but we don't know what's happened in the past 326 00:17:57,040 --> 00:17:58,280 three years of that vulnerability. 327 00:17:58,600 --> 00:18:02,960 Yeah, I'll, I'll leave it to Jay to explain re exploitation, but 328 00:18:02,960 --> 00:18:04,440 that that result is pretty interesting. 329 00:18:05,280 --> 00:18:08,760 Yeah, it's, it gets really complex really quick. 330 00:18:09,360 --> 00:18:12,000 But like the question I had is if you see something on like 331 00:18:12,000 --> 00:18:15,840 Scissor Kev's list and all you know is that it's on the list, 332 00:18:15,840 --> 00:18:18,880 maybe it was put on yesterday or three years ago, but all you 333 00:18:18,880 --> 00:18:21,320 know is it's on there. You don't know when it was last 334 00:18:21,320 --> 00:18:23,040 exploited. What is the probability that it 335 00:18:23,040 --> 00:18:25,160 will be exploited in the next 30 days or something? 336 00:18:25,160 --> 00:18:27,760 And I think when I looked and I was using our data and I did it 337 00:18:27,760 --> 00:18:29,840 over like the last six months, but it was something like 338 00:18:30,280 --> 00:18:35,320 roughly about a 60% chance of exploitation activity for any of 339 00:18:35,320 --> 00:18:38,240 them on the Kev. And then when I started looking 340 00:18:38,240 --> 00:18:40,680 at it very closely though, and I looked at our data and I said 341 00:18:40,680 --> 00:18:43,720 when was the last exploitation activity that we saw? 342 00:18:44,160 --> 00:18:47,440 It was this beautiful little curve that just decayed from 343 00:18:47,440 --> 00:18:49,800 like 100%. So it was exploited yesterday. 344 00:18:50,160 --> 00:18:53,480 You got like a 99.8% chance it'll be exploited today. 345 00:18:54,200 --> 00:18:56,640 And then if it didn't get exploited then you've, you know, 346 00:18:56,680 --> 00:19:00,400 then it's like got a 97% chance and then like a 94% chance and, 347 00:19:00,720 --> 00:19:03,240 and it just drops down and sort of flattens out. 348 00:19:03,240 --> 00:19:06,240 And after about a year it, it's about the same. 349 00:19:06,320 --> 00:19:08,560 If we haven't seen activity in a year, you've got about the same 350 00:19:08,560 --> 00:19:13,000 chance of never been exploited. And so just knowing that recent 351 00:19:13,000 --> 00:19:16,760 exploitation activity was a really key bit of information. 352 00:19:16,760 --> 00:19:19,400 If you know something has been exploited, it's really helpful 353 00:19:19,400 --> 00:19:22,840 to know when was the last time it was observed with activity 354 00:19:22,840 --> 00:19:25,560 being exploited. Yeah, that makes a lot of sense 355 00:19:25,560 --> 00:19:30,120 to me because there are, you know, 10s of thousands of known 356 00:19:30,120 --> 00:19:32,920 vulnerabilities in, you know, I don't know how many are in the 357 00:19:32,920 --> 00:19:37,640 Sisi cab, but there's enough. And, you know, attackers don't 358 00:19:37,640 --> 00:19:39,560 need brand new vulnerabilities all the time. 359 00:19:39,560 --> 00:19:44,360 You know, there's plenty of RC bugs in every firewall sitting 360 00:19:44,360 --> 00:19:46,120 on the Internet and all that kind of stuff. 361 00:19:46,440 --> 00:19:51,240 And the interesting thing to me, and I mean, I don't, there's no 362 00:19:51,240 --> 00:19:55,080 real way to sort of quantify this, but you'll see things get 363 00:19:55,080 --> 00:19:59,560 added to Sis's Cav or other Cavs where, you know, say it was an 364 00:19:59,560 --> 00:20:04,000 iOS bug or a Chrome bug that was used in like one of those 365 00:20:04,000 --> 00:20:07,840 targeted attacks against a journalist in, you know, 366 00:20:08,280 --> 00:20:11,440 Morocco. It was chained with like 9 other 367 00:20:11,480 --> 00:20:14,360 nine other bugs and they add it which is cool. 368 00:20:15,320 --> 00:20:19,040 But like, the chances of that ever being exploited again are 369 00:20:19,040 --> 00:20:22,520 probably pretty low because it took somebody eight months to 370 00:20:22,520 --> 00:20:26,080 build that exploit in the 1st place and Apple just patched it 371 00:20:26,080 --> 00:20:28,040 and you know, that kind of thing. 372 00:20:28,040 --> 00:20:31,000 So there, there's all that, that sort of like variance in there 373 00:20:31,000 --> 00:20:33,280 too. But yeah. 374 00:20:33,280 --> 00:20:34,040 Yeah. I mean, yeah. 375 00:20:34,240 --> 00:20:36,000 And we had. To admit to ourselves at some 376 00:20:36,000 --> 00:20:39,080 point that the technology of the models we use for non targeted 377 00:20:39,080 --> 00:20:43,320 attacks, like the Oracle Weblogic 2019 CV that's 378 00:20:43,320 --> 00:20:47,080 constantly being exploited is not the same model and 379 00:20:47,080 --> 00:20:49,880 technology that you would use to deal with targeted attacks. 380 00:20:51,160 --> 00:20:53,920 Right. And I think that that was not 381 00:20:53,920 --> 00:21:00,560 obvious in 20/15/2019. But to take it back to the 382 00:21:00,560 --> 00:21:02,440 beginning of those organizations that are like, I'm a special 383 00:21:02,440 --> 00:21:04,880 snowflake, my threat vector is different. 384 00:21:05,120 --> 00:21:08,000 Some of them are right. They are special and they have 385 00:21:08,000 --> 00:21:10,240 targeted attacks that might get triggered against them. 386 00:21:11,240 --> 00:21:13,440 That's not going to come up in the data sets that we used to 387 00:21:14,160 --> 00:21:16,440 triage the generalized vulnerabilities that are used 388 00:21:16,440 --> 00:21:18,400 against everybody. Yeah. 389 00:21:18,480 --> 00:21:20,600 And I mean, nor should it really. 390 00:21:20,600 --> 00:21:23,960 I mean, there's kind of outliers or I mean, they're super cool to 391 00:21:23,960 --> 00:21:26,480 look at those exploit chains and all that kind of stuff. 392 00:21:26,480 --> 00:21:29,240 And like, think about the amount of effort that went into that 393 00:21:29,240 --> 00:21:34,200 and that sort of thing and how they're, you know, sort of 394 00:21:34,520 --> 00:21:38,840 driving innovation on the defensive side too, how Apple 395 00:21:38,840 --> 00:21:41,960 and Google and Microsoft and everybody else are working to 396 00:21:41,960 --> 00:21:44,640 break those exploit classes or bug classes. 397 00:21:44,640 --> 00:21:49,960 But yeah, I mean, one crazy snowflake iOS bug shouldn't, 398 00:21:50,960 --> 00:21:55,000 shouldn't, you know, drive a bunch of, I mean, they drive 399 00:21:55,000 --> 00:21:57,840 headlines. That's, you know, we know that. 400 00:21:59,920 --> 00:22:05,560 So the the initial kind of idea behind this this podcast for the 401 00:22:05,560 --> 00:22:09,920 three of us was you guys reached out about the EPSS being 402 00:22:09,920 --> 00:22:15,320 mentioned in anthropics documentation. 403 00:22:15,320 --> 00:22:18,640 I guess documentation is not the right word, but blog post about 404 00:22:18,760 --> 00:22:24,080 the Mythos release and I kind of missed it on first glance 405 00:22:24,080 --> 00:22:28,800 because right above that was a piece of advice that said, I 406 00:22:28,800 --> 00:22:33,320 believe patch everything that's in the Kev, right, Right. 407 00:22:33,320 --> 00:22:37,680 Which is like, have you tried not having vulnerabilities 408 00:22:37,680 --> 00:22:43,040 essentially? Like, yeah, your doctor telling 409 00:22:43,040 --> 00:22:44,600 you, have you tried not being sick? 410 00:22:44,600 --> 00:22:46,920 Sure, yeah, that's excellent advice. 411 00:22:46,960 --> 00:22:51,840 But right below that was essentially pick a threshold, 412 00:22:51,840 --> 00:22:57,600 like determine your threshold for tolerance in EPSS and patch 413 00:22:57,600 --> 00:23:03,520 everything above that. Like, you know, so how did that 414 00:23:03,520 --> 00:23:04,920 come? Were you guys aware that that 415 00:23:04,920 --> 00:23:08,160 was going to be in there? Or was it just sort of like, oh 416 00:23:08,160 --> 00:23:12,840 shit, look at this? Yeah, and it's amazing that they 417 00:23:12,840 --> 00:23:16,200 use the language of threshold because one, it's the right way 418 00:23:16,200 --> 00:23:20,120 to think about any model, but 2, they would know they build 419 00:23:20,120 --> 00:23:24,920 models, but two, that question of what is the right threshold 420 00:23:24,920 --> 00:23:29,720 for me has come up on the EPS CIG no less than 100 times over 421 00:23:29,720 --> 00:23:31,080 the past five years. Oh, I'm sure. 422 00:23:31,560 --> 00:23:36,400 Yeah, and I, I keep trying to punt it because of course I have 423 00:23:36,400 --> 00:23:38,840 no idea. You know, it's all about your 424 00:23:38,840 --> 00:23:41,240 risk tolerance, the security controls everything. 425 00:23:41,240 --> 00:23:44,560 You know, like there's so many factors that go into that 426 00:23:44,560 --> 00:23:48,560 threshold setting for for tolerance and, and early on 427 00:23:48,560 --> 00:23:50,440 people were like, hey, you're producing the score. 428 00:23:50,440 --> 00:23:52,680 Where's critical, high, medium and low. 429 00:23:52,680 --> 00:23:55,480 We want to, we want to know where critical is and, and high. 430 00:23:55,480 --> 00:23:58,360 And, and I kept on saying it's not like we're producing the 431 00:23:58,360 --> 00:24:01,960 score, figure it out like, you know, and it's just the other 432 00:24:01,960 --> 00:24:04,080 thing, like if you think about risk, you know, the, the 433 00:24:04,200 --> 00:24:08,240 pressure, the, the exploitation activity coming in and then your 434 00:24:08,240 --> 00:24:10,960 ability to resist it and then the outcome on the other side. 435 00:24:10,960 --> 00:24:14,280 We're only measuring one of those three things, you know, it 436 00:24:14,280 --> 00:24:17,680 is not the whole risk picture. And so for us to put a 437 00:24:17,680 --> 00:24:20,320 threshold, like we're very concerned. 438 00:24:20,320 --> 00:24:22,880 I'm very concerned that if we say here's critical, then people 439 00:24:22,960 --> 00:24:25,120 are like, great, we're going to do this threshold and above, 440 00:24:25,520 --> 00:24:27,560 then we're done. I'm like, that's not how it 441 00:24:27,560 --> 00:24:30,880 works, you know, because I said like 2% seems low, but you're 442 00:24:30,880 --> 00:24:33,120 still going to see some exploitation activity on the low 443 00:24:33,120 --> 00:24:37,080 end, you know, so I can't tell people don't, you know, don't 444 00:24:37,080 --> 00:24:40,120 patch under this or something. So I've always avoided that. 445 00:24:40,520 --> 00:24:43,680 I want to grandstand about criticals and highs in general a 446 00:24:43,680 --> 00:24:46,680 little bit. So this is a philosophy. 447 00:24:46,680 --> 00:24:49,960 We have an empirical where we don't want to tell people what 448 00:24:49,960 --> 00:24:52,040 higher critical is. We want to give them the tools 449 00:24:52,040 --> 00:24:53,400 to determine that for themselves. 450 00:24:54,280 --> 00:24:55,680 But what Jay saying is absolutely correct. 451 00:24:56,760 --> 00:25:00,000 There's for the typical large organization that's using 10 452 00:25:00,000 --> 00:25:01,760 different security ventures. They have 10 different 453 00:25:01,760 --> 00:25:04,640 definitions of what high and critical is, some of which are 454 00:25:04,640 --> 00:25:07,680 correct and maybe that thing is critical, some of which are not 455 00:25:07,680 --> 00:25:10,280 applicable in that particular instance, some of which have 456 00:25:10,280 --> 00:25:11,920 mitigating controls already in place. 457 00:25:11,920 --> 00:25:15,160 So it was a critical a priori. But isn't that organization? 458 00:25:15,520 --> 00:25:19,680 And so I have met many a person who spends their day-to-day 459 00:25:20,720 --> 00:25:24,880 explaining to a regulator or an IT person why something that is 460 00:25:24,880 --> 00:25:28,960 labeled as read high or critical is not read high or critical for 461 00:25:28,960 --> 00:25:30,160 them. And vice versa, something that 462 00:25:30,160 --> 00:25:32,400 is read labeled low is actually super important for that 463 00:25:32,400 --> 00:25:36,080 organization. Adding another layer of 464 00:25:36,080 --> 00:25:38,360 criticality in the exploit prediction scoring system. 465 00:25:38,360 --> 00:25:41,520 We'll kind of do away with the entire concept of probability. 466 00:25:41,760 --> 00:25:44,360 We want to give people the tools to measure likelihood. 467 00:25:44,360 --> 00:25:47,000 That's EPSS. You know, a little bit of an 468 00:25:47,000 --> 00:25:49,280 empirical plug. We've got models that do take 469 00:25:49,280 --> 00:25:52,200 into account mitigating controls and we can build specific ones 470 00:25:52,200 --> 00:25:54,720 for customers whether more of the risk is measured. 471 00:25:55,320 --> 00:25:58,840 But even in those instances, it would be disingenuous for us to 472 00:25:58,840 --> 00:26:01,440 say we understand your organization's risk tolerance 473 00:26:01,440 --> 00:26:04,160 and posture and capacity. And so we're going to tell you 474 00:26:04,160 --> 00:26:07,920 to patch everything above this. Now we want you to select and 475 00:26:07,920 --> 00:26:12,160 have a discussion at maybe even the board level about what is 476 00:26:12,160 --> 00:26:13,960 our risk tolerance? How much risk do we want to 477 00:26:13,960 --> 00:26:16,960 remove from the organization? How much capacity do we have to 478 00:26:16,960 --> 00:26:19,920 do that? Can we really patch 5% of our 479 00:26:19,920 --> 00:26:22,080 vulnerabilities in any given month? 480 00:26:22,960 --> 00:26:25,160 That is a, that is a measurable thing, right? 481 00:26:25,160 --> 00:26:28,600 Some organizations can that have insane budgets, others can't. 482 00:26:29,600 --> 00:26:32,520 Also 5% is going to change dramatically over the next two 483 00:26:32,520 --> 00:26:34,520 years, but that means the raw numbers. 484 00:26:35,280 --> 00:26:38,600 So just taking a step back from the the entire security industry 485 00:26:38,600 --> 00:26:44,240 has been kind of saying, I found 100,000 things and 30% of them 486 00:26:44,240 --> 00:26:47,040 are high. That determination is actually 487 00:26:47,040 --> 00:26:51,040 something that enterprises and users have to be making rather 488 00:26:51,040 --> 00:26:53,840 than vendors. But I I completely understand 489 00:26:53,840 --> 00:26:56,240 the impetus for it, right? Like, you do need to filter the 490 00:26:56,240 --> 00:26:57,800 signal from the noise in some way. 491 00:26:58,160 --> 00:26:59,920 It's just that maybe the filter's in the wrong place. 492 00:27:01,000 --> 00:27:04,280 Yeah, that's a great point. I, I do think, you know, going 493 00:27:04,280 --> 00:27:07,680 back to your point about a lot of organizations thinking 494 00:27:07,680 --> 00:27:10,600 they're special snowflakes, like in some way every organization 495 00:27:10,600 --> 00:27:13,280 kind of is like they, you know, they have their own risk 496 00:27:13,280 --> 00:27:16,400 tolerance, they have their own budget constraints, they have 497 00:27:16,400 --> 00:27:18,560 their own, you know, resource constraints. 498 00:27:18,640 --> 00:27:22,600 And in terms of personnel and like time to get this stuff done 499 00:27:22,840 --> 00:27:27,120 even without, you know, frontier models dumping a giant pile of 500 00:27:27,120 --> 00:27:32,200 bugs on them in the near future. Just, you know, Patch Tuesday, 501 00:27:32,200 --> 00:27:36,480 Oracle, you know, Chrome, all that kind of stuff, not to 502 00:27:36,480 --> 00:27:39,320 mention whatever other, you know, software they're using. 503 00:27:40,080 --> 00:27:43,320 Even well resourced security teams have a tough time keeping 504 00:27:43,320 --> 00:27:47,360 up with that stuff, you know, and so they do use, you know, we 505 00:27:47,360 --> 00:27:50,960 got to get everything that's critical or high slash important 506 00:27:51,080 --> 00:27:52,840 that that's what we're doing this month. 507 00:27:53,040 --> 00:27:57,680 Like we got to get those done by May 15th because that's what you 508 00:27:57,680 --> 00:28:00,800 know, is in my KPIs. Like, you know, like there's 509 00:28:00,800 --> 00:28:03,360 real world shit that determines that stuff. 510 00:28:04,080 --> 00:28:06,440 Right. Oh yeah, the SLA question is an 511 00:28:06,440 --> 00:28:11,960 entirely different one too. If I say I fix all criticals 512 00:28:11,960 --> 00:28:15,600 within seven days, that seven day period is also a risk and 513 00:28:15,600 --> 00:28:18,720 capacity statement that I'm making about those criticals. 514 00:28:20,560 --> 00:28:24,040 But the definition of the critical doesn't include that 515 00:28:24,040 --> 00:28:27,880 seven day period. So you have two knobs to turn 516 00:28:27,880 --> 00:28:29,840 right? One is how quickly do I do 517 00:28:29,840 --> 00:28:33,800 things in this bucket? The second is what what is 518 00:28:33,800 --> 00:28:35,040 critical? What is that bucket? 519 00:28:35,040 --> 00:28:38,640 How small is it? Because if it's really just the 520 00:28:38,640 --> 00:28:42,560 50 CVS that you know could really 'cause serious 521 00:28:42,560 --> 00:28:44,840 exploitation and are being exploited elsewhere in your 522 00:28:44,840 --> 00:28:47,440 environment, maybe you can't get those done in three days. 523 00:28:48,120 --> 00:28:50,600 But if it's 5000, then you can't. 524 00:28:50,680 --> 00:28:54,280 And that three day SLA is actually just you nicely lying 525 00:28:54,280 --> 00:28:57,840 to yourself and others. Yeah, well, we all have those 526 00:28:57,840 --> 00:29:00,120 little lies we tell ourselves. Yeah. 527 00:29:00,280 --> 00:29:01,880 Some are about bugs, some are about the. 528 00:29:02,440 --> 00:29:04,560 Yeah, our sleep schedules and our diets. 529 00:29:04,560 --> 00:29:08,200 But yeah. So what are you guys 530 00:29:08,200 --> 00:29:16,320 anticipating with, like Mythos and these other Frontier models? 531 00:29:16,320 --> 00:29:20,080 I mean, there's been a lot of like there was a media backlash, 532 00:29:20,120 --> 00:29:22,360 you know, Mythos specifically, then there was kind of backlash 533 00:29:22,360 --> 00:29:24,600 to that of like, no, this is really cool. 534 00:29:24,600 --> 00:29:27,600 There's a lot of like interesting stuff coming out of 535 00:29:27,600 --> 00:29:30,160 this. You know, I, I saw a blog post 536 00:29:30,160 --> 00:29:34,320 from the folks at Mozilla talking about like the, you 537 00:29:35,000 --> 00:29:38,480 know, I think it was 200 and something bugs that it was found 538 00:29:38,480 --> 00:29:40,840 in, in Firefox. Maybe I'm overstating it, but it 539 00:29:40,840 --> 00:29:45,040 was quite a lot. And how, you know, kind of game 540 00:29:45,040 --> 00:29:48,960 changing that was for them. I don't really know where this 541 00:29:48,960 --> 00:29:51,880 is going, but it seems like if we have this conversation, like 542 00:29:51,880 --> 00:29:55,160 even three months from now, we might be looking at a completely 543 00:29:55,160 --> 00:29:58,680 different landscape, you know? Yeah. 544 00:29:59,240 --> 00:30:03,800 And for us like I, I've thought a lot about it and actually it 545 00:30:03,800 --> 00:30:09,040 doesn't matter what I think for for the modeling because the 546 00:30:09,040 --> 00:30:14,040 modeling is is self learning. You know, it like EPSS right 547 00:30:14,040 --> 00:30:18,600 now, I don't think I've looked at the data in like a month. 548 00:30:18,680 --> 00:30:21,920 You know, the underlying data, it is going out and grabbing and 549 00:30:21,920 --> 00:30:25,000 collecting all the stuff and producing the scores and it's 550 00:30:25,000 --> 00:30:28,200 reacting to what's going on and I'm not there doing anything 551 00:30:28,200 --> 00:30:31,320 with it. And so I think what the only 552 00:30:31,320 --> 00:30:34,800 thing that I can say that absolutely will happen is I 553 00:30:34,800 --> 00:30:37,040 think we're dealing with it going to be dealing with a 554 00:30:37,040 --> 00:30:40,200 different system. You know, like our system right 555 00:30:40,200 --> 00:30:43,600 now is, was the one that we've been working with for a long 556 00:30:43,600 --> 00:30:46,200 time is that the attackers have finite resources. 557 00:30:46,720 --> 00:30:50,160 The, you know, their attention is by far the most valuable 558 00:30:50,160 --> 00:30:53,280 resource. And that that balance is going 559 00:30:53,280 --> 00:30:56,320 to be shifting and it already is shifting. 560 00:30:56,600 --> 00:30:59,760 And so the, the rate of exploit, the things that are targeted, I 561 00:30:59,760 --> 00:31:01,480 think all of that is going to be shifting. 562 00:31:01,920 --> 00:31:04,920 And I think the, the probably the hardest thing is for people 563 00:31:04,920 --> 00:31:07,480 who've been in the industry for like 20 years or something going 564 00:31:07,480 --> 00:31:10,200 to be like, no, I know what how vulnerabilities are picked. 565 00:31:10,200 --> 00:31:11,760 This is a bad one. And that's a good one. 566 00:31:11,760 --> 00:31:14,760 You know, like that's going to be really hard for them to shift 567 00:31:14,760 --> 00:31:17,680 their perspective, but for the model, it's going to be like, 568 00:31:17,720 --> 00:31:20,080 oh, I'm seeing this new activity or we're seeing it over here now 569 00:31:20,080 --> 00:31:22,720 we're seeing it over here, You know, so we're going to have to 570 00:31:22,720 --> 00:31:24,680 retrain more often. We're going to have to keep up 571 00:31:24,680 --> 00:31:27,400 on the model, but largely that model is going to react. 572 00:31:28,160 --> 00:31:30,920 So, so the retraining part is where I was going with this. 573 00:31:32,840 --> 00:31:37,480 When we trained DPSS Version 4, which we released in March of 574 00:31:37,480 --> 00:31:42,640 last year, it had been two years since the public UPS had been 575 00:31:42,640 --> 00:31:44,200 retrained. So that's the gap between 3:00 576 00:31:44,200 --> 00:31:46,480 and 4:00, just about two years, OK. 577 00:31:46,480 --> 00:31:50,120 We started measuring model performance decay. 578 00:31:50,360 --> 00:31:53,080 Well, as the first thing that we did before we started trading 579 00:31:53,200 --> 00:31:55,880 version four, we had a lot more resources for Version 4. 580 00:31:55,880 --> 00:31:57,400 We'd raise the seed round of funding. 581 00:31:57,400 --> 00:31:58,880 We had some data infrastructure there. 582 00:31:59,200 --> 00:32:01,240 So we're like, OK, how much worse did it get over 2 years? 583 00:32:01,240 --> 00:32:04,040 And the answer was on median, something like 15%. 584 00:32:04,120 --> 00:32:08,200 Jay, correct me if I'm making numbers up something like that 585 00:32:08,800 --> 00:32:11,160 over 2 year decay, that's actually pretty good for any 586 00:32:11,160 --> 00:32:13,120 deterministic model. I was gonna say like what it 587 00:32:13,120 --> 00:32:17,680 what would you have expected? I honestly so, so a couple of 588 00:32:17,680 --> 00:32:20,680 data points is that 48,000 vulnerabilities were released 589 00:32:20,680 --> 00:32:24,400 this year. In 2021 when EPSL started there 590 00:32:24,400 --> 00:32:26,680 I think were 18,000 vulnerabilities were released. 591 00:32:26,680 --> 00:32:29,120 Yeah. So the growth is already like 592 00:32:29,160 --> 00:32:31,480 exponential, right? Just the exponent is not the 593 00:32:31,480 --> 00:32:33,560 same as it will be with these Frontier models. 594 00:32:33,560 --> 00:32:36,040 Yeah, yeah. So I would have expected 595 00:32:36,040 --> 00:32:38,400 actually more of a decay because the system had changed and the 596 00:32:38,400 --> 00:32:41,440 volumes have changed, but OK. What's interesting is that the 597 00:32:41,440 --> 00:32:43,760 vulnerability and by the way, way more exploits too. 598 00:32:44,280 --> 00:32:46,680 Exploits used to be published and exploit DB and Metasploit. 599 00:32:46,680 --> 00:32:49,680 Now like overwhelmingly they're in GitHub randomly all over the 600 00:32:49,680 --> 00:32:53,360 place. So all of those things are 601 00:32:53,360 --> 00:32:55,720 growing exponentially. But the vulnerabilities that are 602 00:32:55,720 --> 00:32:59,040 being exploited, I think last year we tracked like 900 some 603 00:32:59,800 --> 00:33:04,800 and maybe the maybe 1100 or had active exhibition that we're not 604 00:33:04,800 --> 00:33:06,000 new. Yeah, yeah. 605 00:33:06,000 --> 00:33:08,640 Overall, there's like 18,000 that have been and are being 606 00:33:08,640 --> 00:33:11,280 exploited. But of those 48,000 that were 607 00:33:11,280 --> 00:33:14,080 released that year, only about 1000 had exploitation activity. 608 00:33:14,440 --> 00:33:18,600 OK, yeah, yeah. So I think what we're gonna have 609 00:33:18,600 --> 00:33:22,680 to do and we are currently in the process of training at EPS 610 00:33:22,680 --> 00:33:26,280 version 5, which will account for some of this shift change. 611 00:33:27,040 --> 00:33:30,120 We'll make an announcement about it soon and then release it 612 00:33:30,240 --> 00:33:33,320 shortly thereafter after after sig, sig testing. 613 00:33:33,800 --> 00:33:35,960 But we're going to have to start retraining the model more often 614 00:33:35,960 --> 00:33:39,160 because the decay will probably accelerate until we get to 615 00:33:39,160 --> 00:33:41,800 another steady state where maybe there's some limit to the the 616 00:33:41,800 --> 00:33:43,760 frontier models performance. Yeah. 617 00:33:43,960 --> 00:33:46,840 And before the decay was basically just, you know, the 618 00:33:47,000 --> 00:33:49,440 the market is shifting and it shifts slowly and that's what 619 00:33:49,440 --> 00:33:51,920 the decay is. But now I think that the 620 00:33:51,920 --> 00:33:54,600 underlying system is going to be shifting quicker. 621 00:33:54,720 --> 00:33:57,240 And so that's why that decay I think is going to be a lot more 622 00:33:57,240 --> 00:33:58,800 prominent. So retraining off it was going 623 00:33:58,800 --> 00:34:03,880 to be our key to dealing with the next generation of threats. 624 00:34:04,480 --> 00:34:10,040 Yeah, that next generation, I mean, it's, I don't know, the 625 00:34:10,040 --> 00:34:13,719 next like year to 18 months are going to be really interesting 626 00:34:13,719 --> 00:34:16,800 to watch. I mean, you mentioned that, you 627 00:34:16,800 --> 00:34:22,760 know, we used to think like, OK, attacker attention or time was, 628 00:34:22,800 --> 00:34:25,920 you know, a bounded resource that or bounding resource that 629 00:34:25,920 --> 00:34:28,239 they couldn't you can't really make more time. 630 00:34:28,239 --> 00:34:32,080 You can't make more of yourself. But literally they can now they 631 00:34:32,080 --> 00:34:36,080 can sort of, you know, multiply themselves using AI. 632 00:34:36,080 --> 00:34:41,840 And on the on the defensive side, the scarcity of elite 633 00:34:41,920 --> 00:34:45,360 vulnerability research talent was a real problem too. 634 00:34:45,360 --> 00:34:49,239 You know, that there was a a very finite pool of folks during 635 00:34:49,239 --> 00:34:52,840 doing that top level research, you know, here in the US and 636 00:34:52,840 --> 00:34:56,960 around the world. And now it's just like, I don't 637 00:34:56,960 --> 00:34:59,840 know, do you have enough money to feed into the AI machine? 638 00:35:00,760 --> 00:35:02,480 How many bugs do you want to find? 639 00:35:02,640 --> 00:35:05,080 Like so, money is a finite resource. 640 00:35:05,080 --> 00:35:06,200 Sometimes. Not always. 641 00:35:06,480 --> 00:35:07,720 Sometimes. Yeah. 642 00:35:08,800 --> 00:35:11,920 So I think this brings us to the point of why did Anthropic 643 00:35:11,920 --> 00:35:15,800 recommend a deterministic predictive model that's purpose 644 00:35:15,800 --> 00:35:19,720 built for one task and only does one thing when they literally 645 00:35:19,720 --> 00:35:21,480 build a model. They can do just about anything 646 00:35:21,480 --> 00:35:25,560 in a general purpose model. And the reason they recommended 647 00:35:25,560 --> 00:35:29,840 EPSS is it is a model that can measure that difference in 648 00:35:29,840 --> 00:35:34,560 attacker defender performance. And it is asymmetrically cheaper 649 00:35:34,560 --> 00:35:37,240 for the defender to run a predictive purpose build model 650 00:35:37,240 --> 00:35:42,360 for this task than it is for them to query a Jedi model every 651 00:35:42,440 --> 00:35:44,640 30 minutes to figure out which vulnerability they should focus 652 00:35:44,640 --> 00:35:47,480 on next. So as defenders, we have some of 653 00:35:47,480 --> 00:35:50,680 these advantages where we can have sensor networks that give 654 00:35:50,680 --> 00:35:52,960 us real time telemetry about what attackers are doing and 655 00:35:52,960 --> 00:35:56,120 then use them feed small models that are super efficient and 656 00:35:56,120 --> 00:36:00,200 super cost effective. So they might have infinite 657 00:36:00,200 --> 00:36:02,400 attention, but they don't have infinite money. 658 00:36:02,480 --> 00:36:04,720 And creating that kind of financial asymmetry is also 659 00:36:05,080 --> 00:36:07,960 useful. Yeah, that's a great point. 660 00:36:08,560 --> 00:36:14,160 I mean, yeah, the idea of making attacks more expensive in terms 661 00:36:14,160 --> 00:36:17,200 of money and time for attackers is always good. 662 00:36:17,200 --> 00:36:24,200 But we don't, I guess we have to assume that the apex adversaries 663 00:36:24,200 --> 00:36:27,160 that we're worried about all the time in terms of like state 664 00:36:27,160 --> 00:36:31,600 actors from China, North Korea and Russia are doing exactly 665 00:36:32,120 --> 00:36:36,200 the, you know, have their own labs doing this kind of work as 666 00:36:36,200 --> 00:36:37,400 well. Yeah. 667 00:36:37,760 --> 00:36:41,160 You know, the bug finding and exploit development and all that 668 00:36:41,160 --> 00:36:45,400 kind of stuff. So yeah, yeah, I don't. 669 00:36:45,400 --> 00:36:49,480 Know, and the another thing I'll say like they're really high 670 00:36:49,480 --> 00:36:52,720 profile targets that are typically at the top of the list 671 00:36:52,720 --> 00:36:56,400 for state sponsored activity should be very afraid. 672 00:36:56,400 --> 00:36:59,360 And I'm sure that they are. But I think for those not at the 673 00:36:59,360 --> 00:37:03,280 top of the list for that group, I do want to point out that like 674 00:37:03,840 --> 00:37:08,600 the the number of targets at those places generally was not 675 00:37:08,600 --> 00:37:12,160 the limiting factor, right? I mean, like they had 676 00:37:12,160 --> 00:37:14,160 vulnerabilities. They have vulnerabilities, they 677 00:37:14,160 --> 00:37:15,560 will have vulnerabilities tomorrow. 678 00:37:15,560 --> 00:37:18,840 You know, that that is not the limiting factor for most of the 679 00:37:18,840 --> 00:37:23,440 attacks. And so moving forward, there's 680 00:37:23,480 --> 00:37:26,440 going to be even more, you know, it's not going to change a ton. 681 00:37:26,440 --> 00:37:28,760 So they're they're going to probably have to update and 682 00:37:29,000 --> 00:37:31,600 react a little quicker, but hopefully we can, you know, get 683 00:37:31,600 --> 00:37:34,120 the prioritization better so that they can react smarter. 684 00:37:34,960 --> 00:37:37,880 So if Ed was on this podcast, he'd talk about the Mendoza 685 00:37:37,880 --> 00:37:39,000 Line. Yeah. 686 00:37:40,600 --> 00:37:43,000 The Mendoza Line for security is essentially you want to be 687 00:37:43,000 --> 00:37:45,320 better than an amateur and an amateur used to be. 688 00:37:45,680 --> 00:37:48,440 Don't have anything that Metasploit can reach across your 689 00:37:48,440 --> 00:37:54,600 networks, across that data set of organizations that we worked 690 00:37:54,600 --> 00:37:57,240 with for the prioritization and prediction reports, I don't 691 00:37:57,240 --> 00:38:00,000 think there were any that didn't have at least one Metasploit 692 00:38:00,000 --> 00:38:02,200 exploitable vulnerabilities somewhere across their networks. 693 00:38:03,280 --> 00:38:06,680 This was 2021-2022. The thing that's really 694 00:38:06,680 --> 00:38:09,360 different is that the Mendoza line is shifting. 695 00:38:09,680 --> 00:38:11,960 It's no longer have nothing that's exploitable by 696 00:38:11,960 --> 00:38:14,400 Metasploit. It's now have nothing that is 697 00:38:14,400 --> 00:38:18,400 easily exploitable by an amateur wielding a cheap LLM. 698 00:38:19,840 --> 00:38:24,960 So that line is shifting, but the the problem set and like the 699 00:38:24,960 --> 00:38:26,680 the constraints that organization has to deal with it 700 00:38:26,680 --> 00:38:29,880 really aren't because they were never above that Mendoza line 701 00:38:29,880 --> 00:38:31,720 that would existed in the past anyways. 702 00:38:31,720 --> 00:38:34,600 They had to manage to a different line, but the 703 00:38:34,600 --> 00:38:36,520 management process is still very much the same. 704 00:38:37,560 --> 00:38:38,960 Right. Yeah, In the past you were 705 00:38:38,960 --> 00:38:42,720 worried about, Yeah, somebody that got ahold of Metasploit 706 00:38:42,720 --> 00:38:46,320 and, you know, had some idea how to use it. 707 00:38:47,800 --> 00:38:54,440 One of the other things that kind of worries me now is these 708 00:38:54,440 --> 00:39:00,320 LLMS finding bugs in, you know, cloud platforms or SAS platforms 709 00:39:00,320 --> 00:39:03,200 that are used by everybody and their brother. 710 00:39:04,760 --> 00:39:09,520 And, you know, some universal bug like that that you know, was 711 00:39:09,520 --> 00:39:12,320 found by an adversary and they get an exploit for it. 712 00:39:12,320 --> 00:39:16,680 Like there was a stupid C panel bug this this week that like 713 00:39:16,680 --> 00:39:21,440 effects 10s of millions of like shared hosting instance is and 714 00:39:21,440 --> 00:39:25,240 like the average enterprise that is affected and can't fix that, 715 00:39:25,240 --> 00:39:27,040 they need their hosting provider to fix it. 716 00:39:27,400 --> 00:39:31,560 So, you know, things like that, you know, those happen every 717 00:39:31,560 --> 00:39:34,760 once in a while, but I worry that those could be happening 718 00:39:34,760 --> 00:39:38,920 more and more often with, you know, mythos or whatever other 719 00:39:38,920 --> 00:39:40,720 frontier model decides to go and look. 720 00:39:41,880 --> 00:39:45,240 Yeah, I agree and. Those are the things that 721 00:39:45,240 --> 00:39:47,640 enterprise, you can't really plan for those, you know. 722 00:39:48,240 --> 00:39:49,560 There's a whole class of those, right? 723 00:39:49,560 --> 00:39:54,120 Like it used to be the CV. It was something that the 724 00:39:54,120 --> 00:39:58,560 impetus was on the user of the software to remediate and patch 725 00:39:58,560 --> 00:40:01,040 the system. I'm writing Adobe Reader 91. 726 00:40:01,040 --> 00:40:02,800 I got to apply the patch on my endpoint. 727 00:40:03,960 --> 00:40:07,360 Increasingly, and this is anecdotal from just me reading 728 00:40:07,360 --> 00:40:13,920 CVS day in and day out, there are CVS in Salesforce or some 729 00:40:13,920 --> 00:40:18,720 cloud hosting provider that are there almost as like a record, 730 00:40:19,160 --> 00:40:20,280 but I can't do anything about it. 731 00:40:21,000 --> 00:40:26,160 Nobody is using that software. The ownership of the remediation 732 00:40:26,160 --> 00:40:30,200 is on the provider of the software and that is not the 733 00:40:30,200 --> 00:40:33,800 same CVE as a CVE on a device where I have access and I'm 734 00:40:33,800 --> 00:40:35,280 responsible for that. Right. 735 00:40:35,960 --> 00:40:39,480 Yeah, those are those are the ones that I always, they catch 736 00:40:39,520 --> 00:40:41,400 my attention. You're like, those could be 737 00:40:42,520 --> 00:40:44,480 really bad. You know, when you sort of go 738 00:40:44,480 --> 00:40:47,920 and look at how many people have that, you know, that Salesforce, 739 00:40:48,240 --> 00:40:50,600 you know, they're using that cloud service or whatever it 740 00:40:50,600 --> 00:40:54,160 happens to be. Yeah, but. 741 00:40:54,680 --> 00:40:56,800 At the same time, for the average enterprise, it's like, 742 00:40:56,960 --> 00:40:59,800 would I rather have my security team of three people fixing this 743 00:40:59,800 --> 00:41:04,520 or Amazon security fixing? Yeah, no, that's, I mean, that's 744 00:41:04,520 --> 00:41:07,520 one of the reasons you offload that responsibility to those 745 00:41:07,680 --> 00:41:10,480 folks, right? Like, yeah, Amazon and Microsoft 746 00:41:10,480 --> 00:41:12,760 have all the security talent in the world and all the money in 747 00:41:12,760 --> 00:41:15,120 the world. So hopefully they're doing, you 748 00:41:15,160 --> 00:41:16,560 know, they're spending our money wisely. 749 00:41:21,000 --> 00:41:26,960 Vulnerability tax. Yeah, yes, it comes for us all. 750 00:41:27,080 --> 00:41:29,440 Yeah. All right, Guy. 751 00:41:29,480 --> 00:41:33,920 Anything we forgot to hit, I think we we got a lot of we got 752 00:41:33,920 --> 00:41:37,480 a lot of topics in there. Yeah, everything, I think. 753 00:41:38,440 --> 00:41:41,680 I I I really like anthropics recommendation. 754 00:41:41,760 --> 00:41:45,560 I think it is directionally correct in the sense that 755 00:41:46,400 --> 00:41:49,000 remediate the things you know have active exploitation that 756 00:41:49,000 --> 00:41:51,560 are relevant to you. You can't remediate the entire 757 00:41:51,560 --> 00:41:53,680 capitalist because you don't have every vulnerability on that 758 00:41:53,680 --> 00:41:57,800 capitalist. I think the caveat that Jay's 759 00:41:57,800 --> 00:41:59,720 research with the RE exploitation shows and that 760 00:41:59,840 --> 00:42:04,240 folks can read on the DBIR in the coming weeks is that just 761 00:42:04,240 --> 00:42:06,800 being on the capitalist is not right. 762 00:42:06,800 --> 00:42:09,840 It is an indicator of risk, but it's not the only one, right? 763 00:42:09,920 --> 00:42:12,440 So recency of that exploitation really matters. 764 00:42:12,520 --> 00:42:14,760 I think we're trying to challenge the industry to think 765 00:42:14,760 --> 00:42:19,760 about the acronym Kev as not a binary but a continuous 766 00:42:19,760 --> 00:42:23,040 variable, right? Was it exploited yesterday? 767 00:42:23,040 --> 00:42:25,600 Is it still being exploited? What is it patchy? 768 00:42:25,600 --> 00:42:28,480 Is it bursty? And then the second 769 00:42:28,480 --> 00:42:32,120 recommendation is use a cheap, small purpose built 770 00:42:32,120 --> 00:42:34,760 probabilistic model to forecast what's next. 771 00:42:35,760 --> 00:42:38,680 And that recommendation coming from an LLM lab is really 772 00:42:38,680 --> 00:42:41,040 exciting to me. Like, you know, I don't ask 773 00:42:41,040 --> 00:42:43,360 Anthropic what the weather is. I still go to weather.com. 774 00:42:44,280 --> 00:42:47,400 There's a reason for that. I, I had a really funny thought 775 00:42:47,400 --> 00:42:50,840 and I was wondering like, did they folks at Anthropic like use 776 00:42:50,840 --> 00:42:53,800 Claude to generate that blog and Claude just like shoved it in 777 00:42:53,800 --> 00:42:55,600 there and they never reviewed it and they're like, I whatever 778 00:42:55,600 --> 00:42:57,840 publish, you know, that seems right. 779 00:42:58,400 --> 00:43:00,160 It does seem right. Yeah, it'll be. 780 00:43:00,160 --> 00:43:02,640 Poetic. I think the chances of that are 781 00:43:02,640 --> 00:43:05,120 quite high, yeah. I don't remember if there were 782 00:43:05,120 --> 00:43:07,640 actual names on that blog, but it doesn't matter anyway. 783 00:43:07,960 --> 00:43:09,720 Doesn't matter. Yeah, not at all. 784 00:43:09,720 --> 00:43:13,520 All right, right. Well, guys, thanks so much for 785 00:43:13,520 --> 00:43:16,440 for coming on and explaining the world of math to me. 786 00:43:16,520 --> 00:43:19,320 I really appreciate it. Yeah, great talking to you, 787 00:43:19,320 --> 00:43:21,000 Dennis. Yeah, great to see you guys. 788 00:43:21,480 --> 00:43:23,040 All right, take care all. Right. 789 00:43:23,160 --> 00:43:23,320 Bye bye.