1 00:00:11,480 --> 00:00:13,480 Hey everyone. Welcome back to the Decipher 2 00:00:13,480 --> 00:00:15,880 podcast. I'm Dennis Fisher here with my 3 00:00:15,880 --> 00:00:18,920 old friend Gary McGraw for the first time in a long time. 4 00:00:18,920 --> 00:00:20,760 How are you, Gary? Hi, Dennis. 5 00:00:20,800 --> 00:00:22,120 Great to be here. I'm good. 6 00:00:22,720 --> 00:00:25,560 Yeah, you look good. You seem happy even though 7 00:00:25,560 --> 00:00:27,320 you're bad at retirement, as we've talked. 8 00:00:27,320 --> 00:00:30,520 About that, yeah. Very bad at retirement. 9 00:00:31,200 --> 00:00:33,320 You've tried several times. It doesn't seem to be working. 10 00:00:33,320 --> 00:00:38,320 But the good news for all of us is you're doing cool AI and 11 00:00:38,320 --> 00:00:43,000 machine learning security stuff. Yeah, it's really kind of fun to 12 00:00:43,000 --> 00:00:46,520 be wildly independent and. I used to be. 13 00:00:46,640 --> 00:00:50,800 Wildly independent but also had a corporation and now I have a 14 00:00:50,800 --> 00:00:53,080 non profit and it's even more fun. 15 00:00:54,840 --> 00:00:56,520 You were never. Yeah. 16 00:00:56,520 --> 00:00:59,800 You were the least corporate, corporate person I ever met. 17 00:01:00,040 --> 00:01:01,680 Yeah. Well, it was my corporation, 18 00:01:03,160 --> 00:01:07,040 sort of, until it got bought by Silicon Valley and then it got 19 00:01:07,200 --> 00:01:08,560 really boring fast. Wow. 20 00:01:09,480 --> 00:01:16,040 That does happen, yeah. So what you're doing now is the 21 00:01:16,040 --> 00:01:17,960 Berryville Institute of Machine Learning. 22 00:01:18,400 --> 00:01:20,080 You've been doing that for a few years. 23 00:01:20,120 --> 00:01:23,280 You and a a small group of other folks just doing like as you 24 00:01:23,280 --> 00:01:29,440 said, independent kind of focus on whatever you want to focus on 25 00:01:29,960 --> 00:01:33,280 security research about AI, machine learning and you guys 26 00:01:33,280 --> 00:01:34,800 have some new stuff out this week. 27 00:01:35,680 --> 00:01:38,560 Mostly we're focused on this though, instead of thinking 28 00:01:38,560 --> 00:01:44,200 about using machine learning or AI as the case may be for 29 00:01:44,200 --> 00:01:48,360 security, We're talking about the security of machine learning 30 00:01:48,360 --> 00:01:51,320 itself. So we're focused on, well, is 31 00:01:51,320 --> 00:01:54,400 that model secure? What does it mean for an AI 32 00:01:54,400 --> 00:01:57,880 model to be secure and so on? Yeah, yeah. 33 00:01:58,280 --> 00:02:00,240 And there's a lot of there's, I think there's a bunch of 34 00:02:00,240 --> 00:02:02,800 parameters and like kind of definitional stuff we should do 35 00:02:02,800 --> 00:02:05,800 at the beginning because you guys set this up in the research 36 00:02:05,800 --> 00:02:10,680 that you released this week. And you know, I think for folks 37 00:02:10,680 --> 00:02:15,640 like me and, you know, tech fluent people, but aren't in 38 00:02:15,640 --> 00:02:20,040 the, you know, we're not AI engineers or pH DS like you are, 39 00:02:20,360 --> 00:02:24,440 We essentially use AI and ML like interchangeably, like for 40 00:02:24,440 --> 00:02:29,880 the most part, you know, and you, you separate the terms in 41 00:02:29,880 --> 00:02:32,640 that in the paper that you released. 42 00:02:33,760 --> 00:02:37,600 And it's important to understand that like, what the general 43 00:02:37,600 --> 00:02:42,240 public thinks of is AI in some cases is machine learning. 44 00:02:42,240 --> 00:02:45,120 And you know. Well, I guess, I mean, guess 45 00:02:45,120 --> 00:02:47,920 what, whatever we say, none of that matters. 46 00:02:49,120 --> 00:02:53,760 We're going to call it AI. So I mean, the reason for that, 47 00:02:53,760 --> 00:02:55,880 that I'm a little bit of a stickler for that. 48 00:02:56,160 --> 00:02:59,560 There are two reasons. One is I actually studied 49 00:02:59,840 --> 00:03:05,360 cognitive science and AI when I was a a little tyke in the 90s. 50 00:03:05,600 --> 00:03:10,800 I was one of Doug Hofstetter's PhD students and so I worked on 51 00:03:10,840 --> 00:03:14,360 an AI program that was not a neural network, though I also 52 00:03:14,360 --> 00:03:16,720 wrote my first neural network in 1989. 53 00:03:17,120 --> 00:03:21,720 So you know, AI is a much larger field than just machine 54 00:03:21,720 --> 00:03:25,080 learning. But today basically, you know, 55 00:03:25,080 --> 00:03:30,360 people have Co opted the the phrase AI to mean large language 56 00:03:30,360 --> 00:03:33,840 model, which is also a subset of machine learning. 57 00:03:34,240 --> 00:03:36,920 So, you know, if we want to get all stickler about it, we can 58 00:03:36,920 --> 00:03:40,800 say it's a much larger field. But honestly the most important 59 00:03:40,800 --> 00:03:43,720 thing going on now is large language models and what we're 60 00:03:43,720 --> 00:03:47,920 going to do about them and for them when it comes to security. 61 00:03:47,960 --> 00:03:50,120 As you know, it's just complicated as heck. 62 00:03:51,360 --> 00:03:53,920 Yeah, it's really complicated and we've talked about this 63 00:03:53,920 --> 00:03:58,760 before, I think on the podcast and just in off the podcast is 64 00:03:59,240 --> 00:04:03,080 kind of the observability problem of these models where 65 00:04:04,040 --> 00:04:06,720 almost all of them are black boxes. 66 00:04:06,720 --> 00:04:12,080 We don't really have any insight into their training sets. 67 00:04:12,080 --> 00:04:15,160 We don't really know how they're trained, we don't know how 68 00:04:15,160 --> 00:04:19,120 they're built or maintained or what any of the security 69 00:04:19,120 --> 00:04:22,280 protocols around them are. Yep, sort of reminds me of 70 00:04:22,280 --> 00:04:25,880 software. Yeah, very much so. 71 00:04:26,160 --> 00:04:30,200 That was supposed to be. I I I agree with that 100%. 72 00:04:30,760 --> 00:04:35,360 In 1998, when we started working on software security, people 73 00:04:35,360 --> 00:04:38,440 were treating software applications as a black box. 74 00:04:38,800 --> 00:04:43,360 No doubt web happened and the idea was you just, you know, put 75 00:04:43,360 --> 00:04:45,320 some stuff in the front and see what happens. 76 00:04:46,240 --> 00:04:49,880 We changed the paradigm of software security in the last 30 77 00:04:49,880 --> 00:04:54,720 years by inventing things like threat modelling and static 78 00:04:54,720 --> 00:05:00,240 analysis tools and stuff like that, and we kind of need to do 79 00:05:00,240 --> 00:05:04,840 the same thing for AI. It reminds me, like these days 80 00:05:04,840 --> 00:05:08,960 in AI or machine learning security remind me of the early 81 00:05:08,960 --> 00:05:12,520 days in software security when we were focused on pen testing 82 00:05:12,520 --> 00:05:17,200 and buffer overflows and attacks and patches, and we didn't spend 83 00:05:17,200 --> 00:05:22,000 enough time thinking about what other assurance mechanisms 84 00:05:22,000 --> 00:05:24,160 should we have in the development life cycle to make 85 00:05:24,160 --> 00:05:27,200 sure we build secure software. Great news. 86 00:05:27,400 --> 00:05:29,000 We made a lot of progress on that. 87 00:05:29,360 --> 00:05:32,080 I feel confident we can also make a lot of progress in 88 00:05:32,080 --> 00:05:35,200 machine learning the same way. And one of the ways to do that 89 00:05:35,200 --> 00:05:38,400 is to get inside the model and see what's going on. 90 00:05:38,840 --> 00:05:40,920 I think that's where you were going like. 91 00:05:41,040 --> 00:05:44,240 These. These models, they have millions 92 00:05:44,240 --> 00:05:47,920 of nodes and connections between the nodes and activation States 93 00:05:47,920 --> 00:05:50,920 and attention models and all this stuff. 94 00:05:51,760 --> 00:05:56,920 And it turns out that all of the processing is massively 95 00:05:56,920 --> 00:06:00,040 distributed. It's not like reading a program, 96 00:06:00,400 --> 00:06:05,560 but it is in fact, you know, running a process in there and 97 00:06:05,560 --> 00:06:09,200 you can see activation States and you can see cycles, you can 98 00:06:09,200 --> 00:06:14,760 see circuits and understanding what is going on inside the 99 00:06:14,760 --> 00:06:19,520 network is really, really cool. It's a fun thing to work on. 100 00:06:20,360 --> 00:06:23,560 Let me just give you a, a tangible example of what you can 101 00:06:23,560 --> 00:06:25,760 do though. Like we all know that we can 102 00:06:25,760 --> 00:06:30,000 watch a machine learning model and LLM get jail broken by 103 00:06:30,240 --> 00:06:33,720 putting in a prompt injection attack and you know, goes in the 104 00:06:33,720 --> 00:06:36,520 front door and boom, the thing starts misbehaving. 105 00:06:36,520 --> 00:06:40,920 So it it got jail broken through the front door through some 106 00:06:40,920 --> 00:06:44,920 malicious input. But if you look inside, it's 107 00:06:44,920 --> 00:06:48,360 what at what's going on. You can also see activation 108 00:06:48,360 --> 00:06:51,960 states that indicate that something like a jailbreak is 109 00:06:51,960 --> 00:06:56,920 going on, and it's better to watch inside the brain than to 110 00:06:56,920 --> 00:07:01,280 watch the mouth move on the outside of the brain and do the 111 00:07:01,280 --> 00:07:03,640 thing. You know, in some sense we can. 112 00:07:03,760 --> 00:07:06,480 We can be a mind reader, which is very cool. 113 00:07:06,480 --> 00:07:08,840 So I'm excited about that. But that's the future. 114 00:07:10,000 --> 00:07:14,520 So in that example, does the model, I mean, this might seem 115 00:07:14,520 --> 00:07:17,720 like an insane question. Does the model know that it's 116 00:07:17,720 --> 00:07:22,160 misbehaving and like doing something that it's not meant to 117 00:07:22,160 --> 00:07:25,040 be doing the way that like sometimes humans will know 118 00:07:25,040 --> 00:07:28,000 they're not supposed to be doing something, But hey, man, it's 119 00:07:28,000 --> 00:07:29,360 fun. So you know. 120 00:07:30,400 --> 00:07:33,480 It's hard to know if a model understands anything or is 121 00:07:33,480 --> 00:07:36,520 conscious, and these are thorny philosophical issues that. 122 00:07:36,560 --> 00:07:38,800 I know it. Take on in 20 minutes. 123 00:07:39,120 --> 00:07:44,840 But the but, but the the point is you can interact with these 124 00:07:44,840 --> 00:07:47,960 things as if they were alien intelligences. 125 00:07:47,960 --> 00:07:51,920 Let's just talk it that way. So are the alien intelligences 126 00:07:51,920 --> 00:07:54,640 aware of their own behavior? Well, sort of. 127 00:07:54,640 --> 00:07:57,200 They are a little bit, especially inside of the same 128 00:07:57,240 --> 00:08:00,960 enormous prompt window. You know, you're doing recursive 129 00:08:00,960 --> 00:08:04,520 calls on yourself. There is some sort of 130 00:08:05,120 --> 00:08:08,280 acknowledgement of self, some sort of strange loop in there. 131 00:08:08,880 --> 00:08:13,280 And that's interesting. I mean, I for one, as a guy who, 132 00:08:13,280 --> 00:08:16,920 you know, been doing this for forever, I'm still flabbergasted 133 00:08:16,920 --> 00:08:20,800 by how cool LLMS are. They're cool, they're useful, 134 00:08:20,920 --> 00:08:23,520 they do great stuff. Occasionally they do terrible 135 00:08:23,520 --> 00:08:25,520 things and they make huge mistakes. 136 00:08:25,840 --> 00:08:29,600 And as long as we're prepared to manage those risks 137 00:08:29,600 --> 00:08:33,640 appropriately, that's cool. So I'm, you know, I'm pro AI. 138 00:08:34,280 --> 00:08:38,520 I'm just concerned that these days especially, we're not 139 00:08:38,520 --> 00:08:41,360 measuring AI for security properly. 140 00:08:41,360 --> 00:08:43,720 And that's what our latest piece of work is about. 141 00:08:43,720 --> 00:08:48,920 We released it today, in fact. Yeah, I, I did love one of the 142 00:08:48,920 --> 00:08:52,840 things that made me smile in this, in this work is the 143 00:08:53,600 --> 00:08:57,120 graphic of your Vadnais Ometer in there, which is, you know, 144 00:08:57,120 --> 00:08:59,520 it's like brings me back to the old days. 145 00:08:59,520 --> 00:09:01,240 I love that so much. It you know. 146 00:09:01,600 --> 00:09:04,880 I had to go drag that out of the Wayback Machine, Man, I was 147 00:09:04,880 --> 00:09:06,760 like, I know it was in Doctor Dobbs. 148 00:09:06,760 --> 00:09:10,000 When was it? And of course, Dr. Dobbs is only 149 00:09:10,000 --> 00:09:11,800 in the Wayback Machine. And there it was. 150 00:09:12,360 --> 00:09:16,120 Yep. Yeah, it it goes from what, like 151 00:09:16,280 --> 00:09:18,960 very bad to who knows or something like that. 152 00:09:18,960 --> 00:09:21,080 That, who knows is on the far right side. 153 00:09:21,080 --> 00:09:22,560 Yeah. Deep trouble. 154 00:09:22,560 --> 00:09:24,520 Oh, big deep trouble. Yes, yeah. 155 00:09:24,960 --> 00:09:26,720 So there's no good. Test. 156 00:09:26,760 --> 00:09:29,520 You know when When you pin test and somebody breaks your system, 157 00:09:29,680 --> 00:09:31,160 that means you're in deep trouble? 158 00:09:31,160 --> 00:09:33,600 You better fix it. But when you pin test and 159 00:09:33,600 --> 00:09:36,720 nothing gets found, we're all the way to who knows? 160 00:09:37,200 --> 00:09:38,680 Yeah. How do you know who you don't? 161 00:09:38,800 --> 00:09:41,560 What have you discovered? What have you learned in that 162 00:09:41,960 --> 00:09:43,840 exercise? Yeah. 163 00:09:43,840 --> 00:09:46,960 So we we figured out how to finesse that in software 164 00:09:46,960 --> 00:09:51,360 security by coming up with a whole bunch of kind of assurance 165 00:09:51,360 --> 00:09:55,680 practices that if we put all into a big process, we can 166 00:09:55,680 --> 00:09:59,640 measure with things like the B SIM like threat modeling, static 167 00:09:59,640 --> 00:10:04,400 analysis, security testing, penetration testing, deployment 168 00:10:04,400 --> 00:10:07,760 stuff, S BOM analysis. These are all kinds of 169 00:10:08,160 --> 00:10:12,760 activities that everybody does now in application security and 170 00:10:12,760 --> 00:10:16,160 software security. And you know, when we first 171 00:10:16,160 --> 00:10:19,240 started out, we had to define what those things are. 172 00:10:19,400 --> 00:10:22,120 We're in that same state in AI today. 173 00:10:22,600 --> 00:10:26,480 But there's a further problem with the field, and that is the 174 00:10:26,640 --> 00:10:31,840 AI people love to use benchmarks to measure their own stuff, and 175 00:10:31,840 --> 00:10:35,200 the benchmarks don't do what they think they do. 176 00:10:35,720 --> 00:10:40,360 So, so a lot of people would like to treat an AI benchmark as 177 00:10:40,360 --> 00:10:46,240 kind of a security measurement and go look, it got a 92.4 on 178 00:10:46,240 --> 00:10:49,840 the security benchmark. It must be secure. 179 00:10:49,840 --> 00:10:52,880 And we look at them like, what are you talking about? 180 00:10:53,360 --> 00:10:56,560 So that's why we had to write this, because they're kind of 181 00:10:56,560 --> 00:10:59,880 two aspects that are, you know, there's a lot of white space. 182 00:10:59,880 --> 00:11:02,760 There's the AI people who believe that security engineers 183 00:11:02,760 --> 00:11:06,560 know how to measure security, which we sort of maybe do on 184 00:11:06,560 --> 00:11:09,880 Thursdays. And there's the security people 185 00:11:09,880 --> 00:11:12,920 who think that the AI people know how to measure AI models 186 00:11:13,200 --> 00:11:17,520 with benchmarks and they don't. And so that's a lot of white 187 00:11:17,520 --> 00:11:20,760 space to operate in. What are those benchmarks 188 00:11:20,760 --> 00:11:23,880 usually trying to measure for AI models? 189 00:11:24,560 --> 00:11:27,880 Well, I mean, they have these highfalutin names like, you 190 00:11:27,880 --> 00:11:34,800 know, math or secure or privacy IQ. 191 00:11:35,360 --> 00:11:38,120 And. And those are very misleading. 192 00:11:38,120 --> 00:11:41,320 So what they're doing is a set of canned tests and they're 193 00:11:41,320 --> 00:11:45,160 finding out if the AI model can pass those tests. 194 00:11:45,520 --> 00:11:49,200 Sometimes they're agentic tests that put them into a scenario, 195 00:11:49,400 --> 00:11:54,720 but it's still a canned test. It's like a toy, a security toy. 196 00:11:54,720 --> 00:11:57,560 So you put them in a security puzzle, you say, hey, can you, 197 00:11:57,560 --> 00:11:59,280 can you solve the security puzzle? 198 00:11:59,600 --> 00:12:03,640 And they can't 92% of the time. And you say, OK, 92.5 on that 199 00:12:03,640 --> 00:12:06,160 benchmark. That's what it boils down to. 200 00:12:06,680 --> 00:12:08,880 OK. Here's the thing to know. 201 00:12:09,920 --> 00:12:12,920 They're they're kind of three things that are important. 202 00:12:12,920 --> 00:12:14,560 Let me make sure I get these right. 203 00:12:14,880 --> 00:12:19,920 Number one, that's kind of like the most obvious for us is if 204 00:12:19,920 --> 00:12:27,000 you're measuring whether or not a system can act secure that 205 00:12:27,000 --> 00:12:31,040 doesn't necessarily know whether it is secure. 206 00:12:31,320 --> 00:12:35,400 You see what I mean? So it's like it's being secure, 207 00:12:35,680 --> 00:12:39,560 but AAI system that performs really well on a security 208 00:12:39,560 --> 00:12:42,000 benchmark may itself not really be secure. 209 00:12:42,280 --> 00:12:45,960 So that's kind of the that's one of the things we want to want to 210 00:12:46,320 --> 00:12:48,880 want to focus on, but there are other two. 211 00:12:49,160 --> 00:12:53,440 The other two are passing a specific toy scenario is no 212 00:12:53,440 --> 00:12:55,640 guarantee of general ability at all. 213 00:12:56,440 --> 00:12:59,120 And new attacks and vulnerabilities are released 214 00:12:59,120 --> 00:13:02,120 every day. So are those in there or what? 215 00:13:02,760 --> 00:13:06,120 And, and those three things together mean we need to come up 216 00:13:06,120 --> 00:13:12,000 with a better way of doing, you know, of measuring AI security 217 00:13:12,400 --> 00:13:15,280 that let me let me say this right, we would like to be able 218 00:13:15,280 --> 00:13:19,360 to say that AI model is secure enough for us to use and we can 219 00:13:19,360 --> 00:13:21,160 manage risks appropriately with it. 220 00:13:22,040 --> 00:13:23,760 And that's what we're looking for at Bimble. 221 00:13:25,320 --> 00:13:26,920 There's a couple interesting things to me. 222 00:13:26,920 --> 00:13:31,720 One is in the in the paper you use as an example, Dan farmer 223 00:13:31,720 --> 00:13:35,520 Satan tool, which you know what is great? 224 00:13:36,320 --> 00:13:38,760 And it's I think an apartment example because you're saying 225 00:13:38,760 --> 00:13:43,440 like, look, this is this had a, a Cam set of things and you can 226 00:13:43,440 --> 00:13:47,760 run it against a specific thing and find out if you know if it 227 00:13:47,880 --> 00:13:49,360 did what it was supposed to do or not. 228 00:13:49,760 --> 00:13:52,560 And and in those days it was about configuration. 229 00:13:52,560 --> 00:13:55,360 Remember Satan was. Are you using the right version 230 00:13:55,360 --> 00:13:58,120 of Send mail or not? Yeah, yeah. 231 00:13:58,640 --> 00:14:01,800 But it was a very static thing essentially. 232 00:14:02,360 --> 00:14:06,000 And yeah, I mean IA. Regression test is what it was. 233 00:14:06,680 --> 00:14:07,760 Right. Yeah, typical. 234 00:14:08,120 --> 00:14:11,640 Terms, yeah. So, and I wouldn't, I wonder if 235 00:14:11,640 --> 00:14:15,400 like the the companies that are building these models and you 236 00:14:15,400 --> 00:14:17,800 know, we seem to be getting a new iteration of these things 237 00:14:17,800 --> 00:14:19,280 like almost every week or several. 238 00:14:19,280 --> 00:14:21,600 And they're improving. They really are getting better, 239 00:14:22,040 --> 00:14:25,040 yeah. But how are they testing them? 240 00:14:25,040 --> 00:14:28,040 Do you do you have any sort of like window into the internal 241 00:14:28,040 --> 00:14:30,240 testing? I measurement testing to me seem 242 00:14:30,240 --> 00:14:33,040 to be slightly different things. There's. 243 00:14:33,080 --> 00:14:35,800 Disagreement in AI about that issue. 244 00:14:36,320 --> 00:14:39,640 And there's a there's a person named Melanie Mitchell, who was 245 00:14:39,640 --> 00:14:42,640 also one of Doug Hofstadter's PhD students and she is a 246 00:14:42,640 --> 00:14:44,560 researcher at the Santa Fe Institute. 247 00:14:45,040 --> 00:14:51,080 She gave the keynote at Nurips, the enormous connectionist 248 00:14:51,080 --> 00:14:55,840 conference that runs every year in December, and she talked 249 00:14:55,840 --> 00:15:00,480 about benchmarks, and she talked about the problems with current 250 00:15:00,480 --> 00:15:04,120 benchmarks for AI and what we need to move to if we're going 251 00:15:04,120 --> 00:15:07,960 to treat these things as, quote, alien intelligences. 252 00:15:08,440 --> 00:15:11,280 And I was really struck by her work so much that I was like, 253 00:15:11,280 --> 00:15:13,320 you know what? That absolutely applies to 254 00:15:13,320 --> 00:15:17,480 everything in security, too. So even though we think there's 255 00:15:17,480 --> 00:15:22,400 no security meter for AI, AI has a measurement problem anyway. 256 00:15:22,600 --> 00:15:26,920 And that's just one aspect of our of our challenge in security 257 00:15:26,920 --> 00:15:29,560 and security engineering. Here's the thing though. 258 00:15:29,800 --> 00:15:34,400 Like software security made a huge amount of progress when it 259 00:15:34,400 --> 00:15:40,880 stopped asking is this software secure and instead asked what 260 00:15:40,880 --> 00:15:45,000 specific activities applied at what points in the development 261 00:15:45,000 --> 00:15:49,680 process reliably reduce risk. Now it's a much more complicated 262 00:15:49,680 --> 00:15:52,320 and grown up and a little bit more boring question. 263 00:15:54,120 --> 00:15:58,680 Yes, yeah, it's not as exciting as like, can my team of red 264 00:15:58,680 --> 00:16:03,440 teamers break into your, you know, app or your model? 265 00:16:03,640 --> 00:16:05,880 Yeah. And and so, you know, and so we 266 00:16:05,880 --> 00:16:08,400 came up with stuff like the B SIM and that went for a million 267 00:16:08,400 --> 00:16:13,280 years and still going. And, you know, can we build AB 268 00:16:13,400 --> 00:16:16,920 SIM for AI? Well, maybe we can, but not yet. 269 00:16:16,920 --> 00:16:20,560 We gotta identify all those activities 1st and get pretty 270 00:16:20,560 --> 00:16:23,240 good at them. Remember when the B SIM came 271 00:16:23,240 --> 00:16:26,280 about, we'd been doing software security for a while, for at 272 00:16:26,280 --> 00:16:29,080 least 10 years. And one of the things that was 273 00:16:29,080 --> 00:16:32,560 interesting about the B SIM days was I went to say Steve Lipner 274 00:16:32,560 --> 00:16:35,760 at Microsoft and said, hey, man, you're spending 80 million bucks 275 00:16:35,760 --> 00:16:37,960 on this. Do you sometimes just wake up in 276 00:16:37,960 --> 00:16:40,160 the middle of the night and wonder whether you're doing the 277 00:16:40,160 --> 00:16:43,800 right thing? And he said, yes, I do. 278 00:16:44,000 --> 00:16:47,600 And everybody I talked to said, oh, my God, yes, I do. 279 00:16:47,600 --> 00:16:50,200 Phil Venables at Goldman said, yes, I do. 280 00:16:50,440 --> 00:16:51,040 Like. And. 281 00:16:51,120 --> 00:16:54,160 And so we were like, well, let's just all compare belly buttons 282 00:16:54,200 --> 00:16:58,080 and see, you know, what we have in common and what really works. 283 00:16:58,080 --> 00:16:59,520 And that's where the B SIM came from. 284 00:16:59,960 --> 00:17:02,520 So I think we need one of those for AI, but it's too early for 285 00:17:02,520 --> 00:17:03,120 that. Yeah. 286 00:17:03,640 --> 00:17:06,200 Yeah, it's very early. I was going to mention that. 287 00:17:06,200 --> 00:17:12,319 And also the BSIM relied a lot on the participation of those 288 00:17:12,319 --> 00:17:13,880 third parties that you just mentioned. 289 00:17:13,880 --> 00:17:16,599 And it grew over time. You know, you got more and more 290 00:17:16,599 --> 00:17:21,319 organizations to be measured every year, and it required 291 00:17:21,520 --> 00:17:27,319 people who thought logically and understood the risks and all 292 00:17:27,319 --> 00:17:29,320 that sort of stuff. I'm not positive that we're 293 00:17:29,320 --> 00:17:31,920 there yet in the AI. We're still in the we're still 294 00:17:31,920 --> 00:17:34,480 in the hype cycle. I mean, think about Mythos, 295 00:17:34,520 --> 00:17:38,040 right? So Mythos is too dangerous to 296 00:17:38,040 --> 00:17:41,280 release. And it does find problems. 297 00:17:41,280 --> 00:17:44,480 It really, you know it, it does all the grindy part that's 298 00:17:44,480 --> 00:17:46,800 great. And it's pretty good at coding, 299 00:17:47,040 --> 00:17:49,800 which is also great. But you know what the real 300 00:17:49,800 --> 00:17:52,560 problem is? It's not Mythos AI. 301 00:17:52,560 --> 00:17:53,960 You know what the real problem is? 302 00:17:54,440 --> 00:18:00,920 Bad software mythos is finding out that your software is not 303 00:18:00,920 --> 00:18:02,280 good. Yeah. 304 00:18:02,960 --> 00:18:06,520 Right, which you've been telling people for 30. 305 00:18:06,520 --> 00:18:07,560 Five years. Yeah, When I. 306 00:18:07,640 --> 00:18:10,480 See this thing like Google's like, Oh my God, hey, I found a 307 00:18:10,480 --> 00:18:12,840 problem wasn't found before this morning. 308 00:18:12,960 --> 00:18:16,360 I'm like, oh, your software is broken. 309 00:18:16,360 --> 00:18:19,560 That should be the headline. Google admits software broken 310 00:18:19,560 --> 00:18:23,720 again, you know, and, and it's a, a different way of looking at 311 00:18:23,720 --> 00:18:27,280 this problem. So we really need to build 312 00:18:27,280 --> 00:18:29,200 secure software. I don't know where I've heard 313 00:18:29,200 --> 00:18:32,800 that before. And AI is going to make us do 314 00:18:32,800 --> 00:18:34,560 that, which is actually kind of cool. 315 00:18:34,560 --> 00:18:39,800 So I'm a big fan of, you know, pushing the limits on what these 316 00:18:39,800 --> 00:18:44,000 models can do for security, but I also want to make sure that 317 00:18:44,000 --> 00:18:49,480 they're secure themselves. Yeah, it's a 2 pronged problem 318 00:18:49,480 --> 00:18:51,200 there. Not problem, but, you know, 319 00:18:51,920 --> 00:18:52,840 issue there. Yeah. 320 00:18:52,840 --> 00:18:57,000 I mean the whole idea of Mythos and other models finding new 321 00:18:57,000 --> 00:19:01,040 bugs and, you know, possibly writing exploits and then these 322 00:19:01,040 --> 00:19:03,480 get, you know, pushed out into the world, that's great. 323 00:19:03,480 --> 00:19:06,280 I mean, finding and fixing software is excellent. 324 00:19:07,400 --> 00:19:10,000 And it does seem like that's just going to speed up at kind 325 00:19:10,000 --> 00:19:15,960 of, you know, compounding rate over the next, I don't know, 18 326 00:19:15,960 --> 00:19:18,840 months or something like that. I don't, I don't think we can 327 00:19:18,840 --> 00:19:20,760 estimate. I mean, there have been two 328 00:19:20,760 --> 00:19:25,280 massive changes in AI coding capabilities in the last six 329 00:19:25,280 --> 00:19:27,000 months. One of them happened in 330 00:19:27,000 --> 00:19:28,840 November. You probably remember that with 331 00:19:28,840 --> 00:19:31,520 all the codec stuff. And then the next one happened 332 00:19:31,520 --> 00:19:35,840 with clod code and, and agentic AI stuff. 333 00:19:36,280 --> 00:19:40,000 So things are moving very fast. That's great. 334 00:19:40,320 --> 00:19:44,920 I I, you know, like I'm going to tell you again, I'm a fan of AI 335 00:19:44,920 --> 00:19:48,200 and I think we should use it. I just don't think we should 336 00:19:48,320 --> 00:19:50,720 count on it to do everything right because it's going to 337 00:19:50,720 --> 00:19:54,720 screw up and we know why it's going to make mistakes because 338 00:19:54,960 --> 00:20:00,120 these are what machines, not how machines and the what pile has 339 00:20:00,120 --> 00:20:03,680 some problems in it. So the initial what piles we ate 340 00:20:03,760 --> 00:20:10,160 like everything in, you know, everything in GitHub, everything 341 00:20:10,160 --> 00:20:14,320 in 4 Chan and 8 Chan and 32 Chan or whatever. 342 00:20:14,320 --> 00:20:16,760 And it's just like. What do you expect? 343 00:20:16,760 --> 00:20:20,640 If if, if we trained it statistically on everything, 344 00:20:20,840 --> 00:20:24,640 that means it's a chiroscuro mix of good and bad and everything 345 00:20:24,640 --> 00:20:27,360 in between. Yeah, it's the garbage in, 346 00:20:27,360 --> 00:20:32,200 garbage out problem magnified to, you know, the NTH degree. 347 00:20:32,520 --> 00:20:36,480 Plus it gets recursive because AI is making stuff and putting 348 00:20:36,480 --> 00:20:39,480 it out there and then later AI is eating the stuff that got put 349 00:20:39,480 --> 00:20:42,200 out there. And you know, as long as we 350 00:20:42,200 --> 00:20:47,640 don't get Dumber as humans and rely on the AI to do everything, 351 00:20:47,680 --> 00:20:49,800 we should be OK. Although did you see that 352 00:20:49,800 --> 00:20:53,600 hilarious video go by with the the little dude talking to the 353 00:20:54,040 --> 00:20:59,200 the the council the the town council about AI the other day? 354 00:20:59,240 --> 00:21:01,080 No, I'll. Send you the like, it's 355 00:21:01,120 --> 00:21:05,040 absolutely hilarious. Well, this is the problem like 356 00:21:05,600 --> 00:21:09,920 that, that caveat of it, as long as we don't get Dumber as humans 357 00:21:09,920 --> 00:21:12,680 is, you know, that's a big if. We. 358 00:21:12,840 --> 00:21:15,000 You know. I I think we've learned a lot 359 00:21:15,000 --> 00:21:19,880 about measuring technology in the last 50 years and we can 360 00:21:19,880 --> 00:21:23,440 take advantage of that. So this paper that we just put 361 00:21:23,440 --> 00:21:28,000 out as a reminder that we do in fact know what we're doing and 362 00:21:28,000 --> 00:21:31,520 we just need to do it. You know, there's no easy way 363 00:21:31,520 --> 00:21:34,720 out. We just have to do the security 364 00:21:34,720 --> 00:21:38,520 engineering part now. And that's kind of the way it's 365 00:21:38,520 --> 00:21:42,160 always been. I wonder if, and you would know 366 00:21:42,160 --> 00:21:47,920 better than I would if the current hype cycle that we're in 367 00:21:47,920 --> 00:21:53,240 with AI, it it does remind me of the, you know, move fast and 368 00:21:53,240 --> 00:21:56,920 break things shit that happens every 10 or 12 years in this 369 00:21:56,920 --> 00:21:59,720 industry where somebody has a new idea and they're just like, 370 00:21:59,720 --> 00:22:02,400 go, go, go, go, go. And we'll worry about the, you 371 00:22:02,400 --> 00:22:05,760 know, Detroit is behind us after the fact. 372 00:22:07,040 --> 00:22:08,520 I don't know. I don't know where we are with 373 00:22:08,520 --> 00:22:11,360 that, with AI, but there do seem to be a lot of people like 374 00:22:11,360 --> 00:22:15,960 pushing the AI. Is this answer to everything you 375 00:22:15,960 --> 00:22:19,400 know? I think I fall somewhere in 376 00:22:19,400 --> 00:22:21,840 between. You're you're on the reasonable 377 00:22:21,840 --> 00:22:23,960 side of that, yeah. It's like a super automatic 378 00:22:23,960 --> 00:22:27,520 hammer and there's some stuff you should not be hammering, 379 00:22:27,520 --> 00:22:31,200 stop that, do not handle that. But there is some stuff we 380 00:22:31,200 --> 00:22:33,440 should hammer with a super automatic hammer and we should 381 00:22:33,440 --> 00:22:35,080 absolutely hammer that all day long. 382 00:22:35,480 --> 00:22:41,520 So I'm not a Luddite, but I am a grown up, I suppose, when it 383 00:22:41,520 --> 00:22:44,440 comes to especially security engineering. 384 00:22:44,920 --> 00:22:49,040 And so we just want to remind people, where are we? 385 00:22:49,040 --> 00:22:52,960 Well, we're at this state where nobody's quite measuring 386 00:22:52,960 --> 00:22:55,600 security properly. So let's figure out how to do it 387 00:22:55,600 --> 00:22:58,680 properly so we don't paint ourselves into the corner from a 388 00:22:58,680 --> 00:23:00,160 security engineering perspective. 389 00:23:01,360 --> 00:23:03,120 Yeah. I mean, I guess that's the next 390 00:23:03,120 --> 00:23:07,240 question is do we know what to measure yet with these models 391 00:23:07,240 --> 00:23:09,360 and these systems like in order to the? 392 00:23:09,880 --> 00:23:11,600 Things that we're doing are not bad. 393 00:23:11,600 --> 00:23:13,800 Is red teaming an OK thing to do? 394 00:23:13,800 --> 00:23:16,240 Yeah, absolutely. You should do it all day long. 395 00:23:16,480 --> 00:23:20,120 Just don't count that as the security score by itself. 396 00:23:20,360 --> 00:23:22,440 You need to do that. You need to do some threat 397 00:23:22,440 --> 00:23:24,720 modelling. You need to get inside the white 398 00:23:24,720 --> 00:23:26,280 box. We need to do some data 399 00:23:26,280 --> 00:23:29,640 provenance stuff. We need to think about the what 400 00:23:29,640 --> 00:23:31,880 pile and cleaning up the what pile. 401 00:23:32,240 --> 00:23:37,600 All these things that we've sort of noticed since say 2019 when 402 00:23:37,760 --> 00:23:42,560 Bimal started working on this stuff is still important and we 403 00:23:42,560 --> 00:23:44,480 do have to focus some attention on that. 404 00:23:44,480 --> 00:23:48,000 So that's what Bimal is for, you know, and and since we're not 405 00:23:48,000 --> 00:23:51,360 beholden to any of the big AI companies or any of the big tech 406 00:23:51,360 --> 00:23:54,160 companies or actually anybody at all. 407 00:23:56,440 --> 00:24:00,360 Not as tall. We can say what we think, which 408 00:24:00,360 --> 00:24:03,120 has never been a problem for McGraw, but it's even worse now. 409 00:24:04,280 --> 00:24:08,360 Yeah. Are you getting any feedback, 410 00:24:08,400 --> 00:24:12,200 you know, directly or otherwise from the AI companies on the 411 00:24:12,320 --> 00:24:13,560 stuff that you guys are putting out? 412 00:24:13,920 --> 00:24:17,480 We have talked to them, yes, and we've been, we've given talks 413 00:24:17,480 --> 00:24:20,720 for them. And generally speaking, the 414 00:24:21,400 --> 00:24:24,920 reception for our view of machine learning security is 415 00:24:24,920 --> 00:24:27,880 very high. And we went to unprompted, I 416 00:24:27,880 --> 00:24:29,600 don't know if you saw that go by. 417 00:24:30,000 --> 00:24:33,280 Yeah, yeah, in that. But that was mostly about, you 418 00:24:33,280 --> 00:24:37,440 know, hacking and fixing. So it was kind of old school 419 00:24:38,120 --> 00:24:40,640 Black Hat approach to the problem. 420 00:24:41,240 --> 00:24:45,560 All of these, all of these activities that we're doing 421 00:24:45,560 --> 00:24:50,400 together are going to add up to development life cycle that's 422 00:24:50,400 --> 00:24:54,920 secure for AI, you know, an an operational life cycle that 423 00:24:54,920 --> 00:24:58,400 makes AIB more secure. And I think we are making 424 00:24:58,400 --> 00:25:00,680 forward progress on that. I just want to make sure that we 425 00:25:00,680 --> 00:25:06,240 don't pretend that we have a magic security meter, because we 426 00:25:06,240 --> 00:25:07,920 don't. No, we don't. 427 00:25:08,000 --> 00:25:11,000 We never have. And one of the things that 428 00:25:11,800 --> 00:25:15,680 happens when the security life cycles get implemented is it 429 00:25:15,840 --> 00:25:19,560 slows down development and release schedules. 430 00:25:19,560 --> 00:25:22,360 Like, you know, Microsoft is the extreme example of that when 431 00:25:22,360 --> 00:25:27,880 they sort of stopped work on Windows to to do the Trustworthy 432 00:25:27,880 --> 00:25:30,360 Computing initiative 20 ish years ago now. 433 00:25:31,400 --> 00:25:34,720 And so sometimes, yeah, sometimes companies are 434 00:25:34,720 --> 00:25:37,280 reluctant to do all that stuff because they have these release 435 00:25:37,280 --> 00:25:39,880 schedules and they want to hit those milestones. 436 00:25:39,880 --> 00:25:41,120 So I don't know if we're going to. 437 00:25:41,560 --> 00:25:44,200 Things are moving faster than ever, but oh. 438 00:25:44,200 --> 00:25:46,920 It's crazy that. You know, given all of the 439 00:25:46,920 --> 00:25:52,040 progress we're making with automated coding with AI, that 440 00:25:52,800 --> 00:25:54,480 life cycles are going to speed up. 441 00:25:54,880 --> 00:25:57,400 But what that means is if you're sitting on a big pile of 442 00:25:57,400 --> 00:26:02,520 technical debt, you better start moving because you know, you're 443 00:26:02,880 --> 00:26:06,360 you need to stop creating new technical debt and start working 444 00:26:06,360 --> 00:26:10,480 down the pile from before. And these tools can help you do 445 00:26:10,480 --> 00:26:11,840 that. Let me give you a perfect 446 00:26:11,840 --> 00:26:15,120 example. One of my friends, Bill Pugh, 447 00:26:15,120 --> 00:26:21,240 who is a emeritus professor at Maryland, said, you know, one of 448 00:26:21,240 --> 00:26:25,760 my favorite uses of Claude code is I give it a pile of my code 449 00:26:25,760 --> 00:26:28,040 and I tell it to give me an architecture document. 450 00:26:28,920 --> 00:26:31,520 And I'm just like, of course, that's brilliant. 451 00:26:31,800 --> 00:26:34,560 That's what we need to do. Like, nobody likes writing 452 00:26:34,560 --> 00:26:36,480 those. Make the AI do it. 453 00:26:36,720 --> 00:26:39,600 It's not about, you know, being a code monkey for you, although 454 00:26:39,600 --> 00:26:41,440 it can also do some pretty good coding. 455 00:26:41,760 --> 00:26:47,040 But doing some of the grunt work of real security engineering and 456 00:26:47,040 --> 00:26:51,280 software engineering is awesome. It's really, really cool when 457 00:26:51,280 --> 00:26:53,920 you can, you know, harness it properly. 458 00:26:55,520 --> 00:26:58,920 Yeah, again, you know, I'm always the optimist even though 459 00:26:58,920 --> 00:27:02,040 I'm wildly independent. It's like a wildly independent 460 00:27:02,040 --> 00:27:03,440 optimist. The. 461 00:27:03,440 --> 00:27:06,240 I love it. We can make, we can make 462 00:27:06,240 --> 00:27:09,000 progress because these tools are more powerful. 463 00:27:09,360 --> 00:27:12,680 And if we think about it right, we can put them to work for us 464 00:27:12,720 --> 00:27:16,480 and do amazing stuff. And some of the stuff that we 465 00:27:16,480 --> 00:27:19,560 knew we were supposed to do that we haven't been doing right 466 00:27:19,760 --> 00:27:23,960 because we didn't have time or we didn't have budget, we can do 467 00:27:23,960 --> 00:27:28,200 some of that now because of AI. And in fact, one of the things 468 00:27:28,200 --> 00:27:31,320 to think about with Mythos and the whole glass wing thing is, 469 00:27:31,880 --> 00:27:35,840 hey, how much can we improve software if we spend a billion 470 00:27:35,840 --> 00:27:39,920 dollars? Just think about the cost of the 471 00:27:39,920 --> 00:27:42,480 tokens. Don't don't think about anything 472 00:27:42,520 --> 00:27:44,120 other than that. It's like, well, of course 473 00:27:44,120 --> 00:27:46,880 software's going to get better if we spend a shit ton of money 474 00:27:46,880 --> 00:27:48,400 on it, so let's do it get better. 475 00:27:49,560 --> 00:27:52,840 Yeah, we're about to find out because yeah, people are about 476 00:27:52,840 --> 00:27:54,560 to spend a shit ton of money on this. 477 00:27:54,560 --> 00:27:55,680 You're right. And I mean. 478 00:27:56,240 --> 00:27:59,160 And we and you know, it should be money well spent. 479 00:27:59,160 --> 00:28:03,160 I do think that we're making forward progress, not backwards 480 00:28:03,160 --> 00:28:09,080 progress using this tech. How are we going to know, like 481 00:28:09,080 --> 00:28:12,720 from a public consumer standpoint, if these companies 482 00:28:12,720 --> 00:28:16,280 are actually making progress? Because a lot of this stuff is 483 00:28:16,280 --> 00:28:17,960 opaque. You know, it's all done in the 484 00:28:17,960 --> 00:28:19,280 background. We don't really know what the 485 00:28:19,280 --> 00:28:22,440 security engineering teams are doing or what they're up to. 486 00:28:22,840 --> 00:28:26,480 Yeah, I don't know, maybe we'll need an AI religion. 487 00:28:28,880 --> 00:28:30,960 Don't laugh, it's probably going to. 488 00:28:30,960 --> 00:28:32,440 Happen. I don't want to know about that. 489 00:28:32,840 --> 00:28:36,520 No, thank you. Sorry, Dennis, That's it's 490 00:28:36,720 --> 00:28:39,680 coming, coming soon to a a human race near you. 491 00:28:39,760 --> 00:28:42,960 Oh. God, no, thank you. 492 00:28:44,480 --> 00:28:46,160 Don't want it. No. 493 00:28:46,840 --> 00:28:51,040 Well, if we're, if we do another podcast in say 6 or 8 months and 494 00:28:51,040 --> 00:28:54,600 we're kind of talking about the same topic, what kind of 495 00:28:54,600 --> 00:28:59,880 progress would you hope to see? In terms of some progress in 496 00:29:00,160 --> 00:29:04,560 white box analysis, Anthropic's been doing some really cool work 497 00:29:04,560 --> 00:29:09,720 there. I'd like to see less FUD and 498 00:29:10,040 --> 00:29:14,840 nonsense in the security field and more slightly more serious 499 00:29:14,840 --> 00:29:19,200 approach to real engineering and talk about how these tools can 500 00:29:19,200 --> 00:29:23,040 help us do real engineering instead of the I'm a hacker, I'm 501 00:29:23,040 --> 00:29:29,040 a patcher, you know, dichotomy. And so I hope that in some sense 502 00:29:29,040 --> 00:29:31,320 we grow up. I'd like to see us accelerate 503 00:29:31,520 --> 00:29:37,960 from 1999 software security to 2007 software security in the 504 00:29:37,960 --> 00:29:40,440 next 6 months. And I think we can do it. 505 00:29:40,760 --> 00:29:45,520 I really think we can, but I do. I also think that it's important 506 00:29:45,520 --> 00:29:50,520 for organizations like Bimmel to help point the way and to remind 507 00:29:50,520 --> 00:29:54,920 people that we have to be smart about this and that we have some 508 00:29:54,920 --> 00:29:57,240 battle scars and we need to remember where we got them. 509 00:29:58,080 --> 00:30:01,000 Yeah, I think that's a great point because, you know, a lot 510 00:30:01,000 --> 00:30:03,840 of this stuff that happened with software security in the, you 511 00:30:03,840 --> 00:30:08,800 know, late 90s, early 2000s, twenty 10s was kind of forced on 512 00:30:08,800 --> 00:30:14,760 the vendors through various outside parties, you know, 513 00:30:14,880 --> 00:30:17,320 governments, customers, folks like that. 514 00:30:17,320 --> 00:30:22,240 But I don't know if that's, I don't know, are those same 515 00:30:22,600 --> 00:30:26,360 forces at work for in this case, like the government hasn't 516 00:30:26,360 --> 00:30:30,320 gotten super involved yet, but you know that they're always. 517 00:30:30,320 --> 00:30:32,000 Yeah, yeah, of course they're not. 518 00:30:32,080 --> 00:30:35,080 Getting any less behind with this current administration? 519 00:30:35,400 --> 00:30:39,800 They're just getting more behind so so the don't count on the 520 00:30:39,800 --> 00:30:41,600 government, but I'll tell you this. 521 00:30:41,680 --> 00:30:45,200 What happened is software weaseled its way into everything 522 00:30:45,440 --> 00:30:48,440 and then it had to work like most of the time. 523 00:30:49,040 --> 00:30:51,880 And it's pretty cool. I mean, there's a lot of Kluge's 524 00:30:51,920 --> 00:30:54,400 all over the place and we have all sorts of software, 525 00:30:54,400 --> 00:30:58,480 spectacular software failures, but for the most part, software 526 00:30:58,480 --> 00:31:00,880 works like it's supposed to most of the time. 527 00:31:00,880 --> 00:31:02,760 And it's pretty cool. Like it's. 528 00:31:02,760 --> 00:31:05,360 Kind of amazing. You're running for I don't know 529 00:31:05,360 --> 00:31:07,600 how many days in a row. You don't even have to reboot it 530 00:31:07,600 --> 00:31:09,160 anymore. I remember when you had to 531 00:31:09,160 --> 00:31:11,840 reboot it twice a day. All the time. 532 00:31:12,160 --> 00:31:15,320 And everything was slow. I mean, we've made a great deal 533 00:31:15,320 --> 00:31:18,440 of progress. And because software kind of 534 00:31:18,440 --> 00:31:21,720 sunk into the roots, you know, like a bunch of water being 535 00:31:21,720 --> 00:31:26,520 poured into the plant, we had to make software better. 536 00:31:26,520 --> 00:31:28,640 That's where the pressure was really coming from, from 537 00:31:28,640 --> 00:31:31,320 society. I think the same thing is going 538 00:31:31,320 --> 00:31:33,640 to happen with AI, but it's going to be even faster. 539 00:31:33,920 --> 00:31:36,600 So it's going to sink in because software has already got its 540 00:31:37,040 --> 00:31:39,440 roots in there. And this is going to make 541 00:31:39,440 --> 00:31:43,840 software more powerful than ever and faster and more interesting 542 00:31:44,120 --> 00:31:47,200 and cooler. And we can also make it better 543 00:31:47,640 --> 00:31:51,920 if we do it right. So I think once again, society 544 00:31:51,920 --> 00:31:54,640 is going to kind of cause us to do that. 545 00:31:55,080 --> 00:31:57,720 But it sure would be better not to have to do it the hard way. 546 00:31:57,720 --> 00:32:01,480 Let's do it the easy way. Yeah, let's not learn the same 547 00:32:01,480 --> 00:32:05,680 lessons again. Let's you know over and over and 548 00:32:05,680 --> 00:32:07,440 over. I'm excited. 549 00:32:08,120 --> 00:32:11,160 I, I, I, I, you know, it's actually fun to keep track of 550 00:32:11,160 --> 00:32:13,760 what's going on. I'm sure that you're tearing 551 00:32:13,760 --> 00:32:17,720 your metaphorical hair out over this as a person who has to, who 552 00:32:18,040 --> 00:32:21,800 has to, like cover this space because it's so damn fast. 553 00:32:21,800 --> 00:32:24,680 You wake up in the morning and you're like, whoa, what happened 554 00:32:24,680 --> 00:32:28,240 in tech today? And like, even in the major 555 00:32:28,240 --> 00:32:31,720 press, there's like whole chunks of the New York Times just 556 00:32:31,720 --> 00:32:35,720 devoted to AI that tells you something real's going on. 557 00:32:36,160 --> 00:32:39,040 And it is like the software revolution, but it's going to be 558 00:32:39,040 --> 00:32:41,360 faster. Yeah, no doubt. 559 00:32:41,360 --> 00:32:44,040 I mean, it's, you know, it's gotten to a certain inflection 560 00:32:44,040 --> 00:32:47,160 point when people, you know, in your life that have no 561 00:32:47,160 --> 00:32:49,600 connection to the tech industry at all start asking you 562 00:32:49,600 --> 00:32:52,920 questions about AI and, you know, just kind of out of the 563 00:32:52,920 --> 00:32:56,400 blue at dinner or something, or, you know, you just in a normal 564 00:32:56,400 --> 00:32:58,120 conversation. And that's happened to me all 565 00:32:58,120 --> 00:33:01,960 the time now. Just, you know, yeah. 566 00:33:02,280 --> 00:33:05,960 Which is that's a kind of like tipping point where you're like, 567 00:33:05,960 --> 00:33:10,000 OK, this is kind of, like you said, wormed its way into the 568 00:33:10,000 --> 00:33:14,320 roots and it's like kind of invading everything. 569 00:33:14,600 --> 00:33:16,600 Yeah. And and you know, we're making 570 00:33:16,880 --> 00:33:20,360 so much progress so fast, maybe we will get to something like 571 00:33:20,400 --> 00:33:22,440 AGI. You know, I'm not going to make 572 00:33:22,440 --> 00:33:26,720 any any predictions. But I'll tell you, as a guy who 573 00:33:26,720 --> 00:33:30,840 worked on really early language models 25,000,000 years ago, 574 00:33:31,400 --> 00:33:33,800 it's amazing to see what LLMS can do. 575 00:33:33,920 --> 00:33:37,840 These chat bots, they know a lot of stuff like I was setting up a 576 00:33:37,840 --> 00:33:42,680 trip to go to my friend Matthias Madhu's wedding this summer and 577 00:33:43,080 --> 00:33:45,920 actually just said, let me just pull up Claude and see what 578 00:33:45,920 --> 00:33:49,680 Claude knows about Rotterdam. We're just going to go there and 579 00:33:49,920 --> 00:33:51,560 Claude knows a lot about Rotterdam. 580 00:33:52,720 --> 00:33:54,920 And it was really up. I'm like, well, you know, I'm 581 00:33:54,920 --> 00:33:56,720 going to stay here. Which restaurant? 582 00:33:56,720 --> 00:33:59,840 Like, you know, whatever, just having a conversation as if I 583 00:33:59,840 --> 00:34:04,680 were Googling 50,000 things, but I can just talk to this agent. 584 00:34:05,080 --> 00:34:06,760 And it was really, really helpful. 585 00:34:06,760 --> 00:34:10,920 I, I'm finding these tools, if you know what you're doing now, 586 00:34:11,000 --> 00:34:15,400 you know, I'm in Traveler and, and so you, if you know what 587 00:34:15,400 --> 00:34:20,480 you're doing, these tools can be very helpful enablers. 588 00:34:20,880 --> 00:34:24,920 If you don't know what you're doing, they will also enable you 589 00:34:24,920 --> 00:34:29,560 to screw it up. Yes, really fast. 590 00:34:30,639 --> 00:34:33,800 Very fast, yeah. So my. 591 00:34:34,080 --> 00:34:37,960 My big my biggest concern is where do the architects come 592 00:34:37,960 --> 00:34:39,840 from? Like where do the people who 593 00:34:40,000 --> 00:34:42,600 really know what they're doing come from? 594 00:34:42,960 --> 00:34:47,880 And we need to make sure that we are still producing those in the 595 00:34:47,880 --> 00:34:49,760 world. That's a good point. 596 00:34:49,760 --> 00:34:52,840 Are we like are we? I mean, where are those folks? 597 00:34:53,360 --> 00:34:54,360 I'm a little. Worried about that. 598 00:34:54,560 --> 00:34:56,320 We never really knew where they came from. 599 00:34:56,360 --> 00:34:59,680 They just, you know, one day there was a line outside of one 600 00:34:59,680 --> 00:35:03,640 of your super developers offices and they got the guy got really 601 00:35:03,640 --> 00:35:06,600 mad and he put down his keyboard and picked up a magic marker and 602 00:35:06,600 --> 00:35:08,840 he never could it again. But then he was the architect. 603 00:35:10,960 --> 00:35:15,360 That's you don't go to school to become a software architect. 604 00:35:16,240 --> 00:35:18,520 And I guess not, yeah. Glasses. 605 00:35:18,520 --> 00:35:21,720 Like, no, that's not how it really that's bullshit. 606 00:35:21,720 --> 00:35:25,840 It's all bullshit. So I mean, think about what how 607 00:35:25,880 --> 00:35:29,480 how do you grow from a cubby reporter to a seasoned veteran 608 00:35:29,600 --> 00:35:34,520 who can sniff out the baloney? Where can everybody do it? 609 00:35:34,600 --> 00:35:36,960 No, they cannot. Some people are good at this and 610 00:35:36,960 --> 00:35:40,120 other people are not. And some people have the chops 611 00:35:40,120 --> 00:35:43,760 and some people, you know, develop a career and some people 612 00:35:43,760 --> 00:35:45,840 don't. And that's true in all the 613 00:35:45,840 --> 00:35:48,240 fields. So we got to make sure that we 614 00:35:48,240 --> 00:35:51,280 don't just cut off all the noobs, and I'm not sure 615 00:35:51,280 --> 00:35:53,480 anybody's thinking about that right now. 616 00:35:53,720 --> 00:35:56,920 We need to think about that. I think that's a great point. 617 00:35:56,920 --> 00:35:59,440 I mean, yeah, all those stuff that we hear about in terms of 618 00:35:59,440 --> 00:36:03,520 like, the job stuff is like AI is eliminating developer jobs 619 00:36:03,520 --> 00:36:05,800 because it's so good at coding. OK. 620 00:36:05,800 --> 00:36:08,760 I mean, maybe that's happening in some respects. 621 00:36:08,760 --> 00:36:10,480 It's hard to know what's fact and what's fiction. 622 00:36:10,480 --> 00:36:14,120 But the point you just made, I think is it is valid, too. 623 00:36:14,200 --> 00:36:16,120 You know, like, where are we going to get these folks that 624 00:36:16,120 --> 00:36:18,760 are just going to understand how to architect all of this? 625 00:36:19,040 --> 00:36:20,920 Yeah, it's. Trying I mean. 626 00:36:21,320 --> 00:36:25,200 And humans are still needed. I mean, we are creative, we have 627 00:36:25,240 --> 00:36:30,360 intuition, We have, you know, all of these things are 628 00:36:30,760 --> 00:36:34,600 necessary to guide this technology in the way that's the 629 00:36:34,600 --> 00:36:38,520 most powerful. So the human part is not being 630 00:36:38,520 --> 00:36:41,120 eradicated. In fact, it's getting more 631 00:36:41,120 --> 00:36:46,360 important than ever Now. If you're worried about your job 632 00:36:46,760 --> 00:36:50,320 being replaced by AI, it's you probably got a pretty boring 633 00:36:50,320 --> 00:36:57,080 job, which is maybe most, I don't know, like I never didn't. 634 00:36:57,080 --> 00:36:59,440 I always like my job. I don't know. 635 00:36:59,720 --> 00:37:01,560 Me too like. Your job, yeah. 636 00:37:01,560 --> 00:37:04,800 So, so I when people complain about work, I would just be 637 00:37:04,800 --> 00:37:07,080 like, we'll go do something else, man. 638 00:37:07,080 --> 00:37:09,600 It's like start your own thing, do it right. 639 00:37:10,680 --> 00:37:14,800 Yeah, yeah, I agree. All right, Gary, Well, this is 640 00:37:14,800 --> 00:37:16,400 awesome. It's always good to see you and 641 00:37:16,400 --> 00:37:20,160 have you on the podcast and, and I'd love, love it when you guys 642 00:37:20,160 --> 00:37:21,560 put out new research. So. 643 00:37:22,000 --> 00:37:25,520 We're excited. This is our fourth big thing of 644 00:37:25,520 --> 00:37:29,360 all time, so please take a minute and read it because we we 645 00:37:29,360 --> 00:37:34,080 really worked on it for years. Yeah, you can tell it's not the 646 00:37:34,080 --> 00:37:37,200 slapdash stuff that a lot of places put out. 647 00:37:37,200 --> 00:37:39,280 It's very well done as you would expect. 648 00:37:39,280 --> 00:37:41,480 So All right, man. Great to see you. 649 00:37:41,480 --> 00:37:43,440 Thanks a lot, Gary. Talk to you later. 650 00:37:44,160 --> 00:37:44,480 See you.