1 00:00:00,080 --> 00:00:03,600 Hey, I'm Kimberly with Elixir mentor Jacob recently added 2 00:00:03,600 --> 00:00:06,960 recruitment services to help companies hire pre vetted Elixir 3 00:00:06,960 --> 00:00:09,920 devs. I help with company outreach, so 4 00:00:09,920 --> 00:00:13,200 if you're hiring, I may be reaching out, or you can connect 5 00:00:13,200 --> 00:00:15,400 with us anytime at elixirmentor.com. 6 00:00:17,080 --> 00:00:19,680 Secure your Elixir apps with paraxial IO. 7 00:00:19,840 --> 00:00:22,880 From code analysis and bot defense to dependency checks, 8 00:00:22,880 --> 00:00:26,160 they've got you covered. Elixir first design with GitHub 9 00:00:26,160 --> 00:00:28,560 integration. Elevate your security at 10 00:00:28,560 --> 00:00:39,880 paraxial dot IO testing. What's up? 11 00:00:39,920 --> 00:00:42,760 I'm Jacob Litzo, your Elixir mentor and welcome to another 12 00:00:42,760 --> 00:00:45,120 exciting episode of the Elixir Mentor Podcast. 13 00:00:45,360 --> 00:00:47,480 This is where we discuss everything related to Elixir, 14 00:00:47,480 --> 00:00:49,560 from interviews with enthusiasts, pioneers in the 15 00:00:49,560 --> 00:00:52,800 community to innovative projects and library shaping Elixir's 16 00:00:52,800 --> 00:00:55,720 future. Jamel, founder of Fire Zone 17 00:00:56,080 --> 00:00:59,760 joins me today talking about building AYC backed open source 18 00:00:59,760 --> 00:01:03,360 zero trust access platform with Elixir and Rust. 19 00:01:04,519 --> 00:01:06,960 Welcome, Jamel, and thank you for joining me. 20 00:01:07,560 --> 00:01:10,920 Thanks for having me. I usually like to kick these 21 00:01:10,920 --> 00:01:14,960 things off with like a, just a, an introduction of like who you 22 00:01:14,960 --> 00:01:19,080 are and a little bit about your background and what led you to 23 00:01:19,080 --> 00:01:21,480 Elixir. Yeah, sure. 24 00:01:21,480 --> 00:01:26,800 So let's see, I guess I got my start in in college. 25 00:01:27,240 --> 00:01:32,120 So doing kind of web projects for for, you know, on the side 26 00:01:32,120 --> 00:01:34,080 to make earn a little, earn a little extra money. 27 00:01:34,080 --> 00:01:36,120 I think Rails was getting big around then. 28 00:01:36,120 --> 00:01:40,720 This is the back in like late 2008, 2009, like like Rails 1, 29 00:01:40,720 --> 00:01:44,960 Rails 2 days. And so, yeah, I just started, I 30 00:01:44,960 --> 00:01:47,520 started basically on the on the on the rails train. 31 00:01:47,600 --> 00:01:52,880 And and then from there, you know, after graduation I kind of 32 00:01:52,880 --> 00:01:56,120 got into, I got, I got a job at Cisco. 33 00:01:56,120 --> 00:02:01,160 That's where I got my start in, in security and, and, and sort 34 00:02:01,160 --> 00:02:02,960 of always did the web programming thing on the side. 35 00:02:04,880 --> 00:02:07,200 But at Cisco, I had, I, I worked there over 8 years. 36 00:02:07,200 --> 00:02:10,759 I had, you know, four different, actually four different stents I 37 00:02:10,759 --> 00:02:13,360 would like leave and then some other team would kind of bring 38 00:02:13,360 --> 00:02:15,400 me back in. And then between each one of 39 00:02:15,400 --> 00:02:17,680 those, I would do, you know, go back to sort of web programming, 40 00:02:17,680 --> 00:02:19,720 web application programming and, and Rails. 41 00:02:19,720 --> 00:02:25,360 And, and so the last stint at, at Cisco, it was during COVID 42 00:02:25,560 --> 00:02:29,040 and, you know, I was like 4 years in and I was like just, 43 00:02:29,080 --> 00:02:30,440 you know, kind of getting burned out. 44 00:02:30,880 --> 00:02:34,520 And towards the end there, you know, COVID started and Cisco's 45 00:02:34,520 --> 00:02:37,840 this company where it was hybrid at the time. 46 00:02:38,560 --> 00:02:40,400 And so obviously, you know, everyone had to just go and work 47 00:02:40,400 --> 00:02:44,720 from home immediately, right? And, and so, and so when that 48 00:02:44,720 --> 00:02:48,520 happened, their VPN concentrators sort of fell over. 49 00:02:48,520 --> 00:02:50,440 They got overloaded. And I thought that was 50 00:02:50,440 --> 00:02:52,920 interesting. And so I was like, OK, yeah, 51 00:02:52,920 --> 00:02:55,040 this is kind of the premier, you know, like one of the premier 52 00:02:55,040 --> 00:02:57,400 networking companies and they're having struggles with this. 53 00:02:57,400 --> 00:03:01,360 And so always wanted to, to, to find a project to work on Elixir 54 00:03:01,360 --> 00:03:03,400 with. And, you know, kind of coming 55 00:03:03,400 --> 00:03:05,680 from Rails and was just thought it was interesting. 56 00:03:06,120 --> 00:03:07,560 And so that's how Firezone started. 57 00:03:07,560 --> 00:03:09,600 So it's, yeah, over five years ago now. 58 00:03:10,640 --> 00:03:13,240 But yeah, I, I, yeah. So went from security. 59 00:03:13,240 --> 00:03:15,720 It's kind of like the, you know, the, the intersection of the 60 00:03:15,720 --> 00:03:18,120 Venn diagram between, you know, web programming and security 61 00:03:18,120 --> 00:03:22,800 and, and started Firezone. It's just a side project and 62 00:03:22,920 --> 00:03:26,120 kind of went from there. How did how did you like land 63 00:03:26,120 --> 00:03:30,240 down Elixir being the fit there? I, you know, that's a good 64 00:03:30,240 --> 00:03:32,880 question. I, if I look back at my Git 65 00:03:33,080 --> 00:03:36,040 history, like if I look at my GitHub, you know, my personal 66 00:03:36,040 --> 00:03:38,880 GitHub, I'm like, I had some, some, some side projects there 67 00:03:38,880 --> 00:03:41,800 going back to like 2015. So something piqued my interest 68 00:03:41,800 --> 00:03:42,880 in it. I don't remember what. 69 00:03:42,880 --> 00:03:44,240 It's probably like a Hacker News post. 70 00:03:45,520 --> 00:03:51,360 And, and I, I do remember, so in, you know, we used Ruby and 71 00:03:51,360 --> 00:03:54,160 Rails at Cisco on some of the projects I was on. 72 00:03:54,200 --> 00:03:57,120 And I just remember spending, you know, not, not, not to throw 73 00:03:57,120 --> 00:03:59,960 shade on, on Ruby or Rails, but I just remember spending a long 74 00:03:59,960 --> 00:04:03,880 time kind of debugging some performance issues related to 75 00:04:03,880 --> 00:04:06,880 the threading model. And and, you know, we figured it 76 00:04:06,880 --> 00:04:09,200 out and and it ended up being something, you know, not, not 77 00:04:09,200 --> 00:04:13,440 too hard to fix. But I remember reading sort of 78 00:04:13,440 --> 00:04:17,360 one of the selling points, I think in back in the day with 79 00:04:17,360 --> 00:04:20,000 elixir, I think Jose would talk about how it was like, you know, 80 00:04:20,000 --> 00:04:24,680 kind of his, his, his story about how he how he, you know, 81 00:04:24,680 --> 00:04:27,360 set out the salt, you know, set out to kind of think about 82 00:04:27,360 --> 00:04:30,000 languages, you know, in a multi core, multi core world. 83 00:04:30,440 --> 00:04:32,000 And so that kind of like resonated with me. 84 00:04:32,000 --> 00:04:34,000 And I think, yeah, I just played around with it. 85 00:04:34,000 --> 00:04:35,720 Always, always thought it was interesting. 86 00:04:36,880 --> 00:04:43,120 And we, we had, yeah, Erlang was one of those like no one, no one 87 00:04:43,120 --> 00:04:44,440 really heard about or used, right. 88 00:04:44,440 --> 00:04:46,600 Like, like I didn't, you know, I studied electrical engineering. 89 00:04:46,600 --> 00:04:48,400 We didn't hear about it right in college. 90 00:04:48,480 --> 00:04:53,600 We, we had, we had a guy come on and give a talk, you know, I was 91 00:04:53,600 --> 00:04:56,000 involved with like the networking or the, the I triple 92 00:04:56,000 --> 00:04:58,200 E society. And we had a guy come give a 93 00:04:58,200 --> 00:04:59,960 talk and he's like, oh, I built all this stuff in Erlig. 94 00:04:59,960 --> 00:05:03,840 We're all like, what is Erlig like, you know, and, and so and 95 00:05:03,840 --> 00:05:06,360 so when Elixir came out, I just thought it was interesting and, 96 00:05:06,400 --> 00:05:10,120 and I started looking into it and, and so finally got a chance 97 00:05:10,120 --> 00:05:11,520 to to work with it with Firezone. 98 00:05:11,520 --> 00:05:14,520 In college I turned my back to programming because I was 99 00:05:14,520 --> 00:05:17,760 learning C and Java and I absolutely hated it. 100 00:05:18,440 --> 00:05:20,160 Yeah, I remember those days. Yeah. 101 00:05:20,200 --> 00:05:23,920 I I don't, I don't know why, why they kind of shove Java down our 102 00:05:23,920 --> 00:05:27,040 throats so hard. Yeah, that was a big one. 103 00:05:27,520 --> 00:05:30,920 Yeah, For a minute, I was like in a master's program. 104 00:05:31,040 --> 00:05:34,000 I was like, I'm not going to be a software, you know, software 105 00:05:34,120 --> 00:05:37,320 engineer. And it was all Java. 106 00:05:37,320 --> 00:05:40,080 And I dropped out and, like, built my own projects because I 107 00:05:40,080 --> 00:05:42,960 was, like, so impatient. I wasn't, like, building fast 108 00:05:42,960 --> 00:05:45,400 enough. And I hated Java. 109 00:05:46,360 --> 00:05:49,160 Yeah, They're like, solve this hash table collision in Java. 110 00:05:49,160 --> 00:05:52,200 And it's like, what does this do to it? 111 00:05:52,200 --> 00:05:54,680 How does this help anyone? You know, it's very detached 112 00:05:54,680 --> 00:05:58,880 from the real world, Yeah. I accidentally kind of like I 113 00:05:59,200 --> 00:06:02,200 started writing Elixir for like a marketplace platform that I 114 00:06:02,200 --> 00:06:07,760 was building in like 2020-2021 And I like kind of just picked 115 00:06:07,760 --> 00:06:11,400 it because I was like, it sounds what I like, what I need, but I 116 00:06:11,400 --> 00:06:15,040 didn't like actually understand the full benefits of like the 117 00:06:15,040 --> 00:06:17,640 fault tolerance and like the actor model. 118 00:06:17,640 --> 00:06:20,480 I didn't know what I was getting myself into, but it just ended 119 00:06:20,480 --> 00:06:24,040 up like the more I learned, the more perfect of a fit it ended 120 00:06:24,040 --> 00:06:26,760 up being for me. Yeah, I, you know, now that 121 00:06:26,760 --> 00:06:30,360 you've mentioned that, I do remember we use device, right. 122 00:06:30,360 --> 00:06:34,000 And so like I got, I was like, OK, this is great, right? 123 00:06:34,000 --> 00:06:36,480 You know, in the real worlds, like my favorite gym, right? 124 00:06:36,480 --> 00:06:40,280 And, and so when I yeah, so when I, when I, when I picked up 125 00:06:40,280 --> 00:06:42,480 elixir, I was like, oh, it's by, you know, kind of by the same, 126 00:06:42,520 --> 00:06:44,160 same mind. And I was like, OK, this is, you 127 00:06:44,160 --> 00:06:46,280 know, this is something. I can, yeah. 128 00:06:46,480 --> 00:06:48,840 I fell in love with the syntax before. 129 00:06:48,880 --> 00:06:52,240 I fell in love with the like, the power of like the beam and. 130 00:06:52,640 --> 00:06:54,000 Yeah, exactly. Yeah. 131 00:06:55,720 --> 00:06:59,760 Yeah, I like it. I feel like Fire Zone is like 132 00:06:59,760 --> 00:07:06,960 not not a simple undertaking. Yeah, product it is very it is 133 00:07:06,960 --> 00:07:12,640 very complicated, unfortunately, so yeah. 134 00:07:12,640 --> 00:07:15,800 So I, I, so I maybe I should get into kind of more of the details 135 00:07:15,800 --> 00:07:18,760 of how, you know how Fire Zone was originally architected. 136 00:07:20,480 --> 00:07:23,640 So, so big yeah. So a lot of mistakes were made. 137 00:07:23,880 --> 00:07:26,960 I'll, I'll preface, you know, this, this story with that, but 138 00:07:28,880 --> 00:07:31,600 so, so you know, it's a web app, right? 139 00:07:31,840 --> 00:07:35,440 Like I like, I wanted to learn Elixir now, now and I, and I 140 00:07:35,440 --> 00:07:38,440 wanted to, and so I, I don't know how much you or your 141 00:07:38,440 --> 00:07:40,400 listeners know about Wireguard, but you know, kind of this new 142 00:07:40,400 --> 00:07:43,720 VPN protocol, I know, you know, much simpler. 143 00:07:44,200 --> 00:07:46,320 It's like. Most networking stuff. 144 00:07:47,000 --> 00:07:50,360 There was a time I took two years of a Cisco networking 145 00:07:50,360 --> 00:07:52,360 course in high school. Oh, I'm sorry. 146 00:07:52,920 --> 00:07:56,200 And I did. I did well, but I don't remember 147 00:07:56,200 --> 00:07:58,080 a thing of it. Yeah, yeah. 148 00:07:58,080 --> 00:08:02,320 So, so before Wireguard came along, like the VPN protocols 149 00:08:02,320 --> 00:08:03,760 had sort of been stagnating, right? 150 00:08:03,760 --> 00:08:06,120 There wasn't really any new developments in the space for a 151 00:08:06,120 --> 00:08:08,920 couple decades. And they're pretty heavyweight. 152 00:08:10,960 --> 00:08:13,160 You know, I'm not, I don't claim to be a Wireguard expert, but 153 00:08:13,560 --> 00:08:15,520 you know, the, the protocols before, they're pretty 154 00:08:15,520 --> 00:08:18,840 heavyweight. Very, very, very complicated, 155 00:08:18,840 --> 00:08:21,000 very large. So Wireguard came along as like 156 00:08:21,000 --> 00:08:24,600 this breath of fresh air, right? Easy to configure, perform at 157 00:08:24,600 --> 00:08:29,040 lightweight. And so I had friends here in the 158 00:08:29,040 --> 00:08:30,560 Valley, I, I'm in Silicon Valley. 159 00:08:30,560 --> 00:08:33,880 So I had friends like they're at their, you know, they're working 160 00:08:33,880 --> 00:08:35,480 at their startups, like a hundred 200 people. 161 00:08:35,480 --> 00:08:37,159 And they're like, you know, we need, we need a remote access 162 00:08:37,159 --> 00:08:39,320 like wire guard's great. But like, it doesn't really fit 163 00:08:39,320 --> 00:08:41,440 into the business, right. Like it's, it's, it's like very 164 00:08:41,440 --> 00:08:43,080 bare bones. It's like configuring SSH 165 00:08:43,320 --> 00:08:45,440 configs. You know, you, you, you private 166 00:08:45,440 --> 00:08:47,560 key, public key. Here's your static config. 167 00:08:47,560 --> 00:08:50,800 You know, you got to get on the machine somehow and, you know, 168 00:08:50,800 --> 00:08:53,320 no failover like, and, and not to fault Wireguard. 169 00:08:53,320 --> 00:08:55,320 That's just, you know, it's not meant to do that, right. 170 00:08:56,080 --> 00:08:59,320 So there, so so I had one friend that was like, you know, just 171 00:08:59,320 --> 00:09:01,640 build it AUI for Wireguard to generate configs. 172 00:09:01,640 --> 00:09:04,080 I was like, OK, you know, that's and so that was Fire Zone 173 00:09:04,080 --> 00:09:08,600 initially. And, and, and so it was just an, 174 00:09:08,600 --> 00:09:10,600 it was just a web app and you just like, it would just spit 175 00:09:10,600 --> 00:09:14,120 out Wireguard configs, you know, generate the, I don't even know 176 00:09:14,120 --> 00:09:16,120 if I should admit this, but it would, it would like generate 177 00:09:16,120 --> 00:09:18,880 the, the keys in the browser, like browser JavaScript. 178 00:09:19,600 --> 00:09:22,000 And it was just a toy project. And so it was like a disclaimer, 179 00:09:22,000 --> 00:09:24,320 you know, like this is, you know, just this is a toy 180 00:09:24,320 --> 00:09:26,280 project. Ended up putting it on Hacker 181 00:09:26,280 --> 00:09:29,600 News and it was on the front page all day. 182 00:09:29,640 --> 00:09:33,640 And then like I was like, OK, And then, you know, kind of 183 00:09:33,640 --> 00:09:36,200 always wanted to, you know, in the back of my mind like wanted 184 00:09:36,200 --> 00:09:37,920 to do a startup. You know, I had left Cisco a few 185 00:09:37,920 --> 00:09:41,480 months earlier to take a break and, and sort of recollect as we 186 00:09:41,480 --> 00:09:43,520 do sometimes and. I've been there. 187 00:09:43,680 --> 00:09:46,120 For yeah I've. Taken I've taken a year off from 188 00:09:46,120 --> 00:09:49,920 touching a computer before. We'll have to talk about that 189 00:09:50,720 --> 00:09:54,560 maybe another time. But yeah, they, they, they, they 190 00:09:54,560 --> 00:10:00,040 accepted us. And, and, and so, you know, we 191 00:10:00,040 --> 00:10:05,560 got into YC and, and the architecture was just like, you 192 00:10:05,600 --> 00:10:08,480 know, not, not, not there at the time. 193 00:10:09,560 --> 00:10:12,760 So what it was back then was it was only self hosted. 194 00:10:13,760 --> 00:10:15,080 So you would just take this Elixir. 195 00:10:15,080 --> 00:10:18,080 And so as you know, self hosting, like distributing an 196 00:10:18,080 --> 00:10:20,960 Elixir application is like kind of a challenge in itself. 197 00:10:20,960 --> 00:10:24,040 So we use this this package called Omnibus, which I think 198 00:10:24,040 --> 00:10:27,200 git GitLab uses. It's like way back from the chef 199 00:10:27,200 --> 00:10:28,800 guys, right? And it would just like package 200 00:10:28,800 --> 00:10:32,480 up all these dependencies. I think it came out, you know, 201 00:10:32,480 --> 00:10:35,520 it's from pre Docker days. So nowadays you just ship a 202 00:10:35,520 --> 00:10:37,280 docker container for everyone, right? 203 00:10:37,280 --> 00:10:38,240 I know, that's how I. Roll it. 204 00:10:38,800 --> 00:10:40,080 Yeah, yeah, yeah. So, yeah. 205 00:10:40,120 --> 00:10:43,560 And we added that later on, but yeah, it was, it was built with, 206 00:10:43,760 --> 00:10:46,760 it would package up, you take it, you run Firezone, you got to 207 00:10:46,800 --> 00:10:48,120 manage your database and all that. 208 00:10:48,120 --> 00:10:50,160 And then it would still just spit out configs and you got to 209 00:10:50,160 --> 00:10:52,280 have the Wireguard app to connect. 210 00:10:54,200 --> 00:10:56,400 And we went through YC with that architecture. 211 00:10:56,400 --> 00:10:59,080 And in the back of my mind, I was like, you know, this is not 212 00:10:59,080 --> 00:11:00,600 for, you know, this is not really going to scale the 213 00:11:00,600 --> 00:11:02,320 business. You know, it's not solving the 214 00:11:02,800 --> 00:11:05,880 problems that businesses have when they try to take Wireguard 215 00:11:05,880 --> 00:11:07,840 and bring it into their into their company, right? 216 00:11:07,840 --> 00:11:18,240 So we ended up building onto that architecture for a good 217 00:11:18,240 --> 00:11:21,320 solid. You know, you go through, you go 218 00:11:21,320 --> 00:11:25,160 through YC. For listeners out there who've 219 00:11:25,160 --> 00:11:27,840 never kind of been through an incubator program, it's pretty 220 00:11:27,840 --> 00:11:30,040 intense. Cuz I'm like just like a 221 00:11:30,040 --> 00:11:33,480 bootstrap, you know, small product guy. 222 00:11:33,480 --> 00:11:35,400 So yeah, yeah, yeah, that's the process. 223 00:11:35,840 --> 00:11:36,760 Yeah. So. 224 00:11:37,600 --> 00:11:40,440 It's a different it, you know, it's definitely a different kind 225 00:11:40,440 --> 00:11:42,880 of horse, right? Like, you know, you don't go 226 00:11:42,880 --> 00:11:45,360 through YC to build a sustainable, you know, well, I 227 00:11:45,360 --> 00:11:47,040 shouldn't say sustainable. You don't go through IC to build 228 00:11:47,040 --> 00:11:49,440 like a bootstrap business, but that may seem obvious, right? 229 00:11:49,440 --> 00:11:53,000 But it's it's to get big, right? It's not it's not to go small 230 00:11:53,000 --> 00:11:55,280 and like. It's the IPO, right? 231 00:11:55,480 --> 00:11:57,760 Yeah, it's a it's to have, you got to have an exit at some 232 00:11:57,760 --> 00:12:03,480 point, right. So, so you go through YC and you 233 00:12:03,480 --> 00:12:04,880 know, obviously you got to move fast. 234 00:12:04,920 --> 00:12:08,480 And so when you've got this, like we had this sort of when, 235 00:12:08,480 --> 00:12:10,480 when they say like, do things that don't scale, I wouldn't 236 00:12:10,480 --> 00:12:13,920 recommend applying that to your product architecture, right, if 237 00:12:13,920 --> 00:12:16,360 you can avoid it. And so we, we, we had this sort 238 00:12:16,360 --> 00:12:18,560 of product and it was like growing on GitHub was like 239 00:12:18,560 --> 00:12:20,920 getting GitHub stars, but it was just self osters. 240 00:12:20,920 --> 00:12:24,200 And, and, you know, we, we ended up having a lot of businesses 241 00:12:24,200 --> 00:12:29,520 use that banks and, and whatnot. And, and they would run into 242 00:12:29,520 --> 00:12:32,320 issues and they would e-mail us and they'd find us and e-mail us 243 00:12:32,320 --> 00:12:34,920 and be like, Hey, you know, we're trying to get high. 244 00:12:34,920 --> 00:12:38,480 We're trying to like combined 2 Fire Zone instances on the same 245 00:12:38,480 --> 00:12:40,200 database to like get high availability. 246 00:12:40,200 --> 00:12:41,840 I was like, no, no, no, like what are you doing? 247 00:12:41,840 --> 00:12:45,560 And so after about six months of just growing, right, because, 248 00:12:45,720 --> 00:12:50,480 you know, investors want to see growth and, and, and, you know, 249 00:12:52,280 --> 00:12:54,120 we kind of decided that we were going to go back to the drawing 250 00:12:54,120 --> 00:12:56,320 board and actually build, build the product the correct way. 251 00:12:56,320 --> 00:12:58,880 So, so it took a few months to do that. 252 00:12:58,880 --> 00:13:02,920 Maybe I don't recall how long it was maybe 8 months, nine months. 253 00:13:03,880 --> 00:13:07,960 And before we did that, we, we got all of those people that 254 00:13:07,960 --> 00:13:10,520 were using it. I won't call them customers 255 00:13:10,520 --> 00:13:13,120 because it, because it was, it was just free back then, right? 256 00:13:13,120 --> 00:13:17,880 And we, we got on calls with banks and, and other businesses 257 00:13:18,280 --> 00:13:20,400 and we were like, all right, what would you like to see, 258 00:13:20,480 --> 00:13:22,000 right? Like what, what problems are you 259 00:13:22,000 --> 00:13:24,560 seeing? And we did actual customer 260 00:13:24,560 --> 00:13:26,800 discovery, right? And we had, we had the, we had 261 00:13:26,800 --> 00:13:28,520 the kind of the privilege to do that because they're already 262 00:13:28,520 --> 00:13:33,120 using Fire Zone, right? So we, we took, yeah, we took 263 00:13:33,120 --> 00:13:36,520 lots of calls and we sort of collected everything and put 264 00:13:36,520 --> 00:13:40,520 that into the new architecture. And so we, we, we, yeah, we, we, 265 00:13:40,560 --> 00:13:43,040 that's the complicated thing that you see in our repository 266 00:13:43,040 --> 00:13:46,280 today. But yeah, if you go look at like 267 00:13:46,280 --> 00:13:49,680 the legacy branch, that's where we, we just like stopped 268 00:13:49,680 --> 00:13:52,080 development on the other thing 'cause it was just, yeah, it 269 00:13:52,080 --> 00:13:55,080 was, it wasn't gonna scale so. Was it already in elixir? 270 00:13:55,120 --> 00:13:57,120 You just re architected things or? 271 00:13:57,120 --> 00:13:59,160 Yeah. We, we, we sort of rebuilt 272 00:13:59,160 --> 00:14:03,080 everything we knew, we knew we wanted to have a cloud kind of a 273 00:14:03,080 --> 00:14:06,200 major cloud component to, to manage all the configuration, 274 00:14:06,200 --> 00:14:09,360 right. So the, so with the new 275 00:14:09,360 --> 00:14:11,840 architecture and then we know we need that we needed our own 276 00:14:11,840 --> 00:14:14,400 client applications, right. So, you know, you need to do 277 00:14:14,400 --> 00:14:16,560 like the user authentication and all of that tie that to the 278 00:14:16,560 --> 00:14:20,960 keys. So we went back and we built all 279 00:14:20,960 --> 00:14:24,200 of that. We kind of started with a Mac, I 280 00:14:24,200 --> 00:14:27,640 think it was Mac OS client and sort of, you know, started 281 00:14:27,640 --> 00:14:31,320 building everything in parallel. A lot of stuff to, you know, a 282 00:14:31,320 --> 00:14:32,440 lot. A lot of code to write. 283 00:14:32,440 --> 00:14:37,360 As you can imagine, and are. You writing a native apps for 284 00:14:37,360 --> 00:14:40,440 everything or? Yeah, we have native app for so, 285 00:14:40,480 --> 00:14:41,560 yeah. So I'll get into the, the 286 00:14:41,560 --> 00:14:44,560 architecture a bit. So the so so Fire Zone is kind 287 00:14:44,560 --> 00:14:47,120 of two, two components, right? It's a control plane and then 288 00:14:47,120 --> 00:14:50,520 the data plane. The control plane is all 289 00:14:50,920 --> 00:14:53,360 essentially Elixir. So that's, you know, a big 290 00:14:53,360 --> 00:14:59,160 Phoenix app up in the cloud. And and then the data plane is 291 00:14:59,160 --> 00:15:01,840 the client application on one end of the tunnel. 292 00:15:01,840 --> 00:15:04,600 And then you have what we call the gateway, the VPN gateway on 293 00:15:04,600 --> 00:15:06,720 the other end of the tunnel. And those the gateway's 294 00:15:06,720 --> 00:15:08,680 completely in Rust. The client applications are, you 295 00:15:08,680 --> 00:15:12,480 know, it's a Rust core and then you've got a on Apple and in 296 00:15:12,480 --> 00:15:15,200 Android we do Kotlin and you know, Swift and Kotlin, right? 297 00:15:15,200 --> 00:15:17,960 OK. Can you for for the listeners, 298 00:15:17,960 --> 00:15:20,480 can you kind of explain the the reason? 299 00:15:20,480 --> 00:15:23,280 Like I'm guessing it's for like computation reasons like why 300 00:15:23,280 --> 00:15:25,560 you're using Rust? Yeah. 301 00:15:25,560 --> 00:15:30,920 So that's a great question. So I think memory safety was was 302 00:15:30,920 --> 00:15:33,960 kind of one of the first things that that that pulled me in that 303 00:15:33,960 --> 00:15:35,520 direction. It was it was kind of on, it was 304 00:15:35,520 --> 00:15:36,760 up and coming. So it was a little bit of a 305 00:15:36,760 --> 00:15:39,440 risky bet when we chose it. This is maybe like 3-4 years ago 306 00:15:39,440 --> 00:15:43,560 now, OK, but you know you've got to have a systems language, 307 00:15:43,720 --> 00:15:45,600 right? You like we couldn't do, you 308 00:15:45,600 --> 00:15:48,240 can't do, you can't maybe you can't, I don't know. 309 00:15:48,240 --> 00:15:50,240 But I wouldn't recommend building AVPN tunnel with 310 00:15:50,240 --> 00:15:54,880 Elixer, at least not yet, right? Because you've got to you've got 311 00:15:54,920 --> 00:15:56,600 to you've got to interface closely with a system. 312 00:15:56,600 --> 00:15:59,400 You got to control routes, you got to open, open what's called 313 00:15:59,400 --> 00:16:03,680 the Tung device. You got to do obviously packet 314 00:16:03,680 --> 00:16:06,720 encryption and decryption. So yeah, those need to be 315 00:16:06,720 --> 00:16:08,760 performant. So, so you do need the systems 316 00:16:08,760 --> 00:16:13,680 language and and so, yeah, Russ just seemed like a good fit at 317 00:16:13,680 --> 00:16:15,640 the time. Yeah, yeah. 318 00:16:15,640 --> 00:16:18,520 I've heard that a lot. I have never like I've only been 319 00:16:18,520 --> 00:16:21,240 on like strictly Elixir like backends. 320 00:16:21,240 --> 00:16:25,240 So I have never like implemented both and it's it's cool. 321 00:16:26,000 --> 00:16:29,920 Yeah, I, I, yeah, my Russ skills are, I feel like I'm, I'm a 322 00:16:29,920 --> 00:16:31,400 forever student in that language. 323 00:16:31,400 --> 00:16:33,800 Like no matter how much I don't, I, you know, I don't get the 324 00:16:33,800 --> 00:16:37,400 opportunity to, to code as much anymore, but and, and don't 325 00:16:37,400 --> 00:16:39,360 really have the time to kind of learn as much as I would like. 326 00:16:39,360 --> 00:16:42,280 But yeah. Well, our, our Rust lead, he 327 00:16:42,280 --> 00:16:45,080 just did another podcast. He did a Rust podcast on Fire 328 00:16:45,080 --> 00:16:46,280 Zone. Nice, cool. 329 00:16:46,480 --> 00:16:49,320 Yeah, yeah. So he's, yeah, he's a, he's in 330 00:16:49,320 --> 00:16:51,520 Sydney, down in down in Australia. 331 00:16:51,920 --> 00:16:54,280 But yeah. We'll have to bring one of your 332 00:16:54,280 --> 00:16:56,120 one of your Elixir devs over here too. 333 00:16:56,400 --> 00:17:00,320 For a yeah. So I'm curious like you have a, 334 00:17:00,440 --> 00:17:03,840 you have a Co founder, right? No, I don't. 335 00:17:03,840 --> 00:17:06,520 Not anymore. OK, yeah, I was gonna be. 336 00:17:06,720 --> 00:17:09,960 I was curious about like how how that relationship started and 337 00:17:09,960 --> 00:17:13,160 like how you balance like skills and things at the beginning. 338 00:17:14,160 --> 00:17:16,560 Let's see, at the beginning, I can talk a little bit about it. 339 00:17:16,560 --> 00:17:21,280 So yeah, we, we had a mutual friend. 340 00:17:21,280 --> 00:17:26,720 Let's see. So I, I had a friend and he was 341 00:17:26,720 --> 00:17:30,520 applying to IC. This is like summer of 2021 and 342 00:17:30,520 --> 00:17:32,960 he was like, he was going to do a security something in the 343 00:17:32,960 --> 00:17:36,240 security space. And so he got referred to me 344 00:17:36,240 --> 00:17:38,680 just to bounce ideas off of he's like, Hey, can you validate 345 00:17:38,680 --> 00:17:40,400 this? It was, I think it was like 346 00:17:40,400 --> 00:17:43,120 analyzing it was like doing static analysis of, of source 347 00:17:43,120 --> 00:17:46,280 code for kind of like security vulnerabilities right before 348 00:17:46,280 --> 00:17:47,640 they before they go and get shipped, right. 349 00:17:47,760 --> 00:17:51,120 It's like, it sounds like such a hot thing now, but back in 2021, 350 00:17:51,120 --> 00:17:53,720 you know, pre ChatGPT, it was like cutting edge. 351 00:17:55,080 --> 00:17:57,880 And so he's like, Oh yeah, we're applying to YC. 352 00:17:57,880 --> 00:18:01,840 And then a couple of months later when we did the, you know, 353 00:18:01,840 --> 00:18:05,160 we put Firezone out there. I guess I put Firezone out on 354 00:18:05,160 --> 00:18:09,040 GitHub and he introduced me to, to another guy. 355 00:18:09,080 --> 00:18:12,680 And so we worked together and actually applied to YC together. 356 00:18:13,840 --> 00:18:16,800 And his background was in growth and sort of business, right? 357 00:18:17,080 --> 00:18:19,520 And he had worked on it at A at this company before. 358 00:18:19,520 --> 00:18:24,360 Like I've made the mistake of partnering with too many like 359 00:18:24,360 --> 00:18:29,720 minded same skill sets people and it ends up just like you 360 00:18:29,720 --> 00:18:33,000 have no one to help you on the business market inside when 361 00:18:33,800 --> 00:18:36,560 you're both technical. Yeah, there's, well, there's 362 00:18:36,560 --> 00:18:39,040 like the time element and then there's like the skills element 363 00:18:39,040 --> 00:18:42,080 of it too. And I guess they're related. 364 00:18:43,480 --> 00:18:46,440 But yeah, it was very, it's very helpful to have someone to just 365 00:18:46,440 --> 00:18:49,200 be like, hey, you know, are you thinking about this? 366 00:18:49,200 --> 00:18:50,800 Hey, you know, are we getting, are we thinking about putting 367 00:18:50,800 --> 00:18:54,120 posts here and here and here on Reddit or you know what not, or 368 00:18:54,400 --> 00:18:56,840 we should organize our GitHub read me to look like this so 369 00:18:56,840 --> 00:18:59,840 that it like, you know, ranks a little bit better in SEO. 370 00:18:59,840 --> 00:19:03,120 And it's very helpful to have someone like that for sure. 371 00:19:03,840 --> 00:19:06,600 So what happened was, I think when, when we went through our, 372 00:19:08,440 --> 00:19:11,000 you know, so I mentioned we had this thing that that wasn't 373 00:19:11,000 --> 00:19:14,160 really ready to sell, I would say, and it was going to take a 374 00:19:14,160 --> 00:19:17,080 long time to rebuild. So when we went back to the 375 00:19:17,080 --> 00:19:19,280 drawing board, it just didn't make sense to have, I think to 376 00:19:19,280 --> 00:19:22,680 have really any, any business people, you know, on, on at that 377 00:19:22,680 --> 00:19:25,680 point, 'cause it was like we, we needed all the engineering 378 00:19:25,680 --> 00:19:27,120 talent we could get right at that point. 379 00:19:28,360 --> 00:19:35,400 So, yeah, so, yeah, we, we, so, so now we're at a play. 380 00:19:35,440 --> 00:19:39,680 We're still tiny. We, we shipped that. 381 00:19:39,680 --> 00:19:43,560 I think we shipped the one at O that was like summer of 2024. 382 00:19:44,920 --> 00:19:48,080 OK. Had a lot of, you know, as, as 383 00:19:48,080 --> 00:19:50,760 you can imagine with any sort of one dot O product, you had a lot 384 00:19:50,760 --> 00:19:55,280 of interesting problems to figure out and things to learn. 385 00:19:55,280 --> 00:19:59,080 But yeah, I would say it's it's got that mostly ironed out 386 00:19:59,080 --> 00:20:02,920 within a few months and tacking on features and what not, you 387 00:20:02,920 --> 00:20:05,000 know, doing the whole product iteration thing sense. 388 00:20:05,800 --> 00:20:07,200 Awesome. How do you how do you balance 389 00:20:07,720 --> 00:20:10,560 like features versus like product improvement at this 390 00:20:10,560 --> 00:20:13,160 point? That's a good question. 391 00:20:13,160 --> 00:20:19,040 So, so, so my day is like I wake up, I see, OK, what fires should 392 00:20:19,040 --> 00:20:21,240 I put out right, Like what fires need to be put out? 393 00:20:22,000 --> 00:20:25,880 Customer support and any sort of critical bug fixes tend to take 394 00:20:25,880 --> 00:20:29,640 priority. And with Fire Zone, you know, 395 00:20:29,640 --> 00:20:34,040 luckily the elixir side is like has been completely trouble, you 396 00:20:34,040 --> 00:20:36,640 know, like worry free, like any, any problems we've had with 397 00:20:36,640 --> 00:20:38,680 elixir have been our own doing right, our own bugs. 398 00:20:38,680 --> 00:20:41,640 We haven't hit like, Oh my gosh, you know, you know, the VMS out 399 00:20:41,640 --> 00:20:44,400 of memory, like for no reason are like garbage collector lock 400 00:20:44,400 --> 00:20:46,520 UPS, you know, like, you know, nothing like that. 401 00:20:46,520 --> 00:20:49,520 No random crashes. It's just been super rock solid. 402 00:20:52,240 --> 00:20:56,800 And so all of the problems that we face typically are either, 403 00:20:57,800 --> 00:21:00,120 you know, like maybe the database got overloaded, right? 404 00:21:00,120 --> 00:21:02,560 We didn't, we didn't structure a query the right way. 405 00:21:02,600 --> 00:21:07,480 And or more more commonly, you know, you have these client 406 00:21:07,480 --> 00:21:11,760 applications and we're doing all of this like we, we do this, we 407 00:21:11,760 --> 00:21:14,640 do this thing called, you know, DNS interception. 408 00:21:14,840 --> 00:21:17,720 So we see all the DNS queries on the system whenever you sign in 409 00:21:17,720 --> 00:21:21,800 to Fire Zone. OK, to do that, well, requires 410 00:21:21,800 --> 00:21:24,320 you to like read the system resolvers, right? 411 00:21:25,200 --> 00:21:27,760 And some platforms. One layer is the DNS stuff 412 00:21:27,800 --> 00:21:30,160 happening on. Is that that's that's in the 413 00:21:30,160 --> 00:21:31,080 Rust layer? Yeah. 414 00:21:31,080 --> 00:21:31,400 OK. Yeah. 415 00:21:31,760 --> 00:21:35,000 So, so that's, so that goes through the, you know, the FFI 416 00:21:35,000 --> 00:21:39,000 boundary and we've got, you know, it communicates the 417 00:21:39,600 --> 00:21:42,720 depending on the platforms for Apple and Android, they've got, 418 00:21:43,440 --> 00:21:46,200 it's a very sort of platform specific way to do it. 419 00:21:46,200 --> 00:21:49,160 So we do that on the Swift and the, the Kotlin side and then we 420 00:21:49,160 --> 00:21:53,520 send the system resolvers down into the into the Rust side. 421 00:21:53,520 --> 00:21:56,600 And and then we know like kind of how to do this DNS 422 00:21:57,000 --> 00:22:02,320 orchestration dance or DNS routing dance rather. 423 00:22:02,320 --> 00:22:07,360 And, and yeah, so, so we so there's a lot of bugs that just 424 00:22:07,360 --> 00:22:12,160 sort of come up because like someone's got like a, you know, 425 00:22:12,200 --> 00:22:15,480 is running like a ADNS, another DNS program on their computer, 426 00:22:15,480 --> 00:22:16,840 right? And there's like fighting over 427 00:22:16,840 --> 00:22:19,920 who has control, right? Or someone's like, you know, 428 00:22:19,920 --> 00:22:22,240 running another VPN. And, you know, they may not 429 00:22:22,240 --> 00:22:24,760 because the people that are using Firezone meant much of the 430 00:22:24,760 --> 00:22:26,360 time, like they're not technical, right? 431 00:22:26,360 --> 00:22:28,800 They just they just have to use it because they're IT admin, 432 00:22:28,800 --> 00:22:31,680 they're saying right, right. We need a VBN client, right, 433 00:22:32,080 --> 00:22:33,680 Yeah, yeah. So they. 434 00:22:33,800 --> 00:22:35,360 Don't know if they're running 2 VPNs. 435 00:22:36,120 --> 00:22:37,640 They they sometimes they don't know. 436 00:22:37,640 --> 00:22:39,520 Sometimes you know they don't. They thought they disconnected 437 00:22:39,520 --> 00:22:40,880 the old one and it's still connected. 438 00:22:41,680 --> 00:22:44,680 Sometimes it got like I'm. Curious, does like Apple's like 439 00:22:44,680 --> 00:22:48,040 relay system does that affect Fire Zone? 440 00:22:48,480 --> 00:22:51,400 That's a good question. I don't we've never had issues 441 00:22:51,400 --> 00:22:54,640 with it. I believe that that is, yeah, 442 00:22:54,640 --> 00:22:56,120 that's a good question. I'm not sure. 443 00:22:56,880 --> 00:23:00,160 I think that I think because that's an Apple technology, it's 444 00:23:00,760 --> 00:23:03,320 it's like, it might be, it might be like, yeah, it just plays. 445 00:23:03,320 --> 00:23:06,240 It's like shielded offset, like a lower level maybe I I don't 446 00:23:06,240 --> 00:23:08,680 know. I mean, if it's not an issue and 447 00:23:08,680 --> 00:23:10,680 you don't know about it, that's probably a good thing. 448 00:23:11,000 --> 00:23:15,640 Yeah, but also like typically, you know, Sally from marketing 449 00:23:15,640 --> 00:23:18,720 or you know, whoever is like not gonna use the private relay and 450 00:23:18,720 --> 00:23:20,800 then also their work on their work computer I guess at the 451 00:23:20,800 --> 00:23:23,240 same time. So yeah, yeah, yeah. 452 00:23:23,240 --> 00:23:27,120 So, yeah, typically the issues that we see it's it's not so 453 00:23:27,120 --> 00:23:29,320 much other VPN software. I'd say most of the issues that 454 00:23:29,320 --> 00:23:33,600 we see are like Windows, you know, a Windows laptop plugged 455 00:23:33,600 --> 00:23:36,440 into like an, a dock. I won't name any brands here. 456 00:23:36,840 --> 00:23:39,880 And you know, that you like the like the network card driver of 457 00:23:39,880 --> 00:23:43,800 the dock is like, doesn't, doesn't properly respond to, you 458 00:23:43,800 --> 00:23:45,320 know, some of the system calls that we make. 459 00:23:46,000 --> 00:23:48,600 And so then we've got to handle that gracefully. 460 00:23:48,600 --> 00:23:52,320 And you know, and, and it's like if Fire Zone goes down, like if, 461 00:23:52,400 --> 00:23:55,520 if the user can't connect to their stuff, it's a big problem, 462 00:23:55,640 --> 00:23:57,120 right? It's like, it's like your 463 00:23:57,120 --> 00:24:00,240 Internet going out or like your, your, it's like where utility at 464 00:24:00,240 --> 00:24:03,400 that point in a lot of cases. And so that's the stuff that we 465 00:24:03,400 --> 00:24:06,240 really have to get right. And so that's where we tend to 466 00:24:06,240 --> 00:24:09,080 have the most fires. I feel like those are also very 467 00:24:09,080 --> 00:24:11,720 difficult bugs to track down. You probably have to have 468 00:24:11,880 --> 00:24:16,000 specific hardware and like be able to replicate it on device 469 00:24:16,000 --> 00:24:19,800 or not really you. So, yeah, this is a this is a 470 00:24:19,800 --> 00:24:21,880 whole, Yeah, this is a whole, whole topic. 471 00:24:22,640 --> 00:24:25,280 Observability, yes, is super critical for our client 472 00:24:25,280 --> 00:24:28,160 applications and for anything the customer's running on Prem. 473 00:24:29,120 --> 00:24:31,360 What we use now is Sentry Sentry's been great. 474 00:24:32,120 --> 00:24:34,000 I love Century. Yeah, we're not we're not 475 00:24:34,000 --> 00:24:38,560 sponsored by them or anything, but yeah, Sentry, Sentry, once 476 00:24:38,560 --> 00:24:42,800 we added Sentry and and we turned it on, it just like lit 477 00:24:42,800 --> 00:24:44,680 up like a Christmas tree. We're like, oh, we're getting 478 00:24:44,680 --> 00:24:46,200 these different warnings in these places. 479 00:24:46,200 --> 00:24:48,560 Oh, this user has this issue. You know, we never would have 480 00:24:48,560 --> 00:24:50,640 suspected that, right. All of those things that you 481 00:24:50,640 --> 00:24:53,440 think like, you know, your software is never going to hit 482 00:24:53,640 --> 00:24:57,840 it hit like, you know, you see it with system software. 483 00:24:57,840 --> 00:25:01,640 So, so sentry's been super helpful. 484 00:25:01,640 --> 00:25:06,160 We we used to and now it's got log streaming. 485 00:25:06,160 --> 00:25:09,080 So we can on demand. We'll, you know, if a customer's 486 00:25:09,080 --> 00:25:13,120 having an issue, we'll will remotely enable debug log 487 00:25:13,120 --> 00:25:16,480 streaming just for that user. And then we'll and then yeah. 488 00:25:16,480 --> 00:25:18,920 And so we'll just see like, and then we'll see, we'll say like, 489 00:25:18,920 --> 00:25:22,040 OK, go reproduce the issue and then we'll see basically trace 490 00:25:22,040 --> 00:25:24,040 logs of all the packets coming in and out. 491 00:25:26,280 --> 00:25:28,600 Yeah, we don't do that for everyone, but you know, it's 492 00:25:28,600 --> 00:25:31,040 like a on a case by case basis when they're having an issue. 493 00:25:31,040 --> 00:25:37,320 So the opt in to it, yeah. Yeah, having sensory and a good 494 00:25:37,320 --> 00:25:41,640 logging system is so important. Yeah, it's, yeah. 495 00:25:41,640 --> 00:25:46,560 It's been super helpful. So not to like, I know we kind 496 00:25:46,560 --> 00:25:50,400 of were just going kind of down architecture and how things work 497 00:25:50,400 --> 00:25:52,440 rabbit hole. I want to like kind of circle 498 00:25:52,440 --> 00:25:57,560 back, like what Like when you apply to like YC and stuff like 499 00:25:57,560 --> 00:26:00,360 that, Like I don't like your background. 500 00:26:00,360 --> 00:26:04,640 You're kind of just a day job Cisco developer, yeah. 501 00:26:05,440 --> 00:26:10,560 Well, I, I would. So I worked at Cisco over a 502 00:26:10,680 --> 00:26:13,720 total span of eight years, but I worked there four times. 503 00:26:14,160 --> 00:26:18,200 So I had an internship and then I had to go back to school, 504 00:26:18,200 --> 00:26:21,680 right? So I left and then I worked as a 505 00:26:21,720 --> 00:26:24,840 network security analyst on the incident response team. 506 00:26:25,520 --> 00:26:30,360 And that was actually a really fun job and, but, but I always 507 00:26:30,360 --> 00:26:32,560 knew like I wanted to try to start something, right? 508 00:26:32,560 --> 00:26:35,080 Like I like I like I wanted to. Kind of like try the same bug 509 00:26:35,480 --> 00:26:37,560 dude. It's like I so I, so I would 510 00:26:37,560 --> 00:26:39,680 just like leave, you know, I would, I put in a notice and 511 00:26:39,680 --> 00:26:41,200 everything and leave responsibly, right? 512 00:26:41,200 --> 00:26:44,640 But like I would always just like leave and, and then, you 513 00:26:44,640 --> 00:26:46,200 know, just start building, right? 514 00:26:46,200 --> 00:26:51,280 And, and, and then I would take on various contracts, like part 515 00:26:51,280 --> 00:26:55,200 time contracts, you know, in the meantime, right? 516 00:26:55,200 --> 00:26:57,960 Like kind of in between, right. So I was always kind of 1 foot 517 00:26:57,960 --> 00:27:01,520 in the startup space and then one foot at and Cisco and I 518 00:27:01,520 --> 00:27:04,320 never took the full leap. And then we'll be like when you 519 00:27:04,320 --> 00:27:06,800 get into YC, like they give you enough money to kind of like, 520 00:27:07,200 --> 00:27:09,040 you know, you, you got to do it right? 521 00:27:09,040 --> 00:27:11,680 Like you can't, you can't like have a new YC at the same time. 522 00:27:11,680 --> 00:27:17,440 So, so yeah, it's, it's, it's a good sort of onus to to get, to 523 00:27:17,440 --> 00:27:20,480 get started, but. I've I've I've always been the 524 00:27:20,520 --> 00:27:24,760 same way like like 1 foot in one foot out and then like I'll 525 00:27:24,760 --> 00:27:29,440 usually jump ship too soon and then just like add financial 526 00:27:29,440 --> 00:27:31,000 stress to my life on top of using the. 527 00:27:31,120 --> 00:27:34,600 Product oh and I've been there so many times and I gotta say 528 00:27:34,600 --> 00:27:39,400 like I think yeah, definitely at least you know, just coming 529 00:27:39,400 --> 00:27:43,240 coming from my my perspective like I wish I had done it 530 00:27:43,240 --> 00:27:47,480 sooner, right like I, I'm in I guess I'm in my late 30s now and 531 00:27:47,480 --> 00:27:51,160 like, you know kind of like I, I applied to YC in the past like, 532 00:27:51,200 --> 00:27:53,080 you know, got rejected. I think a lot of people don't 533 00:27:53,080 --> 00:27:54,360 don't get in their first time, but. 534 00:27:54,360 --> 00:27:57,040 Man, I'm sitting here. I'm almost 41. 535 00:27:57,520 --> 00:28:02,920 Oh, OK, not too far, but you know, I think the right way to 536 00:28:02,920 --> 00:28:05,640 do it or I don't want to say right, there's no right or wrong 537 00:28:05,640 --> 00:28:07,080 way to do it. I think what would have, what 538 00:28:07,080 --> 00:28:10,280 would have worked better for me is to probably do a little bit 539 00:28:10,280 --> 00:28:12,800 earlier and then yeah, and just go straight on and do and do the 540 00:28:12,800 --> 00:28:16,280 YC thing or do some, some kind of incubator And then because 541 00:28:16,280 --> 00:28:18,520 the financial stress thing was just like unnecessary. 542 00:28:18,520 --> 00:28:20,520 That that's a big bonus for sure. 543 00:28:20,560 --> 00:28:23,160 I'm like what, what was your experience like? 544 00:28:23,320 --> 00:28:26,440 Do they on board you well or like how do you what do you know 545 00:28:26,440 --> 00:28:29,400 what like what do you know what to expect when you join a 546 00:28:29,880 --> 00:28:34,480 incubator program? That's, well, it's, it's a lot 547 00:28:34,480 --> 00:28:38,480 of fun. Unfortunately for us, ours was 548 00:28:38,480 --> 00:28:41,680 the last batch that was fully remote because of COVID. 549 00:28:43,160 --> 00:28:46,840 But what I hear from other YC founders is that one of the big 550 00:28:47,240 --> 00:28:51,360 sort of value adds or, or I guess pros they get out of it is 551 00:28:51,360 --> 00:28:52,840 that they, they meet other people, right? 552 00:28:52,840 --> 00:28:54,280 It's like the whole social aspect. 553 00:28:54,480 --> 00:28:56,280 A bunch of people doing really cool stuff. 554 00:28:56,560 --> 00:29:00,440 To you meet all kind of amazing, amazingly talented people, right 555 00:29:00,560 --> 00:29:03,680 and from all over the world. And so we had a little bit of 556 00:29:03,680 --> 00:29:07,080 that because we would do sort of informal meetings, like outside 557 00:29:07,080 --> 00:29:10,400 of the day-to-day batch activities, but everything we 558 00:29:10,400 --> 00:29:13,120 did was remote. So yeah, so I'm sure it's a lot 559 00:29:13,120 --> 00:29:17,400 better now with the in person thing, but yeah, it was I have 560 00:29:17,400 --> 00:29:20,000 to say like it was probably one of the best. 561 00:29:20,600 --> 00:29:24,120 I mean it's a it's a you know, it's a it's a boot camp for 562 00:29:24,120 --> 00:29:27,000 business, right so it's probably one of the best experiences I've 563 00:29:27,000 --> 00:29:29,640 had. Obviously, you know, there's 564 00:29:29,640 --> 00:29:33,280 some stress involved, you know, and you're not, you're not. 565 00:29:33,280 --> 00:29:40,080 You don't want to be like, yeah, you want to try to have your, 566 00:29:40,080 --> 00:29:42,520 your life like free of other distractions. 567 00:29:42,560 --> 00:29:46,400 I'll say you don't want to have competing things going on. 568 00:29:46,400 --> 00:29:49,200 I would kind of sabotage your your time in NYC. 569 00:29:49,640 --> 00:29:54,320 But it was, yeah, it was, it was great. 570 00:29:54,680 --> 00:30:01,000 Would I do it again? Maybe if I, you know, not having 571 00:30:01,000 --> 00:30:03,640 done it, I would definitely do it one time, but you know what, 572 00:30:03,720 --> 00:30:05,040 where I would go back and do it again. 573 00:30:05,120 --> 00:30:06,680 I'm not. I have a huge fear of taking 574 00:30:06,720 --> 00:30:08,680 outside money. I don't know why but. 575 00:30:08,760 --> 00:30:12,000 So, yeah, I don't know how. Yeah. 576 00:30:12,040 --> 00:30:14,800 So I have I have opinion, opinions here. 577 00:30:14,800 --> 00:30:19,360 I won't share all of them. So I will say that Y CS great in 578 00:30:19,360 --> 00:30:24,280 the sense of they're not going to tell you what to do, right? 579 00:30:24,280 --> 00:30:27,240 Like they're like, you're going to get money, you're gonna, you 580 00:30:27,240 --> 00:30:31,120 know, you need to work on, you need to work on your thing, but 581 00:30:31,120 --> 00:30:34,080 you're not gonna get a call from, you know, from YC saying 582 00:30:34,080 --> 00:30:36,720 like, hey, why haven't you sold? You know, why aren't you at 583 00:30:36,720 --> 00:30:38,320 1,000,000 ARR yet? Right. 584 00:30:39,520 --> 00:30:44,960 So, so the so yeah, I, I would say, yeah. 585 00:30:44,960 --> 00:30:48,000 I mean, at least for me, that that part wasn't too bad when 586 00:30:48,000 --> 00:30:50,720 you. The thing about YC, though, is 587 00:30:51,280 --> 00:30:54,240 for, for those of you who I guess aren't, aren't too 588 00:30:54,240 --> 00:30:56,880 familiar with it, you, it's a 10 week program. 589 00:30:58,240 --> 00:31:03,240 You, you start and then each week you're sort of moving along 590 00:31:03,240 --> 00:31:07,080 towards this like demo day. And the way the whole program is 591 00:31:07,080 --> 00:31:09,640 set up and the way the demo day is set up is so that you sort of 592 00:31:09,640 --> 00:31:12,600 raise a round of investment, right? 593 00:31:12,600 --> 00:31:15,480 You, you raise, usually you're, you're, I guess your seed round, 594 00:31:17,800 --> 00:31:19,960 but it's, it could be different for, you know, depending on what 595 00:31:19,960 --> 00:31:23,280 stage your company's at, but the, the sort of like condensed 596 00:31:23,280 --> 00:31:25,000 to, you know, to hit that goal, right? 597 00:31:25,320 --> 00:31:27,720 And so, and that's what everyone's doing. 598 00:31:27,840 --> 00:31:30,200 And that's what we did, right? You, you go and you raise this 599 00:31:30,200 --> 00:31:32,600 money. Now when you raise that money, 600 00:31:32,640 --> 00:31:35,400 right? So if you got 2,000,003 million, 601 00:31:35,400 --> 00:31:38,800 4,000,005, whatever it is your, your seed round, you gotta start 602 00:31:38,800 --> 00:31:40,920 putting that to use, right? You can't just be like, oh, we 603 00:31:40,920 --> 00:31:44,200 need to redo our architecture for eight months, right? 604 00:31:44,760 --> 00:31:46,360 Yeah. So, so just think about that 605 00:31:46,360 --> 00:31:48,920 when you go through like try to understand that, that that's the 606 00:31:48,920 --> 00:31:51,360 advice I wish I had given my, my, you know, my former self. 607 00:31:51,360 --> 00:31:54,840 Like make sure you have something that or you, you at 608 00:31:54,840 --> 00:31:56,800 least know where you're going or you know, where your product's 609 00:31:56,800 --> 00:31:59,200 at before you raise that money. You don't have to raise money, 610 00:31:59,560 --> 00:32:01,000 right. When you, when you go through 611 00:32:01,000 --> 00:32:05,360 YC, you can, you can take the YC money and then and, and, and 612 00:32:05,360 --> 00:32:07,280 kind of like, you know, decide when you want to raise your next 613 00:32:07,280 --> 00:32:09,920 round. Just YC is a huge boost to to 614 00:32:09,920 --> 00:32:11,240 getting that first. Yeah. 615 00:32:12,240 --> 00:32:16,040 Like is it like milestone based or is it just when you get 616 00:32:16,040 --> 00:32:21,040 accepted into the program you get a chunk of seed, seed money 617 00:32:21,040 --> 00:32:24,240 essentially? So I think nowadays they give 618 00:32:24,240 --> 00:32:30,440 you it's half a million and yeah, as soon as you're 619 00:32:30,440 --> 00:32:33,400 accepted, they'll start setting you up and. 620 00:32:33,400 --> 00:32:37,120 That's a nice kick start. It's yeah, it feels, feels 621 00:32:37,120 --> 00:32:39,280 really good, Yeah. I mean coming from someone who 622 00:32:39,280 --> 00:32:41,280 who tried to bootstrap everything himself. 623 00:32:41,280 --> 00:32:43,560 Like it feels good. You get accepted and then you 624 00:32:43,560 --> 00:32:44,920 just get this chunk of money to go. 625 00:32:45,600 --> 00:32:50,080 You basically don't have to worry about making ends meet I 626 00:32:50,080 --> 00:32:52,120 guess for the next few years. Kind of wild. 627 00:32:52,800 --> 00:32:54,760 I like that. All right I'm gonna just quick 628 00:32:54,760 --> 00:33:01,080 rate a sponsor and then we'll continue How do I let's see so 629 00:33:02,080 --> 00:33:04,400 this is for gito. Sorry Mike, still haven't pre 630 00:33:04,400 --> 00:33:07,080 recorded. So I'm just talking coding 631 00:33:07,080 --> 00:33:10,440 Elixir with AI or building AI into your Elixir apps. 632 00:33:10,680 --> 00:33:14,440 The Elixir AI collective discord is your community for both Link 633 00:33:14,440 --> 00:33:19,080 is in the description below go to agent gito dot XYZ forward 634 00:33:19,080 --> 00:33:24,120 slash discord go check it out. It's a good group of guys all 635 00:33:24,240 --> 00:33:28,840 super obsessed with LLMS and all the coding. 636 00:33:28,840 --> 00:33:31,560 So it's a it's a cool discord to hang out with and Mike 637 00:33:31,600 --> 00:33:34,600 Hostetler, he's a cool dude if you guys don't know him. 638 00:33:35,760 --> 00:33:42,400 All right, we're back. All right, let's see. 639 00:33:43,360 --> 00:33:47,600 Oh I'm kind of like. So this stuff I feel like is so 640 00:33:47,600 --> 00:33:50,880 over my head. Like DNS layer, whatever it may 641 00:33:50,880 --> 00:33:52,160 be. Can you kind? 642 00:33:52,160 --> 00:33:58,640 Of explain like what? What's like snownet and how does 643 00:33:58,640 --> 00:34:01,840 it work and why? Why is it important? 644 00:34:04,200 --> 00:34:08,000 Yeah. So Thomas is, is our lead on, on 645 00:34:08,000 --> 00:34:12,040 all of that stuff and on on the Russ, on the Russ side and he 646 00:34:12,040 --> 00:34:14,600 architected that. So let me let me kind of back up 647 00:34:14,600 --> 00:34:18,679 a little bit. I guess I won't get into what 648 00:34:18,679 --> 00:34:20,480 AVPN is. That's that's a that's a that's 649 00:34:20,480 --> 00:34:23,360 a longer, that's a whole, you know, it's a longer story, but 650 00:34:25,320 --> 00:34:27,800 so well, I think. We have like a macro like. 651 00:34:27,800 --> 00:34:32,360 What is AVPN like? I'll yeah, so so the Internet, 652 00:34:34,080 --> 00:34:36,840 yeah. So I have a post on our blog 653 00:34:36,840 --> 00:34:39,520 about this that goes way back to the history of the Internet. 654 00:34:41,120 --> 00:34:46,400 But so, so AVPN essentially at least in for for our purposes, 655 00:34:46,400 --> 00:34:47,600 right. So there's, there's two main 656 00:34:47,600 --> 00:34:49,440 types of VPNs. There's like a personal VPN 657 00:34:49,760 --> 00:34:53,400 which is mostly too exposed or to, not to expose, but to, to 658 00:34:53,400 --> 00:34:56,520 hide to prevent you from exposing your, your traffic, 659 00:34:56,520 --> 00:35:00,480 your net traffic until it reaches some end of destination, 660 00:35:00,520 --> 00:35:02,360 right? So it's like protects that first 661 00:35:02,360 --> 00:35:06,240 hop typically if you're working from a cafe or something like 662 00:35:06,240 --> 00:35:07,520 that, right? Just make sure all of your 663 00:35:07,520 --> 00:35:10,200 traffic is default encrypted. Sorry to like derail this. 664 00:35:10,400 --> 00:35:13,480 What's the difference between like AVPN and a proxy? 665 00:35:15,280 --> 00:35:17,920 So a proxy's typically application layer based, so that 666 00:35:17,920 --> 00:35:22,560 would be like an Https://proxy. So you've got your OSI model. 667 00:35:23,040 --> 00:35:25,840 That would be like a layer seven thing all the way. 668 00:35:25,840 --> 00:35:28,400 Yeah. So, so if you're using any other 669 00:35:28,400 --> 00:35:31,600 application protocol that's not HTTPS, like if you're connecting 670 00:35:31,600 --> 00:35:34,440 to your Postgres database that's not gonna go through the VPS, 671 00:35:34,560 --> 00:35:37,280 that's not gonna be proxy. You could do, well, you could do 672 00:35:37,320 --> 00:35:40,120 a Postgres proxy, but it's not gonna be your typical, you know, 673 00:35:40,120 --> 00:35:42,480 standard Https://proxy. OK. 674 00:35:42,920 --> 00:35:46,200 And then AVPN sits all the way down in layer three, layer 4, 675 00:35:46,200 --> 00:35:52,000 sometimes layer 2. That's going to be your, that is 676 00:35:52,000 --> 00:35:54,880 your, you know, that's going to just route and tunnel all 677 00:35:54,880 --> 00:35:56,560 traffic, right. So it doesn't matter what 678 00:35:56,560 --> 00:35:59,480 application is trying to, you know, communicate out there, 679 00:35:59,880 --> 00:36:01,760 it's going to get wrapped up in in the VPN. 680 00:36:02,400 --> 00:36:03,000 OK. Right. 681 00:36:03,000 --> 00:36:04,120 Yeah, OK. Yeah. 682 00:36:04,520 --> 00:36:07,720 So, so the personal VPN is typically concerned with 683 00:36:08,120 --> 00:36:12,120 protecting your traffic right, from from prying eyes, I would 684 00:36:12,120 --> 00:36:16,200 say. And then a corporate VPN is to 685 00:36:16,200 --> 00:36:20,560 get is it does that as well, but it also sort of authenticates 686 00:36:20,560 --> 00:36:24,000 you or ties your identity to what used to be, you know, the 687 00:36:24,000 --> 00:36:26,880 corporate Internet, right. So you get into the perimeter. 688 00:36:26,880 --> 00:36:30,080 So for these like hybrid workforces, you've got the 689 00:36:30,200 --> 00:36:32,880 office network. So in Cisco, we had the, you 690 00:36:32,880 --> 00:36:36,320 know, over in like Milpitas, right, We had the, the HQ you 691 00:36:36,320 --> 00:36:38,840 come in, you're connected to the, you know, when you badge in 692 00:36:38,840 --> 00:36:42,600 you're, you're in this sort of perimeter, so to speak, You're 693 00:36:42,600 --> 00:36:45,400 in the protected zone and you're in the Internet. 694 00:36:45,560 --> 00:36:48,320 And typically, you know, you wouldn't use a VPN then, but 695 00:36:48,320 --> 00:36:53,080 then as soon as you went home, you would need some way to get, 696 00:36:53,720 --> 00:36:55,360 since you're not physically in the office, you need some 697 00:36:55,360 --> 00:36:57,680 virtual way to get to that network, right? 698 00:36:58,000 --> 00:37:00,200 And so you're sending these packets through. 699 00:37:00,200 --> 00:37:03,920 And so the, the corporate VPN is, is a way to put, you know, 700 00:37:03,920 --> 00:37:07,120 it's like putting an envelope inside a bigger package and then 701 00:37:07,120 --> 00:37:09,080 like sending that. And then it gets unwrapped and 702 00:37:09,160 --> 00:37:12,440 you know, it's it that's, that's, that's all of AVPN as a 703 00:37:12,440 --> 00:37:15,960 technology is packets and packets. 704 00:37:16,640 --> 00:37:20,880 So, so those in a corporate setting, those packets to get to 705 00:37:20,880 --> 00:37:25,560 the firewall, you know, the VPN, they they're unwrapped, it says, 706 00:37:25,560 --> 00:37:27,240 you know, this is Jameel or, or whoever. 707 00:37:27,320 --> 00:37:30,040 And then you're, it's like you're sitting on that Internet. 708 00:37:32,120 --> 00:37:33,640 So how does this play into Fire Zone? 709 00:37:33,640 --> 00:37:36,440 So Fire Zone. So, so the model I just 710 00:37:36,440 --> 00:37:40,680 described is what's known as you could, you could say perimeter 711 00:37:40,680 --> 00:37:42,800 based security. But as soon as you're in the 712 00:37:42,800 --> 00:37:44,720 Internet like you can, you can sort of talk to everyone. 713 00:37:44,720 --> 00:37:48,240 You're, it's, you're, I guess I wouldn't say authenticated, but 714 00:37:48,240 --> 00:37:52,000 you're, you're, you're in a more trusted, coming from a more 715 00:37:52,000 --> 00:37:52,600 trusted zone. OK. 716 00:37:54,080 --> 00:37:59,840 The, the model that Firezone that we're, that I would, you 717 00:37:59,920 --> 00:38:03,600 know, the, the, the vision for Firezone, you could say, or the, 718 00:38:03,800 --> 00:38:08,000 you know what, what we call Firezone as a product is a zero 719 00:38:08,000 --> 00:38:12,160 trust access. And so the term zero trust just 720 00:38:12,160 --> 00:38:19,960 comes from this notion that you don't have this trusted, you 721 00:38:19,960 --> 00:38:22,200 don't have a trusted, A trusted network anymore, right? 722 00:38:22,200 --> 00:38:24,640 There's no, there's no perimeter or you're moving the perimeter 723 00:38:24,680 --> 00:38:27,160 much closer to the resources that you're trying to access. 724 00:38:28,040 --> 00:38:33,880 And, and so as part of that, there are many more, potentially 725 00:38:34,320 --> 00:38:37,440 many more little VPN tunnels that you need to make to get 726 00:38:37,440 --> 00:38:38,720 closer to those resources, right? 727 00:38:38,720 --> 00:38:41,480 So let's say you've got a database server and a web server 728 00:38:41,480 --> 00:38:45,920 and a, you know, a file server. So the old way of doing it with 729 00:38:45,920 --> 00:38:48,800 the perimeters, you put one firewall, let's say in front of 730 00:38:48,800 --> 00:38:50,200 it. You've got all of these things 731 00:38:50,200 --> 00:38:52,480 in the, in the giant intranet. And then you've got one VPN 732 00:38:52,480 --> 00:38:55,440 tunnel to that firewall. And it may be doing like full 733 00:38:55,440 --> 00:38:59,000 tunnel, you know, into the old, you know, kind of typical cases, 734 00:38:59,000 --> 00:39:00,000 you'd be doing full tunnel drive. 735 00:39:00,000 --> 00:39:02,360 You're sending all of your YouTube video watching and 736 00:39:02,360 --> 00:39:03,720 everything like through this tunnel. 737 00:39:04,200 --> 00:39:08,520 And, and then you would get to the the intranet and then you'd 738 00:39:08,520 --> 00:39:13,560 access those resources behind. And so with zero trust at at 739 00:39:13,560 --> 00:39:15,720 least the way that we're approaching it is you move that 740 00:39:15,720 --> 00:39:19,400 perimeter, which is run by the Fire Zone gateway, you move that 741 00:39:19,400 --> 00:39:21,880 closer to the resources. So you'd put, you know, you may 742 00:39:21,880 --> 00:39:24,880 have a gateway near the the file server, the web server, the 743 00:39:24,880 --> 00:39:26,960 Postgres server, the database server. 744 00:39:26,960 --> 00:39:28,600 And then those would be 3 tunnels, right? 745 00:39:30,080 --> 00:39:35,560 Yeah. So so so so I'm I'm not quite to 746 00:39:35,560 --> 00:39:38,920 where to what snow net is yet, but but. 747 00:39:39,680 --> 00:39:41,680 But I'm learning a lot so. Yeah, yeah. 748 00:39:41,680 --> 00:39:46,520 So, so, so you have and So what? So it kind of plays into, you 749 00:39:46,520 --> 00:39:49,200 know, Wireguard helps enable this because it's so 750 00:39:49,200 --> 00:39:52,120 lightweight. You can open even thousands of 751 00:39:52,120 --> 00:39:53,880 tunnels, right. You couldn't do this. 752 00:39:53,880 --> 00:39:56,400 You can do this with with a, with another VPN protocol, by 753 00:39:56,400 --> 00:39:59,000 the way. So so you can't so you know you 754 00:39:59,000 --> 00:40:04,520 can't micro segment your your network to reach each resource. 755 00:40:04,960 --> 00:40:10,280 I'm I'm assuming that in like plummets like latency issues and 756 00:40:10,440 --> 00:40:13,840 like it keeps speed up because everything has got a purpose. 757 00:40:13,880 --> 00:40:15,280 Right. But the latency breaking out 758 00:40:15,360 --> 00:40:18,760 traffic, yeah. So you've so instead of doing 759 00:40:18,760 --> 00:40:21,360 like a full tunnel route to send all of your traffic right to 760 00:40:21,360 --> 00:40:26,400 your corporate concentrator, you're doing what's called split 761 00:40:26,880 --> 00:40:30,920 tunnel routing and you're sending just the intended 762 00:40:30,920 --> 00:40:33,680 traffic through, right. So that's Fire Zone by default 763 00:40:33,680 --> 00:40:39,280 is a split tunnel PPN. And so, yeah, why we, you know, 764 00:40:39,280 --> 00:40:40,880 Wireguard sort of helps enable that. 765 00:40:42,200 --> 00:40:45,680 And so, you know, from a, from a high level, that's really all 766 00:40:45,680 --> 00:40:48,160 Fire Zone is, is like, you know, it, you configure the routing 767 00:40:48,160 --> 00:40:51,040 rules, we call them other things, we call them policies 768 00:40:51,040 --> 00:40:52,880 and groups and, and things like that. 769 00:40:52,880 --> 00:40:55,720 But you, you know, at the end of the day, your, your client is 770 00:40:55,720 --> 00:41:00,520 receiving a, you know, a routing table from Firezone or routing 771 00:41:00,520 --> 00:41:02,640 rules from Firezone, applying them to the routing table. 772 00:41:03,000 --> 00:41:05,720 And that decides like whether your packets go over here to 773 00:41:05,720 --> 00:41:07,880 this gateway, this corporate gateway or this corporate 774 00:41:07,880 --> 00:41:13,280 gateway. And, and yeah, so, so, so that's 775 00:41:13,280 --> 00:41:17,000 sort of at a, at a high level, what the control plane's primary 776 00:41:17,000 --> 00:41:22,920 responsibility is in Firezone now getting into Snownet and 777 00:41:22,920 --> 00:41:28,600 sort of the Rust layer. So one of the big things with 778 00:41:28,880 --> 00:41:33,120 that we learned with our with with that legacy product that we 779 00:41:33,120 --> 00:41:40,760 had was so a couple things. So this is, this is, you know, 780 00:41:41,360 --> 00:41:46,800 some, I guess more technical drivers, but you had to open a 781 00:41:46,800 --> 00:41:49,440 firewall port right now Wireguard is, you know, it's 782 00:41:49,440 --> 00:41:54,040 like provably secure. It's, it's, you know, most 783 00:41:54,040 --> 00:41:56,320 people, I guess don't have a problem opening a port for, for 784 00:41:56,320 --> 00:41:57,600 wire. Some people still want to 785 00:41:57,600 --> 00:41:59,200 protect it, right? They, they want to keep it 786 00:41:59,200 --> 00:42:06,960 closed off, keep it invisible. And so we had, we had and then 787 00:42:06,960 --> 00:42:12,480 so that's, that's sort of #1. And then number 2 was failover 788 00:42:12,480 --> 00:42:16,120 and high availability, right? So if you can imagine your 789 00:42:16,120 --> 00:42:18,520 packets have to go to a destination, right? 790 00:42:18,600 --> 00:42:23,400 Typically, like if you just in a sort of basic network, if you 791 00:42:23,400 --> 00:42:26,920 open a, a port for a service, they have to go to that port and 792 00:42:26,920 --> 00:42:28,640 typically there's a service behind it, right? 793 00:42:31,200 --> 00:42:34,200 If you then want the packets to fail over to a different 794 00:42:34,400 --> 00:42:36,320 service, right? Like let's say that server goes 795 00:42:36,320 --> 00:42:40,120 down, your wire guard server goes down, you need something to 796 00:42:40,120 --> 00:42:42,400 to steer the packets over, right? 797 00:42:42,400 --> 00:42:45,880 Like, so you either got to put a load balancer in front of it, or 798 00:42:45,880 --> 00:42:47,760 you've got to configure the clients and then talk to 799 00:42:47,760 --> 00:42:55,400 another, another one, right? And so in Fire Zone, we, we 800 00:42:55,400 --> 00:42:58,440 chose the latter. And So what this meant was that 801 00:42:58,600 --> 00:43:04,440 we had to build, we had to build mechanisms to sort of steer the 802 00:43:04,440 --> 00:43:07,000 packets and set up connections on the fly, right? 803 00:43:07,000 --> 00:43:09,360 It wasn't just going to be, hey, here's your static wire guard 804 00:43:09,360 --> 00:43:10,720 config. You're going to always talk to 805 00:43:10,720 --> 00:43:15,000 this end point and. Because that's not guaranteed to 806 00:43:15,000 --> 00:43:15,960 always be there. Right. 807 00:43:15,960 --> 00:43:17,600 It's not guaranteed to always be there. 808 00:43:17,600 --> 00:43:20,800 It requires you to open a port. And so, so that, that kind of 809 00:43:20,800 --> 00:43:24,880 leads me to the second part of this, which is the the Nat 810 00:43:24,880 --> 00:43:27,280 traversal or or hole punching, right. 811 00:43:27,280 --> 00:43:30,360 So there's a trick you can do with UDP. 812 00:43:30,440 --> 00:43:33,760 It works for TCP too, but most commonly used with UDP. 813 00:43:33,760 --> 00:43:37,560 And actually on our call, we're probably using it as well via 814 00:43:37,560 --> 00:43:40,520 web RTC. So it's a way there's a trick 815 00:43:40,520 --> 00:43:45,040 you can do where you can, you can whole bunch connection from 816 00:43:45,080 --> 00:43:47,840 from one peer behind a firewall on the Internet to another peer 817 00:43:47,840 --> 00:43:50,440 behind the firewall on the Internet so that they're talking 818 00:43:50,440 --> 00:43:55,520 peer-to-peer, right which? It is a video ninja, I'm pretty 819 00:43:55,520 --> 00:43:57,680 sure, yeah. Yeah, it's, well, it's every 820 00:43:57,680 --> 00:44:00,920 browser, you know, web RTC uses this, this bag of tricks, right? 821 00:44:00,920 --> 00:44:05,400 So, so it's a, it's a collection of, of what of it's a collection 822 00:44:05,400 --> 00:44:08,840 of, of techniques. It's collectively known as ICE. 823 00:44:09,880 --> 00:44:12,240 So there's Rs CS for this. You know, we didn't invent this. 824 00:44:12,240 --> 00:44:15,920 We so Thomas, Thomas is our, is our rust lead and he implemented 825 00:44:15,920 --> 00:44:17,760 all of this and he calls it snow. 826 00:44:17,880 --> 00:44:20,680 You know, we we named the night of the library Snownet because 827 00:44:20,680 --> 00:44:23,520 it's it's, it implements ice, the ICE standard. 828 00:44:23,520 --> 00:44:24,880 Oh, I like that, I think. OK. 829 00:44:24,960 --> 00:44:27,640 ICE is, I think, interactive connectivity establishment. 830 00:44:28,000 --> 00:44:33,440 OK. So, so happy to let's see I can, 831 00:44:33,600 --> 00:44:36,080 I can talk about how that works. That might be a little too deep, 832 00:44:36,080 --> 00:44:39,480 but but it, but essentially it allows you to establish a 833 00:44:39,480 --> 00:44:44,720 connection and end 95 to 90% of the time it'll be peer-to-peer 834 00:44:45,440 --> 00:44:49,160 just depending on the firewall or the Nat devices on either 835 00:44:49,160 --> 00:44:52,280 side, if either one of them misbehaves, then you've got to 836 00:44:52,280 --> 00:44:54,600 go through a relay. And so that that's the whole 837 00:44:54,600 --> 00:44:57,200 thing. And so that that's basically the 838 00:44:57,200 --> 00:45:00,320 purpose that Snownet serves. No, I think that's plenty deep, 839 00:45:00,320 --> 00:45:01,800 especially just for a conversation. 840 00:45:01,800 --> 00:45:05,160 You know, yeah, OK. Yeah, no, I appreciate it. 841 00:45:05,160 --> 00:45:08,320 And I like I, I, you know, I saw these words, these trigger 842 00:45:08,320 --> 00:45:10,600 words, and I'm like, I'm going to just sound like an idiot, but 843 00:45:10,680 --> 00:45:13,200 I'm going to ask. No, that's OK. 844 00:45:13,240 --> 00:45:16,720 Yeah, that's. Now. 845 00:45:16,720 --> 00:45:18,840 Yeah, I, I, I'm in this stuff day-to-day, so. 846 00:45:19,360 --> 00:45:21,520 No, I love it. I love hearing about it. 847 00:45:21,520 --> 00:45:25,280 I'm kind of curious like, all right, how does like the actor 848 00:45:25,280 --> 00:45:29,360 model and like, you know, the concurrency layer and all that, 849 00:45:29,360 --> 00:45:32,840 how does that fit in? Like how does Elixir fit in 850 00:45:32,960 --> 00:45:38,200 after the Rust layer? Yeah, that's a great question. 851 00:45:39,000 --> 00:45:46,000 So, so, so that's a, it's actually, yeah, that's a good 852 00:45:46,000 --> 00:45:48,080 segue. So I mentioned, you know, we set 853 00:45:48,080 --> 00:45:49,640 up connections on the fly, right. 854 00:45:49,640 --> 00:45:55,920 So the way the Fire Zone client works is let's say you wanna 855 00:45:55,920 --> 00:45:59,480 talk to like google.com or maybe that's a bad example. 856 00:45:59,480 --> 00:46:01,800 Let's say you know, let's go back to the file server example. 857 00:46:01,800 --> 00:46:04,040 So let's say you're, you know, you want to reach this file 858 00:46:04,040 --> 00:46:07,520 server and it's running at like 10.1 dot 1.1. 859 00:46:07,600 --> 00:46:11,640 Let's say something like that. In the Fire Zone client, we'll 860 00:46:11,640 --> 00:46:17,480 see a packet for that server and we'll set up a connection to the 861 00:46:17,480 --> 00:46:19,720 gateway. Now the way the connection setup 862 00:46:19,720 --> 00:46:23,960 works is we will send what's called a, we call it a resource 863 00:46:23,960 --> 00:46:26,640 intent. So it's an API message from the 864 00:46:26,640 --> 00:46:30,000 client to our control plane, and this is over a web socket. 865 00:46:30,000 --> 00:46:34,160 OK, so that goes up. We run this through what we call 866 00:46:34,160 --> 00:46:38,720 the policy engine, which approves or denies the request. 867 00:46:39,640 --> 00:46:43,040 If the request is approved, we'll then talk to the gateway 868 00:46:43,040 --> 00:46:44,800 and say, hey, this client's trying to connect to you. 869 00:46:44,920 --> 00:46:47,320 And then Snownet kicks in and it'll do the dance. 870 00:46:47,320 --> 00:46:51,080 Now, as part of that, the keys need to be distributed, right? 871 00:46:51,080 --> 00:46:55,920 The wire guard keys themselves. So, so, so the connection will 872 00:46:55,920 --> 00:47:00,480 be will be established on the data plane, we'll log it 873 00:47:00,520 --> 00:47:05,120 basically in the control plane and all of that happens sort of 874 00:47:05,120 --> 00:47:08,320 within about 500 milliseconds anywhere you are in the world. 875 00:47:08,720 --> 00:47:15,760 OK, so where Elixir comes in is we do all of this with actually 876 00:47:15,760 --> 00:47:19,920 Phoenix presence. So we have a web socket. 877 00:47:19,920 --> 00:47:23,000 Pub sub essentially. Yeah, we use pub sub exactly. 878 00:47:23,000 --> 00:47:26,520 And so cuz we've got these, we've got app servers all over 879 00:47:26,520 --> 00:47:30,160 the world. So you may, as a client device, 880 00:47:30,640 --> 00:47:34,680 my web socket may be connected to an app server in San Jose and 881 00:47:34,680 --> 00:47:37,560 then the Gateways app server may be in, you know, Sao Paulo or 882 00:47:37,560 --> 00:47:40,960 something like that, right. And, and so, yeah, so we've got 883 00:47:40,960 --> 00:47:43,680 a giant Erlang cluster. And so whenever I need to send 884 00:47:43,680 --> 00:47:46,160 this resource intent, this connection intent that actually 885 00:47:46,160 --> 00:47:48,520 the keys actually get distributed through our Erlang 886 00:47:48,520 --> 00:47:53,240 cluster over to the gateway. And so we use Elixir as our 887 00:47:53,240 --> 00:47:56,440 signaling layer for ice as well. OK, cool. 888 00:47:56,440 --> 00:47:58,320 So it's in the hot path of connection setup. 889 00:47:58,320 --> 00:48:01,080 So it needs to be. Passing, you're just like. 890 00:48:01,360 --> 00:48:03,480 Exactly. Yeah, exactly. 891 00:48:03,480 --> 00:48:06,800 And, you know, and, and, and there's some, there's some logic 892 00:48:06,800 --> 00:48:09,280 in there to like, you know, it finds a healthy gateway that's 893 00:48:09,280 --> 00:48:10,640 running. And so that's how we do our load 894 00:48:10,640 --> 00:48:13,320 balancing and our failover. So whenever you're trying to, 895 00:48:13,600 --> 00:48:15,800 you set up a connection to, to something you want to talk to. 896 00:48:15,800 --> 00:48:17,800 Yeah, we do it. We all, we, we handle that with 897 00:48:17,800 --> 00:48:19,720 Elixir too. OK, cool. 898 00:48:20,280 --> 00:48:21,760 Yeah. And so we do all of the 899 00:48:21,760 --> 00:48:26,040 configuration, you know, if, if, if an admin goes in and says 900 00:48:26,040 --> 00:48:28,360 like, hey, you no longer, you know, are allowed to talk to 901 00:48:28,360 --> 00:48:32,160 this database server or what have you, we, we, we push those 902 00:48:32,160 --> 00:48:35,240 messages down to the. You're really kind of just 903 00:48:36,000 --> 00:48:39,360 you're solving like the phone system problem that Erlang 904 00:48:39,360 --> 00:48:43,680 solved, but just at like network connectivity level instead. 905 00:48:43,680 --> 00:48:44,800 That's a great way to look at it. 906 00:48:44,800 --> 00:48:45,920 Yeah. That, that is it. 907 00:48:46,280 --> 00:48:47,680 Yeah, that's a good way to look at it. 908 00:48:48,560 --> 00:48:53,800 Yeah, that's pretty cool. I'm curious like cuz like I've 909 00:48:53,800 --> 00:48:58,240 said, I'm like been part of just like small bootstrap, you know, 910 00:48:58,240 --> 00:49:00,920 a couple 1000 users. I usually don't even need a 911 00:49:00,920 --> 00:49:03,480 distributed system like 1 server is enough. 912 00:49:03,760 --> 00:49:05,360 What does that look like for you? 913 00:49:05,360 --> 00:49:09,040 Like how many clusters or nodes do you need? 914 00:49:09,200 --> 00:49:12,360 That's in the world, yeah. This has been fresh in our mind. 915 00:49:12,360 --> 00:49:15,280 So we've been we've been well, I wouldn't say we are taking we've 916 00:49:15,280 --> 00:49:18,440 been we've been migrating clouds. 917 00:49:18,440 --> 00:49:27,120 So, so this is all sort of fresh, but we, we've got so, so 918 00:49:27,120 --> 00:49:29,480 because connection set up is in the hot, what we call the hot 919 00:49:29,480 --> 00:49:32,280 path, right? Anything you slow down there is 920 00:49:32,280 --> 00:49:34,320 going to sort of ruin the experience for your end users, 921 00:49:34,760 --> 00:49:36,520 right? So you don't want to slow down 922 00:49:36,520 --> 00:49:38,800 packets as much as you can, right, data packets. 923 00:49:38,800 --> 00:49:41,520 So, so we try to make the connection set up as fast as 924 00:49:41,520 --> 00:49:43,160 possible. And to do that, you know, we've 925 00:49:43,160 --> 00:49:45,560 got to have app servers pretty close. 926 00:49:46,920 --> 00:49:49,960 We want, we want to make sure that the total, the round trip 927 00:49:49,960 --> 00:49:53,440 time from a client through an app server to its nearest 928 00:49:53,440 --> 00:49:58,440 gateway is as short as possible. So I guess I yeah, so we're 929 00:49:58,440 --> 00:50:01,760 moving, we're in the process of moving from GCP to Azure. 930 00:50:01,760 --> 00:50:04,360 But really it doesn't matter which cloud provider. 931 00:50:05,800 --> 00:50:09,600 The only important thing is that we have to use a hosting 932 00:50:09,600 --> 00:50:12,480 provider that's got essentially servers everywhere. 933 00:50:12,760 --> 00:50:15,600 Servers everywhere. And so they're hyperscalers, 934 00:50:15,720 --> 00:50:17,400 yeah. Is it like auto scaling? 935 00:50:17,400 --> 00:50:20,240 Like if you have a demand, you kind of spin up a node. 936 00:50:20,880 --> 00:50:23,720 No, we, we, we don't really need that yeah. 937 00:50:23,720 --> 00:50:27,560 We don't use we, we, we've got we just use virtual machines and 938 00:50:27,560 --> 00:50:30,360 and we deploy with Docker. We just got, you know, kind of 939 00:50:30,360 --> 00:50:33,840 basic Ubuntu, you know, so you're. 940 00:50:33,840 --> 00:50:37,120 Manually spinning things up depending. 941 00:50:37,120 --> 00:50:39,640 Yeah, we've, we've got a manual, yeah, 'cause I mean, our, our 942 00:50:39,640 --> 00:50:42,560 load, at least on the application servers is quite it. 943 00:50:42,800 --> 00:50:44,640 You know, we're not like Netflix where we've got, you know, 944 00:50:44,640 --> 00:50:47,720 hordes of people just kind of coming online, like depending on 945 00:50:47,720 --> 00:50:51,280 the region, I guess we do. But from a, from a general 946 00:50:51,280 --> 00:50:54,200 cluster perspective, it's pretty even load. 947 00:50:55,440 --> 00:50:59,840 And so yeah, rather than deal with the the churn of like 948 00:50:59,840 --> 00:51:02,240 adding new nodes to the cluster and then like suddenly removing 949 00:51:02,240 --> 00:51:05,120 them and like, like that creates some overhead, right? 950 00:51:05,120 --> 00:51:08,360 Like, you know, and, and like the, you know, the, the, the 951 00:51:08,360 --> 00:51:10,840 presence state track is great. Like we haven't really had any 952 00:51:10,840 --> 00:51:13,680 issues with it, but we'd just avoid that if we can. 953 00:51:14,720 --> 00:51:22,280 So we have a consistent number of of Elixir nodes, what we call 954 00:51:22,280 --> 00:51:25,800 our relays that are built in Rust and those are those are 955 00:51:25,800 --> 00:51:29,360 everywhere, right. Those are in every region we can 956 00:51:29,360 --> 00:51:31,680 we put we put a relay because those are those are very 957 00:51:31,680 --> 00:51:34,480 critical to to latency. Yeah, OK. 958 00:51:34,520 --> 00:51:36,960 Yeah, that makes sense, especially cuz you just you 959 00:51:36,960 --> 00:51:40,200 mentioned like 500 milliseconds and I'm guessing that you don't 960 00:51:40,200 --> 00:51:42,960 wanna go above that cuz then. Well, that's just for the setup. 961 00:51:42,960 --> 00:51:45,240 So once we set up the connection, then the client 962 00:51:45,480 --> 00:51:47,520 talks directly to the gateway and then it's just whatever 963 00:51:47,520 --> 00:51:49,480 they're they're linking with each other. 964 00:51:49,480 --> 00:51:51,040 Yeah. So just kind of pass through at 965 00:51:51,040 --> 00:51:52,800 that, then at that. Point, it's just like when you 966 00:51:52,800 --> 00:51:55,680 go to like, yeah, you go to likegithub.com, enter, it's like 967 00:51:55,800 --> 00:51:58,120 500 milliseconds. And then, yeah, we'll start, and 968 00:51:58,120 --> 00:52:00,480 then the page will start. Yeah, that's pretty good. 969 00:52:00,520 --> 00:52:04,440 That's awesome. I like I so collects. 970 00:52:04,880 --> 00:52:11,080 We I'm a back end engineer there and we have like 4 million users 971 00:52:11,080 --> 00:52:13,360 now. And so it's like it's the first 972 00:52:13,360 --> 00:52:16,600 platform I've been on where like query, like as you mentioned, 973 00:52:16,600 --> 00:52:19,640 query performance and things like it's a different, it's a 974 00:52:19,640 --> 00:52:22,400 different ball game. Like I remember like week 1 of 975 00:52:22,400 --> 00:52:26,520 joining the team. I I pushed out of my first fix 976 00:52:26,520 --> 00:52:31,200 and I like crashed production all because I built a nasty 977 00:52:31,200 --> 00:52:35,480 query and like wasn't used to dealing with like a ton of 978 00:52:35,480 --> 00:52:39,760 requests in a second and then just like large data tables, you 979 00:52:39,760 --> 00:52:43,560 know? And yeah, we, yeah, we've done, 980 00:52:44,640 --> 00:52:47,600 yeah, we've done similar things. I mean, we. 981 00:52:47,720 --> 00:52:51,600 So yeah, I could talk about that a little bit like, and So what 982 00:52:51,600 --> 00:52:54,840 we've recently done is we've moved to having more read 983 00:52:54,840 --> 00:52:58,320 replicas around near 1 of near each of these app clusters. 984 00:52:58,320 --> 00:53:01,920 And they've been super useful because our first, our, our 985 00:53:01,920 --> 00:53:05,000 heaviest query is when a client first connects and it needs to 986 00:53:05,000 --> 00:53:07,760 get all of the list of resources that it has that it can connect 987 00:53:07,800 --> 00:53:09,640 to. So that requires that's like a 988 00:53:09,640 --> 00:53:11,280 join across like 3 or 4 different tables. 989 00:53:11,280 --> 00:53:13,080 And we call it our hydration query. 990 00:53:13,080 --> 00:53:15,720 From there it's cached. And then we only update the 991 00:53:15,720 --> 00:53:18,880 cache from that point forward. But so those now hit the 992 00:53:18,880 --> 00:53:21,880 replicas and so yeah, that that that's been a lot better. 993 00:53:21,880 --> 00:53:27,560 But our architecture before this was a little bit different. 994 00:53:28,000 --> 00:53:37,200 And so, so, so how it worked before was we had, it was we, we 995 00:53:37,200 --> 00:53:39,400 have to continuously solve this problem of. 996 00:53:40,480 --> 00:53:43,680 So the way that we, we do access control in Firezone is based on 997 00:53:43,680 --> 00:53:45,480 what group you're in. So if you're in like, if, if 998 00:53:45,480 --> 00:53:48,880 your company uses like Google Workspace or like intra right, 999 00:53:48,880 --> 00:53:52,560 or Okta or something like that, you're going to have users and 1000 00:53:52,560 --> 00:53:57,440 groups. And in Firezone, we say you need 1001 00:53:57,440 --> 00:53:59,960 to create a policy to get access to anything. 1002 00:53:59,960 --> 00:54:02,600 And that what a policy is, is basically just a join. 1003 00:54:02,600 --> 00:54:06,440 It's a fancy join table between a group and a resource. 1004 00:54:07,320 --> 00:54:10,720 OK, So depending on what membership you have, like in in 1005 00:54:10,720 --> 00:54:14,720 which groups determines what resources you have access to. 1006 00:54:15,360 --> 00:54:21,160 And let's say that, you know, we do like an identity sync and you 1007 00:54:21,160 --> 00:54:23,600 got removed from you like you switched teams or something like 1008 00:54:23,600 --> 00:54:24,680 that, like whatever it is, right? 1009 00:54:24,680 --> 00:54:27,120 And you got removed from the DevOps group. 1010 00:54:27,120 --> 00:54:30,800 And so we've got to evaluate, you know, we've got to re re re 1011 00:54:30,800 --> 00:54:32,960 evaluate what resources you have access to. 1012 00:54:33,640 --> 00:54:35,880 And we've got to push that down to the client and push it down 1013 00:54:35,880 --> 00:54:38,000 to the gateway, right? Very quickly. 1014 00:54:39,480 --> 00:54:44,600 And so rather than what we used to do was like anytime each one 1015 00:54:44,600 --> 00:54:47,760 of these things changed, right? Anything that could potentially 1016 00:54:47,760 --> 00:54:50,520 change your, what we call your resource list and your client, 1017 00:54:50,520 --> 00:54:51,800 your list of resource you have access to. 1018 00:54:52,440 --> 00:54:55,080 Any time 1 of that, one of those changed, we would just go and 1019 00:54:55,080 --> 00:54:59,480 just rerun the, you know, the the four join right four table 1020 00:54:59,480 --> 00:55:04,760 join query and and we started to hit problems pretty quickly, as 1021 00:55:04,760 --> 00:55:08,960 you might imagine with that. And so we moved to using, looked 1022 00:55:08,960 --> 00:55:13,640 into a few different solutions like listen, notify, you know, 1023 00:55:13,680 --> 00:55:18,160 we use Postgres as our as our database and found out. 1024 00:55:18,160 --> 00:55:22,360 And then we discovered, I don't remember how I, I found out 1025 00:55:22,360 --> 00:55:28,200 about I, I think I, I don't remember, I think it was, I 1026 00:55:28,200 --> 00:55:30,800 might have been Superbase. I know Superbase uses this, but 1027 00:55:30,800 --> 00:55:32,840 yeah, so logical decoding, right? 1028 00:55:33,000 --> 00:55:36,920 Or change data capture. And, and so we move to that 1029 00:55:36,920 --> 00:55:39,360 system. And so we, we essentially we 1030 00:55:39,360 --> 00:55:41,760 take all of the updates that that flow through that we 1031 00:55:41,880 --> 00:55:44,400 previously would have had to query for and we push them out 1032 00:55:46,440 --> 00:55:47,280 that way instead. OK. 1033 00:55:48,160 --> 00:55:51,280 Cool. So yeah, on the subject of slow 1034 00:55:51,280 --> 00:55:55,280 queries, yeah. Well, we got add away in the 1035 00:55:55,280 --> 00:55:58,320 chat and now Peter Ulrich just joined I. 1036 00:56:00,000 --> 00:56:06,360 Don't. Know but one add way said wall 1037 00:56:06,360 --> 00:56:10,320 events WAL is. That yes, yeah, we, we streamed 1038 00:56:10,320 --> 00:56:12,440 the wall. So we've got, we've got 2 1039 00:56:12,440 --> 00:56:14,800 replication connections essentially we've got one to do 1040 00:56:14,800 --> 00:56:17,280 and handle all this called what we call the changes. 1041 00:56:18,280 --> 00:56:21,000 So it's, you know, change data capture, we take the change. 1042 00:56:21,480 --> 00:56:25,400 We, we, we have one consumer, it basically reads all of these. 1043 00:56:25,400 --> 00:56:28,720 So these are insert, update, delete events coming from the 1044 00:56:28,720 --> 00:56:31,720 database. And then we, we, we form the 1045 00:56:31,720 --> 00:56:33,800 struct out of that and we just broadcast the struct out to the 1046 00:56:33,800 --> 00:56:36,240 client and gateway. The channel pits, right the 1047 00:56:36,240 --> 00:56:39,160 channel project, wherever they're whatever servers they're 1048 00:56:39,160 --> 00:56:40,320 running on. Whoever's listening. 1049 00:56:40,840 --> 00:56:43,720 Whoever's listening, right? So we broadcast that out and 1050 00:56:43,920 --> 00:56:45,920 let's say it was like the resource got deleted. 1051 00:56:45,920 --> 00:56:49,240 Well, that'll get turned into a resource deleted event on the 1052 00:56:49,240 --> 00:56:52,720 client and, and yeah, Bob's your uncle and, you know, everything 1053 00:56:52,720 --> 00:56:56,880 stays updated. So the second replication 1054 00:56:56,880 --> 00:57:01,200 connection is actually for audit log. 1055 00:57:01,480 --> 00:57:05,280 So we haven't shipped this yet, but it's a, it's an up and 1056 00:57:05,280 --> 00:57:10,320 coming feature. But yeah, it's, it's a, you 1057 00:57:10,320 --> 00:57:13,240 know, you, you when you get these, you get this, this wall 1058 00:57:13,240 --> 00:57:14,840 processing, right? You get it. 1059 00:57:14,840 --> 00:57:16,760 It makes for a very, very nice audit log. 1060 00:57:17,800 --> 00:57:20,480 And so we can embed like subject data into there. 1061 00:57:21,200 --> 00:57:24,560 And yeah, you can just get a, you get a history of what's 1062 00:57:24,560 --> 00:57:27,960 changed in the portal where like, hey, did this user, well, 1063 00:57:27,960 --> 00:57:30,720 you know who, who added this user to this group, right? 1064 00:57:30,720 --> 00:57:32,520 Like, why do they suddenly have access to the production 1065 00:57:32,520 --> 00:57:35,920 database? So we we can track an audit log 1066 00:57:36,600 --> 00:57:38,520 using the the wall stream as well. 1067 00:57:38,800 --> 00:57:41,680 OK nice. Is that like internal or like 1068 00:57:41,760 --> 00:57:43,760 user access? Can see logs as well. 1069 00:57:45,200 --> 00:57:47,360 So there's two types of audit logs that we'll be shipping in 1070 00:57:47,360 --> 00:57:50,080 Fire Zone. There is the what we call like 1071 00:57:50,080 --> 00:57:52,320 the configuration change logs. Those are in the those are, 1072 00:57:52,480 --> 00:57:54,200 those are basically what's changed in the database. 1073 00:57:54,200 --> 00:57:58,000 And that's where this replication stream comes in. 1074 00:57:59,080 --> 00:58:03,120 And then the second flavour is what we call flow logs or yeah, 1075 00:58:03,120 --> 00:58:04,480 those would be like the access logs. 1076 00:58:04,480 --> 00:58:06,640 And that's like, what did this user access? 1077 00:58:07,040 --> 00:58:09,880 Like what are, you know, how many bytes did they transfer to 1078 00:58:09,880 --> 00:58:12,040 the, you know, when did they talk to this database? 1079 00:58:12,960 --> 00:58:15,120 And so that's at the. Those are captured at the 1080 00:58:15,120 --> 00:58:19,680 gateway. And then we, we, we, we emit the 1081 00:58:19,680 --> 00:58:21,000 metadata from there. Yeah. 1082 00:58:21,200 --> 00:58:26,560 OK, nice. All right, let's see 10. 1083 00:58:27,280 --> 00:58:29,120 I have some questions, but we'll see. 1084 00:58:32,000 --> 00:58:35,400 Do you are you using Phoenix or Live view in any part of your 1085 00:58:35,400 --> 00:58:38,800 project? Like, yeah, we, we, we are, we 1086 00:58:38,800 --> 00:58:41,920 should, yeah, we, we want to use it more and more. 1087 00:58:41,920 --> 00:58:45,840 So we've got, we do use live view pretty heavily. 1088 00:58:47,240 --> 00:58:54,560 And it's been awesome. We we've, let's see, we've got, 1089 00:58:55,000 --> 00:58:57,840 we've got like, so imagine we get, you have like a table view. 1090 00:58:59,200 --> 00:59:01,880 You got one administrator using the product on one on, you know, 1091 00:59:01,880 --> 00:59:05,120 using what we call the admin portal, using it in one screen. 1092 00:59:05,120 --> 00:59:08,600 And then you got another admin using, you know, using the admin 1093 00:59:08,600 --> 00:59:09,600 portal. Let's say they're looking at the 1094 00:59:09,600 --> 00:59:12,920 same screen or the same view, and then one goes and like 1095 00:59:12,920 --> 00:59:16,880 deletes a resource, right? So it's nice just to be able to 1096 00:59:16,920 --> 00:59:20,560 detect that and be like, you know, we, you know, we, we're 1097 00:59:20,560 --> 00:59:23,240 receiving resource events in this, in this live view. 1098 00:59:23,240 --> 00:59:26,440 And then we can say like, hey, the table table data has 1099 00:59:26,440 --> 00:59:27,760 changed. Would you like to reload it 1100 00:59:28,080 --> 00:59:29,720 right you? Don't auto. 1101 00:59:29,720 --> 00:59:33,920 You don't want auto reload it. We, we, yeah, this is like a we 1102 00:59:33,920 --> 00:59:36,920 had a discussion about this. We decided not to just because 1103 00:59:37,120 --> 00:59:39,760 you know, how many times do you like you're using a product and 1104 00:59:39,760 --> 00:59:41,560 then you try to click on something and then it just like 1105 00:59:41,600 --> 00:59:42,960 swaps out from. You. 1106 00:59:43,080 --> 00:59:44,520 Yeah. And then you click on the wrong 1107 00:59:44,560 --> 00:59:46,760 thing. Yeah, it I feel like that 1108 00:59:46,760 --> 00:59:48,600 happens to me like 5 times a day now. 1109 00:59:49,840 --> 00:59:52,200 So we don't do stuff like, we try not to do stuff like that. 1110 00:59:52,840 --> 00:59:53,600 That makes sense. We'll just. 1111 00:59:54,000 --> 00:59:55,480 We'll just show a little notification. 1112 00:59:55,760 --> 00:59:58,560 We show online, offline status in real time. 1113 00:59:58,560 --> 01:00:01,320 So when a client connects, we'll show like you're looking at the 1114 01:00:01,320 --> 01:00:03,240 clients table or the gateways table. 1115 01:00:03,520 --> 01:00:04,920 We'll show all of that stuff in real time. 1116 01:00:05,600 --> 01:00:07,000 Yeah, we're using presence, Yeah. 1117 01:00:07,000 --> 01:00:11,080 So those that live view module will subscribe to the presence 1118 01:00:11,800 --> 01:00:14,720 tracking for that. Yeah, it's been great. 1119 01:00:14,720 --> 01:00:17,680 I love I love live view. I know some people have harsh 1120 01:00:17,680 --> 01:00:19,720 opinions of it, but it's been perfect for everything. 1121 01:00:19,720 --> 01:00:21,720 Really, what are the What are the criticisms? 1122 01:00:21,760 --> 01:00:25,920 I mean, a lot of people it's just like it, they don't like 1123 01:00:25,920 --> 01:00:29,120 how like the access to the Dom and like things like that. 1124 01:00:29,120 --> 01:00:32,360 So they want like react and like more control. 1125 01:00:32,680 --> 01:00:38,200 But like I I'm not a front end guy so maybe I don't really I 1126 01:00:38,200 --> 01:00:41,080 have the same pain points or I refuse to accept I have pain 1127 01:00:41,080 --> 01:00:42,680 points and I just make live view work I. 1128 01:00:42,880 --> 01:00:46,880 Don't Yeah, live view seems like perfect for us, you know, full 1129 01:00:46,880 --> 01:00:50,240 stack engineers who lean back in, who are just like, I just 1130 01:00:50,240 --> 01:00:52,800 wanna yeah, like I have. The front end together, you 1131 01:00:52,800 --> 01:00:54,120 know. Exactly like I don't have to 1132 01:00:54,120 --> 01:00:58,400 write JavaScript. Mostly I will say we had to 1133 01:00:58,400 --> 01:01:03,360 think a little bit more about, you know, as with any, as of any 1134 01:01:03,360 --> 01:01:06,360 project or any web app these days, like if you're doing, if 1135 01:01:06,360 --> 01:01:08,320 you've got global users, you gotta think about latency and 1136 01:01:08,320 --> 01:01:11,920 like interactivity. So by default, just putting 1137 01:01:11,920 --> 01:01:15,320 something up with live view, you know, you, you could like you, 1138 01:01:15,360 --> 01:01:18,600 you may need to think about like how to handle interactivity and 1139 01:01:18,600 --> 01:01:21,880 what state needs to live actually in the browser or on 1140 01:01:21,880 --> 01:01:24,520 the app server or, and what actually needs to go back. 1141 01:01:24,760 --> 01:01:26,760 Yeah. It's a difficult balance right? 1142 01:01:26,760 --> 01:01:30,000 Like so. I recently just launched a a SAS 1143 01:01:30,000 --> 01:01:32,000 product called Kill Switch. Encrypted. 1144 01:01:32,000 --> 01:01:35,280 Document storage with dead man switches. 1145 01:01:35,280 --> 01:01:39,240 So you can like if you don't check in you can send documents 1146 01:01:39,240 --> 01:01:44,120 to people automatically, right? And but I really like so it's 0. 1147 01:01:44,120 --> 01:01:45,160 Knowledge. Encryption. 1148 01:01:45,160 --> 01:01:50,120 So I have a lot of JavaScript on the client side taking care of 1149 01:01:50,120 --> 01:01:53,200 encryption. And then just a lot, there's a 1150 01:01:53,200 --> 01:01:55,560 lot of JavaScript. And I feel like you have to have 1151 01:01:55,560 --> 01:01:58,360 that fine balance. You need that speed with 1152 01:01:58,360 --> 01:02:02,560 JavaScript and you have to be really mindful of what's taking 1153 01:02:02,560 --> 01:02:04,480 the trip there and back to the server. 1154 01:02:04,840 --> 01:02:07,200 Do you, do you, do you use live view for that? 1155 01:02:07,600 --> 01:02:09,080 I. Do I mean the JS side? 1156 01:02:09,080 --> 01:02:12,200 Yeah, the little JS, yeah. I'm I have a lot of JavaScript 1157 01:02:12,200 --> 01:02:13,280 hooks. Essentially, yeah. 1158 01:02:13,600 --> 01:02:17,280 Yeah, yeah, yeah. Yeah, it's been great. 1159 01:02:17,600 --> 01:02:21,840 I'm happy with it. Like I don't, I don't find any. 1160 01:02:21,840 --> 01:02:24,360 Limitations at least. Yet, you know, we'll see what 1161 01:02:24,360 --> 01:02:27,320 happens. Yeah, it's been. 1162 01:02:29,600 --> 01:02:32,920 We have so, so sometimes the Rust guys on our team will come 1163 01:02:32,920 --> 01:02:37,200 and, you know, to ship a ship a feature like we, we try to, you 1164 01:02:37,200 --> 01:02:39,200 know, encourage, you know, people to take ownership over a 1165 01:02:39,200 --> 01:02:41,360 whole feature, right. And so like to get a feature 1166 01:02:41,360 --> 01:02:44,760 shipped, they may need to make some changes to the web UI or to 1167 01:02:44,960 --> 01:02:46,560 the channel modules or something. 1168 01:02:46,560 --> 01:02:50,040 And surprisingly, they've picked it up like they've picked picked 1169 01:02:50,040 --> 01:02:52,720 up both Elixir and Phoenix in live view like pretty quickly. 1170 01:02:52,720 --> 01:02:54,240 Like they don't have any issues, you know? 1171 01:02:54,640 --> 01:02:58,400 Yeah, I haven't written anything in Rust, honestly, so I don't 1172 01:02:58,400 --> 01:03:01,720 know. Like is there is syntax similar 1173 01:03:01,720 --> 01:03:02,800 or not? Not at all. 1174 01:03:02,960 --> 01:03:08,000 Oh, Rust is. Rust is. 1175 01:03:08,520 --> 01:03:10,560 Rust is a bigger I feel like. It's a bigger language than 1176 01:03:10,560 --> 01:03:12,680 Elixir. That's just, that's just my, my 1177 01:03:12,680 --> 01:03:14,480 feeling, right. It's it's there's definitely a 1178 01:03:14,480 --> 01:03:17,160 learning curve there. Yeah. 1179 01:03:17,160 --> 01:03:21,000 You, yeah, you, you get, you get sort of beat into submission by 1180 01:03:21,000 --> 01:03:25,280 the borrow checker very quickly, but. 1181 01:03:25,760 --> 01:03:28,240 But once you get that. Once you get over that hump, I 1182 01:03:28,240 --> 01:03:29,680 hear I don't think I'm over the hump. 1183 01:03:29,680 --> 01:03:32,720 I'm somewhere, you know, still pushing, pushing that boulder up 1184 01:03:32,720 --> 01:03:34,680 the hill. But once you get over that, I 1185 01:03:34,680 --> 01:03:37,520 hear it's it's it's a good feeling. 1186 01:03:37,960 --> 01:03:41,080 Yeah, OK. I mean, I feel like the. 1187 01:03:41,160 --> 01:03:45,080 Rust devs, they love Rust, so it can't be horrible, right? 1188 01:03:45,120 --> 01:03:47,800 Or is it more of a is that a kool-aid cult? 1189 01:03:47,800 --> 01:03:51,440 I don't know. No, I mean, I think it has its 1190 01:03:51,440 --> 01:03:52,840 merits. Yeah. 1191 01:03:53,120 --> 01:03:55,720 But yeah, we, yeah, there's there, Yeah, there can be a 1192 01:03:55,720 --> 01:03:56,920 little bit of Kool-aid there, I guess. 1193 01:03:57,000 --> 01:04:03,680 A little bit of Kool-aid. Let's I'm so I, I know there's 1194 01:04:03,680 --> 01:04:06,760 like a lot of like AI and LLM fatigue out there. 1195 01:04:07,080 --> 01:04:10,800 What, what's your approach to developers using it in their 1196 01:04:10,800 --> 01:04:13,680 workflows and like, how do you feel about it? 1197 01:04:15,120 --> 01:04:20,240 So I don't think. From a so I so like we. 1198 01:04:20,240 --> 01:04:22,000 Encourage it. At the end of the day, you own 1199 01:04:22,000 --> 01:04:24,720 the code that you push, right? So if you're doing, if you're 1200 01:04:24,720 --> 01:04:27,840 doing or if you're doing a review, you need to make sure to 1201 01:04:27,840 --> 01:04:29,760 do a good pull request review, right? 1202 01:04:29,760 --> 01:04:32,480 You can't, you know, you can use, you can use it on both 1203 01:04:32,480 --> 01:04:33,840 sides. You can use it both for writing 1204 01:04:33,840 --> 01:04:36,800 and for reading. But at the end of the day, I 1205 01:04:36,800 --> 01:04:39,440 think it means you, we all, we all are reading more code now 1206 01:04:39,440 --> 01:04:43,040 than we're writing. And so some developers, you 1207 01:04:43,040 --> 01:04:45,120 know, they're, you know, feel better about that or some feel 1208 01:04:45,120 --> 01:04:50,840 worse about that. But yeah, I think the, you know, 1209 01:04:50,840 --> 01:04:56,040 I think the lever, yeah. Where the, the kind of the thing 1210 01:04:56,040 --> 01:04:59,040 to focus on is like is, you know, are, are you reading the 1211 01:04:59,040 --> 01:05:00,880 code that you're writing and like, are you still taking 1212 01:05:00,880 --> 01:05:03,120 ownership over it? And as long as that's the case 1213 01:05:03,120 --> 01:05:04,800 and you feel like that, you know, you feel like your 1214 01:05:04,800 --> 01:05:07,560 engineers are still embodying that, then I think, you know, 1215 01:05:07,560 --> 01:05:09,680 whatever helps them go faster. Yeah. 1216 01:05:10,640 --> 01:05:12,360 Yeah. So Colex takes a like. 1217 01:05:12,480 --> 01:05:16,760 Our whole back end team takes a very similar approach because at 1218 01:05:16,760 --> 01:05:19,760 the end of the day, like whatever I submit, I have to 1219 01:05:19,760 --> 01:05:22,000 have ownership of. So if I'm submitting garbage, 1220 01:05:22,000 --> 01:05:26,920 that's on me. And and yeah, and if we're 1221 01:05:26,920 --> 01:05:31,240 wasting people's time P doing poll requests and reviews, like 1222 01:05:31,240 --> 01:05:32,480 that's not fair either. So. 1223 01:05:32,520 --> 01:05:34,040 You have to be. Very mindful of what you're 1224 01:05:34,040 --> 01:05:36,200 pushing up. Yeah, exactly. 1225 01:05:36,200 --> 01:05:37,880 Yeah. Like you know that you can't 1226 01:05:37,880 --> 01:05:39,320 you? Can't push up anything that you 1227 01:05:39,320 --> 01:05:41,200 didn't come over yourself, right? 1228 01:05:41,200 --> 01:05:46,320 Like you can't request review Yeah like with with Elixir 1229 01:05:46,320 --> 01:05:49,400 mentor, I do like. Pre vetted recruitment as well. 1230 01:05:49,400 --> 01:05:53,240 So like I'll send like take home technical projects, you know, to 1231 01:05:53,240 --> 01:05:58,520 people and I like I don't I mean if they specifically ask, I say 1232 01:05:58,520 --> 01:06:00,600 look, I don't care. Use your normal developer 1233 01:06:00,600 --> 01:06:03,520 workflow, like if it's using an LLM, great. 1234 01:06:03,520 --> 01:06:05,600 Like I want to talk about your workflow. 1235 01:06:05,920 --> 01:06:08,720 But then there's also like if there's over engineered parts of 1236 01:06:08,720 --> 01:06:11,680 the application, you better fully understand that shit 1237 01:06:11,680 --> 01:06:15,280 because we're diving into it. Because that is one thing that I 1238 01:06:15,280 --> 01:06:18,400 found like people will go over the top with like logging 1239 01:06:18,400 --> 01:06:21,920 systems and like admin and like extras. 1240 01:06:22,240 --> 01:06:25,400 And I'm like, I know you probably didn't write this or 1241 01:06:25,400 --> 01:06:28,200 think about this feature. You were just like, make it 1242 01:06:28,200 --> 01:06:29,600 better. Like make it more. 1243 01:06:29,600 --> 01:06:31,880 Impactive. Exactly. 1244 01:06:31,880 --> 01:06:36,000 So like if I catch you not? Knowing what code is in a 1245 01:06:36,000 --> 01:06:38,240 project you submitted, it's not going to go. 1246 01:06:38,280 --> 01:06:39,560 It's not going to look good for you. 1247 01:06:40,120 --> 01:06:41,520 Worse than just not having it, right? 1248 01:06:41,720 --> 01:06:44,960 Exactly. I'd rather a feature be missing. 1249 01:06:44,960 --> 01:06:47,680 Than have a feature rich, something you didn't write or 1250 01:06:47,680 --> 01:06:50,480 understand. I mean, yeah, yeah. 1251 01:06:50,480 --> 01:06:52,720 I mean, at the end of the day. Like all code is tech debt, 1252 01:06:52,720 --> 01:06:53,920 right? It's all going to get going to 1253 01:06:53,920 --> 01:06:56,560 get changed and be obsolete eventually, right? 1254 01:06:56,560 --> 01:06:58,440 So isn't that kind of sad to think about? 1255 01:06:58,440 --> 01:07:00,040 But then they're like, we're so happy when we. 1256 01:07:00,040 --> 01:07:03,440 Delete more lines of code than we've added on a refactor. 1257 01:07:03,560 --> 01:07:05,880 Yeah, yeah. That is true. 1258 01:07:07,240 --> 01:07:08,200 Yeah. We just did it. 1259 01:07:08,200 --> 01:07:11,400 We shipped a really a really large refactor recently. 1260 01:07:11,400 --> 01:07:13,760 We we had to refactor essentially our whole identity 1261 01:07:13,760 --> 01:07:18,520 model and it was like it was, you know, I, I'm gonna, I have, 1262 01:07:18,560 --> 01:07:23,840 I have scars from it, like PTSD. It was, it was, it was, I think 1263 01:07:23,840 --> 01:07:26,400 the PR, it was one of those PRS where you're like, I'll just 1264 01:07:26,400 --> 01:07:28,560 start this one off small. And then you're like, you know, 1265 01:07:28,560 --> 01:07:30,280 I can't really like split this up easily. 1266 01:07:30,280 --> 01:07:33,760 So I'll just like add into it. And then it grew to be like, it 1267 01:07:33,760 --> 01:07:35,360 grew into this thing where it's like, all right, we're gonna 1268 01:07:35,360 --> 01:07:38,800 have to do this in an atomic cut over way because it was like 1269 01:07:38,800 --> 01:07:40,720 shipping AV 2 of something, right? 1270 01:07:40,720 --> 01:07:43,720 And so we had like 7080 thousand lines of code at the end of the 1271 01:07:43,720 --> 01:07:45,080 day. And then we just had to merge it 1272 01:07:45,080 --> 01:07:46,760 in and then schedule the roll out for that. 1273 01:07:46,760 --> 01:07:50,480 So I'm curious, I like, do you have a staging? 1274 01:07:50,480 --> 01:07:52,680 Environment. I feel like, yeah, we do OK, 1275 01:07:53,280 --> 01:07:54,640 because I feel like that like is that. 1276 01:07:54,640 --> 01:07:57,040 Difficult. To kind of mirror production, 1277 01:07:57,040 --> 01:08:00,680 how does that work? It's not difficult at. 1278 01:08:00,680 --> 01:08:03,800 All, it's all just terraform, you know, So you, yeah, you 1279 01:08:03,800 --> 01:08:05,240 just, it's just a renaming of things. 1280 01:08:05,240 --> 01:08:11,800 But we have it identical to to production and it's, it's 1281 01:08:11,800 --> 01:08:15,160 essential for us because you know, you know, like I mentioned 1282 01:08:15,160 --> 01:08:19,200 earlier, broken code we if we're down for 30 seconds. 1283 01:08:19,359 --> 01:08:21,520 Then. We start hearing complaints, 1284 01:08:21,520 --> 01:08:24,840 right, like just 3030 seconds, like or a minute, right, like 1285 01:08:24,840 --> 01:08:27,359 people. Yeah, it's so, so there's, so 1286 01:08:27,359 --> 01:08:30,680 there's yeah, you've got to, you've got to make it reliable. 1287 01:08:30,680 --> 01:08:33,120 And yeah, we, we have a staging environment for that. 1288 01:08:33,600 --> 01:08:35,479 OK. And then like you mentioned, 1289 01:08:35,760 --> 01:08:40,040 like a rollout. Plan do you slowly roll changes 1290 01:08:40,040 --> 01:08:43,680 out to so we do blue greens yeah we we do we. 1291 01:08:43,680 --> 01:08:46,800 Do sort of, I guess you could say blue-green. 1292 01:08:48,319 --> 01:08:52,520 What we do now, it's evolved over the, over the, over the 1293 01:08:52,520 --> 01:08:55,359 past year or so in a few different times. 1294 01:08:56,560 --> 01:09:02,319 What, what we do now is we ship Docker containers to a, a pinned 1295 01:09:02,319 --> 01:09:04,200 image. So let's call it, you know, 1296 01:09:04,479 --> 01:09:08,960 whatever our imagery repo is production, then we've got a job 1297 01:09:08,960 --> 01:09:12,920 on the on our VMS that looks for new images. 1298 01:09:13,000 --> 01:09:16,640 So when it finds them, it'll pull it down and then it run it 1299 01:09:16,640 --> 01:09:18,520 spins up the image, but in isolation. 1300 01:09:18,520 --> 01:09:20,880 So it's not actually connected as a service yet. 1301 01:09:21,520 --> 01:09:23,880 And then it'll run a series of health checks on it, right? 1302 01:09:23,880 --> 01:09:28,600 So it'll join it to the cluster. It'll make sure you know, so 1303 01:09:28,600 --> 01:09:31,479 there's you know, it'll it'll make sure that everything's good 1304 01:09:31,680 --> 01:09:34,600 and proper and then it'll start returning 200 from its I guess 1305 01:09:35,000 --> 01:09:37,840 we have a we we do a red easy endpoint and then you kind of 1306 01:09:37,840 --> 01:09:39,960 don't have to do, you don't have to deal with hot reload. 1307 01:09:40,120 --> 01:09:43,279 From the Elixir side, yeah. So that I mean we've talked 1308 01:09:43,279 --> 01:09:48,479 about it. The, the way that we've 1309 01:09:48,479 --> 01:09:54,760 architected our, our, our control plane is such that it 1310 01:09:54,760 --> 01:09:57,640 can go down or it can get reconnected without really any, 1311 01:09:57,680 --> 01:09:59,560 any packet loss on the data plane side. 1312 01:09:59,960 --> 01:10:03,200 So those are kind of like that. I feel like I need to implement 1313 01:10:03,200 --> 01:10:07,000 something similar like right? Now I have like a few seconds of 1314 01:10:07,880 --> 01:10:11,160 intermittent connection, like when I push a change to 1315 01:10:11,160 --> 01:10:12,680 production. Yeah. 1316 01:10:12,680 --> 01:10:16,560 And it's because I'm just, you know, I'm, I'm basically just 1317 01:10:16,560 --> 01:10:20,120 Docker composing a building. And then so it goes down from, 1318 01:10:20,120 --> 01:10:21,880 oh, it restarts. Yeah, yeah, yeah. 1319 01:10:21,880 --> 01:10:23,600 So, So what? I do. 1320 01:10:23,600 --> 01:10:25,960 So yeah, this is actually fairly simple to do. 1321 01:10:25,960 --> 01:10:27,880 What you could do is you could spin up the container on 1322 01:10:27,880 --> 01:10:31,320 different ports and then you have an IP tables, IP tables 1323 01:10:31,320 --> 01:10:33,280 command that just swaps the ports with Nat. 1324 01:10:33,920 --> 01:10:36,120 And then now the packets are gonna and then you spin down the 1325 01:10:36,120 --> 01:10:40,320 old container right, 'cause so that way the, yeah, it's, it's 1326 01:10:40,320 --> 01:10:42,160 been, yeah, it's been pretty reliable for us. 1327 01:10:42,360 --> 01:10:49,280 Just a little trick you can do. And yeah, but no, we've we've 1328 01:10:49,280 --> 01:10:51,000 talked. About doing the hot code. 1329 01:10:51,000 --> 01:10:53,440 Reloading thing. I mean, it's nice to know that 1330 01:10:53,920 --> 01:10:56,560 if we really need to hold the socket open, the web socket 1331 01:10:56,560 --> 01:11:00,280 open, we can we could do that in the future and the. 1332 01:11:00,280 --> 01:11:01,360 Language supports it, right? The. 1333 01:11:01,360 --> 01:11:03,400 Runtime. Yeah, that's another like. 1334 01:11:03,480 --> 01:11:06,480 I've never dealt with it. Like, I've read some blog posts 1335 01:11:06,480 --> 01:11:09,480 on it and maybe I'd just go that route too because it's pretty 1336 01:11:09,480 --> 01:11:12,000 cool. Yeah, yeah, we've got. 1337 01:11:12,000 --> 01:11:15,120 It's just a lot of state. I I'm sure too. 1338 01:11:15,160 --> 01:11:16,400 Yeah. We've, we haven't done it 1339 01:11:16,400 --> 01:11:18,000 either. It'd be great to talk to someone 1340 01:11:18,000 --> 01:11:19,960 who has a lot of experience in that regard. 1341 01:11:20,120 --> 01:11:23,080 Yeah. Who has production code? 1342 01:11:23,280 --> 01:11:26,240 And they're using hot reload. Reach out, I'll have you on. 1343 01:11:26,800 --> 01:11:30,400 Yeah, because yeah. Like I've heard like. 1344 01:11:31,640 --> 01:11:33,720 Once you understand how to set it up. 1345 01:11:33,720 --> 01:11:35,240 It's relatively like. Easy. 1346 01:11:35,640 --> 01:11:38,880 But I don't know if it's like the most straightforward thing 1347 01:11:38,880 --> 01:11:40,760 to get functioning. Well, I don't know. 1348 01:11:40,760 --> 01:11:43,080 I mean I, I think, yeah, I mean I think the the setup in the 1349 01:11:43,080 --> 01:11:44,640 boilerplate is not. The hard part, right? 1350 01:11:44,640 --> 01:11:47,640 It's like, you know, you, you, you do a mixed release, you get 1351 01:11:47,640 --> 01:11:49,960 the the bundle and then you just send it out to the server and 1352 01:11:49,960 --> 01:11:52,640 swap it over. I think the hard part is to 1353 01:11:52,640 --> 01:11:56,080 reason about, OK, you have this old state, right, that your 1354 01:11:56,080 --> 01:12:00,080 application's in, it's running and now you've, let's say you've 1355 01:12:00,080 --> 01:12:02,800 done a refactor where you've just, you know, you've added new 1356 01:12:02,800 --> 01:12:05,000 fields to a schema module or something like that. 1357 01:12:06,400 --> 01:12:08,960 You know, like now you've got to like get it into this new state 1358 01:12:08,960 --> 01:12:12,040 and then so every. So it may not be possible prefer 1359 01:12:12,040 --> 01:12:14,840 that balance like yeah, so. You've got to migrate. 1360 01:12:14,880 --> 01:12:16,400 I think it depends on the module, right? 1361 01:12:16,480 --> 01:12:18,720 But you've got to migrate, you've got to migrate state over 1362 01:12:18,720 --> 01:12:20,120 yourself, right? I see. 1363 01:12:20,200 --> 01:12:21,520 Because you're not. You're not. 1364 01:12:21,520 --> 01:12:22,760 You're not taking the supervision. 1365 01:12:22,760 --> 01:12:24,080 Tree. Down and then bringing it back 1366 01:12:24,080 --> 01:12:26,840 up right that stays that all stays online. 1367 01:12:26,880 --> 01:12:30,560 So each module needs to know how to migrate state. 1368 01:12:30,720 --> 01:12:33,880 So like yeah the bigger the changes the more. 1369 01:12:33,920 --> 01:12:36,320 State demand. So do you write like I'm, I'm, 1370 01:12:36,560 --> 01:12:40,320 I'm wondering, do you just write some like migration files for 1371 01:12:40,320 --> 01:12:43,280 the changes Then yeah, I wouldn't know. 1372 01:12:43,320 --> 01:12:46,280 Yeah, I think. I don't think they're migrations 1373 01:12:46,280 --> 01:12:48,400 in the sense of database migrations. 1374 01:12:48,400 --> 01:12:51,000 State migrations, not database migrations I see. 1375 01:12:51,600 --> 01:12:54,240 I yeah, I think now we're getting onto onto. 1376 01:12:54,240 --> 01:12:55,600 The edges of my Elixir knowledge. 1377 01:12:55,600 --> 01:12:57,640 But I think now, I think, I think you've got a right. 1378 01:12:57,720 --> 01:12:59,920 Yeah, yeah, I think you write, you write hooks for this and 1379 01:12:59,920 --> 01:13:02,760 they live in the modules. I could be wrong on that, but 1380 01:13:02,760 --> 01:13:05,840 yeah, they're definitely not other separate files from what I 1381 01:13:05,840 --> 01:13:06,200 know. Yeah. 1382 01:13:06,200 --> 01:13:07,720 Now I want to kind of dig into it. 1383 01:13:07,800 --> 01:13:10,520 I'm curious. But the, you know, the, the sort 1384 01:13:10,520 --> 01:13:11,760 of, it's like the Holy Grail, right? 1385 01:13:11,840 --> 01:13:13,960 You don't even drop a website connection, right? 1386 01:13:13,960 --> 01:13:15,760 It's like everything stays connected. 1387 01:13:15,920 --> 01:13:19,400 Yeah, which I kind of would like like from a. 1388 01:13:19,440 --> 01:13:22,320 From a live view. Experience on the front end that 1389 01:13:22,320 --> 01:13:24,440 would that would make it incredible. 1390 01:13:24,840 --> 01:13:31,400 We we do that is 1 big area. Where we do, where we are 1391 01:13:31,400 --> 01:13:32,960 interested in using something like that. 1392 01:13:32,960 --> 01:13:37,360 But even then, the workaround I think we'll be employing, which 1393 01:13:37,360 --> 01:13:42,000 we do a little bit are ready of is when you ship a new, 1394 01:13:42,000 --> 01:13:43,680 basically just drain the connection. 1395 01:13:43,680 --> 01:13:46,680 So like when you ship a new release, pop up a little 1396 01:13:46,680 --> 01:13:48,600 dialogue. If whoever's got an active 1397 01:13:48,600 --> 01:13:50,920 session, just pop up a little dialogue saying like, hey, you 1398 01:13:50,920 --> 01:13:53,920 know, the application's going to be upgraded in 5 minutes or 1399 01:13:53,920 --> 01:13:55,560 something like that. They got to, they have like a 1400 01:13:55,560 --> 01:13:57,600 countdown, they can finish whatever they're doing and then 1401 01:13:57,600 --> 01:14:00,240 they can, you know, if they're filling out a big form. 1402 01:14:00,440 --> 01:14:03,160 Yeah, 'cause, 'cause like, yeah, this whole, the whole hot code 1403 01:14:03,160 --> 01:14:06,720 reloading thing is like, you know, from from what, what what 1404 01:14:06,720 --> 01:14:10,240 I know about it. I it seems like a lot of it, it 1405 01:14:10,240 --> 01:14:12,840 just seems like it could get very complicated very, very 1406 01:14:12,840 --> 01:14:15,040 quickly. So if you can avoid it, it's 1407 01:14:15,040 --> 01:14:17,080 like metaprogramming. Like if you can avoid it, just 1408 01:14:17,200 --> 01:14:19,560 just avoid it. You know, dude, metaprogramming 1409 01:14:19,560 --> 01:14:21,440 hurts my brain every time. Yeah, I. 1410 01:14:21,440 --> 01:14:26,280 Can like I can like I always. Feel like I have my head. 1411 01:14:26,280 --> 01:14:30,740 Wrapped around it and then I don't you're like quote UN 1412 01:14:30,740 --> 01:14:32,800 quote, like like where, which, which one am I? 1413 01:14:32,840 --> 01:14:37,040 In yeah, like I I did like a elixir. 1414 01:14:37,040 --> 01:14:42,520 Introductory course and I touch on metaprogramming and even 1415 01:14:42,520 --> 01:14:46,800 through that course, like I struggled through it like like 1416 01:14:46,800 --> 01:14:48,440 I'm teaching it and I struggle through it. 1417 01:14:48,440 --> 01:14:52,960 This is like it's very hard to stay focused on what you're 1418 01:14:52,960 --> 01:14:54,400 actually. Intending it is. 1419 01:14:54,800 --> 01:14:56,440 Don't get me wrong, I think it's awesome. 1420 01:14:56,440 --> 01:15:00,800 It just really hurts my brain. Yeah, yeah, yeah, we, we've 1421 01:15:00,800 --> 01:15:04,520 recently. Yeah, we've done a lot with just 1422 01:15:04,800 --> 01:15:07,280 kind of trying to get rid of some of some meta programming in 1423 01:15:07,280 --> 01:15:09,560 our code base, just just with the with the action of like 1424 01:15:09,560 --> 01:15:12,400 LLMS. So that's that's one thing. 1425 01:15:12,400 --> 01:15:15,880 Like it's hard for LS PS to kind of like integrate with, right? 1426 01:15:15,880 --> 01:15:18,120 It breaks your go to definition, right? 1427 01:15:18,160 --> 01:15:22,000 Like so we've just sort of set a rule in our code base to avoid 1428 01:15:22,000 --> 01:15:25,320 it as much as possible. And and so then the places that 1429 01:15:25,320 --> 01:15:28,760 we do use it, it's like very clear and very beneficial. 1430 01:15:28,760 --> 01:15:31,640 Like it's a clear benefit to have it in there. 1431 01:15:31,640 --> 01:15:34,480 And then, you know, the, the pro out the pros outweigh the cons. 1432 01:15:34,760 --> 01:15:38,840 But yeah, we were doing it. We we used to you. 1433 01:15:38,840 --> 01:15:40,160 Know we we were trying to be clever. 1434 01:15:40,160 --> 01:15:43,320 In some places and and just ended up just making it 1435 01:15:43,480 --> 01:15:47,080 explicit. Yeah, yeah, sometimes cleverness 1436 01:15:47,320 --> 01:15:49,080 and over. Engineering just hurts you in 1437 01:15:49,080 --> 01:15:51,800 the long run. Yeah, yeah. 1438 01:15:54,840 --> 01:15:57,160 I. I'm curious, is there? 1439 01:15:57,160 --> 01:16:00,400 Any like other than you said, you mentioned your logging 1440 01:16:00,400 --> 01:16:03,160 system, are there any like big features you can talk about that 1441 01:16:03,160 --> 01:16:09,400 you're excited about implementing in 2026 or let's 1442 01:16:09,400 --> 01:16:10,960 see? Let me take a look at our road 1443 01:16:10,960 --> 01:16:13,400 map. We do have a public road map. 1444 01:16:13,440 --> 01:16:15,920 We try to keep it mostly up to date. 1445 01:16:18,760 --> 01:16:20,240 I mean, we're. We are open source so. 1446 01:16:20,760 --> 01:16:24,640 So there's not much there's, you know, there's, there's, there's 1447 01:16:24,640 --> 01:16:30,320 not much I guess on the product road map that is closed off. 1448 01:16:30,320 --> 01:16:33,920 But yeah, I mean, we've got the the audit logs. 1449 01:16:33,920 --> 01:16:38,480 Features of is a really. Big one, other products in the 1450 01:16:38,480 --> 01:16:44,360 space don't quite give you the level of detail that the this 1451 01:16:44,360 --> 01:16:47,880 like logical decoding can can give you. 1452 01:16:47,880 --> 01:16:51,600 So you know just every insert, update and delete, seeing what 1453 01:16:51,600 --> 01:16:55,880 user you know is attributed to that and whatnot like. 1454 01:16:55,880 --> 01:16:56,880 What you know all the way down to their. 1455 01:16:56,880 --> 01:16:58,920 IP address. So we're really excited about 1456 01:16:58,920 --> 01:17:04,000 audit logs. Another one is so we we just 1457 01:17:04,000 --> 01:17:06,680 like I mentioned, we just shipped that IDP refactor we do 1458 01:17:06,960 --> 01:17:08,760 now we sync your users and groups. 1459 01:17:08,920 --> 01:17:10,680 We refactored how we do all of that. 1460 01:17:11,800 --> 01:17:13,760 We're going to so we built the sync engine for that. 1461 01:17:13,760 --> 01:17:16,600 We're going to use that sync engine to then pull in like 1462 01:17:16,600 --> 01:17:20,000 your. So pull in your your 1463 01:17:20,000 --> 01:17:21,000 organization. 'S. 1464 01:17:21,160 --> 01:17:24,960 Device asset list, so this is for like MDM mobile device 1465 01:17:24,960 --> 01:17:29,120 manager like Intune, Kanji, JAMP, any of those ring a bell 1466 01:17:30,080 --> 01:17:33,520 IA little bit pull in, so we'll be pulling in your. 1467 01:17:33,520 --> 01:17:36,480 Devices basically. Your organization has an asset 1468 01:17:36,480 --> 01:17:38,640 list. We'll pull those devices in and 1469 01:17:38,640 --> 01:17:40,240 then we'll be able to authenticate that. 1470 01:17:40,240 --> 01:17:42,800 Hey, this, this user's connecting only from an 1471 01:17:42,800 --> 01:17:44,560 organization owned device, right. 1472 01:17:45,000 --> 01:17:48,080 So we call that MDM better MDM integration. 1473 01:17:50,800 --> 01:17:53,000 We've got a few on here. Those those are kind of the big 1474 01:17:53,000 --> 01:17:56,200 ones that we're focused on now finishing our cloud migration, 1475 01:17:56,200 --> 01:17:59,000 although that's not really user facing feature, but gotcha. 1476 01:17:59,600 --> 01:18:04,480 I'm curious like with with open. Source is Is your full feature 1477 01:18:04,480 --> 01:18:10,720 list available open source or do you pay all parts of it so? 1478 01:18:11,920 --> 01:18:14,880 The, the full product is, is, is open source. 1479 01:18:14,880 --> 01:18:18,360 Now some of the purists out there may take issue with with 1480 01:18:18,360 --> 01:18:20,880 that. So we, our control plane, the 1481 01:18:20,880 --> 01:18:24,360 Elixir stuff is elastic. So if you're going to run it, 1482 01:18:25,160 --> 01:18:28,320 you can run it yourself or your organization unrestricted, 1483 01:18:28,320 --> 01:18:29,760 right? No, no feature gating or 1484 01:18:29,760 --> 01:18:34,040 anything like that. The reason we did that was 1485 01:18:34,040 --> 01:18:37,160 because we used to be and then the, the rest of it's Apache 1486 01:18:37,160 --> 01:18:41,520 two, we used to be fully Apache 2 and we had some issues with 1487 01:18:41,520 --> 01:18:46,120 like people taking that and just running it and selling it on, on 1488 01:18:46,120 --> 01:18:48,480 the cloud marketplaces. Oh gosh, of course. 1489 01:18:48,480 --> 01:18:50,040 And it's like, OK, but that was like that. 1490 01:18:50,040 --> 01:18:51,400 That's fine. Like, that's one thing. 1491 01:18:52,320 --> 01:18:54,520 But then we would get the support request for it, right? 1492 01:18:54,520 --> 01:18:57,120 And so, so that's like we're supporting someone else's 1493 01:18:57,120 --> 01:19:00,400 business at that point. And so we, we thought like, you 1494 01:19:00,400 --> 01:19:03,400 know, going forward, we'd still like to make the code open 1495 01:19:03,400 --> 01:19:05,520 source and available and kind of like free to use as long as 1496 01:19:05,520 --> 01:19:07,680 you're not trying to sell it, sell yourself. 1497 01:19:07,680 --> 01:19:10,880 How do you mitigate though your policies? 1498 01:19:10,880 --> 01:19:13,880 Like that, like I feel like that would be difficult alone. 1499 01:19:14,400 --> 01:19:16,960 We had, I mean, we, we find we. It's funny. 1500 01:19:16,960 --> 01:19:20,840 Because like we typically find out because they're, they're 1501 01:19:20,840 --> 01:19:23,600 not, you know, they, they may take our control plane and go 1502 01:19:23,600 --> 01:19:26,320 host it, but they're not taking our client apps and like 1503 01:19:26,320 --> 01:19:29,280 rebuilding them. You can't take Firezone like re 1504 01:19:29,480 --> 01:19:31,640 rebrand it and distribute it on the Apple App Store. 1505 01:19:31,640 --> 01:19:33,400 Like Apple will typically catch that, right? 1506 01:19:33,400 --> 01:19:33,960 Yeah. OK. 1507 01:19:33,960 --> 01:19:37,200 So, so unless you're, unless you have like an enterprise system 1508 01:19:37,200 --> 01:19:40,160 set up like MDM, well, they probably see the binaries are 1509 01:19:40,160 --> 01:19:42,360 exactly the same, right? And then they're just, well, a 1510 01:19:42,920 --> 01:19:47,240 white label over it. Yeah, Like I, I, yeah, I that's 1511 01:19:47,240 --> 01:19:47,800 a good. Question. 1512 01:19:47,800 --> 01:19:51,120 I don't know if they would detect it like that, but but 1513 01:19:51,120 --> 01:19:54,240 they typically what happens is like they're just using our apps 1514 01:19:54,360 --> 01:19:59,560 And so we get all their sentry logs and we're like, OK, you 1515 01:19:59,560 --> 01:20:02,560 know, like we, we, we, this happened actually with one, with 1516 01:20:02,560 --> 01:20:05,440 one entity and we had to send them, we had to send them a 1517 01:20:05,440 --> 01:20:08,760 letter like, hey, please stop. You know, like, you know, you're 1518 01:20:08,760 --> 01:20:11,640 violating our terms or our our license. 1519 01:20:11,640 --> 01:20:16,880 So, but yeah, it hasn't been, I mean, I think most. 1520 01:20:17,280 --> 01:20:19,520 Players in the world are pretty good act are are are good 1521 01:20:19,520 --> 01:20:20,880 actors. And you know, no one's gonna go 1522 01:20:20,880 --> 01:20:23,880 blatantly try to violate your yeah. 1523 01:20:24,200 --> 01:20:27,080 Do you find like since like I like? 1524 01:20:27,280 --> 01:20:31,520 You're AVPN service it's like based around security. 1525 01:20:31,520 --> 01:20:35,000 Do you think making it open source is essential for like for 1526 01:20:35,000 --> 01:20:37,760 trust? That's a great question. 1527 01:20:38,120 --> 01:20:39,880 I. I personally do. 1528 01:20:41,560 --> 01:20:44,240 I guess it's not always cut and dried, right? 1529 01:20:44,240 --> 01:20:46,240 Some, some people, I guess, still believe in this. 1530 01:20:46,240 --> 01:20:50,480 Like security through obscurity. Yeah, I mean it helps, but 1531 01:20:50,480 --> 01:20:53,040 you're probably more. Secure because you get scrutiny 1532 01:20:53,400 --> 01:20:55,680 it it could it could go both ways. 1533 01:20:55,680 --> 01:20:59,600 I mean, we definitely. Get drive by researchers sort of 1534 01:20:59,600 --> 01:21:01,760 being like, hey, I found a problem in your product, but I 1535 01:21:01,760 --> 01:21:04,200 won't tell you what it is unless you pay me, you know, some some 1536 01:21:04,200 --> 01:21:09,880 bounty right and and so we you know, we're stocked too 1537 01:21:09,880 --> 01:21:11,400 compliant. You know, we get, we get pin 1538 01:21:11,400 --> 01:21:16,640 tested and all of that. And so I will say overall it is 1539 01:21:16,640 --> 01:21:19,520 a, you know, you, you can get people out there who might find 1540 01:21:19,520 --> 01:21:21,840 something and never report it. But I think you, you generally 1541 01:21:21,840 --> 01:21:24,520 get, there's many more people out there who will find 1542 01:21:24,520 --> 01:21:29,960 something, either report it or you know, we've yeah, they'll, 1543 01:21:30,040 --> 01:21:34,280 they'll, they'll report it. Or generally by having it open 1544 01:21:34,280 --> 01:21:36,320 source, I think you you. You tend to squash those. 1545 01:21:36,320 --> 01:21:39,440 A lot quicker right? Do you get random contributors 1546 01:21:39,440 --> 01:21:42,720 or is it mostly your team? It's mostly, it's mostly just 1547 01:21:42,720 --> 01:21:43,880 our team. Yeah, we, I mean we. 1548 01:21:43,880 --> 01:21:45,840 We welcome contributions. Before you're going to go work 1549 01:21:45,840 --> 01:21:48,200 on something really big, though, probably check in with us, 1550 01:21:48,200 --> 01:21:49,880 right? We've had, we've had a few 1551 01:21:49,880 --> 01:21:52,280 larger PRS that we, we couldn't accept because they just weren't 1552 01:21:52,280 --> 01:21:54,760 in the right direction for where we were going to go. 1553 01:21:54,760 --> 01:21:56,760 Or we were going to be, you know, they were on some area of 1554 01:21:56,760 --> 01:21:58,840 the product we were about to completely refactor. 1555 01:21:59,080 --> 01:22:02,600 That happened recently. So but like bug fixes and things 1556 01:22:02,600 --> 01:22:04,960 like that, we've had, we've had a number of and yeah, super 1557 01:22:04,960 --> 01:22:08,240 welcome. That's awesome. 1558 01:22:08,680 --> 01:22:10,400 All right. So being a. 1559 01:22:11,640 --> 01:22:15,960 Single like SAS founder like no employees. 1560 01:22:16,320 --> 01:22:20,440 I'm curious what what do you do as far as like marketing and 1561 01:22:20,600 --> 01:22:24,160 getting eyes on your product? What's like the best Ave. for 1562 01:22:24,160 --> 01:22:27,480 you? That's a great question and it's 1563 01:22:27,480 --> 01:22:29,880 something I need to do. More of right it's AI feel like 1564 01:22:29,880 --> 01:22:34,680 we technical founders don't technical founders it's well 1565 01:22:34,680 --> 01:22:36,360 it's. Like yeah, the product. 1566 01:22:36,360 --> 01:22:38,920 Has to work right so a lot of our a lot of my time is making 1567 01:22:38,920 --> 01:22:41,760 sure that everything you know the grease the big figures are 1568 01:22:41,760 --> 01:22:51,800 greased when I'm not doing that we we typically will we do a 1569 01:22:51,800 --> 01:22:54,120 number of so we we. Have blog posts? 1570 01:22:54,400 --> 01:22:57,400 That we'll write from time to time that some of them have 1571 01:22:57,400 --> 01:23:00,600 landed well, some of them have been complete flops, but we get 1572 01:23:00,600 --> 01:23:03,920 a good amount of inbound and we get a good amount from actually 1573 01:23:03,920 --> 01:23:06,160 our GitHub repository. Oh, nice. 1574 01:23:06,200 --> 01:23:09,360 OK, that's actually so, yeah. So we'll be spinning up a number 1575 01:23:09,360 --> 01:23:10,480 of different. Marketing efforts. 1576 01:23:10,480 --> 01:23:14,360 I guess in this year 2026, so, so stay tuned. 1577 01:23:15,640 --> 01:23:19,040 But yeah, a lot of this, a lot of a lot of the stuff that we've 1578 01:23:19,040 --> 01:23:22,840 been working on, especially on the data plane side, I think has 1579 01:23:22,840 --> 01:23:26,160 been really interesting for for that community. 1580 01:23:26,560 --> 01:23:30,480 And then in addition, you know, better kind of detailing some of 1581 01:23:30,480 --> 01:23:31,880 the stuff we've been doing on the Elixir side. 1582 01:23:32,200 --> 01:23:35,160 So we want to write some some blog posts going over all of 1583 01:23:35,160 --> 01:23:36,600 that. What do you think? 1584 01:23:36,600 --> 01:23:39,120 What lands better there? Like very technical. 1585 01:23:39,360 --> 01:23:44,680 Blog posts or more feature based or like we do, we like what 1586 01:23:44,680 --> 01:23:46,440 you're doing. We've been doing these. 1587 01:23:46,440 --> 01:23:48,320 Devlog posts. That have been. 1588 01:23:48,320 --> 01:23:50,000 Interesting. Yeah. 1589 01:23:50,000 --> 01:23:53,480 And and yeah. So, so it's nice just to look 1590 01:23:53,480 --> 01:23:56,960 back on the previous month and go like, Oh yeah, we did work on 1591 01:23:56,960 --> 01:24:00,400 that and we shipped that. And you know, here's how, you 1592 01:24:00,400 --> 01:24:02,880 know, we we should probably remind, you know, such and so 1593 01:24:02,880 --> 01:24:04,720 that this was shipped because they were waiting on it, right? 1594 01:24:04,720 --> 01:24:07,240 It's a, it's a nice little ecap at the end of the month. 1595 01:24:07,720 --> 01:24:10,440 That's a good idea. I should do like a dev log. 1596 01:24:10,800 --> 01:24:14,360 I so I just started doing blog posts with kill switch but more 1597 01:24:14,360 --> 01:24:19,040 and more like feature related or why you would want kill switch 1598 01:24:19,040 --> 01:24:23,360 in your life kind of thing. But maybe more dev based ones 1599 01:24:23,360 --> 01:24:26,040 that explain how the encryption works and things like that might 1600 01:24:26,040 --> 01:24:30,360 be beneficial what you could do that we've started doing 1601 01:24:30,360 --> 01:24:32,080 recently. Is just do AI. 1602 01:24:33,600 --> 01:24:35,960 Don't know if you're working with anyone on on Killswitch. 1603 01:24:35,960 --> 01:24:38,680 But. We've started doing week weekly. 1604 01:24:38,680 --> 01:24:41,640 Well, you maybe you could still do it, but we've started doing 1605 01:24:41,640 --> 01:24:44,640 weekly vlogs. So we'll, so I'll get on a call 1606 01:24:44,640 --> 01:24:48,280 with someone else and at the end of the week and we'll just talk 1607 01:24:48,280 --> 01:24:50,640 about, we'll block off 30 minutes and we'll just have a 1608 01:24:50,640 --> 01:24:53,280 casual chat about what we worked on, what we were worried about, 1609 01:24:53,280 --> 01:24:56,200 what issues we faced. And then it's in a, and then 1610 01:24:56,200 --> 01:24:58,440 it's in a video format, right? You don't have to think of, you 1611 01:24:58,440 --> 01:25:01,920 know, you don't have to type it all out, pick your favorite LLM 1612 01:25:01,920 --> 01:25:04,800 tool, transcribe it right. And then you've got, and then 1613 01:25:04,800 --> 01:25:06,240 you've got content for your, for your. 1614 01:25:06,240 --> 01:25:08,600 I like that. I mean, I could just like, talk 1615 01:25:08,600 --> 01:25:10,880 to myself. About what I did that week for 1616 01:25:10,880 --> 01:25:12,920 Killswitch and do that. I like that. 1617 01:25:12,960 --> 01:25:17,200 Yeah, yeah, it's a good idea. Do. 1618 01:25:17,440 --> 01:25:19,800 Do does your blog? I haven't checked it out but. 1619 01:25:19,960 --> 01:25:22,480 Do you can people log in and interact? 1620 01:25:22,480 --> 01:25:26,400 Can they comment like, Oh no, not, we don't have that. 1621 01:25:26,400 --> 01:25:29,240 Yeah, we it. So the website is something I 1622 01:25:29,240 --> 01:25:32,480 built in like 2 weeks I think it was like a couple summers ago. 1623 01:25:32,480 --> 01:25:35,600 I was like, oh we need a launch, we need a website and it doesn't 1624 01:25:35,600 --> 01:25:38,840 have any of that stuff. No, it's my I'm not throwing any 1625 01:25:38,840 --> 01:25:40,600 shed. My blog post is the same way. 1626 01:25:41,040 --> 01:25:44,960 Yeah, it's just you read it. Yeah, Yeah. 1627 01:25:45,240 --> 01:25:46,960 And then I was pretty. Proud I built a pretty. 1628 01:25:46,960 --> 01:25:50,040 Cool like dynamic? Site map so I don't have to 1629 01:25:50,040 --> 01:25:52,840 update it with my blog post. Slugs. 1630 01:25:53,600 --> 01:25:56,000 Yeah, site maps are critical, you know. 1631 01:25:56,040 --> 01:25:59,360 Do you have? Your your site optimized for 1632 01:25:59,360 --> 01:26:01,960 what do they call it the LLMS text? 1633 01:26:02,320 --> 01:26:04,840 Yeah, the. There's that one, but there's 1634 01:26:04,840 --> 01:26:07,520 like SEO. For it's called like Geo or 1635 01:26:07,520 --> 01:26:09,840 something like that. Oh no, I don't have that. 1636 01:26:09,920 --> 01:26:11,400 I may I may I may be getting that. 1637 01:26:11,400 --> 01:26:12,880 Acronym wrong. It's. 1638 01:26:12,960 --> 01:26:18,800 Like SEO for LLMS. Let me see if I can find. 1639 01:26:18,800 --> 01:26:22,040 This generative engine. Optimization Geo. 1640 01:26:22,800 --> 01:26:27,160 Yeah, I put an. LLMS dot text route in I don't. 1641 01:26:27,160 --> 01:26:29,000 Know if that's what this is? Let's see. 1642 01:26:33,120 --> 01:26:40,640 Just tell me more Google. Dang it, so much to learn. 1643 01:26:40,640 --> 01:26:42,120 Now I got to do something with this. 1644 01:26:45,520 --> 01:26:49,160 But yeah, it's it's wild because you start hearing everyone's 1645 01:26:49,240 --> 01:26:53,440 just. Using LMS to do research now and 1646 01:26:53,440 --> 01:26:56,320 what's funny is I actually used. Claude to. 1647 01:26:56,320 --> 01:27:01,960 Find I have an affiliate partner program now and I found the 1648 01:27:01,960 --> 01:27:05,800 Stripe app with Claude because I was like, I need to be able to 1649 01:27:05,800 --> 01:27:10,000 pay people, you know, worldwide and that's like not it's not an 1650 01:27:10,000 --> 01:27:13,760 easy thing to track because you have to give them all the proper 1651 01:27:13,760 --> 01:27:17,920 like tax paperwork and things for like the EU and then the 1652 01:27:18,280 --> 01:27:21,080 that's and they handle all that, which is kind. 1653 01:27:21,080 --> 01:27:26,920 Of nice and I'm using. Dubdub.co OK, is this is your? 1654 01:27:26,920 --> 01:27:29,280 Selling direct to consumer. Yeah, yeah. 1655 01:27:29,560 --> 01:27:30,680 OK. Yeah, Yeah, yeah, that. 1656 01:27:30,680 --> 01:27:32,320 Yeah, they're a. Little that you use a little 1657 01:27:32,320 --> 01:27:33,560 more. Lenient if you're doing like B 1658 01:27:33,560 --> 01:27:36,040 to B transactions, I think. Oh, are they? 1659 01:27:36,320 --> 01:27:38,320 Yeah, you just yeah, it's a different tax. 1660 01:27:38,440 --> 01:27:41,480 I've had some people bring up, I don't know if it's. 1661 01:27:41,480 --> 01:27:47,880 Worth my my time yet? But like a what would you call 1662 01:27:47,880 --> 01:27:48,360 them? Like a a. 1663 01:27:48,920 --> 01:27:54,080 Financial advisor was like, he wants, he wants a plan that 1664 01:27:54,080 --> 01:27:57,560 would allow him to kind of like oversee and control clients 1665 01:27:57,560 --> 01:28:00,360 stuff. And I was like that like defeats 1666 01:28:00,360 --> 01:28:03,240 the purpose because my point is like you don't have to trust any 1667 01:28:03,240 --> 01:28:06,800 third party. And so I don't know, maybe 1668 01:28:06,800 --> 01:28:07,400 there's like. A. 1669 01:28:07,400 --> 01:28:10,400 Way I have like. Different vaults of files and 1670 01:28:10,400 --> 01:28:14,720 you can give people access to specific data, but not all your 1671 01:28:14,720 --> 01:28:15,720 data. I don't know. 1672 01:28:15,720 --> 01:28:19,520 I got to figure that out because the second you start. 1673 01:28:20,120 --> 01:28:24,920 Sharing like. 00 knowledge encryption, it's no longer that 1674 01:28:24,920 --> 01:28:27,200 right? Because now you're you're 1675 01:28:27,560 --> 01:28:31,080 allowing access to the encrypted file. 1676 01:28:31,080 --> 01:28:34,600 So you have to kind of you unwrap your private key 1677 01:28:34,600 --> 01:28:36,120 essentially. And it just doesn't. 1678 01:28:36,120 --> 01:28:41,480 Work the same way and it it opens things up but I don't 1679 01:28:41,480 --> 01:28:44,680 know. So you're using 00 knowledge 1680 01:28:44,680 --> 01:28:45,880 encryption. For for. 1681 01:28:46,280 --> 01:28:48,840 Yeah, yeah, yeah. Is. 1682 01:28:49,240 --> 01:28:50,640 It is there like a. JavaScript. 1683 01:28:50,640 --> 01:28:54,400 Library for that that you've. Been working with yeah kind of. 1684 01:28:54,400 --> 01:28:58,280 So I'm using JavaScript. And I I generate like a private 1685 01:28:58,280 --> 01:29:00,480 key. I'm I'm doing basically like 1686 01:29:00,480 --> 01:29:02,440 what 1 password and bit warden do. 1687 01:29:03,200 --> 01:29:08,840 And so basically your password client side will unwrap your 1688 01:29:08,840 --> 01:29:11,960 private key. Like I wrap the private key 1689 01:29:12,160 --> 01:29:14,360 client side, send it to my server and store it. 1690 01:29:14,920 --> 01:29:16,680 And so I can't do anything with it. 1691 01:29:16,680 --> 01:29:20,440 But then you get it back, you can unwrap it, encrypt, decrypt 1692 01:29:20,440 --> 01:29:24,160 your files client side. And then it's all like 1693 01:29:24,200 --> 01:29:27,440 relatively safe. And then the, the share links 1694 01:29:27,440 --> 01:29:31,120 were the hard part, right, 'cause this is where you kind of 1695 01:29:31,120 --> 01:29:33,960 like you kind of open up a layer of vulnerability. 1696 01:29:34,720 --> 01:29:41,800 But I I basically create a share link with a new encrypted key 1697 01:29:42,520 --> 01:29:48,360 that you can remove access to. So let's say this share link 1698 01:29:48,360 --> 01:29:50,760 goes out through a deadman switch to a client. 1699 01:29:50,760 --> 01:29:53,080 It was an accident like you don't want it to hit that 1700 01:29:53,080 --> 01:29:57,400 beneficiary. You can revoke that that off 1701 01:29:57,440 --> 01:29:59,720 token essentially. And so they lose. 1702 01:29:59,720 --> 01:30:05,440 Access to the file and all that. But the caveat to that is both 1703 01:30:05,440 --> 01:30:07,760 parties can now read your encrypted file. 1704 01:30:07,760 --> 01:30:10,920 So, but you can remove it. You can remove access, but 1705 01:30:11,160 --> 01:30:14,360 there's a balance. It's kind of, but I tried to 1706 01:30:14,360 --> 01:30:17,720 follow best practices and I went down the rabbit hole of like, 1707 01:30:18,040 --> 01:30:19,960 how does 1 password share vaults? 1708 01:30:19,960 --> 01:30:23,880 How does this work? And yeah, we're all doing a 1709 01:30:24,000 --> 01:30:26,360 combination of symmetric and asymmetric. 1710 01:30:26,360 --> 01:30:28,560 Key encryption, yes. Yeah, exactly. 1711 01:30:29,120 --> 01:30:32,800 Yeah, Yeah. But. 1712 01:30:32,800 --> 01:30:35,520 I kind of like I went into. It not knowing that. 1713 01:30:35,520 --> 01:30:37,600 Much. And I learned a lot about 1714 01:30:37,600 --> 01:30:42,880 encryption, so that's cool. It was like a service I. 1715 01:30:42,880 --> 01:30:44,800 Wanted for myself, I was like I want a dead. 1716 01:30:44,800 --> 01:30:48,600 Man, switch and I turned it into. 1717 01:30:48,600 --> 01:30:52,520 A product is it? Which can I ask? 1718 01:30:52,520 --> 01:30:54,400 Which cloud it's running on? Are you running on a on the 1719 01:30:54,400 --> 01:30:57,640 cloud or are you using like fly? I use. 1720 01:30:57,680 --> 01:31:01,360 I use Digitalocean. I run on my own droplet and just 1721 01:31:01,600 --> 01:31:05,280 docker compose. Yeah, yeah, yeah. 1722 01:31:07,160 --> 01:31:10,480 And then everything is stored. On their S3 buckets. 1723 01:31:11,160 --> 01:31:14,000 Encrypted so I can't open any of it. 1724 01:31:15,080 --> 01:31:16,160 Nice. Yeah. 1725 01:31:17,520 --> 01:31:19,120 And that was like, that's, that was my. 1726 01:31:19,120 --> 01:31:22,120 Whole goal, I was like, if I ever have engineers, I don't 1727 01:31:22,120 --> 01:31:26,000 want any of my engineers to be able to go and read a user's 1728 01:31:27,040 --> 01:31:30,160 files, you know? Yeah, you say that. 1729 01:31:30,160 --> 01:31:32,160 Now until you get. Support issues that are. 1730 01:31:32,160 --> 01:31:37,160 Like ah lost by login slash browser storage slash browser. 1731 01:31:37,160 --> 01:31:40,400 So that is, that is a concern and. 1732 01:31:40,480 --> 01:31:45,560 Right now I don't expose the private key to the client and I 1733 01:31:45,560 --> 01:31:49,400 do like recovery codes. I generate recovery codes that 1734 01:31:49,400 --> 01:31:51,000 they can download on registration. 1735 01:31:51,440 --> 01:31:55,160 And I have considered I may be revealing their private key to 1736 01:31:55,160 --> 01:31:58,480 them in that same step and give them a way to save it. 1737 01:31:58,480 --> 01:32:01,560 Because if they have that, they don't even need my service to 1738 01:32:01,560 --> 01:32:05,680 decrypt their files. So I could still allow them to 1739 01:32:05,680 --> 01:32:08,720 download their encrypted files and then if they have their 1740 01:32:08,720 --> 01:32:10,480 private key, they could still decrypt it. 1741 01:32:11,040 --> 01:32:13,520 I don't know though. I feel like that adds a layer of 1742 01:32:14,760 --> 01:32:16,280 complexity that I don't really need. 1743 01:32:17,960 --> 01:32:20,720 Yeah, you know the. Let's see. 1744 01:32:22,080 --> 01:32:24,800 The at least. On Apple. 1745 01:32:24,800 --> 01:32:30,040 Systems, you know you've got these if you're using, so it's 1746 01:32:30,040 --> 01:32:34,920 not like a cross-platform, cross browser solution, but you can 1747 01:32:34,920 --> 01:32:37,880 get into the you know you can, you can integrate with the I 1748 01:32:37,880 --> 01:32:42,200 guess the secure enclave or key chain access to to help out with 1749 01:32:42,200 --> 01:32:43,320 all of that stuff. So you're not. 1750 01:32:43,320 --> 01:32:46,200 Having to do it in. In JavaScript, yeah. 1751 01:32:47,880 --> 01:32:50,560 Yeah, that's. An option I've also like batted 1752 01:32:50,560 --> 01:32:55,920 around the idea of creating like an Electron app version or some 1753 01:32:56,400 --> 01:32:59,800 version where you can locally have it to kind of protect that 1754 01:32:59,800 --> 01:33:01,920 side a little bit. Little menu bar applet that's 1755 01:33:01,920 --> 01:33:03,840 like running with a browser extension. 1756 01:33:03,840 --> 01:33:06,680 And yeah, yeah, I think that could be cool. 1757 01:33:06,680 --> 01:33:08,720 And then just like. You just get a local 1758 01:33:08,720 --> 01:33:11,920 notification for check insurance and just another layer. 1759 01:33:12,560 --> 01:33:14,440 So we'll see. I have a lot of ideas. 1760 01:33:14,440 --> 01:33:17,760 It's just where does my time go? Is it open source? 1761 01:33:18,400 --> 01:33:20,240 It's not. Should it be? 1762 01:33:23,000 --> 01:33:24,600 Yeah. I think it should you. 1763 01:33:24,600 --> 01:33:25,840 Should you should open source it? 1764 01:33:26,480 --> 01:33:28,880 Yeah, you can do little pieces at a time. 1765 01:33:28,880 --> 01:33:34,040 Just do like, you know, just do the web, the web UI first, OK? 1766 01:33:34,440 --> 01:33:40,000 My like my fear is. Properly open sourcing things 1767 01:33:40,280 --> 01:33:44,120 like making sure everything is is clean. 1768 01:33:44,120 --> 01:33:46,320 Do I have any? Default. 1769 01:33:46,360 --> 01:33:48,640 Keys in there that I shouldn't you should. 1770 01:33:48,640 --> 01:33:51,080 I mean those should be. Those should be easy to. 1771 01:33:51,080 --> 01:33:53,240 Find, yeah, those should be easy to find. 1772 01:33:53,240 --> 01:33:56,360 But yeah, you won't get, you'll get. 1773 01:33:56,840 --> 01:34:00,360 I'm sure if you've got any major issues, someone will come by and 1774 01:34:00,360 --> 01:34:01,760 fix it for you. That is true. 1775 01:34:01,760 --> 01:34:03,880 It's kind of like you get a free audit, right? 1776 01:34:05,520 --> 01:34:07,440 Yeah, but I mean nowadays you can run. 1777 01:34:07,480 --> 01:34:09,560 You can, you know, there's a million different audit tools 1778 01:34:09,560 --> 01:34:12,240 you can run like code QL and yeah, yeah, for sure. 1779 01:34:12,240 --> 01:34:15,000 Some of the stuff like so below I think is another one just for 1780 01:34:15,000 --> 01:34:17,600 basic I I use paraxial on it too. 1781 01:34:17,600 --> 01:34:22,320 And that is so blow. OK, so as far as I know. 1782 01:34:22,560 --> 01:34:26,280 Everything is secure. Yeah, yeah. 1783 01:34:26,280 --> 01:34:27,680 I mean. Like I think you could. 1784 01:34:27,680 --> 01:34:29,840 Also just put a big disclaimer and say like, hey, you know, 1785 01:34:29,840 --> 01:34:32,800 don't, yeah, maybe don't offer it as a service. 1786 01:34:32,800 --> 01:34:35,720 I don't know if you're offering it as a service now, but it is a 1787 01:34:35,720 --> 01:34:37,160 service. Oh, it is a service now. 1788 01:34:37,160 --> 01:34:37,760 So. Yeah. 1789 01:34:37,760 --> 01:34:39,120 So you. Yeah. 1790 01:34:39,120 --> 01:34:41,480 So if you put it out there, yeah. 1791 01:34:41,480 --> 01:34:43,480 You, you probably, you know, you could, you could also get like 1792 01:34:43,480 --> 01:34:47,720 an independent review. That's not a bad idea. 1793 01:34:47,960 --> 01:34:50,080 Yeah. I think like the hard part. 1794 01:34:50,080 --> 01:34:52,960 Is people have to trust I'm. Handling their data in the right 1795 01:34:52,960 --> 01:34:54,040 way, right? Yeah. 1796 01:34:54,040 --> 01:34:56,480 So maybe like the third party review would be a good 1797 01:34:57,560 --> 01:34:59,880 disclaimer or you know, a good review. 1798 01:34:59,880 --> 01:35:02,200 To have. Have you looked into like the 1799 01:35:02,640 --> 01:35:06,360 the whole trusted? Computing movement with to to 1800 01:35:06,360 --> 01:35:08,800 because like you can have your. Because the problem there is 1801 01:35:08,800 --> 01:35:13,120 like you, you put your code, put your code out there in a repo, 1802 01:35:13,320 --> 01:35:15,840 but there's no real guarantee that what you're running in your 1803 01:35:15,840 --> 01:35:17,560 service is the same, right? Yeah. 1804 01:35:17,640 --> 01:35:21,640 So if you're looking to build trust, that's a good need to. 1805 01:35:21,920 --> 01:35:24,760 Yeah, I'll have to go down that because I think it's like. 1806 01:35:24,760 --> 01:35:27,040 One of those things you have to prove and people have to know 1807 01:35:27,080 --> 01:35:30,600 you're doing it the right way. But yeah, cool. 1808 01:35:31,800 --> 01:35:35,960 All right. I really appreciate you joining 1809 01:35:35,960 --> 01:35:37,200 me this. Was a lot of fun. 1810 01:35:37,560 --> 01:35:40,440 Yeah, it's great. Fire Zone is a cool. 1811 01:35:40,440 --> 01:35:41,360 Product, it's over. My. 1812 01:35:41,360 --> 01:35:42,840 Head, but it's. Impressive. 1813 01:35:42,840 --> 01:35:45,920 Like it's cool to. It's cool to see something I 1814 01:35:45,920 --> 01:35:48,800 feel like at that scale working so well. 1815 01:35:48,800 --> 01:35:51,760 Like, it's cool. And yeah, thanks for having me. 1816 01:35:51,760 --> 01:35:54,920 Yeah, it was a lot of fun. Yeah, I appreciate it. 1817 01:35:54,920 --> 01:35:57,000 Thank you so much. Jamelle. 1818 01:35:57,040 --> 01:35:59,800 And. I have all your contact info in 1819 01:35:59,800 --> 01:36:04,120 the description and tomorrow after this process is on YouTube 1820 01:36:04,320 --> 01:36:09,400 for 24 hours, it'll be pushed up to all the podcast platforms so 1821 01:36:09,400 --> 01:36:13,600 people can catch this conversation wherever they want. 1822 01:36:14,760 --> 01:36:17,920 Awesome, the future is now. And for everyone that tuned. 1823 01:36:17,920 --> 01:36:21,040 In Thank you so much and. I'll see you guys next week. 1824 01:36:21,680 --> 01:36:25,360 All righty, bye. Testing.