1 00:00:00,040 --> 00:00:02,200 This episode is brought to you by Beam OPS. 2 00:00:02,360 --> 00:00:05,480 If your Elixir deployments are painful, or your infrastructure 3 00:00:05,480 --> 00:00:07,920 is a mess, or you just need experience hands to help you 4 00:00:07,920 --> 00:00:11,840 scale, these are your people. Beam OPS specializes in scalable 5 00:00:11,840 --> 00:00:14,800 Elixir development, automated deployment pipelines, and 6 00:00:14,800 --> 00:00:18,000 infrastructure migrations. They will help you work smarter 7 00:00:18,040 --> 00:00:21,480 and ship faster. As authors of the Pragprog book 8 00:00:21,560 --> 00:00:24,480 Engineering Elixir Applications, they know their stuff. 9 00:00:24,720 --> 00:00:30,520 Check them out at beamops.co.uk. Secure your Elixir apps with 10 00:00:30,520 --> 00:00:33,240 paraxial IO. From code analysis and bot 11 00:00:33,240 --> 00:00:35,840 defense to dependency checks, they've got you covered. 12 00:00:36,000 --> 00:00:38,800 Elixir first design with GitHub integration. 13 00:00:38,920 --> 00:00:46,680 Elevate your security at paraxial dot IO testing. 14 00:00:50,720 --> 00:00:53,120 What's up? I'm Jacob Litzo, your Elixir 15 00:00:53,120 --> 00:00:55,880 mentor, and welcome to another exciting episode of the Elixir 16 00:00:55,880 --> 00:00:58,080 Mentor Podcast. This is where we discuss 17 00:00:58,080 --> 00:01:00,240 everything related to Elixir, from interviews with 18 00:01:00,240 --> 00:01:03,040 enthusiasts, pioneers in the community, to innovative 19 00:01:03,040 --> 00:01:05,560 projects and library shaping. Elixir's future. 20 00:01:06,200 --> 00:01:09,880 Thomas joins me today about building systems that don't hold 21 00:01:09,880 --> 00:01:12,880 you hostage and keeping knowledge that makes your team 22 00:01:12,880 --> 00:01:16,240 irreplaceable from ever walking out the door. 23 00:01:17,520 --> 00:01:20,880 I'm sure we'll talk about plenty more, but thanks for joining me 24 00:01:20,880 --> 00:01:22,880 today, Thomas. I'm excited to have you. 25 00:01:24,040 --> 00:01:26,240 Thanks for having me, looking forward to this a bunch. 26 00:01:27,080 --> 00:01:29,400 Yeah. So do you want to just give a 27 00:01:29,400 --> 00:01:33,240 like a quick version of how do you end up as head of 28 00:01:33,240 --> 00:01:36,440 engineering at level all and like kind of what's your journey 29 00:01:36,920 --> 00:01:42,920 there to that to to this moment? To this moment, I have been with 30 00:01:42,960 --> 00:01:47,520 Oval all for three years. I started as a senior, moved to 31 00:01:47,520 --> 00:01:52,480 staff and then we had a departure above me who wanted to 32 00:01:52,480 --> 00:01:56,240 return to being an IC and it was just kind of like a natural fit 33 00:01:56,240 --> 00:02:00,720 that I slotted myself into. I'm a Co head of engineering 34 00:02:00,720 --> 00:02:03,880 with another gentleman on our team and we've been tag teaming 35 00:02:03,960 --> 00:02:07,280 us into success. Awesome, very cool. 36 00:02:08,280 --> 00:02:11,840 I recently I met you down at X Max. 37 00:02:11,840 --> 00:02:16,280 You did a talk down there and I know like we've talked in the 38 00:02:16,280 --> 00:02:18,800 past, we Elixir community is small. 39 00:02:18,800 --> 00:02:24,760 I like that. And you're incredibly passionate 40 00:02:24,840 --> 00:02:30,520 about like deploying your own code, not getting locked into 41 00:02:30,600 --> 00:02:36,320 solutions that make it hard to be flexible and kind of like 42 00:02:36,320 --> 00:02:41,600 adjust your needs as, you know, fast-paced start-ups change. 43 00:02:43,200 --> 00:02:48,120 What's kind of your experience with deploying production 44 00:02:48,200 --> 00:02:53,280 applications and what made you not like solutions that exist 45 00:02:53,280 --> 00:02:57,400 out there and kind of do your own bare metal deployments? 46 00:02:58,720 --> 00:03:04,280 Of course you can call it passion, some people may call it 47 00:03:04,280 --> 00:03:10,040 delusional. It's really like AII want to 48 00:03:10,040 --> 00:03:15,240 keep all available options open. And I feel like when you are 49 00:03:15,840 --> 00:03:25,720 deploying to a pass like Heroku, RIP Gig Elixir Fly, while they 50 00:03:25,920 --> 00:03:29,800 take care of a lot of things for you, they also tie your hands at 51 00:03:29,800 --> 00:03:35,080 the exact same time. And that's not necessarily the 52 00:03:35,080 --> 00:03:37,000 flexibility that people may want. 53 00:03:38,000 --> 00:03:40,280 Hands off is great until you run into a problem. 54 00:03:41,120 --> 00:03:44,800 And last year we ran into a problem. 55 00:03:45,240 --> 00:03:49,320 We were deployed, we had a couple services on Fly and then 56 00:03:49,320 --> 00:03:54,880 our main production application was on Gigalixer and we had a 57 00:03:54,880 --> 00:03:59,920 bunch of traffic and we could not throw enough hardware at the 58 00:03:59,920 --> 00:04:02,800 problem to settle all the traffic. 59 00:04:04,440 --> 00:04:09,960 We took the bar and slid it all the way over the that node that 60 00:04:09,960 --> 00:04:12,560 we were on at Gigalixxer. We had multiple, but it like our 61 00:04:12,560 --> 00:04:19,160 bill was $7000 a month. And that just seems nutty to me 62 00:04:19,760 --> 00:04:24,640 because I know with the shadow of a doubt that our application 63 00:04:24,640 --> 00:04:28,800 doesn't need that, right? We all instinctually know that 64 00:04:30,280 --> 00:04:32,560 Elixir doesn't. Elixir's really powerful, but it 65 00:04:32,560 --> 00:04:34,480 doesn't also need a really powerful machine. 66 00:04:35,360 --> 00:04:38,200 I feel like most Elixir production applications could be 67 00:04:38,200 --> 00:04:40,400 hosted on a Raspberry Pi and be totally Vine. 68 00:04:41,480 --> 00:04:46,120 But I think that's where the lie of performance and cloud kind of 69 00:04:46,480 --> 00:04:52,000 conflate because you are paying for inflexibility and bad 70 00:04:52,000 --> 00:04:53,720 performance. And I know that's a hot take. 71 00:04:53,960 --> 00:04:59,280 I'm going to hear about it, but it's what I feel so. 72 00:04:59,640 --> 00:05:01,280 Hot takes are the best takes man. 73 00:05:03,320 --> 00:05:08,800 Sometimes we migrated away from Fly and Gigalix are still in the 74 00:05:08,800 --> 00:05:16,640 cloud at Google, but using Google directly, we are as bare 75 00:05:16,640 --> 00:05:20,440 metal as you can get on AVM mix release. 76 00:05:20,480 --> 00:05:23,000 There's no Docker, there's no nothing like that. 77 00:05:23,440 --> 00:05:26,440 Raising instance templates and Migs. 78 00:05:28,760 --> 00:05:31,760 I deploy a little bit differently, but the first time 79 00:05:31,760 --> 00:05:36,160 that I heard of this deployment mechanism for Elixir was 80 00:05:36,160 --> 00:05:39,880 actually Log Flare. There's a blog post on how they 81 00:05:39,880 --> 00:05:42,960 deployed and so I kind of mimicked it, but I do a whole 82 00:05:42,960 --> 00:05:46,800 lot more in our cloud in it than they're doing. 83 00:05:48,160 --> 00:05:51,240 But I'm happy to say I finally got approval for actual bare 84 00:05:51,240 --> 00:05:53,280 metal. We need to wait till our Sock 2 85 00:05:53,280 --> 00:05:55,880 audit window closes in the middle of June and then we're 86 00:05:55,920 --> 00:05:58,560 out of there. We are going actually bare metal 87 00:05:58,560 --> 00:06:00,680 on rented hardware getting out of. 88 00:06:00,680 --> 00:06:08,760 I have no idea what that was. We're, we're going to actually, 89 00:06:08,760 --> 00:06:11,520 yeah, I guess not. It's not surprising to be 90 00:06:11,520 --> 00:06:16,280 honest. We're, we're going to take that 91 00:06:16,280 --> 00:06:19,160 actual leap. If you, if you've paid attention 92 00:06:19,160 --> 00:06:23,800 at all, I, I feel like there's a kind of symbiotic relationship 93 00:06:23,800 --> 00:06:26,320 between Elixir and Ruby. But if you paid attention to 94 00:06:26,320 --> 00:06:29,360 what 37 signals talks about all the time with getting out of the 95 00:06:29,360 --> 00:06:33,880 cloud, exfilling, that's what we plan on doing too. 96 00:06:34,000 --> 00:06:38,400 The, the insanity of the amount of money that we're paying for 97 00:06:38,440 --> 00:06:40,720 like hosted Postgres and things like that. 98 00:06:41,760 --> 00:06:48,160 An $1800 a month server with 64 physical cores, a TB and a half 99 00:06:48,160 --> 00:06:53,160 of RAM, 16 terabytes of storage. That's lickety split fast. 100 00:06:53,680 --> 00:06:58,840 I mean, we pay $6000 a month for our databases. 101 00:06:58,840 --> 00:07:01,600 Across all databases. There's more than one, but we 102 00:07:01,600 --> 00:07:07,200 pay for 30,000 provision IOPS. If you're on an NVM me drive, 103 00:07:07,240 --> 00:07:11,160 you're getting millions of IOPS just because that's a massive 104 00:07:11,160 --> 00:07:14,640 difference. So yeah, you lose the benefit of 105 00:07:15,120 --> 00:07:20,360 point and click backups and things like that, but. 106 00:07:21,040 --> 00:07:23,320 When you say like you're gonna rent space, like, are you 107 00:07:23,320 --> 00:07:26,080 actually gonna just rent a server or do you rent Rackspace 108 00:07:26,080 --> 00:07:29,320 to put your own server in? Like what does that look like? 109 00:07:29,320 --> 00:07:33,480 We're we're going to rent the the server, the company that 110 00:07:33,480 --> 00:07:35,400 will go with this called latitude. 111 00:07:35,520 --> 00:07:42,680 It's Latitude dot SH and they, they were really big and the the 112 00:07:42,680 --> 00:07:45,960 hosting of blockchain initially, it's how they got started is 113 00:07:45,960 --> 00:07:48,400 hosting really powerful servers for Solana. 114 00:07:49,240 --> 00:07:55,160 But Equinix Metal, which bought deploy.net, they bought it, they 115 00:07:55,160 --> 00:07:58,160 ran it for three or four years and as acquisitions go, they 116 00:07:58,160 --> 00:08:02,600 just sunset it. And so Latitude is been picking 117 00:08:02,600 --> 00:08:10,080 up the baton where Equinix Metal I guess left it and they are 118 00:08:10,080 --> 00:08:12,080 growing like crazy. They have a ton of available 119 00:08:12,080 --> 00:08:15,600 hardware. They've got things that where 120 00:08:15,600 --> 00:08:17,800 it's not you're just not running just a computer. 121 00:08:18,000 --> 00:08:22,120 You can set up your networks. You can immediately deploy 5 122 00:08:22,120 --> 00:08:24,360 second bare metal deploys like that's. 123 00:08:24,360 --> 00:08:26,720 That's pretty wild. That's amazing. 124 00:08:27,360 --> 00:08:33,120 I'm I I'm inspired by your just the mixed release and getting 125 00:08:33,120 --> 00:08:35,320 rid of Docker. I haven't pulled the trigger of 126 00:08:35,320 --> 00:08:39,679 getting rid of Docker in my own deployments, but I really want 127 00:08:39,679 --> 00:08:41,039 to. I should probably spend the time 128 00:08:41,039 --> 00:08:46,280 to do it. It's it's an interesting choice. 129 00:08:46,280 --> 00:08:48,960 Docker provides you a lot of ease. 130 00:08:49,560 --> 00:08:52,040 You set up your container exactly the way that it needs to 131 00:08:52,040 --> 00:08:55,240 run. Logging or metrics are really 132 00:08:55,240 --> 00:08:57,720 easy because there's like a single outlet for it. 133 00:08:59,360 --> 00:09:03,840 I feel though, that the overhead, which is crazy to say 134 00:09:03,840 --> 00:09:06,480 because there's not that much, it's just not worth it. 135 00:09:07,200 --> 00:09:10,720 Beam really likes being on hardware. 136 00:09:11,480 --> 00:09:16,040 So I wanna I wanna give it it. And I feel VM is like when 137 00:09:16,040 --> 00:09:18,520 you're running in the cloud. AVM is already kind of, you 138 00:09:18,520 --> 00:09:20,880 know, it's hypervisor. It's not exactly bare metal. 139 00:09:21,080 --> 00:09:24,720 But that's where when I was running benchmarks to convince 140 00:09:24,720 --> 00:09:28,920 upper management last year to move out of Gig Elixir, I was 141 00:09:28,920 --> 00:09:32,920 load testing our application there and I was comparing it 142 00:09:32,920 --> 00:09:38,800 against a 2011 Intel Mac mini old. 143 00:09:39,480 --> 00:09:43,440 And I was beating the pants off of whatever gig Elixer was with 144 00:09:43,640 --> 00:09:49,720 a piece of junk Mac mini at least 10X the load on a $7000 145 00:09:49,720 --> 00:09:51,880 Gigalixer node. And I'm not here to like throw 146 00:09:51,880 --> 00:09:54,680 shade at gigs lickers or they they obviously provide a great 147 00:09:55,000 --> 00:09:58,920 hands off pass for people. I I like what they offer, just 148 00:09:58,920 --> 00:10:00,200 like fly. I really like them. 149 00:10:00,720 --> 00:10:04,280 But for our needs, we just need to be different. 150 00:10:04,840 --> 00:10:08,920 I feel like a lot of people say, you know, if I build it, traffic 151 00:10:08,920 --> 00:10:13,760 will come yes or no. But our traffic is very bursty. 152 00:10:14,720 --> 00:10:20,360 Our application is for education and you'll have a classroom of 153 00:10:20,360 --> 00:10:24,160 students who all use the product at the exact same time. 154 00:10:24,680 --> 00:10:29,320 Or in the case where you're like no traffic or all traffic like 155 00:10:29,360 --> 00:10:32,640 you don't have to. Exactly. 156 00:10:32,640 --> 00:10:37,200 So we right now, as the school year is coming to a close, 157 00:10:37,200 --> 00:10:39,360 people are doing all the testing and things like that. 158 00:10:40,520 --> 00:10:44,160 Right now is the worst time for traffic for us because we won't 159 00:10:44,160 --> 00:10:47,840 just have a classroom, we'll have an entire school sitting in 160 00:10:47,840 --> 00:10:50,680 a cafeteria or an assembly like in their pack. 161 00:10:51,880 --> 00:10:54,800 And then it won't just be 1 school, it will be 12 schools in 162 00:10:54,800 --> 00:10:57,360 an entire district. So you'll go from zero traffic 163 00:10:57,360 --> 00:11:00,440 to 50,000 concurrent users all at once. 164 00:11:00,760 --> 00:11:03,560 And that thundering herd problem is kind of hard. 165 00:11:03,800 --> 00:11:07,720 Yeah, that's crazy. So rather than worrying about 166 00:11:08,040 --> 00:11:12,520 like there is no time to say oh we're gonna auto fail on memory 167 00:11:12,520 --> 00:11:16,640 or CPU or request per second and then wait for the server to come 168 00:11:16,640 --> 00:11:19,760 up, we just have to be over provisioned all the time in 169 00:11:19,760 --> 00:11:23,240 order to handle it. How do you, how do you handle 170 00:11:23,240 --> 00:11:25,400 that? Do you know when like events are 171 00:11:25,560 --> 00:11:29,560 going to happen or like a influx of users? 172 00:11:29,560 --> 00:11:33,560 Do you know or like, how do you prepare for things like that? 173 00:11:34,440 --> 00:11:38,400 Sometimes, sometimes our sales team can give us a heads up when 174 00:11:38,400 --> 00:11:41,920 an onboarding is happening, but most of the time it's kind of 175 00:11:41,920 --> 00:11:45,520 like, oh, all of our charts are going up and to the right, I 176 00:11:45,520 --> 00:11:48,000 guess something's happening. Yeah. 177 00:11:48,560 --> 00:11:52,640 And then do you do you auto scale or do you just have to 178 00:11:52,640 --> 00:11:57,920 have nodes ready we? Have auto scaling turned on 179 00:11:57,920 --> 00:12:02,560 right now in Google but we are over provisioned so much that we 180 00:12:02,560 --> 00:12:10,200 rarely hit it and some could say well that's a waste. 181 00:12:11,080 --> 00:12:15,800 Sure not using docker, not been packing your compute could be a 182 00:12:15,800 --> 00:12:20,160 waste too. But that extra few $100 a month 183 00:12:20,160 --> 00:12:25,720 of overhead for our entire businesses like stay up ability, 184 00:12:25,720 --> 00:12:28,560 I I don't feel like is a thing like no, of course you're just 185 00:12:28,560 --> 00:12:31,600 going to pay the extra few $100 to have an over provision 186 00:12:31,600 --> 00:12:34,080 machine and not worry about it. Yeah, I, I feel like the trade 187 00:12:34,080 --> 00:12:37,600 off is probably worth not having the headaches. 188 00:12:37,600 --> 00:12:42,360 Of course, like over a couple $100 when you're talking about 189 00:12:42,680 --> 00:12:44,760 server bill, that's 6 grand, you know. 190 00:12:45,680 --> 00:12:48,040 Exactly. Yeah, it doesn't really matter 191 00:12:48,040 --> 00:12:50,720 at that point. What do you do? 192 00:12:50,720 --> 00:12:56,360 How do you, how do you like, I know this is just like a lot 193 00:12:56,480 --> 00:12:59,560 more difficult of a problem and a lot of people don't talk about 194 00:12:59,560 --> 00:13:04,600 it, but like sharding and scaling databases are it's not 195 00:13:04,600 --> 00:13:09,520 an easy thing to do. Do you just have one really big 196 00:13:09,520 --> 00:13:13,080 database or do you actually have? 197 00:13:14,080 --> 00:13:17,520 Do you actually distribute your your data as well? 198 00:13:19,200 --> 00:13:28,560 Today it's all one big database. It's the worst type of database 199 00:13:28,560 --> 00:13:33,120 to have in the sense that if there's anyone listening who's 200 00:13:33,120 --> 00:13:35,880 in Operations or ADBA, they'll know what I'm talking about. 201 00:13:36,760 --> 00:13:41,120 Very high read and writes. You have both sides of the 202 00:13:41,120 --> 00:13:44,000 problem. Our entire database is only like 203 00:13:44,000 --> 00:13:49,720 200 and 52150 gigs big, but our wall file is like 8 terabytes. 204 00:13:50,080 --> 00:13:53,080 Oh Dang, why? Well, it's because we have so 205 00:13:53,080 --> 00:13:55,920 much coming in and out and and editing and changing and 206 00:13:55,920 --> 00:14:00,320 updating all the time that like it's always being touched. 207 00:14:00,320 --> 00:14:05,720 So we haven't sharded yet. We haven't done anything out of 208 00:14:05,720 --> 00:14:08,480 the blue. We have failover in place, but 209 00:14:08,480 --> 00:14:10,280 we're not even using Reed replicas. 210 00:14:11,560 --> 00:14:13,040 I just don't feel like we need to. 211 00:14:13,360 --> 00:14:16,080 I was just going to ask if you do Reed replicas, but no. 212 00:14:16,920 --> 00:14:19,840 No, they're, they're only there for hot spares. 213 00:14:20,080 --> 00:14:27,000 I when we go true bare metal, I could look into it, but so far I 214 00:14:27,000 --> 00:14:28,640 don't think we I don't think we need it. 215 00:14:30,200 --> 00:14:33,800 Live view is actually the the thing that uses the most amount 216 00:14:33,800 --> 00:14:35,800 of like resources. Or compute if you want to. 217 00:14:36,080 --> 00:14:39,760 If you want to be like super honest about it, there's just 218 00:14:39,760 --> 00:14:41,960 like. That of internal sockets. 219 00:14:42,880 --> 00:14:46,200 Yeah, exactly. Yep. 220 00:14:46,760 --> 00:14:50,920 But we are moving away from live view going into API only mode 221 00:14:51,200 --> 00:14:55,840 using a view JS front end because we have iOS client now 222 00:14:55,920 --> 00:14:59,960 and we don't want to dual right API and live view. 223 00:15:01,360 --> 00:15:03,960 That change is a little bit different. 224 00:15:04,000 --> 00:15:06,640 I like live view, I don't really want to leave it, but at the 225 00:15:06,640 --> 00:15:11,400 same time it's like it's kind of a necessity now and I don't feel 226 00:15:11,400 --> 00:15:14,920 like we're going to leave. In fact, we'll probably be able 227 00:15:14,920 --> 00:15:17,080 to de provision a little bit because live view is pretty 228 00:15:17,080 --> 00:15:20,320 heavy and when we're just going standard controllers, it won't 229 00:15:20,320 --> 00:15:22,160 be a big deal. I see. 230 00:15:22,560 --> 00:15:25,400 Do you are you gonna keep live view at all for like admin 231 00:15:25,680 --> 00:15:27,280 dashboards or anything like that? 232 00:15:27,280 --> 00:15:30,560 Or are you completely removing it? 233 00:15:31,520 --> 00:15:34,080 We're completely removing it. OK. 234 00:15:35,200 --> 00:15:38,880 That's a big, big step. You you use Ashe framework as 235 00:15:38,880 --> 00:15:40,880 well right? Yes. 236 00:15:41,280 --> 00:15:44,840 Nice. And does that, does that help 237 00:15:44,840 --> 00:15:50,400 you at all with opening up like API endpoints versus not and how 238 00:15:50,400 --> 00:15:55,640 are you? It it did, but here's another 239 00:15:55,640 --> 00:15:58,280 hot take. We're we're getting rid of ash. 240 00:15:58,680 --> 00:16:00,720 Oh, interesting. How come? 241 00:16:00,720 --> 00:16:05,760 Yeah. We, we, we, we jumped in two 242 00:16:05,760 --> 00:16:09,560 feet first and just started running as fast as we could with 243 00:16:09,560 --> 00:16:12,160 Ash. The problem is finding people 244 00:16:12,160 --> 00:16:15,400 that understand Ash and it's interesting. 245 00:16:15,400 --> 00:16:21,280 If you talk to Zach or anyone from Olympic, they will show you 246 00:16:21,280 --> 00:16:25,680 a bell curve. Of I'm having Josh Price on next 247 00:16:25,680 --> 00:16:27,920 week actually. Great guy. 248 00:16:28,760 --> 00:16:31,240 I've talked to him quite talked to him quite a lot. 249 00:16:31,800 --> 00:16:34,560 I sat next to him at an elixir confluence and he's he's he's 250 00:16:34,560 --> 00:16:41,480 hilarious. The The Bell Curve talks about 251 00:16:41,960 --> 00:16:46,320 you and the progression of being a developer between new and 252 00:16:46,320 --> 00:16:49,040 super senior. And they talk about how the 253 00:16:49,040 --> 00:16:53,640 people in the middle who are not quite senior but are very 254 00:16:53,640 --> 00:16:58,280 dangerous, will push back a lot against Ash because they're 255 00:16:58,280 --> 00:17:00,000 like, well, I can write this better, I don't need a 256 00:17:00,000 --> 00:17:03,360 framework. And for the most part they're 257 00:17:03,360 --> 00:17:09,520 wrong. But where, where Ash is wrong is 258 00:17:10,040 --> 00:17:15,000 there are so many of those sorts in the Elixir community that it 259 00:17:15,000 --> 00:17:18,240 becomes very, very hard to hire for it. 260 00:17:18,560 --> 00:17:21,400 I can see that. And so there's people who come 261 00:17:21,400 --> 00:17:23,880 in who are gung ho about like, I don't know Ash, but I'll learn 262 00:17:23,880 --> 00:17:27,960 it when 80% of stuff is written in Ash. 263 00:17:27,960 --> 00:17:30,160 Like I, we don't, we just don't have time for you to learn it. 264 00:17:31,560 --> 00:17:37,320 That's it's, it's just a pure business CTO level decision that 265 00:17:37,320 --> 00:17:41,320 you just have to contend with. And so although Ash has helped 266 00:17:41,320 --> 00:17:45,640 us a bunch with going fast, we have to pull it out because it 267 00:17:45,640 --> 00:17:48,360 hiring is just too hard. Interesting. 268 00:17:49,240 --> 00:17:54,880 I feel like I have like I've, I have a bunch of developers that 269 00:17:54,880 --> 00:17:58,960 like always ask me if if they can write my like recruitment 270 00:17:58,960 --> 00:18:01,520 projects in Ash, which is kind of funny. 271 00:18:02,040 --> 00:18:03,880 Like the people that love it, love it. 272 00:18:05,920 --> 00:18:08,320 People who love it, love it. That's a really good way of 273 00:18:08,320 --> 00:18:11,400 saying it. And I feel like there is a, if 274 00:18:11,400 --> 00:18:15,000 you were to draw a Venn diagram of people who love Ash and just 275 00:18:15,000 --> 00:18:18,440 passionate people in general, they're like right on top of one 276 00:18:18,440 --> 00:18:23,160 another because I mean, I, that was me at the beginning and I 277 00:18:23,160 --> 00:18:25,400 still love Ash. I just don't. 278 00:18:25,800 --> 00:18:27,600 Love the idea of that for. Hiring. 279 00:18:27,880 --> 00:18:30,240 Like when I first discovered Ash, I was like, oh, I love the 280 00:18:30,280 --> 00:18:35,200 idea of this, but I was running into like I got frustrated 281 00:18:35,200 --> 00:18:39,640 because I was moving too slow learning the DSL and I would, I 282 00:18:39,640 --> 00:18:42,320 ended up like ripping it out of a couple of my projects. 283 00:18:42,320 --> 00:18:46,800 But then I reinvested my time into learning it and I'm using 284 00:18:46,800 --> 00:18:52,040 Ash again across like most of my projects because it I do enjoy 285 00:18:52,040 --> 00:18:53,840 the speed. Like especially like being the 286 00:18:53,840 --> 00:18:56,400 lone developer on like my own projects. 287 00:18:56,640 --> 00:19:00,920 You can move so fast. Absolutely. 288 00:19:01,880 --> 00:19:04,560 If you know what you're doing, it's very quick. 289 00:19:04,720 --> 00:19:07,680 If you don't, it's an exercise in frustration. 290 00:19:08,040 --> 00:19:10,400 Yeah, yeah. No, for sure, it definitely is 291 00:19:10,400 --> 00:19:13,000 because you're like, I know how to do this without ash, but how 292 00:19:13,000 --> 00:19:16,520 do I do this the ash way? Exactly. 293 00:19:16,520 --> 00:19:20,240 And so I feel like when people start to try to convert stuff in 294 00:19:20,240 --> 00:19:22,920 their head, they'll be like, oh, I need this and this and this. 295 00:19:22,920 --> 00:19:28,920 And just because of how Ash is written, very DSL heavy errors 296 00:19:28,960 --> 00:19:31,360 are not as good as they could be. 297 00:19:31,480 --> 00:19:34,360 If you're not doing it correctly, well then you're 298 00:19:35,160 --> 00:19:38,040 going to hold the phone wrong and it's your fault and you're 299 00:19:38,040 --> 00:19:41,520 going to go slow. Yeah, Yep, yeah. 300 00:19:41,520 --> 00:19:44,880 I ran into that a lot. Like I was like, dude, I know 301 00:19:44,880 --> 00:19:49,240 how to just, you know, I, I'm trying to think I just went 302 00:19:49,240 --> 00:19:50,800 blank on what I was about to say. 303 00:19:52,440 --> 00:19:54,280 Oh yeah. I was like, I was building a 304 00:19:54,280 --> 00:19:57,360 multi tenancy app and I was like, I know how to manually set 305 00:19:57,360 --> 00:20:02,640 this up and I freaking couldn't get it to work right with Ash 306 00:20:02,680 --> 00:20:06,320 and it was so frustrating. And then it was literally a one 307 00:20:06,320 --> 00:20:09,800 line change and like Zach helped me and I was like, are you 308 00:20:09,800 --> 00:20:13,680 kidding me? Like so like once you unlock the 309 00:20:13,680 --> 00:20:16,840 power, it's amazing. Absolutely. 310 00:20:18,000 --> 00:20:24,360 I mean, Josh may talk about this next week, but Olympic has a Ash 311 00:20:24,400 --> 00:20:28,600 APS, ASH Premium support. It's a thing where you pay 312 00:20:28,600 --> 00:20:31,680 monthly to a limbic and they'll create a shared Slack channel 313 00:20:31,960 --> 00:20:35,200 and you can create tickets for the people who write ASH. 314 00:20:35,840 --> 00:20:38,720 We did that and it was incredibly helpful because 315 00:20:38,720 --> 00:20:41,360 there's sometimes where you're like, well, this is ambiguous. 316 00:20:41,720 --> 00:20:43,800 I could do this a couple different ways and ASH, I don't 317 00:20:43,800 --> 00:20:48,160 know what's best or one of the big ones that we found is if you 318 00:20:48,160 --> 00:20:56,840 have really weird esoteric queries where you you need to 319 00:20:57,000 --> 00:21:00,400 like a good example is you need to reach inside of a Jason B 320 00:21:00,400 --> 00:21:03,680 column to pull something out. You don't want the whole column 321 00:21:03,680 --> 00:21:06,960 because you know, it's gonna be like a mega of Jason. 322 00:21:07,920 --> 00:21:11,280 Ash by default trips over itself because it's like, oh, let me 323 00:21:11,280 --> 00:21:13,680 load the whole thing and then let me go select the thing that 324 00:21:13,680 --> 00:21:15,800 you want. It's like, no, you're I don't 325 00:21:15,800 --> 00:21:18,600 want 1000 megs of junk coming out of the database. 326 00:21:18,600 --> 00:21:21,760 I just want the one title of whatever I want out. 327 00:21:22,120 --> 00:21:25,160 So questions like that are really good for APS because 328 00:21:25,160 --> 00:21:27,400 although the community is helpful, their Discord is 329 00:21:27,400 --> 00:21:30,000 helpful. Going directly to the people who 330 00:21:30,000 --> 00:21:34,280 write it is a lot more helpful. And then if you run into like a 331 00:21:34,320 --> 00:21:37,760 bug, they'll confirm it's a bug. They'll fix it immediately, 332 00:21:37,800 --> 00:21:40,800 almost same day or not signing you up for anything Josh. 333 00:21:41,520 --> 00:21:45,200 But if there's some feature that is just basically missing, like 334 00:21:45,200 --> 00:21:48,560 I wish I could do this, it gets added to the road map and and 335 00:21:48,800 --> 00:21:52,080 merged in pretty quickly. So for those of you listening 336 00:21:52,120 --> 00:21:55,440 who really like Ash that want to take it to the next step, not a 337 00:21:55,440 --> 00:21:58,200 shell, but I would go with APS because it's really helpful. 338 00:21:58,320 --> 00:22:01,120 That's awesome. So how do you feel about ripping 339 00:22:01,120 --> 00:22:03,160 it out of your project? Are you sad about it? 340 00:22:04,640 --> 00:22:07,320 Mixed feelings. Totally mixed feelings. 341 00:22:08,000 --> 00:22:14,120 I have to change my hat. If I put head of engineering hat 342 00:22:14,120 --> 00:22:17,680 on, it's kind of a no brainer because of how hard it is to 343 00:22:17,680 --> 00:22:20,240 hire. If I put my developer hat on, 344 00:22:20,240 --> 00:22:23,280 yeah, I'm sad because it does a lot of things. 345 00:22:23,280 --> 00:22:27,560 It's declaratively magical, as Zach would say. 346 00:22:27,560 --> 00:22:31,880 It reminds me of Rails in a lot of ways, although there's 347 00:22:32,840 --> 00:22:35,760 circuit Breakers and peel off points everywhere. 348 00:22:35,760 --> 00:22:38,520 If you need to do something custom, it allows you to do it. 349 00:22:38,520 --> 00:22:42,480 So I, I, I'm, I'm sad, but I think it will be a good change 350 00:22:42,480 --> 00:22:46,480 for us because at the end of the day, again, putting the head of 351 00:22:46,480 --> 00:22:49,680 engineering cap back on every team is going to deal with 352 00:22:49,680 --> 00:22:53,080 churn. People move on with their lives, 353 00:22:53,080 --> 00:22:56,240 want something different. And when you lose developers 354 00:22:56,960 --> 00:23:01,280 that understand the code base, understand how Ash works, you 355 00:23:01,280 --> 00:23:03,120 got to hire someone and train them up. 356 00:23:03,120 --> 00:23:05,960 And it's like, it's just one more thing that you have to 357 00:23:05,960 --> 00:23:08,760 train these people on. And it's like, we, we just don't 358 00:23:08,760 --> 00:23:11,560 have the time. We're a tiny team, including our 359 00:23:11,560 --> 00:23:13,000 project manager. We're 11. 360 00:23:13,040 --> 00:23:16,760 That's that's not a lot of room to to handle churn every every 361 00:23:16,760 --> 00:23:20,360 year. Yeah, how, how do you guys feel 362 00:23:20,360 --> 00:23:25,040 about like Claude code and LLMS these days with ash? 363 00:23:25,040 --> 00:23:28,000 Are you are you using it or not really? 364 00:23:30,840 --> 00:23:35,520 Our team has gone through ebbs and flow with AI in general. 365 00:23:36,880 --> 00:23:42,240 Last year I feel like we leaned in really heavy and then it bit 366 00:23:42,240 --> 00:23:46,360 everyone in the butt hard because I don't think last year, 367 00:23:46,360 --> 00:23:49,760 like early last year, the models just weren't ready for that. 368 00:23:50,400 --> 00:23:54,800 And so we we pulled way back and said stop doing this. 369 00:23:55,040 --> 00:23:59,760 You can use it to augment what you're doing, but don't, please 370 00:23:59,760 --> 00:24:02,720 don't add a bunch of AI generated nonsense into our code 371 00:24:02,720 --> 00:24:06,440 base. But over the last few months, 372 00:24:06,680 --> 00:24:11,840 the models have been coming so good that I feel like it's just 373 00:24:11,840 --> 00:24:15,680 an accelerator. And as I explained to our team 374 00:24:16,360 --> 00:24:20,520 and now the general public who's listening, there should be like 375 00:24:20,680 --> 00:24:25,080 a one to one relationship almost with your planning phase as your 376 00:24:25,080 --> 00:24:30,000 coding phase. You need to spend an hour 377 00:24:30,000 --> 00:24:34,440 planning, going back and forth, making sure you know exactly 378 00:24:34,440 --> 00:24:37,400 what that model is about to write before it does. 379 00:24:38,800 --> 00:24:42,240 I don't really have a problem with one shotting things because 380 00:24:42,240 --> 00:24:44,920 of that. And I think when people read on 381 00:24:44,920 --> 00:24:51,640 X or and blog post about these developers that we look up to, 382 00:24:51,640 --> 00:24:58,600 like DHH or or Mitchell who's doing stuff with Ghosty or 383 00:24:58,600 --> 00:25:02,120 anything like that, how can you one shot stuff? 384 00:25:02,120 --> 00:25:03,920 I've never been able to get my model do that. 385 00:25:03,920 --> 00:25:08,480 It's because they're planning endlessly and making sure that 386 00:25:08,800 --> 00:25:11,640 every line of that plan, it's like, yeah, that's exactly how I 387 00:25:11,640 --> 00:25:14,440 would write it. Because once it matches what you 388 00:25:14,440 --> 00:25:17,520 would do, then it's going to do the thing that you expect. 389 00:25:17,960 --> 00:25:20,640 Otherwise you're going to end up with hilarity. 390 00:25:21,080 --> 00:25:26,480 The amount of times you get with a an unplanned model writing 391 00:25:26,480 --> 00:25:32,600 nested IFS conditionals inside of a with macro is like Oh my 392 00:25:32,600 --> 00:25:34,080 goodness, who would ever write that? 393 00:25:34,080 --> 00:25:36,160 Well, no one would, but AI will if you let it. 394 00:25:36,240 --> 00:25:40,600 I, I found it doesn't do it so much to me anymore. 395 00:25:40,600 --> 00:25:44,640 And I maybe it's because my planning phase and like my like 396 00:25:45,640 --> 00:25:51,000 my agent MD is just more organized, but I used to like, 397 00:25:51,480 --> 00:25:56,040 for whatever reason, Claude just loved conditionals and like 398 00:25:56,040 --> 00:25:59,480 nested if statements over pattern matching and I always 399 00:25:59,480 --> 00:26:01,880 had to yell at it. I was like always use pattern 400 00:26:01,880 --> 00:26:05,320 matching like over, you know, a messy block like this. 401 00:26:06,320 --> 00:26:13,040 And I, I feel like I pretty much can one shot like new features 402 00:26:13,040 --> 00:26:15,360 now. And I mean like I do spend a lot 403 00:26:15,360 --> 00:26:19,280 of time planning it and like I get a detailed markdown file of 404 00:26:19,280 --> 00:26:21,320 my feature that I want to implement. 405 00:26:21,880 --> 00:26:25,800 And then I don't know if you've have you heard of beads? 406 00:26:26,920 --> 00:26:30,040 Yes. So that whole project has kind 407 00:26:30,040 --> 00:26:31,880 of gone to shit and it's like kind of garbage. 408 00:26:31,880 --> 00:26:34,400 They've tried to make it like too fancy in my opinion. 409 00:26:34,840 --> 00:26:39,440 And actually one of my Co workers, Jason, he he built his 410 00:26:39,440 --> 00:26:43,720 own version of beads now and it's called bead bead beads 411 00:26:43,720 --> 00:26:47,640 work. And it uses like it uses git and 412 00:26:47,640 --> 00:26:52,960 stuff and will create its own like unattached branches. 413 00:26:52,960 --> 00:26:54,920 And you can do it's like it's awesome. 414 00:26:54,920 --> 00:26:57,040 And I use that for extending my context. 415 00:26:57,040 --> 00:27:00,440 And then I just make like a feature that I want to do. 416 00:27:00,720 --> 00:27:04,280 It creates it as an epic and then it breaks everything down 417 00:27:04,280 --> 00:27:06,120 into issues. And it's pretty awesome. 418 00:27:07,480 --> 00:27:10,840 I enjoy it and like, yeah, I one shot things all the time and it 419 00:27:10,840 --> 00:27:15,360 just keeps track of its progress with with issues. 420 00:27:15,360 --> 00:27:20,560 So it's pretty sweet. The planned phase is really 421 00:27:20,560 --> 00:27:23,400 important, I think for people that are new to AI who can't 422 00:27:23,400 --> 00:27:24,960 figure out how people are using it. 423 00:27:24,960 --> 00:27:29,400 That's that's the secret sauce they're missing is don't, don't 424 00:27:29,400 --> 00:27:31,920 assume the AI is going to do the right thing without planning 425 00:27:31,920 --> 00:27:34,720 because it won't, it will not write it like you would. 426 00:27:34,760 --> 00:27:38,920 And that's what we're paying you for your brain, not its brain. 427 00:27:39,320 --> 00:27:41,360 Right. Well, like, I feel like it's 428 00:27:41,360 --> 00:27:44,400 really important, like you need a language understanding, You 429 00:27:44,400 --> 00:27:47,920 need to understand how your project works so you can 430 00:27:47,920 --> 00:27:53,680 properly coach the. Like I look at Claude as like my 431 00:27:53,680 --> 00:27:56,440 little junior developer and knows how to write code right 432 00:27:56,440 --> 00:27:59,800 and knows how to write really good code as long as you guide 433 00:27:59,800 --> 00:28:03,120 it properly. And yeah. 434 00:28:05,000 --> 00:28:10,200 Absolutely. I feel like the inexperienced 435 00:28:11,000 --> 00:28:14,680 people with AI are actually the most senior people around 436 00:28:15,360 --> 00:28:17,680 because they're like, it's got to take my job. 437 00:28:17,880 --> 00:28:20,160 Well, you could think about it that way. 438 00:28:20,160 --> 00:28:24,240 Or you can think about how you've got the best rubber ducky 439 00:28:24,240 --> 00:28:28,000 in the world. I've had really angry, like 440 00:28:28,000 --> 00:28:31,640 probably really smart senior level engineers that are like, 441 00:28:32,320 --> 00:28:36,920 how dare you, like, you know, just chill AI and then say that 442 00:28:36,920 --> 00:28:39,600 everyone that's not using it is going to fall behind. 443 00:28:39,600 --> 00:28:43,600 Like they don't like opinionated like opinions like that, but 444 00:28:43,600 --> 00:28:47,440 like it's, it's a new tool and it's entered our lives and I 445 00:28:47,440 --> 00:28:50,440 don't think it's going anywhere. So like if you don't know how to 446 00:28:50,440 --> 00:28:53,680 use it, you are gonna fall behind 'cause you're gonna be 447 00:28:53,800 --> 00:28:58,360 relatively a lot slower at getting stuff out than like a 448 00:28:58,360 --> 00:29:02,360 mid level engineer who knows how to use AI joining your team. 449 00:29:03,360 --> 00:29:07,920 Of course, yeah. If you wanna dip your toes into 450 00:29:07,920 --> 00:29:12,320 it without having it write everything for you. 451 00:29:12,840 --> 00:29:17,040 The suggestion that I've given people is use it as a a really 452 00:29:17,040 --> 00:29:24,400 fancy To Do List and if anyone out there has executive function 453 00:29:24,400 --> 00:29:29,160 issues, because it seems like everyone around here has ADHD. 454 00:29:30,120 --> 00:29:34,120 It's really easy to queue up multiple things that you're 455 00:29:34,120 --> 00:29:39,120 working on all at the same time. If your project or your company 456 00:29:39,120 --> 00:29:41,560 has multiple services or multiple applications. 457 00:29:42,520 --> 00:29:45,320 What I typically do day-to-day is I've got multiple things 458 00:29:45,320 --> 00:29:48,960 running all at the same time and I'm not waiting for AIAI is 459 00:29:48,960 --> 00:29:51,400 waiting for me because by the time I've done something, I'll 460 00:29:51,400 --> 00:29:54,240 switch to a different project or a different branch or something 461 00:29:54,800 --> 00:30:00,440 and then go do something. It's, it's a way of always 462 00:30:00,440 --> 00:30:04,600 knowing what's next because context switching is really hard 463 00:30:04,600 --> 00:30:07,560 for all of us. Sometimes if you get, I feel 464 00:30:07,560 --> 00:30:10,600 like this is a difference between an IC and A and 465 00:30:10,600 --> 00:30:14,280 management, but I get bugged all day long, every day, pulled into 466 00:30:14,280 --> 00:30:15,840 meetings that I wasn't aware of, etcetera. 467 00:30:16,560 --> 00:30:21,440 And so I, I don't really have the time to deal with the 10 or 468 00:30:21,440 --> 00:30:23,600 15 minutes it takes me to get up to speed. 469 00:30:23,960 --> 00:30:28,160 But if there's a markdown file waiting for me saying, oh, hey, 470 00:30:28,160 --> 00:30:30,240 you did this, this and this, we were going to work on that. 471 00:30:30,240 --> 00:30:31,800 How about you do this next? Great. 472 00:30:32,240 --> 00:30:36,760 I it's like a built in. It's a built in somebody behind 473 00:30:36,760 --> 00:30:39,360 me, behind me coaching me on what to do next. 474 00:30:39,360 --> 00:30:44,920 And I that's a great way. If you don't want to like just 475 00:30:44,920 --> 00:30:47,000 go all in with it coding for you, great. 476 00:30:47,000 --> 00:30:48,520 Don't use it like that. Use it as a coach. 477 00:30:48,520 --> 00:30:50,880 Use it as someone who's like cheerleading you on. 478 00:30:51,120 --> 00:30:54,440 Yeah, rubber duck with it. Like make a plan like you don't 479 00:30:54,440 --> 00:30:59,560 have to let it write your code. I mean for like real redundant 480 00:30:59,560 --> 00:31:00,760 here. I'm going to put this on hold 481 00:31:00,760 --> 00:31:04,480 for a second and we'll be right back in just like 30 seconds. 482 00:31:05,800 --> 00:31:09,320 Hey, I'm Kimberly with Elixir mentor Jacob recently added 483 00:31:09,320 --> 00:31:12,680 recruitment services to help companies hire pre bedded Elixir 484 00:31:12,680 --> 00:31:15,640 devs. I help with company outreach, so 485 00:31:15,640 --> 00:31:18,960 if you're hiring, I may be reaching out, or you can connect 486 00:31:18,960 --> 00:31:21,160 with us anytime at elixirmentor.com. 487 00:31:22,960 --> 00:31:26,080 Coding Elixir with AI or building AI into your Elixir 488 00:31:26,080 --> 00:31:28,800 apps? The Elixir AI Collective Discord 489 00:31:28,800 --> 00:31:31,680 is your community for both. Link in the description below 490 00:31:37,280 --> 00:31:39,600 back on. All right. 491 00:31:40,640 --> 00:31:44,640 So to go back to like a management position, like I feel 492 00:31:44,640 --> 00:31:48,680 like once you're like not just an engineer at a company, you're 493 00:31:48,680 --> 00:31:51,160 not just focusing like feature at a time, right? 494 00:31:51,160 --> 00:31:52,560 You're bouncing all over the place. 495 00:31:52,560 --> 00:31:56,880 So it's probably nice to have the ability to just have 496 00:31:56,880 --> 00:32:01,080 markdown to do lists like, hey, this is where you left off and 497 00:32:01,440 --> 00:32:06,200 welcome back. Absolutely, especially. 498 00:32:06,200 --> 00:32:09,360 If you're working close with product and you know that 499 00:32:09,360 --> 00:32:11,720 there's a feature that's scheduled on the road map for 500 00:32:11,720 --> 00:32:15,680 six months out and you really need to start thinking about the 501 00:32:15,680 --> 00:32:18,600 changes that I'm making today, Are they helping or hurting what 502 00:32:18,600 --> 00:32:20,640 I know is coming six months down the road? 503 00:32:21,680 --> 00:32:26,320 That's really easy to just take that nugget out of your head and 504 00:32:26,320 --> 00:32:29,600 shove it into an agent, say remember this. 505 00:32:29,680 --> 00:32:32,640 And every time we talk about a feature that is anywhere related 506 00:32:32,640 --> 00:32:36,440 to the thing that's in six months, remind me or say, oh, 507 00:32:36,720 --> 00:32:39,280 you know, if we added this column here right now, we would 508 00:32:39,280 --> 00:32:40,880 be collecting this data for the future. 509 00:32:41,480 --> 00:32:49,360 So I feel like there's just so much low hanging fruit here that 510 00:32:49,360 --> 00:32:51,840 for people who are really against AI writing code, great, 511 00:32:51,840 --> 00:32:54,400 don't use it for that. But it is a great personal 512 00:32:54,400 --> 00:32:59,640 assistant. You know, our, our team at 513 00:32:59,640 --> 00:33:04,640 Collects, we, we lean pretty heavy into AI, but like into our 514 00:33:04,640 --> 00:33:06,800 workflow. One of my favorite things that 515 00:33:06,800 --> 00:33:12,640 we've done is we created a skill called Neckbeard and you, you 516 00:33:12,640 --> 00:33:16,240 know, you can call hey, neckbeard, can you just go go 517 00:33:16,240 --> 00:33:19,960 through the code and tell me exactly how like our 518 00:33:19,960 --> 00:33:23,760 authentication flow works And it will go through it. 519 00:33:23,760 --> 00:33:25,240 It's not allowed to write any code. 520 00:33:25,240 --> 00:33:27,800 All it does is give you a summary and then it will 521 00:33:27,800 --> 00:33:32,760 footnote actual modules that it's referencing at the bottom. 522 00:33:33,080 --> 00:33:36,600 And dude, it saved me so much. I could have just written a 523 00:33:36,600 --> 00:33:39,680 feature and I like forgot something, but someone's asking 524 00:33:39,680 --> 00:33:42,520 me a question about it. And I just, you know, call up 525 00:33:42,520 --> 00:33:47,320 our neckbeard and he, he'll, he'll run through it and give me 526 00:33:47,320 --> 00:33:49,960 a summary of how everything works. 527 00:33:49,960 --> 00:33:53,680 And it's like the most powerful thing that I use. 528 00:33:53,680 --> 00:33:56,920 And it's not even writing code. It's just evaluating code for 529 00:33:56,920 --> 00:34:02,280 me. I, I hear things like that and 530 00:34:02,280 --> 00:34:08,840 then I start thinking like the people who are so against AI 531 00:34:09,960 --> 00:34:14,000 could be using that stuff and it's just going to accelerate 532 00:34:14,000 --> 00:34:18,760 your workflow. And I'm not as big on AI as I 533 00:34:18,760 --> 00:34:22,880 would say most of our team is. I just know that there is a new 534 00:34:22,880 --> 00:34:24,800 tool. It's not going away. 535 00:34:25,080 --> 00:34:27,880 It's the worst that could possibly be today. 536 00:34:28,000 --> 00:34:31,159 It's only going to get better. So you might as well learn it 537 00:34:31,239 --> 00:34:34,760 because I think he will fall behind if you don't. 538 00:34:35,320 --> 00:34:37,760 Like a good example, like especially like in a role like 539 00:34:37,760 --> 00:34:40,520 yours where you're, you're jumping around, you get invited 540 00:34:40,520 --> 00:34:46,000 to meetings last minute. Like if like just for example, I 541 00:34:46,000 --> 00:34:48,600 just like implemented like an OTP feature, right? 542 00:34:49,199 --> 00:34:53,520 And I had a question come in on Discord as the front end team is 543 00:34:53,520 --> 00:34:56,120 implementing it. They're like, hey, what actions 544 00:34:56,880 --> 00:35:00,160 require OTP? Like what mutations are we going 545 00:35:00,160 --> 00:35:03,040 to hit? And like, off the top of my 546 00:35:03,040 --> 00:35:06,880 head, I think I can remember most of them, but with looking 547 00:35:06,880 --> 00:35:09,640 through and verifying in the code base could take me like an 548 00:35:09,640 --> 00:35:12,720 hour, right? But asking neckbeard, I get an 549 00:35:12,720 --> 00:35:17,360 answer in like a minute. And I have every single thing 550 00:35:17,360 --> 00:35:20,960 that I just added an OTP requirement into the middleware 551 00:35:20,960 --> 00:35:24,640 for. Like, dude, that's, that's huge. 552 00:35:27,200 --> 00:35:32,480 It's big, yeah. It's real big that I I want to 553 00:35:32,480 --> 00:35:36,480 borrow that idea or steal it. It's amazing for it, yeah. 554 00:35:37,760 --> 00:35:40,960 I'll, I am more than happy to share this the, the markdown 555 00:35:40,960 --> 00:35:44,480 with you and you can edit it for your, your own project. 556 00:35:45,360 --> 00:35:49,520 But yeah, it's, it's the it's my favorite feature of using 557 00:35:49,520 --> 00:35:52,720 Claude. Just explain this feature to me. 558 00:35:52,920 --> 00:35:56,600 How does it work? And then like well we have one 559 00:35:56,600 --> 00:36:01,280 that we have like a coverage 1 and I can like be on a current 560 00:36:01,280 --> 00:36:04,280 feature and be like hey, where is my test coverage low on this? 561 00:36:04,280 --> 00:36:09,800 How can I improve it? And it's only skill is to find 562 00:36:10,120 --> 00:36:14,400 low test coverage with. Testing. 563 00:36:14,560 --> 00:36:20,400 I feel like it's amazing because you can say, what edge cases am 564 00:36:20,400 --> 00:36:23,320 I not thinking about? And it's like, oh, here's like 565 00:36:23,320 --> 00:36:25,600 40 different things that you obviously didn't think about and 566 00:36:25,600 --> 00:36:27,960 you're like, oh, of course we would have run into these things 567 00:36:27,960 --> 00:36:30,000 in the future. Go write test for all of that 568 00:36:30,000 --> 00:36:33,240 stuff. Oh, hey, now we're doing TDD and 569 00:36:33,240 --> 00:36:35,440 stuff broke because we weren't thinking about that when we 570 00:36:35,440 --> 00:36:37,160 wrote the feature. Let's go fix that and be a 571 00:36:37,160 --> 00:36:39,200 little bit more defensive in our code. 572 00:36:39,360 --> 00:36:41,520 Great. You just helped yourself by 573 00:36:41,520 --> 00:36:45,640 asking a question real easy. Yeah, Yeah, It's so awesome. 574 00:36:45,640 --> 00:36:49,280 I used to hate writing tests, and now it's, like, enjoyable. 575 00:36:49,840 --> 00:36:54,200 Like, I feel like writing tests, though, it requires almost more 576 00:36:54,200 --> 00:36:58,000 attention to detail than the actual code itself now, because 577 00:36:58,000 --> 00:37:01,560 you don't want just garbage tests going in that just pass. 578 00:37:01,560 --> 00:37:04,720 Because I have, you know, every once in a while, I feel like in 579 00:37:04,720 --> 00:37:06,480 the last couple months, not so much. 580 00:37:06,480 --> 00:37:09,760 But when Claude just writes a test to pass and you're like, 581 00:37:11,400 --> 00:37:15,760 that's not going to fly. It will absolutely be lazy with 582 00:37:15,760 --> 00:37:20,840 testing or it will mock stuff where it's like just returning 583 00:37:20,840 --> 00:37:25,160 OK's no matter what happens and you're like bro, don't do that. 584 00:37:25,160 --> 00:37:27,480 So sometimes you have to catch it on itself. 585 00:37:27,560 --> 00:37:32,880 Yeah, 100%. I, I kind of want to go back to 586 00:37:34,720 --> 00:37:38,880 your transition from past systems to like your own, you 587 00:37:38,880 --> 00:37:41,200 know, your own bare metal server. 588 00:37:41,680 --> 00:37:45,080 What was that transition like? Did you have downtime? 589 00:37:45,080 --> 00:37:48,640 How did you handle it? How much time did you have from 590 00:37:48,920 --> 00:37:53,400 getting off of Gig Elixir to your own your own box? 591 00:37:54,000 --> 00:37:59,200 Oh boy, I had one week. Less than one week, actually. 592 00:37:59,200 --> 00:38:04,280 I had five working days. We ran into a problem. 593 00:38:04,960 --> 00:38:09,000 It was on a Monday. That was the third Monday in a 594 00:38:09,000 --> 00:38:12,400 row that we were having issues with the amount of traffic. 595 00:38:12,960 --> 00:38:16,240 It just so happens that Monday around lunchtime is when the 596 00:38:16,240 --> 00:38:20,960 state of New Mexico uses our product and we fell over. 597 00:38:21,360 --> 00:38:27,400 And when our upper management are fielding phone calls from 598 00:38:27,760 --> 00:38:30,680 the people in charge of education in New Mexico. 599 00:38:32,200 --> 00:38:35,200 We have to fix it. We can't, we can't deal with 600 00:38:35,200 --> 00:38:36,720 going down or anything like that. 601 00:38:36,960 --> 00:38:43,000 And so it was a week and that was a very, that was a very 602 00:38:43,000 --> 00:38:46,040 challenging week. It was myself and another 603 00:38:46,280 --> 00:38:52,720 developer who acted as as my second and we stayed up all 604 00:38:52,720 --> 00:38:59,000 hours and just moved everything. And at the time, I think we're 605 00:38:59,000 --> 00:39:05,840 at 3 or 400,000 users and 0 downtime. 606 00:39:06,600 --> 00:39:10,040 The only downtime is if you got caught potentially in a 607 00:39:10,040 --> 00:39:13,720 situation where DNS wasn't being authoritative. 608 00:39:13,880 --> 00:39:17,640 And we had pointed it at two things and one thing. 609 00:39:17,640 --> 00:39:21,360 And it's like maybe if you had like a really slow ISP that go 610 00:39:21,360 --> 00:39:23,240 really catch things, you would have downtime. 611 00:39:23,240 --> 00:39:25,880 But realistically we had 0 downtime. 612 00:39:26,400 --> 00:39:28,560 That's impressive. That was because that's one of 613 00:39:28,680 --> 00:39:30,000 those. Situations. 614 00:39:30,480 --> 00:39:35,680 It'd be a lot easier just to down things and you know, of 615 00:39:35,680 --> 00:39:37,040 course sold it back up, you know? 616 00:39:38,120 --> 00:39:40,480 And I would, you know, I was given permission to do that. 617 00:39:41,560 --> 00:39:45,480 And I wanted to take it as a personal challenge to say, how 618 00:39:45,480 --> 00:39:49,560 can I take services from two different past companies and put 619 00:39:49,560 --> 00:39:54,440 it all into Google without any downtime and just a hard cut 620 00:39:54,440 --> 00:39:57,960 over. And I don't know, maybe I like 621 00:39:57,960 --> 00:40:01,960 pain, but it was it was fun. I'm sure people are like, why 622 00:40:01,960 --> 00:40:04,400 would you want to like just 100 hour we can stay. 623 00:40:04,960 --> 00:40:08,120 I I had a lot of fun. That's good. 624 00:40:10,360 --> 00:40:12,360 I don't. Think I don't think anyone would 625 00:40:12,360 --> 00:40:16,680 do what we do if you weren't having some sort of fun. 626 00:40:17,280 --> 00:40:18,320 Right. Well, I think. 627 00:40:18,320 --> 00:40:21,520 Being a developer in general is incredibly hard and challenging 628 00:40:21,520 --> 00:40:23,600 and frustrating. If you're not a team of one, if 629 00:40:23,600 --> 00:40:25,680 you don't have your own product, if you're inside of an 630 00:40:25,680 --> 00:40:30,000 organization, everything is your fault, no matter if it's good or 631 00:40:30,000 --> 00:40:33,520 bad. Tensions run high, things are 632 00:40:33,520 --> 00:40:37,680 needed last minute or three weeks ago, and it's it's hard. 633 00:40:38,160 --> 00:40:41,960 But if you weren't having fun, and if you didn't enjoy that 634 00:40:41,960 --> 00:40:44,600 challenge, you'd go do something else. 635 00:40:44,640 --> 00:40:48,520 What I think is like a good learning experience, like before 636 00:40:48,520 --> 00:40:51,760 I worked at collects, I didn't really experience like a large 637 00:40:51,760 --> 00:40:55,680 user base, you know, tons of queries per second, things like 638 00:40:55,680 --> 00:40:59,840 that. And you, you kind of like when 639 00:40:59,840 --> 00:41:02,360 you're in like a start up mode, you only have a couple 1000 640 00:41:02,360 --> 00:41:05,160 users, traffic's not high. Like, you know what I mean? 641 00:41:05,520 --> 00:41:10,360 And you always think like, I'm gonna get to a point where all 642 00:41:10,360 --> 00:41:14,360 these growing pains are gone and you forget that there's like a 643 00:41:14,360 --> 00:41:18,080 whole set of new problems you have once you start having to 644 00:41:18,080 --> 00:41:22,280 scale. And I feel like like level all 645 00:41:22,320 --> 00:41:26,920 like are you, can we talk about the most recent like growing 646 00:41:26,920 --> 00:41:30,360 pain that you've had that you're working on right now? 647 00:41:30,680 --> 00:41:33,240 And it's another like good experience of like kind of 648 00:41:33,240 --> 00:41:36,440 vendor lock in problem, right? You want to explain the issue? 649 00:41:37,640 --> 00:41:43,120 Yeah, absolutely. On Monday, another New Mexico 650 00:41:43,120 --> 00:41:48,520 onboarding and we we fixed our systems last year. 651 00:41:48,520 --> 00:41:52,800 So we're extremely fast. But that unlocked a new problem. 652 00:41:52,840 --> 00:41:59,160 A new challenger has arrived and no, we got rate limited at auth 653 00:41:59,160 --> 00:42:04,280 0. So when that happens, your users 654 00:42:04,280 --> 00:42:07,560 don't see your login screen, they see AI don't I don't 655 00:42:07,560 --> 00:42:09,280 remember. It was so cheeky and cheesy. 656 00:42:09,280 --> 00:42:11,880 It was like, oh, this is a popular application wait a 657 00:42:11,880 --> 00:42:13,440 minute till you can get in or something. 658 00:42:13,440 --> 00:42:16,920 And yeah, we immediately heard from educators like, hey, we're 659 00:42:16,920 --> 00:42:20,600 we're reaching this error screen and it's like, Oh, no, like 660 00:42:20,920 --> 00:42:23,120 that's something completely out of our control. 661 00:42:23,840 --> 00:42:29,760 And that was Monday, so. That's a great way to start. 662 00:42:29,800 --> 00:42:32,520 The week, what's what's, what's the plan? 663 00:42:32,520 --> 00:42:35,920 What do we do? We immediately spoke to customer 664 00:42:35,920 --> 00:42:40,960 support and I mean, we pay all sorts of money to all zero. 665 00:42:41,120 --> 00:42:44,760 Anyone else there knows that they are extremely expensive and 666 00:42:44,760 --> 00:42:47,480 we pay for enterprise and enterprise support. 667 00:42:47,480 --> 00:42:50,400 And luckily we were able to get some answers pretty quickly, but 668 00:42:50,400 --> 00:42:53,240 the answers were ambiguous at best. 669 00:42:54,160 --> 00:42:57,720 Well, here's why it happened. OK, how are we going to fix it? 670 00:42:57,720 --> 00:42:58,960 Oh, well. But here's how. 671 00:42:58,960 --> 00:43:01,880 This is why it happened. Great. 672 00:43:02,440 --> 00:43:04,040 How are we going to fix it? Are you going to raise our 673 00:43:04,040 --> 00:43:07,360 rates, the limits? What are we doing here? 674 00:43:07,360 --> 00:43:10,120 And we have yet to get a good answer out of them. 675 00:43:11,360 --> 00:43:17,880 The biggest problem that we have is with Albuquerque Public 676 00:43:17,880 --> 00:43:24,360 School system owns public IP space and all of their students 677 00:43:24,360 --> 00:43:31,120 come out of like 4 IP addresses all at once, Dang, across the 678 00:43:31,120 --> 00:43:34,560 entire district. And so it's a lot of users 679 00:43:34,560 --> 00:43:38,680 coming out of a single IP. That's, that's an attack 680 00:43:39,320 --> 00:43:42,640 according to most systems. When you have 10s of thousands 681 00:43:42,640 --> 00:43:47,000 of people just crushing on a single IP address, like, yeah, I 682 00:43:47,120 --> 00:43:49,000 I would probably turn you off too. 683 00:43:49,880 --> 00:43:53,240 But that's just the way that our traffic is very bursty. 684 00:43:53,640 --> 00:43:58,320 And so, I mean, this is a real live issue that as soon as I'm 685 00:43:58,320 --> 00:44:00,160 off this call, I'm going to go back and fix. 686 00:44:00,160 --> 00:44:04,640 But we decided to completely RIP out all zero and put in a 687 00:44:04,640 --> 00:44:10,040 different all system in a matter of days because it's Friday. 688 00:44:10,040 --> 00:44:14,760 It's got to work by the end of the day and to make sure that we 689 00:44:14,760 --> 00:44:19,200 can be ready for Monday and we don't have 3 or 400,000 users 690 00:44:19,200 --> 00:44:20,720 anymore. We've got over 1,000,000 in our 691 00:44:20,720 --> 00:44:22,160 database. It's a lot to do. 692 00:44:23,200 --> 00:44:28,320 That is how do you so you're switching to a new service and 693 00:44:28,520 --> 00:44:33,440 have they don't rate limit or they are willing to make a 694 00:44:33,440 --> 00:44:36,520 custom solution for your circumstance. 695 00:44:36,520 --> 00:44:40,720 So how's that working? Yep, we are moving to Fusion 696 00:44:40,720 --> 00:44:43,640 Auth and I've used Fusion Auth in the past. 697 00:44:44,200 --> 00:44:46,640 The nice thing about their product is you can self host it 698 00:44:48,040 --> 00:44:51,080 OK so you can give it all the hardware, You can give it all 699 00:44:51,080 --> 00:44:55,440 the hardware you want. So back to, you know, no vendor 700 00:44:55,440 --> 00:44:59,640 lock in your yourself in your solution exactly. 701 00:44:59,640 --> 00:45:01,920 We own the database, we own the software. 702 00:45:02,120 --> 00:45:06,760 It's just there, we control what how much that's load balanced. 703 00:45:06,800 --> 00:45:10,320 It's right now being deployed in the same mechanism as our Elixir 704 00:45:10,640 --> 00:45:12,240 server. So there's a template and a 705 00:45:12,240 --> 00:45:16,400 managed incense group. It also has auto scaling turned 706 00:45:16,400 --> 00:45:18,960 on. So I'm just not worried at all 707 00:45:19,160 --> 00:45:22,440 about of of thundering herd problem because we'll auto scale 708 00:45:22,440 --> 00:45:27,720 on that. That is a big part of vendor 709 00:45:27,720 --> 00:45:29,800 lock in. And so some people would say, 710 00:45:29,800 --> 00:45:33,000 but didn't you just move from one vendor to another? 711 00:45:33,640 --> 00:45:38,280 Yeah, sure. But for the price, could we 712 00:45:38,400 --> 00:45:41,440 handle what we need to handle? Not at all. 713 00:45:41,680 --> 00:45:44,160 We we aren't just doing username and password here. 714 00:45:44,440 --> 00:45:46,960 We have lots of people with username and password or e-mail 715 00:45:46,960 --> 00:45:50,280 and password. We also have a ton of people 716 00:45:50,280 --> 00:45:56,160 coming in through OIDC, SAML. We don't use it today, but skim 717 00:45:56,160 --> 00:46:01,040 might be a thing if you, we say students and educators, 718 00:46:01,040 --> 00:46:04,400 districts, states, but really, if you squint, they're 719 00:46:04,400 --> 00:46:06,600 enterprises. They use enterprise software. 720 00:46:06,600 --> 00:46:09,280 They expect an enterprise connection, they expect 721 00:46:09,440 --> 00:46:11,800 directory sync. There's a lot of things that we 722 00:46:11,800 --> 00:46:13,920 do that are actually enterprise everything. 723 00:46:14,240 --> 00:46:17,360 And although, yeah, we could write it ourselves, it's just 724 00:46:17,360 --> 00:46:20,440 not worth it because we have so many other really cool features 725 00:46:20,440 --> 00:46:22,360 that help students out that we want to get to. 726 00:46:23,080 --> 00:46:29,160 Yeah, yeah, if I feel like, like that problem alone is just 727 00:46:29,160 --> 00:46:33,240 something you'd wouldn't even worry about and then all of a 728 00:46:33,240 --> 00:46:36,120 sudden it's just knocking on your door and it's a huge 729 00:46:36,120 --> 00:46:39,400 emergency because no one can get in your system, of course, from 730 00:46:39,400 --> 00:46:42,600 that IP at least. I never expected it. 731 00:46:42,600 --> 00:46:47,520 And I think on on one of the calls that we were on with on 732 00:46:47,600 --> 00:46:51,880 zero, my CEO said something and it's really true. 733 00:46:52,000 --> 00:46:54,080 Well, we should have known the rate limit. 734 00:46:54,800 --> 00:46:59,040 And it's like, oh, you're absolutely right, that's my 735 00:46:59,040 --> 00:47:03,640 fault. We should have known and I did 736 00:47:03,640 --> 00:47:07,760 it so. And it's really low. 737 00:47:07,760 --> 00:47:13,920 By the way, default rate limit is 500 per minute per IP 738 00:47:13,920 --> 00:47:16,760 address. Which is it's not going. 739 00:47:16,760 --> 00:47:18,800 To work at all for us. Right. 740 00:47:19,000 --> 00:47:21,800 Well, how many students are in a school district? 741 00:47:22,040 --> 00:47:27,040 Couple thousand at least. Albuquerque has 25,000 there. 742 00:47:27,080 --> 00:47:30,760 You go for for for 888 through 12th or something like that. 743 00:47:30,760 --> 00:47:32,480 They've got a ton. There's a ton of people in that 744 00:47:32,480 --> 00:47:33,800 district. Wow. 745 00:47:34,040 --> 00:47:38,280 Yeah, 500 won't do it. Yeah, No, we were. 746 00:47:38,560 --> 00:47:44,760 They told us that we had 60,000 requests in a 20 minute window. 747 00:47:45,440 --> 00:47:48,320 So some of those were retries because they couldn't get in. 748 00:47:48,320 --> 00:47:51,480 But it's like, yeah, we had 10s of thousands of people trying to 749 00:47:51,480 --> 00:47:54,800 get in and they were essentially told no. 750 00:47:55,160 --> 00:47:59,400 And it, I can't say it wasn't our fault because of what my CEO 751 00:47:59,400 --> 00:48:00,240 said. It makes total sense. 752 00:48:00,280 --> 00:48:02,400 I, I should have known. And I feel, I feel really bad 753 00:48:02,400 --> 00:48:04,320 about that because that's something that I would have 754 00:48:04,320 --> 00:48:08,480 highlighted when we moved to Google last year. 755 00:48:08,960 --> 00:48:11,880 I haven't seen for months. Like, I'm really concerned about 756 00:48:11,880 --> 00:48:15,440 the performance of gig Elixir. I'm really concerned about the 757 00:48:15,440 --> 00:48:18,080 performance of fly. Can we please move to something 758 00:48:18,080 --> 00:48:20,280 that is a little bit more scalable for usage? 759 00:48:21,000 --> 00:48:24,760 And essentially I was, I was told not by the CEO as someone 760 00:48:24,760 --> 00:48:28,480 else, it's not a problem until it's a problem there. 761 00:48:28,520 --> 00:48:30,160 That's a conservative way of looking at it. 762 00:48:30,160 --> 00:48:31,440 It's not a problem until it's a problem. 763 00:48:31,720 --> 00:48:34,560 But when it's a problem, it's a really bad problem. 764 00:48:35,880 --> 00:48:38,640 When you're dealing with unavailability, that's the worst 765 00:48:38,640 --> 00:48:42,720 type of problem you can have. So I felt like I was seeing the 766 00:48:42,720 --> 00:48:46,360 future with our hardware, but I never once thought we're going 767 00:48:46,360 --> 00:48:49,280 to get rate limited by our auth provider, never thought about 768 00:48:49,280 --> 00:48:51,360 it. I mean, I think this is just a 769 00:48:51,360 --> 00:48:55,280 good example for everyone. Like especially like junior 770 00:48:55,280 --> 00:48:59,760 level people that are learning, like you're never going to see 771 00:48:59,760 --> 00:49:02,960 it all, know it all until you experience it. 772 00:49:02,960 --> 00:49:07,880 Like a lot of times you learn your best lessons through a 773 00:49:07,880 --> 00:49:10,040 really hard, hard way of learning. 774 00:49:10,040 --> 00:49:13,360 You know, like, this is a huge pain point for you, right? 775 00:49:13,360 --> 00:49:17,480 Your whole week, whatever you had planned for this week got 776 00:49:17,480 --> 00:49:20,480 thrown out, right? And this is your main problem 777 00:49:20,480 --> 00:49:22,600 now and you had to solve it immediately. 778 00:49:23,960 --> 00:49:27,240 Absolutely. And it being in the management 779 00:49:27,240 --> 00:49:31,320 side of things, it's not just the code level, it's getting on 780 00:49:31,320 --> 00:49:34,920 sales calls, it's walking the contract through, making sure 781 00:49:34,920 --> 00:49:38,160 that we're OK with terms, sending our red line version 782 00:49:38,160 --> 00:49:40,240 back. Like there's so many steps to 783 00:49:40,280 --> 00:49:44,560 things that if you're just an IC, it just happens magically. 784 00:49:44,560 --> 00:49:48,520 And it's like, there's a lot of stuff that we do to just get 785 00:49:48,520 --> 00:49:52,000 prepared to sign a contract, then we sign it and we still 786 00:49:52,000 --> 00:49:57,200 have to do all of the work. So there's it's a good point of 787 00:49:57,200 --> 00:49:59,400 like, sometimes you don't know until you don't know. 788 00:50:01,040 --> 00:50:03,200 I'm sure a lot of people out there would say like, well, 789 00:50:03,200 --> 00:50:04,880 don't beat yourself up about it. You didn't know. 790 00:50:04,880 --> 00:50:08,760 It's like, yeah, you're right, but I wish I would have known. 791 00:50:08,840 --> 00:50:11,920 And that's just me trying to be as best as I can be at what I 792 00:50:11,920 --> 00:50:16,600 do, knowing where the pitfalls or the things that could go 793 00:50:16,600 --> 00:50:22,760 wrong exist. But yeah, fusion auth it fusion 794 00:50:22,760 --> 00:50:24,480 auth. The integration so far has been 795 00:50:24,480 --> 00:50:27,520 going pretty smooth. I'm a little tired tonight. 796 00:50:27,520 --> 00:50:29,560 I was up till Oh, you look great. 797 00:50:29,560 --> 00:50:32,840 I can't. Even tell thank you. 798 00:50:33,600 --> 00:50:40,760 The I fake it really well. I think the the the the biggest 799 00:50:40,760 --> 00:50:43,720 challenge of just between the two, if anyone cares about the 800 00:50:43,720 --> 00:50:48,360 tech is all zero makes some really interesting choices where 801 00:50:48,840 --> 00:50:52,360 they're erring on the side of ease of integration. 802 00:50:52,680 --> 00:50:57,240 So they do things like their view SDK as storing tokens and 803 00:50:57,240 --> 00:51:00,280 local storage. It's not the most secure thing 804 00:51:01,080 --> 00:51:03,200 there there there are some risks involved. 805 00:51:03,200 --> 00:51:04,760 They're low, but there are risks there. 806 00:51:05,400 --> 00:51:08,840 Fusion auth is like, well, why would you even take that risk? 807 00:51:09,200 --> 00:51:12,840 Don't do that. Store them as session HTTP only 808 00:51:12,840 --> 00:51:14,160 cookies. Then you don't have to worry 809 00:51:14,160 --> 00:51:16,160 about JavaScript having access to your token. 810 00:51:16,840 --> 00:51:21,280 OK, but then that necessitates a rewrite of how our API off flow 811 00:51:21,280 --> 00:51:25,560 works in multiple stages. And it's it's one of those like 812 00:51:25,560 --> 00:51:29,480 it's not it's not as easy as like, oh, we'll just turn this 813 00:51:29,480 --> 00:51:31,600 one off and this one on and away you go. 814 00:51:31,600 --> 00:51:34,520 There's there's some hard things that you have to contend with. 815 00:51:35,080 --> 00:51:39,360 And then later today I will we already we already have the 816 00:51:39,360 --> 00:51:42,360 export of users out of all serial, but when we import them 817 00:51:42,440 --> 00:51:45,120 got it. Make sure that the B crypt hash 818 00:51:45,120 --> 00:51:47,240 goes incorrectly, gets converted correctly. 819 00:51:47,240 --> 00:51:51,040 There's a lot of, there's a lot of, there's a lot of minutia 820 00:51:51,040 --> 00:51:53,400 that you got to make sure it goes correctly in order for 821 00:51:53,800 --> 00:51:57,640 again, I, I want to hit deploy and all of our services roughly 822 00:51:57,640 --> 00:51:59,800 at the same time and I don't want anyone to experience 823 00:51:59,800 --> 00:52:01,200 downtime. That is a personal goal. 824 00:52:02,280 --> 00:52:05,640 How do you do you have a staging environment set up where you can 825 00:52:05,960 --> 00:52:08,120 at least test this? Yeah, that's good. 826 00:52:08,280 --> 00:52:10,880 Yep, we will. We will test it very thoroughly 827 00:52:10,880 --> 00:52:13,920 before it goes to. Production, I'm always like, I 828 00:52:13,920 --> 00:52:17,880 feel like it's becoming more and more common where people push 829 00:52:17,880 --> 00:52:21,520 straight to prod these days. And I just think like, that's, 830 00:52:22,080 --> 00:52:29,880 that's brave. It it's, it's brave, but I also 831 00:52:29,880 --> 00:52:33,600 feel like I'm dating myself that that's, that's what we used to 832 00:52:33,600 --> 00:52:38,800 do, right? You would live edit over an FTP 833 00:52:39,320 --> 00:52:41,040 connection. I mean, I'm not gonna lie, I 834 00:52:41,040 --> 00:52:46,360 have, I have changed code on the fly in an IAX instance, of 835 00:52:47,520 --> 00:52:50,560 course. But, but you know, I don't like 836 00:52:50,560 --> 00:52:55,680 to do that stuff. It's not ideal, but I like to be 837 00:52:55,680 --> 00:53:01,160 able to. Of course, yeah, we our code, I 838 00:53:01,200 --> 00:53:05,800 mean we deal with education and students and our, our PII is 839 00:53:06,800 --> 00:53:12,000 very valuable because we are bound by Copa and FERPA sock 840 00:53:12,000 --> 00:53:17,400 two, we have a lot of regulatory bodies who oversee what we do 841 00:53:17,400 --> 00:53:20,800 and don't do. So we can't be as cavalier as 842 00:53:21,240 --> 00:53:24,200 other people. Everything is security around 843 00:53:24,200 --> 00:53:26,400 here and we take it really seriously. 844 00:53:27,120 --> 00:53:30,600 But there are definitely times where it's like, man, I wish I 845 00:53:30,600 --> 00:53:34,080 could just pop into a a server and just edit this file because 846 00:53:34,080 --> 00:53:37,760 like now I gotta wait for the whole CICD thing to run and all 847 00:53:37,760 --> 00:53:40,200 that. That was, that's a good point. 848 00:53:40,280 --> 00:53:42,040 Actually. That was one of the biggest 849 00:53:42,040 --> 00:53:45,000 gains from moving from the multiple passes to what we're 850 00:53:45,000 --> 00:53:50,520 doing ourselves is we used to have, because of the way that 851 00:53:50,920 --> 00:53:54,320 Gigalixer used to run where it didn't use releases. 852 00:53:54,320 --> 00:53:59,120 It's different now, but when it was in its build pack era, it 853 00:53:59,120 --> 00:54:03,760 would take close to 27 minutes or so to from merge to 854 00:54:03,760 --> 00:54:07,440 deployment. And that is if you have a hot 855 00:54:07,440 --> 00:54:10,720 fix that needs to go out. That is an excruciatingly long 856 00:54:10,720 --> 00:54:11,960 time. Yes, it is. 857 00:54:11,960 --> 00:54:16,600 We're down to 5 or 7 minutes now and when we go to bare metal I'd 858 00:54:16,600 --> 00:54:18,040 like to get that down to 30 seconds. 859 00:54:18,320 --> 00:54:23,400 Does that include like your CI like running tests and 860 00:54:23,400 --> 00:54:26,840 everything 5 to 7 minutes? Oh, OK, no. 861 00:54:26,880 --> 00:54:31,680 Well, so CI will run on the PR and once it's good when you 862 00:54:31,680 --> 00:54:34,680 merge, it doesn't rerun the thing that it just ran. 863 00:54:34,920 --> 00:54:37,200 We don't we don't do that. I feel like that's a waste. 864 00:54:37,560 --> 00:54:41,080 So when you merge. Into main I actually I should 865 00:54:41,080 --> 00:54:45,280 change that in my processes. So many people do it because I 866 00:54:45,280 --> 00:54:47,800 feel like someone wrote a blog post, you know, 10 years ago 867 00:54:47,800 --> 00:54:49,640 that that's what you're supposed to do and everyone does it. 868 00:54:49,640 --> 00:54:51,600 But it's like, but you just ran the stuff. 869 00:54:51,600 --> 00:54:53,320 It's. Really annoying the way to the 870 00:54:53,320 --> 00:54:56,040 yeah. Of course, yeah, no, get rid of 871 00:54:56,040 --> 00:54:57,560 it. The only thing that it should do 872 00:54:57,560 --> 00:54:59,880 is if you're. If you're on ACD system, it's 873 00:54:59,880 --> 00:55:02,360 going to deploy to staging or production of any on what branch 874 00:55:02,360 --> 00:55:08,160 you've just merged into the. Way it goes, Yep, I like, I like 875 00:55:08,160 --> 00:55:09,560 that. I'm trying to think how long my 876 00:55:09,560 --> 00:55:13,880 tanks to deploy right now, my kill switch product, at least I 877 00:55:13,880 --> 00:55:17,200 bet it's about. It's probably like 10 minutes, 878 00:55:17,320 --> 00:55:19,400 honestly. Yeah, yeah. 879 00:55:20,880 --> 00:55:22,840 I want to get it down to 30 seconds. 880 00:55:23,040 --> 00:55:28,760 Our view applications take about 5 to 8 seconds to deploy, which 881 00:55:28,760 --> 00:55:30,880 is great because it's just moving some files. 882 00:55:31,440 --> 00:55:35,960 But I want to get I want to get our elixir down to that we are 883 00:55:36,320 --> 00:55:40,960 I've I haven't decided if level all is going to open source this 884 00:55:40,960 --> 00:55:44,080 or I am personally, but I'm working on a a deployment 885 00:55:44,080 --> 00:55:46,680 mechanism for bare metal or bare VM. 886 00:55:47,360 --> 00:55:50,080 Have you written any like blog posts about it or anything? 887 00:55:51,040 --> 00:55:53,640 I haven't. I am so terrible with like 888 00:55:54,120 --> 00:55:57,320 personal brand stuff. I'm just I'm bad at it. 889 00:55:57,480 --> 00:56:00,800 I should start a blog and write about it, but the software 890 00:56:00,800 --> 00:56:07,280 essentially is a, it's a, it's a server agent set up where the 891 00:56:07,280 --> 00:56:10,040 agents are running on your deployment targets. 892 00:56:10,040 --> 00:56:14,400 So kind of think about like Ansible or Kamal or something. 893 00:56:14,400 --> 00:56:18,480 But there's instead of having just a a singular server that's 894 00:56:18,480 --> 00:56:23,480 just doing things, the server agent are talking to each other 895 00:56:23,480 --> 00:56:25,720 over Erlang. It moves files that are 896 00:56:25,720 --> 00:56:30,720 deployable over Erlang and the agent is declarative so the 897 00:56:30,720 --> 00:56:32,920 server will say this is what you need to do. 898 00:56:33,080 --> 00:56:35,760 The agent is like oh OK I'm going to change from this 899 00:56:35,760 --> 00:56:38,040 version to this version. I'm stuck there. 900 00:56:38,400 --> 00:56:41,600 The agent will do its own stuff to deploy so that if the 901 00:56:41,600 --> 00:56:44,800 connection is severed between server and agent, it doesn't 902 00:56:44,800 --> 00:56:47,360 matter. It can continue to run and 903 00:56:47,960 --> 00:56:54,600 typically people would reach for Go for this, but I want to do it 904 00:56:54,600 --> 00:56:58,120 in Elixir because I want to take advantage of Erlang talking to 905 00:56:58,120 --> 00:57:00,840 itself. So there's no APIs to write, 906 00:57:01,040 --> 00:57:03,200 you're just remote calling functions. 907 00:57:03,520 --> 00:57:06,720 The files will happen over Erlang, very similar to how 908 00:57:06,720 --> 00:57:12,760 flame works on fly. Yeah, I that's that's when we 909 00:57:13,160 --> 00:57:16,880 when CICD will run the CI portion will run all the tests 910 00:57:16,880 --> 00:57:20,520 and create the the packaged build. 911 00:57:21,040 --> 00:57:24,240 Once it hits CD, that server will take over. 912 00:57:24,280 --> 00:57:27,320 Tell the agent, agents like, let me download this, spin it up, 913 00:57:27,600 --> 00:57:29,480 change the system D file around it. 914 00:57:29,480 --> 00:57:31,840 Away you go. It should be about. 30 seconds. 915 00:57:32,280 --> 00:57:33,640 That's awesome. Very cool. 916 00:57:34,080 --> 00:57:37,160 Yeah, I want this. You got to get this this open 917 00:57:37,160 --> 00:57:40,560 source project out. Absolutely I will. 918 00:57:41,080 --> 00:57:42,440 I'll let you know when it's ready. 919 00:57:42,840 --> 00:57:46,240 It's like that'll be my push to get off docker. 920 00:57:48,440 --> 00:57:53,520 I, I think once I and I thought really long and hard about just 921 00:57:53,520 --> 00:57:57,000 doing it for ourselves because there's definitely some 922 00:57:57,560 --> 00:58:00,480 particulars that I think we will make. 923 00:58:01,320 --> 00:58:04,000 And so I actually had a discussion a really, it was like 924 00:58:04,000 --> 00:58:07,680 a multi day discussion with AI on like, what are the merits of 925 00:58:07,680 --> 00:58:10,000 doing this internally versus open source? 926 00:58:10,440 --> 00:58:15,720 And a part of that is like, well, do you feel it's asking me 927 00:58:15,720 --> 00:58:17,800 this? Do you feel like you've consumed 928 00:58:17,800 --> 00:58:19,600 more than produced for open source? 929 00:58:20,800 --> 00:58:24,640 Yeah, that's my entire career. I've edited a couple of things 930 00:58:24,640 --> 00:58:26,960 here and there for open source, but I feel like they're like 931 00:58:26,960 --> 00:58:31,640 small peanuts compared to how much I've actually benefited 932 00:58:31,880 --> 00:58:33,600 from open source. It's ridiculous. 933 00:58:33,880 --> 00:58:36,600 It's what I've done is a rounding error. 934 00:58:36,640 --> 00:58:38,480 Yeah, my only open source. Now. 935 00:58:38,920 --> 00:58:41,320 Tutorials. Exactly. 936 00:58:41,320 --> 00:58:46,200 So I want to, I want to give back and so I hope the AI helped 937 00:58:46,200 --> 00:58:51,840 me think about like where to be opinionated and where to allow 938 00:58:51,840 --> 00:58:56,960 for customization hooks or plug insurance, but still have you 939 00:58:56,960 --> 00:58:59,880 know what, if this doesn't work for you, that's fine. 940 00:58:59,880 --> 00:59:01,160 It's not going to be for everyone. 941 00:59:01,160 --> 00:59:03,280 And that's basically what Kamal is. 942 00:59:04,280 --> 00:59:07,480 You, you want a Ruby based Docker deployment system, use 943 00:59:07,480 --> 00:59:08,880 it. It's amazing, it's great, works. 944 00:59:08,880 --> 00:59:10,960 Great. What do you, how do you handle 945 00:59:10,960 --> 00:59:14,360 like your database when you're just, you know, using mixed 946 00:59:14,360 --> 00:59:16,680 release? Do you just have have like 947 00:59:16,800 --> 00:59:21,400 Postgres installed on the server and it's just there or how do 948 00:59:21,400 --> 00:59:26,160 you how do you handle that? Are you asking this like what 949 00:59:26,160 --> 00:59:30,000 happens when you go to? 'Cause like I'm thinking about 950 00:59:30,000 --> 00:59:34,600 like I spin up a Postgres instant in a Docker container, 951 00:59:34,600 --> 00:59:36,880 right? Yeah. 952 00:59:36,880 --> 00:59:40,400 So how are how do you handle that on like a bare metal 953 00:59:40,960 --> 00:59:50,040 deployment? Run it yourself and Postgres. 954 00:59:50,760 --> 00:59:53,480 Postgres is one of those things that people think it's big and 955 00:59:53,480 --> 00:59:55,680 scary and you're going to lose all of your data if you run it 956 00:59:55,680 --> 01:00:00,040 yourself. No, no, no you won't. 957 01:00:02,200 --> 01:00:05,400 It can happen. Disclaimer, data loss is a 958 01:00:05,400 --> 01:00:09,840 thing, but no, you won't. Postgres is unbelievably stable. 959 01:00:10,080 --> 01:00:12,600 Modern hardware is unbelievably stable. 960 01:00:13,120 --> 01:00:16,480 As long as you're backing up, I just really wouldn't worry about 961 01:00:16,480 --> 01:00:18,280 it. I feel like most people think 962 01:00:18,280 --> 01:00:23,360 their application or their project is it's got to have 900 963 01:00:23,360 --> 01:00:25,040 nines of uptime. No it doesn't. 964 01:00:25,600 --> 01:00:30,480 Yeah. Look how many times you know 965 01:00:30,480 --> 01:00:34,320 Azure or Google or Amazon has problems and then half the 966 01:00:34,320 --> 01:00:38,480 Internet doesn't work anymore. Stuff happens and it happens at 967 01:00:38,480 --> 01:00:42,360 least once or twice a year. If they can go down and it's OK, 968 01:00:42,360 --> 01:00:44,120 then you can go down and it's OK. 969 01:00:44,360 --> 01:00:49,480 That's not your goal obviously. But for Postgres, I really stick 970 01:00:49,480 --> 01:00:54,920 to as simple as possible. App get install tenant to a 971 01:00:54,920 --> 01:00:57,760 version. And then there's a couple of 972 01:00:57,760 --> 01:01:02,760 tweaks that I'll do If you Google Postgres configuration 973 01:01:02,760 --> 01:01:05,520 performance tweaks or whatever, there's a lot of blog articles 974 01:01:05,520 --> 01:01:10,320 on like how to change the config from whatever bases to something 975 01:01:10,320 --> 01:01:12,480 that's a little bit more reasonable taking into 976 01:01:12,480 --> 01:01:15,040 consideration how much memory and cores the machine you're 977 01:01:15,040 --> 01:01:19,080 running on has. And then the biggest tool out 978 01:01:19,080 --> 01:01:22,560 there for making sure that you have backups and if you want to 979 01:01:22,560 --> 01:01:27,240 get fancy a hot spare things like that is Oh my goodness, I'm 980 01:01:27,240 --> 01:01:30,960 blinking on it. PG backrest I think is what it 981 01:01:31,000 --> 01:01:36,800 is. That's that's something that I'm 982 01:01:37,200 --> 01:01:40,840 I'm blinking on the names, completely forgot the name of 983 01:01:40,840 --> 01:01:44,800 the people that that run that. But it's a great system where 984 01:01:45,880 --> 01:01:48,840 you you have a wall file that's continuously there. 985 01:01:48,840 --> 01:01:53,400 You can do nightly backups to multiple S3 buckets or Google 986 01:01:53,400 --> 01:01:55,840 buckets or whatever just to make sure that you've got multiple 987 01:01:55,840 --> 01:01:57,760 backups like it. It you just, I don't know, you 988 01:01:57,840 --> 01:01:59,840 just don't worry about it as. Much I I should like? 989 01:01:59,840 --> 01:02:01,680 Things are less fragile than you realize. 990 01:02:01,680 --> 01:02:02,880 I think that's the biggest thing. 991 01:02:03,040 --> 01:02:06,120 Things are less fragile. Well, I feel like databases are 992 01:02:06,120 --> 01:02:10,720 just robust systems too. Like they're impressive pieces 993 01:02:10,720 --> 01:02:14,200 of code. Absolutely. 994 01:02:15,120 --> 01:02:17,480 Think about all the garbage queries that are getting written 995 01:02:17,480 --> 01:02:20,680 by AI and the database is like here you go. 996 01:02:21,080 --> 01:02:23,760 Imagine writing like a database right? 997 01:02:23,760 --> 01:02:30,680 And you pretty much just have to accept everything and not break. 998 01:02:31,840 --> 01:02:33,240 Yep. That's amazing. 999 01:02:34,200 --> 01:02:36,320 It is. It is. 1000 01:02:37,040 --> 01:02:42,440 My database backup is. I have like a like I do like a a 1001 01:02:42,440 --> 01:02:46,240 postgres dump and then I push it up to S3 all through just like a 1002 01:02:46,240 --> 01:02:48,440 bash script. But I could probably clean that 1003 01:02:48,440 --> 01:02:51,160 up and then I have a Quran job. Just run it every day. 1004 01:02:52,000 --> 01:02:56,680 Yeah, for the most part it's good enough. 1005 01:02:57,280 --> 01:03:00,880 If you need point in time recovery wall files the way to 1006 01:03:00,880 --> 01:03:03,240 do it, PG backrest will be able to help you. 1007 01:03:03,360 --> 01:03:09,040 It's real easy to set up there, and when you go bare metal you 1008 01:03:09,040 --> 01:03:11,640 don't even have to separate the database from where the 1009 01:03:11,640 --> 01:03:13,440 application runs. Put it on the same server. 1010 01:03:13,440 --> 01:03:16,040 Talk about latency overhead. Just none. 1011 01:03:16,400 --> 01:03:18,480 Just there. That's actually pretty cool. 1012 01:03:22,000 --> 01:03:24,760 I need to do this. I need your open source project 1013 01:03:24,760 --> 01:03:27,600 too. We do too. 1014 01:03:27,880 --> 01:03:31,760 It's our sock. Our Sock 2 window closes mid 1015 01:03:31,760 --> 01:03:33,280 June. I think that's when we're going 1016 01:03:33,280 --> 01:03:36,880 to start migrating into bare metal, so it'll be between now 1017 01:03:36,880 --> 01:03:38,080 and then that it will get written. 1018 01:03:38,600 --> 01:03:39,520 Awesome. Very cool. 1019 01:03:39,720 --> 01:03:41,680 That's exciting. I like that. 1020 01:03:44,080 --> 01:03:50,000 All right, let's see. All right. 1021 01:03:50,000 --> 01:03:55,000 I kind of want to talk about like what you think like like 1022 01:03:55,000 --> 01:03:58,280 founder mode thinking means to you as like being a leader 1023 01:03:58,280 --> 01:04:02,320 inside of a company versus just being a developer. 1024 01:04:02,400 --> 01:04:05,800 Like what I know we've kind of touched on this a little bit 1025 01:04:05,800 --> 01:04:09,400 with like the Oauth problems and rate limiting, like things that 1026 01:04:09,400 --> 01:04:12,640 you have to think about that other engineers don't until 1027 01:04:13,080 --> 01:04:17,200 they're told to think about it. Like, was that a hard shift for 1028 01:04:17,200 --> 01:04:19,960 you? No. 1029 01:04:20,560 --> 01:04:23,240 No. And that's just because of my 1030 01:04:23,440 --> 01:04:32,240 career trajectory I guess. This is my first W2 job in 17 1031 01:04:32,240 --> 01:04:33,320 years. OK. 1032 01:04:34,240 --> 01:04:40,280 Yeah, I was a consultant before, had my own business. 1033 01:04:41,400 --> 01:04:47,400 So there's a lot of, there's a lot that I bring to my W2 1034 01:04:47,400 --> 01:04:51,920 position that maybe a normal career trajectory person 1035 01:04:51,920 --> 01:04:55,240 wouldn't just because when you are the, when you're in the hot 1036 01:04:55,240 --> 01:04:58,720 seat running your own company and you have, I mean, I've run 1037 01:04:58,880 --> 01:05:01,640 plenty of companies, but when people are dependent on you, 1038 01:05:01,640 --> 01:05:05,760 like you can't just say, oh, well, it'll happen unless you do 1039 01:05:05,760 --> 01:05:08,840 it. So there's, there's things that 1040 01:05:08,840 --> 01:05:12,360 I just, I think I handle myself completely differently than 1041 01:05:12,560 --> 01:05:17,080 other people might. Do you think it's a personality 1042 01:05:17,080 --> 01:05:21,880 trait or is it a learnable trait to be more in management leader 1043 01:05:21,880 --> 01:05:26,120 position where you you just absorb? 1044 01:05:27,640 --> 01:05:30,320 You know what's the word I'm looking for? 1045 01:05:34,760 --> 01:05:37,080 I don't know. I hate when I go blank live on a 1046 01:05:37,080 --> 01:05:41,560 stream, but like you kind of just absorb whatever job 1047 01:05:41,560 --> 01:05:44,280 requirements you have to in that moment, right? 1048 01:05:44,320 --> 01:05:48,800 And I, I've met some people that are just really good engineers. 1049 01:05:48,800 --> 01:05:51,800 You give them a task, they get it done, but then they also 1050 01:05:51,800 --> 01:05:55,960 can't like high level think and like, you know, think six months 1051 01:05:55,960 --> 01:05:58,840 from now as well. Like that's a skill in itself or 1052 01:05:58,840 --> 01:06:01,800 just feature planning, product planning. 1053 01:06:02,720 --> 01:06:05,960 A lot of people lack those skill sets I feel like. 1054 01:06:11,920 --> 01:06:17,240 I want to say that it's probably teachable, but there's a part of 1055 01:06:17,240 --> 01:06:23,280 me that thinks it it might be teachable, but you might not be 1056 01:06:23,280 --> 01:06:25,560 good at it because you might not want to do it, period. 1057 01:06:26,840 --> 01:06:34,720 I am the most motivated person ever when someone tells me no or 1058 01:06:34,920 --> 01:06:36,960 you're not good enough. You can't do that. 1059 01:06:37,840 --> 01:06:41,800 And I'm now on a warpath to prove you wrong. 1060 01:06:42,880 --> 01:06:45,080 That's personality from when I was a little kid. 1061 01:06:46,080 --> 01:06:53,080 And I feel like, I feel like that is a, there's a lot of 1062 01:06:53,080 --> 01:06:55,800 people that are not like that. You can't do that, OK. 1063 01:06:56,040 --> 01:06:57,840 And then life goes on and they don't care. 1064 01:06:57,840 --> 01:07:01,160 And sometimes I, I, I have envy for that because it's like, how 1065 01:07:01,160 --> 01:07:05,000 can you not care? Oh my gosh, I've got to go have 1066 01:07:05,000 --> 01:07:07,520 an Internet fight now 'cause someone said I was wrong. 1067 01:07:07,760 --> 01:07:15,760 But that's that innate sense of fighting for what you believe 1068 01:07:15,760 --> 01:07:22,000 in, fighting for yourself. Proving people wrong is why I 1069 01:07:22,000 --> 01:07:28,440 feel like I'm successful when I put a, a view upon everything 1070 01:07:28,440 --> 01:07:31,800 that's on the table, whether it's immediate need six months, 1071 01:07:31,920 --> 01:07:34,640 a year down the road. I'm looking at all things from 1072 01:07:34,640 --> 01:07:36,840 all angles, all the exact same time. 1073 01:07:36,840 --> 01:07:40,240 And I'm proud and I'm like, I'm proud that I can do it. 1074 01:07:40,520 --> 01:07:43,840 I pride myself on the ability to do it, that it's like, no, I 1075 01:07:43,840 --> 01:07:47,600 have visibility on everything. And what's interesting is that 1076 01:07:47,600 --> 01:07:52,080 wasn't just now that I transitioned into this role. 1077 01:07:52,080 --> 01:07:53,800 That was day one three years ago. 1078 01:07:54,480 --> 01:07:56,720 I just as just as a normal. I see. 1079 01:07:57,760 --> 01:07:59,720 That's why I'm sitting where I am now though. 1080 01:08:01,320 --> 01:08:05,000 Like when people can trust you and rely on you and when you say 1081 01:08:05,680 --> 01:08:08,960 don't worry, I got it. And you've proven yourself 1082 01:08:08,960 --> 01:08:12,000 multiple times that you can actually deliver and bring 1083 01:08:12,000 --> 01:08:16,560 receipts, then yeah, you are afforded opportunities in your 1084 01:08:16,560 --> 01:08:18,640 career path that otherwise wouldn't be there. 1085 01:08:19,240 --> 01:08:21,600 And I think that. I want to say that there's a lot 1086 01:08:21,600 --> 01:08:24,880 of people out there that may say, well, I'm just a developer, 1087 01:08:24,880 --> 01:08:28,200 I'm just gonna stay in my lane, don't do that. 1088 01:08:28,760 --> 01:08:31,840 Yeah, I think, I mean, I think it's really important. 1089 01:08:32,520 --> 01:08:35,960 Like I'm thinking back like I, I was late to the game in the 1090 01:08:35,960 --> 01:08:39,120 software world. Like I didn't start software 1091 01:08:39,120 --> 01:08:43,439 development until I was 30. So like I spent like 10 years of 1092 01:08:43,439 --> 01:08:47,359 my life in manufacturing and I was, I was an engineer, like I 1093 01:08:47,359 --> 01:08:51,640 was building automated systems. I was doing like ladder logic, 1094 01:08:51,640 --> 01:08:53,240 you know, but drag and drop things. 1095 01:08:53,240 --> 01:08:58,560 Like I didn't really understand software development or anything 1096 01:08:58,560 --> 01:09:01,399 above it. So I was self-taught and I 1097 01:09:01,399 --> 01:09:06,080 learned iOS development first. Like Swift was my first language 1098 01:09:06,080 --> 01:09:10,800 and I didn't know that much, but I got, you know, I was like very 1099 01:09:10,800 --> 01:09:16,479 motivated and I got my foot in the door at a startup and I was 1100 01:09:16,479 --> 01:09:20,359 just willing to, I think like I was a little naive to like, 1101 01:09:20,399 --> 01:09:23,240 like, you know, the Dunning Kruger curve. 1102 01:09:23,520 --> 01:09:25,359 I was at the top of Mount stupid. 1103 01:09:26,000 --> 01:09:29,399 But I think that benefited me at the beginning because I was 1104 01:09:29,399 --> 01:09:33,279 willing to jump in on any problem and I would solve it and 1105 01:09:33,279 --> 01:09:35,399 get it done. And it like accelerated, like I 1106 01:09:35,399 --> 01:09:38,640 got promoted very fast within this company because I would 1107 01:09:38,640 --> 01:09:43,000 just take responsibility and ownership over anything that 1108 01:09:43,279 --> 01:09:47,560 fell on the floor was handed to me, you know, And I think that 1109 01:09:47,560 --> 01:09:50,080 helped me. I mean, I learned the hard way, 1110 01:09:50,080 --> 01:09:52,319 like I was very naive with security. 1111 01:09:53,520 --> 01:09:57,960 My first encounter was our app had a man in the middle attack 1112 01:09:58,000 --> 01:10:01,640 and I had no idea what that was. And I learned very quickly what 1113 01:10:01,640 --> 01:10:05,560 it was and then how to fix it. And then I kind of got obsessed 1114 01:10:05,560 --> 01:10:10,600 with being better and like, I hated the fact that someone got 1115 01:10:10,600 --> 01:10:15,960 like one up on me and, you know, could circumvent my security on 1116 01:10:15,960 --> 01:10:19,920 my app. And then I just powered my drive 1117 01:10:19,920 --> 01:10:26,280 to not let that happen again. That, I believe is the perfect 1118 01:10:26,280 --> 01:10:28,360 way of describing founder mode mentality. 1119 01:10:28,360 --> 01:10:30,640 Is that total ownership of the problem? 1120 01:10:32,040 --> 01:10:35,560 It's it's I see it. What am I going to do about it? 1121 01:10:36,080 --> 01:10:37,960 I want to get better so it doesn't happen again. 1122 01:10:38,200 --> 01:10:40,880 I want to understand it so that I know what I'm doing in the 1st 1123 01:10:40,880 --> 01:10:44,400 place. And for people who are at the 1124 01:10:44,400 --> 01:10:48,680 top of Mount Stupid, founder mode mentality is going off the 1125 01:10:48,680 --> 01:10:51,960 other side and learning all the things that you don't currently 1126 01:10:51,960 --> 01:10:53,360 know to get. Better. 1127 01:10:53,880 --> 01:10:57,720 Yeah, Yep, yeah, yeah. I think there's like there's 1128 01:10:57,720 --> 01:11:00,880 definitely a benefit to the whole the Dunning Kruger curve, 1129 01:11:00,880 --> 01:11:04,440 right? Because you get to a point like 1130 01:11:04,440 --> 01:11:07,520 when you first start building something and you're like, Dang, 1131 01:11:07,520 --> 01:11:11,000 I know it all. Like I can build an app, I can, 1132 01:11:11,120 --> 01:11:14,880 I can build a REST API, you know, I can, I can do CRUD 1133 01:11:14,880 --> 01:11:17,480 operations. And you think that's it, right. 1134 01:11:19,280 --> 01:11:23,520 And then you start having to learn how the production side of 1135 01:11:23,520 --> 01:11:25,640 things works. How does deployment work? 1136 01:11:25,640 --> 01:11:30,880 How does security work? How do you secure things and and 1137 01:11:30,880 --> 01:11:34,200 then life goes on and you just keep realizing how little you 1138 01:11:34,200 --> 01:11:38,600 know and then it humbles you a little bit, but you still have 1139 01:11:38,600 --> 01:11:43,640 all the the scars from learning all those things. 1140 01:11:46,880 --> 01:11:53,680 For a long time, I always worried about what if I don't 1141 01:11:55,480 --> 01:11:59,200 understand what I don't know? What if I never come on the 1142 01:11:59,200 --> 01:12:01,240 other side of Mount Stupid, as you put it. 1143 01:12:01,880 --> 01:12:06,400 And I think there's a sense of freedom when you when you 1144 01:12:06,400 --> 01:12:10,560 finally understand that no, imposter syndrome's never going 1145 01:12:10,560 --> 01:12:11,960 away, bro. No, it's not. 1146 01:12:11,960 --> 01:12:14,800 You're just gonna feel it. You just have to embrace it, and 1147 01:12:14,800 --> 01:12:18,600 in fact, if you take yourself less seriously, then it's OK to 1148 01:12:18,600 --> 01:12:21,440 be wrong. The worst part to be the worst 1149 01:12:21,440 --> 01:12:27,480 part of that chart is when you are confidently incorrect and 1150 01:12:27,480 --> 01:12:30,480 you, you just know better and, and no one's gonna tell you 1151 01:12:30,480 --> 01:12:32,240 otherwise. And I feel like that's when we 1152 01:12:32,240 --> 01:12:34,200 were talking about AI and super senior people. 1153 01:12:34,200 --> 01:12:36,760 It's like that's them. They just don't know because 1154 01:12:36,760 --> 01:12:41,920 they're unwilling to learn. And for, for founders, no, 1155 01:12:42,560 --> 01:12:44,920 you're gonna be wrong. And that's a part of it. 1156 01:12:44,920 --> 01:12:48,200 I tell my kids all the time, like, oh, oh, and they, they'll 1157 01:12:48,200 --> 01:12:49,600 say, well, I don't know how to do this. 1158 01:12:49,600 --> 01:12:50,800 I'm frustrated. Great. 1159 01:12:50,800 --> 01:12:53,040 You don't know how to do it. And that's a part of learning. 1160 01:12:54,800 --> 01:12:57,480 Understand that you don't know. Understand you're going to get 1161 01:12:57,480 --> 01:13:00,960 the problem wrong. Nobody gets everything right 1162 01:13:00,960 --> 01:13:03,400 100% of the time. That's an anomaly. 1163 01:13:03,400 --> 01:13:08,640 It doesn't happen. So be OK with the ebb and flow 1164 01:13:08,640 --> 01:13:12,080 of your own success. You're gonna fail. 1165 01:13:12,080 --> 01:13:14,280 And that's a part of life. That's a part of your career. 1166 01:13:14,280 --> 01:13:16,240 It's part of everything, especially if you're doing your 1167 01:13:16,240 --> 01:13:18,400 own thing. You're, you're, you're a true 1168 01:13:18,400 --> 01:13:21,240 founder and you're your own company or you're leading A-Team 1169 01:13:21,240 --> 01:13:24,760 or if you were a junior IC, you're going to get it wrong. 1170 01:13:24,760 --> 01:13:25,960 And that's OK. It's part of the problem. 1171 01:13:25,960 --> 01:13:26,880 It's part of the. Process. 1172 01:13:27,800 --> 01:13:33,000 Yeah, I, I was part of a, a, a pretty funny conversation at my 1173 01:13:33,000 --> 01:13:36,760 gym yesterday, actually. And one of the one of the 1174 01:13:36,760 --> 01:13:40,280 people, like young, young girl in college, right, complaining 1175 01:13:40,280 --> 01:13:42,920 about, you know, exams and how hard school is. 1176 01:13:43,600 --> 01:13:46,840 And then, you know, one of the one of the older people that 1177 01:13:46,840 --> 01:13:50,560 runs a company and all of that goes, oh, just wait till you 1178 01:13:50,560 --> 01:13:53,160 graduate. It only gets harder. 1179 01:13:54,920 --> 01:13:59,080 Like, dude, it's never over like you're, you continue learning. 1180 01:13:59,080 --> 01:14:01,120 You'll have to learn more. You have to learn at a higher 1181 01:14:01,120 --> 01:14:05,320 level and just get ready. Strap in for the ride. 1182 01:14:08,280 --> 01:14:12,600 Level all as a company I feel like is walking on to the set of 1183 01:14:12,600 --> 01:14:21,120 suits where everyone around you is like Ivy League educated, 1184 01:14:21,360 --> 01:14:23,800 unbelievably smart. Everyone is there for the 1185 01:14:23,800 --> 01:14:27,920 mission ready to help America with post high school choices. 1186 01:14:28,720 --> 01:14:35,520 And then there's me, someone who dropped out of college and 1187 01:14:35,520 --> 01:14:41,160 became a developer and I always felt the sense of like, man, it 1188 01:14:41,200 --> 01:14:44,720 feels weird to be around all of education all the time and, and 1189 01:14:44,720 --> 01:14:48,000 whatever. So it's interesting that you put 1190 01:14:48,000 --> 01:14:51,320 it that way from the lady from your gem, because it's like, 1191 01:14:51,680 --> 01:14:57,120 sometimes you just have to know, yeah, after college it gets 1192 01:14:57,120 --> 01:14:58,640 harder. Or when you're in your adult 1193 01:14:58,640 --> 01:15:02,280 life, it's just harder. Since then I went and, and while 1194 01:15:02,280 --> 01:15:05,040 I was working at level all, I finished my undergrad and got an 1195 01:15:05,040 --> 01:15:07,440 MBA. So I I fixed that problem 1196 01:15:08,600 --> 01:15:10,000 sometimes. Do you? 1197 01:15:10,000 --> 01:15:13,040 Feel like you needed to do that or was that just like personal 1198 01:15:13,080 --> 01:15:14,520 con? Yeah, I was going to say like. 1199 01:15:14,520 --> 01:15:17,600 It's a it's a personal thing and I feel like I feel like that's 1200 01:15:17,600 --> 01:15:20,920 my personality is like I, I need to, I got to collect them all. 1201 01:15:20,920 --> 01:15:23,680 I've got to accomplish every possible thing that I want. 1202 01:15:24,000 --> 01:15:27,040 Yeah. So it was really important that 1203 01:15:27,040 --> 01:15:29,320 I, I don't want to just walk around the office even though 1204 01:15:29,320 --> 01:15:33,200 we're remote and be like, yeah, drop out. 1205 01:15:33,200 --> 01:15:36,360 No, not when people are coming from Harvard and Stanford and 1206 01:15:36,360 --> 01:15:38,440 stuff. It's like, no, I'm, I'm going to 1207 01:15:38,440 --> 01:15:40,840 go do my thing. And I really wanted my MBA to 1208 01:15:40,840 --> 01:15:43,600 say it wasn't enough just to get the undergrad. 1209 01:15:43,600 --> 01:15:46,600 I wanted the full thing. Yeah, I always think like 1210 01:15:46,600 --> 01:15:49,400 hindsight, I like, I tell myself this all the time. 1211 01:15:49,400 --> 01:15:52,240 Like I probably wouldn't have gone to school, but then at the 1212 01:15:52,240 --> 01:15:56,920 same time, maybe I would feel like you and you're like, well, 1213 01:15:57,400 --> 01:16:00,600 I need to accomplish that. Like I do have a piece of paper 1214 01:16:00,600 --> 01:16:05,320 that says I'm a I'm an engineer. That's pretty cool, you know, So 1215 01:16:05,320 --> 01:16:07,800 I don't know, like, am I using today? 1216 01:16:08,720 --> 01:16:11,440 And maybe it's a lot, maybe it's not. 1217 01:16:12,040 --> 01:16:15,400 I feel like the math side of things like calculus and that 1218 01:16:15,400 --> 01:16:18,640 stuff sneaks up randomly and you need to know how to do it right. 1219 01:16:18,880 --> 01:16:21,160 But it's not right. It's definitely not a daily 1220 01:16:21,160 --> 01:16:23,040 problem I have as a software engineer. 1221 01:16:25,160 --> 01:16:27,120 The NBA program is really interesting. 1222 01:16:27,160 --> 01:16:28,920 Because there's a whole side of HR. 1223 01:16:29,160 --> 01:16:31,520 And laws around that, that I was just unaware of. 1224 01:16:31,880 --> 01:16:34,320 So that was a really interesting learning experience. 1225 01:16:34,320 --> 01:16:38,320 But the for the most part, I've run multiple businesses. 1226 01:16:38,320 --> 01:16:42,280 I've done accounting and taxes and books since 2009 for 1227 01:16:42,280 --> 01:16:45,800 multiple of them. That on the job experience 1228 01:16:46,160 --> 01:16:49,160 obviously is a lot more valuable than what you'll learn in a 1229 01:16:49,160 --> 01:16:53,640 class when you're actually doing it and understanding PNLS and 1230 01:16:53,640 --> 01:16:59,600 all that stuff. But I I talking about. 1231 01:16:59,600 --> 01:17:00,520 Founder mode. It's like. 1232 01:17:00,520 --> 01:17:03,160 Yeah, owning the problem. If you feel like an imposter. 1233 01:17:03,160 --> 01:17:05,960 Because you're working with a bunch of wicked smart people 1234 01:17:05,960 --> 01:17:09,640 with college degrees, and you work at a company who's, like, 1235 01:17:09,640 --> 01:17:12,840 one of the focus of the software is to get kids into college, 1236 01:17:12,840 --> 01:17:15,480 like, and you don't have any like, come on, let's fix that 1237 01:17:15,480 --> 01:17:16,240 problem. So I did. 1238 01:17:16,640 --> 01:17:17,200 Yeah. Fair. 1239 01:17:17,320 --> 01:17:18,600 I like that. That's cool. 1240 01:17:20,000 --> 01:17:23,840 Yeah. An NBA could probably benefit me 1241 01:17:23,880 --> 01:17:27,560 as well, just there's a lot of gaps in my knowledge on the 1242 01:17:27,560 --> 01:17:30,800 business side of things and it's just school of Hard Knocks. 1243 01:17:30,800 --> 01:17:36,360 Currently it's and then when you look for. 1244 01:17:36,360 --> 01:17:38,280 Help out there, I feel like when you're on. 1245 01:17:38,280 --> 01:17:39,920 YouTube looking for a business help. 1246 01:17:39,920 --> 01:17:42,520 You never know who's trying to sell you something. 1247 01:17:43,560 --> 01:17:45,880 And so I think that's the biggest problem right now. 1248 01:17:45,880 --> 01:17:53,200 Everyone wants to sell you. Like I don't coaching like yeah, 1249 01:17:53,240 --> 01:17:55,040 well, why are you selling coaching? 1250 01:17:55,240 --> 01:17:57,880 Like are you literally just a coach? 1251 01:17:57,880 --> 01:18:01,160 Like is the only business you've built your coaching business? 1252 01:18:01,160 --> 01:18:02,840 Because they're not. I'm not interested. 1253 01:18:05,960 --> 01:18:07,560 Yep. And I feel like. 1254 01:18:07,560 --> 01:18:08,760 Yes, what? What gives? 1255 01:18:08,760 --> 01:18:11,520 You the qualifications to be. A business coach. 1256 01:18:11,520 --> 01:18:13,360 Where are all of your successful businesses? 1257 01:18:13,360 --> 01:18:16,840 It's like people who are Uber successful in business aren't 1258 01:18:16,840 --> 01:18:20,320 coaches, they're not businesses. You want to just be their friend 1259 01:18:20,320 --> 01:18:22,280 and take him out for a drink and. 1260 01:18:22,280 --> 01:18:24,600 Talk to him, you know. Yep. 1261 01:18:25,960 --> 01:18:27,840 Yeah. I always find it funny when 1262 01:18:27,840 --> 01:18:29,240 like. I get a like a. 1263 01:18:29,240 --> 01:18:32,760 LinkedIn request and it's like a 20 something year old kid 1264 01:18:32,960 --> 01:18:38,560 selling coaching services. I'm like, bro, good luck, all 1265 01:18:38,840 --> 01:18:41,080 right? I respect the hustle, but wrong 1266 01:18:41,080 --> 01:18:44,360 audience, Yeah. Like it's not, it's like it's 1267 01:18:44,360 --> 01:18:46,240 like if you're hiring a personal. 1268 01:18:46,240 --> 01:18:49,000 Trainer, are you going to hire someone that's like your same 1269 01:18:49,000 --> 01:18:51,200 fitness level or worse? Probably not. 1270 01:18:52,520 --> 01:18:55,040 You're going to want someone that you're like, oh, I'd like 1271 01:18:55,040 --> 01:18:57,960 to look like them. Absolutely. 1272 01:18:58,400 --> 01:19:01,280 I always laugh at the Super huge NFL coaches. 1273 01:19:01,280 --> 01:19:04,600 It's like the least you could do is eat right and get in the gym 1274 01:19:04,600 --> 01:19:07,000 with your with your team. For real. 1275 01:19:07,080 --> 01:19:12,120 Like dude, come on. I have a big problem with that 1276 01:19:12,120 --> 01:19:15,760 and I I feel like you see a lot, especially in like personal 1277 01:19:15,760 --> 01:19:20,000 fitness, you see a lot of like coaches that don't look like 1278 01:19:20,000 --> 01:19:26,200 they, they follow what they preach, of course, which. 1279 01:19:26,200 --> 01:19:27,480 Kind of leads back. To. 1280 01:19:28,080 --> 01:19:31,440 The importance like I. Like you got your degree right 1281 01:19:32,480 --> 01:19:36,120 because you felt like you need to practice what you preach like 1282 01:19:36,200 --> 01:19:39,800 at level all. That's kind of that's the point 1283 01:19:39,800 --> 01:19:41,960 exactly. With my with my own children, 1284 01:19:42,280 --> 01:19:45,080 college is really important. Daddy works at a company that 1285 01:19:45,080 --> 01:19:48,040 helps people get into college, but Daddy doesn't have a degree. 1286 01:19:48,040 --> 01:19:50,960 Like, yeah, yeah, do that. Yeah. 1287 01:19:51,120 --> 01:19:52,920 Fair. Well, you could. 1288 01:19:54,040 --> 01:19:56,400 You could. Yeah, but I could I? 1289 01:19:56,680 --> 01:19:59,360 I get why you don't want to lead by. 1290 01:19:59,360 --> 01:20:03,840 Example right? I think that's what makes. 1291 01:20:04,120 --> 01:20:08,000 Really good. Leaders inside of technology 1292 01:20:08,000 --> 01:20:12,760 companies is the people who will get into the trenches with their 1293 01:20:12,760 --> 01:20:16,520 developers. Lead by example, be be a player, 1294 01:20:16,520 --> 01:20:19,360 coach, get in there, get get your hands dirty. 1295 01:20:19,360 --> 01:20:23,520 Because just orchestrating from afar is a really good way to 1296 01:20:23,520 --> 01:20:27,920 isolate yourself. Not not wanting or willing to do 1297 01:20:27,920 --> 01:20:30,080 the work that's required to get something done. 1298 01:20:30,760 --> 01:20:33,120 It just causes a lack of respect for you. 1299 01:20:33,240 --> 01:20:35,480 If you're telling someone to do something that you won't do 1300 01:20:35,480 --> 01:20:37,960 yourself, there's a problem. Yeah. 1301 01:20:38,680 --> 01:20:41,160 Yeah. Like at Collects, our CTO. 1302 01:20:41,160 --> 01:20:43,240 He's always in the trenches with us. 1303 01:20:43,240 --> 01:20:46,560 He like, I'll wake up to messages from, you know, 3:00 1304 01:20:46,560 --> 01:20:51,880 AM, like, like, so you know, he's just always there and it, 1305 01:20:52,720 --> 01:20:56,120 it, it helps a lot. I feel like it makes you respect 1306 01:20:56,120 --> 01:21:00,480 people at a a new level, you know, of course, absolutely 1307 01:21:01,600 --> 01:21:02,840 like. You're engineers, you're up till 1308 01:21:02,840 --> 01:21:04,560 3:00 AM. Fixing a problem, right? 1309 01:21:04,880 --> 01:21:07,480 Like probably every day this week because it's a pretty big 1310 01:21:07,480 --> 01:21:10,760 problem and everyone sees that, right? 1311 01:21:10,760 --> 01:21:15,280 So it's good. I, I I. 1312 01:21:15,280 --> 01:21:17,920 Am blessed to have such. An incredible team of people to 1313 01:21:17,920 --> 01:21:19,880 work with. It's it's fun. 1314 01:21:19,920 --> 01:21:24,320 It's I don't not not every not every company or not every 1315 01:21:24,320 --> 01:21:28,160 engineering team has this and we have a really good thing going 1316 01:21:28,160 --> 01:21:30,720 because we have incredible world cast people. 1317 01:21:31,680 --> 01:21:35,280 Yeah, that's awesome. I like when you like the people 1318 01:21:35,280 --> 01:21:37,480 you. Work with and you feel like you 1319 01:21:37,600 --> 01:21:43,280 like I like I personally, I work with people that are way smarter 1320 01:21:43,280 --> 01:21:45,360 than me at least like that's. My. 1321 01:21:46,120 --> 01:21:48,880 Opinion right, And that's the room I want to be in. 1322 01:21:48,880 --> 01:21:52,000 I want to be in a room of people I feel like I'm learning from 1323 01:21:52,000 --> 01:21:56,920 constantly and that's a cool, that's a good team to be on. 1324 01:21:59,040 --> 01:22:01,040 So I know just probably in the room. 1325 01:22:01,120 --> 01:22:02,840 Find a different room. You're in the wrong. 1326 01:22:02,840 --> 01:22:07,200 Room, right? I feel like this might. 1327 01:22:07,200 --> 01:22:10,720 Be a difficult question for you right now just because you're in 1328 01:22:10,720 --> 01:22:13,120 the midst of a like a, a current fire. 1329 01:22:13,600 --> 01:22:18,280 But like, what do you foresee being like your next big hurdle 1330 01:22:18,280 --> 01:22:21,000 or issue that you're going to have to deal with and you want 1331 01:22:21,000 --> 01:22:22,800 to deal with it in a like a clean manner. 1332 01:22:22,800 --> 01:22:25,640 Like you're growing, you're a growing company. 1333 01:22:25,640 --> 01:22:28,720 What's the next like hurdle that you kind of see coming? 1334 01:22:31,360 --> 01:22:33,440 The biggest one that we're going to deal with. 1335 01:22:33,520 --> 01:22:37,360 Is. Our code base, just how it was 1336 01:22:37,360 --> 01:22:42,000 formed over the last four years, decisions that were made. 1337 01:22:42,320 --> 01:22:46,440 The product that we have today is nothing like the product that 1338 01:22:46,440 --> 01:22:51,680 was started four years ago. But foundationally we can only 1339 01:22:51,680 --> 01:22:55,600 make so many changes here or there because it was envisioned 1340 01:22:55,600 --> 01:22:59,760 as a different product and the product a year from now is even 1341 01:22:59,760 --> 01:23:04,640 way different than today. So we are having to do massive 1342 01:23:04,640 --> 01:23:08,600 rewrites of the core functionality of what we do and 1343 01:23:08,600 --> 01:23:14,240 doing that in a way that it's not downtime as much as it's 1344 01:23:14,240 --> 01:23:15,200 like. Resetting. 1345 01:23:15,200 --> 01:23:21,440 Progression of students progress through our application. 1346 01:23:21,480 --> 01:23:25,800 We don't want to have to. Say, yeah, we moved away from 1347 01:23:25,800 --> 01:23:29,040 whatever we're doing before all the stuff you did before it's 1348 01:23:29,040 --> 01:23:30,520 all gone. You can start again. 1349 01:23:30,520 --> 01:23:33,640 And it's like, that's not not ideal. 1350 01:23:33,640 --> 01:23:35,480 So that's going to be a very hard problem. 1351 01:23:35,560 --> 01:23:38,360 Is. You're talking about shifting 1352 01:23:38,360 --> 01:23:41,800 tables and columns around, adding, removing. 1353 01:23:41,840 --> 01:23:46,520 But you still have to restore the old state of things at at in 1354 01:23:46,520 --> 01:23:48,960 some way. Exactly. 1355 01:23:49,760 --> 01:23:55,760 At least there's a. Benefit that are the core 1356 01:23:55,760 --> 01:23:59,760 functionality of what we do. If you start and you start with 1357 01:23:59,760 --> 01:24:02,760 us in 6th grade, you're out at 12. 1358 01:24:03,280 --> 01:24:08,000 Or if if you start with us as a college student, you're out in 1359 01:24:08,000 --> 01:24:12,000 4. There's, there's some, there's 1360 01:24:12,000 --> 01:24:14,240 some niceties there because people. 1361 01:24:14,240 --> 01:24:16,760 Typically have a time. Frame in which they'll use the 1362 01:24:16,760 --> 01:24:19,720 product and then hopefully you're successful with our help 1363 01:24:21,280 --> 01:24:25,080 where there's like a bounding box of like, well, if we just 1364 01:24:25,080 --> 01:24:28,800 take care of this time frame, we can put everyone else in a 1365 01:24:28,800 --> 01:24:31,440 different box completely, but make sure that we have 1366 01:24:31,440 --> 01:24:34,400 compatibility for those folks. I see. 1367 01:24:34,480 --> 01:24:37,200 And then it's kind of like you only have to maintain it. 1368 01:24:37,200 --> 01:24:43,320 For a certain amount of year window too, because then all the 1369 01:24:45,560 --> 01:24:48,480 all the old. Stuff. 1370 01:24:48,760 --> 01:24:50,840 Moves on, right? People move on with their life. 1371 01:24:50,840 --> 01:24:53,040 They graduate, they stop using your software. 1372 01:24:54,080 --> 01:24:55,320 Exactly. Yeah. 1373 01:24:55,320 --> 01:24:57,120 So I think that I think that's a blessing. 1374 01:24:57,120 --> 01:25:00,320 There is that we we have time boxes, but definitely the 1375 01:25:00,320 --> 01:25:03,360 biggest problem is just looking for that word, but couldn't come 1376 01:25:03,360 --> 01:25:05,120 to it. Yeah. 1377 01:25:06,120 --> 01:25:08,320 The other is is another vendor lock thing. 1378 01:25:08,640 --> 01:25:14,440 Are the way that our the the content that teaches. 1379 01:25:14,440 --> 01:25:17,760 People how to do things and. Our software is controlled by 1380 01:25:17,760 --> 01:25:19,960 Sanity. Sanity is a. 1381 01:25:21,520 --> 01:25:23,560 It's basically a data lake with a. 1382 01:25:23,720 --> 01:25:26,400 React front end you want. To think about it as like an 1383 01:25:26,400 --> 01:25:32,280 enterprise version of WordPress in a way that that that's 1384 01:25:32,280 --> 01:25:36,160 holding us back a little bit. You can do infinite amount of 1385 01:25:36,160 --> 01:25:38,840 customizations inside of sanity, but the problem is, is that we 1386 01:25:38,840 --> 01:25:41,760 don't own our content. It gets synced to us. 1387 01:25:42,560 --> 01:25:46,480 And because we don't fully own that we everything we do, you 1388 01:25:46,480 --> 01:25:48,760 have to change inside of sanity and our product. 1389 01:25:49,160 --> 01:25:51,120 We have to make sure that that never breaks. 1390 01:25:51,320 --> 01:25:54,240 We have to make sure that as we progress forward with whatever 1391 01:25:54,240 --> 01:25:56,680 we're making from product choices that we're always 1392 01:25:56,680 --> 01:26:00,080 including sanity. And I think the next big hurdle, 1393 01:26:00,080 --> 01:26:03,680 which won't be any time now, I know probably next year, is how 1394 01:26:03,680 --> 01:26:06,600 do we replicate what we're doing in Sanity but in our own 1395 01:26:06,600 --> 01:26:10,000 product? Or how do we make our connection 1396 01:26:10,000 --> 01:26:13,600 with Sanity as flexible as possible so that the way that it 1397 01:26:13,600 --> 01:26:15,800 works today isn't holding us back for the future. 1398 01:26:16,200 --> 01:26:22,480 Yeah, Fair, do you think? It's pretty common practice. 1399 01:26:23,080 --> 01:26:24,400 Like, you know, you start, you're. 1400 01:26:24,400 --> 01:26:26,600 A fresh start up. You don't know if you're going 1401 01:26:26,600 --> 01:26:31,080 to be around next week, right? Do you find like the more 1402 01:26:31,080 --> 01:26:35,160 established you become in level all the more you start building 1403 01:26:35,160 --> 01:26:38,680 your own systems to get out of the vendor lock situations? 1404 01:26:38,680 --> 01:26:41,600 Do you think like that's a pretty normal process, 1405 01:26:44,200 --> 01:26:51,160 Completely normal? It's so easy to say the things 1406 01:26:51,160 --> 01:26:53,800 that we do like don't get. Vendor locked, Build it 1407 01:26:53,800 --> 01:26:57,000 yourself. When you're in a business that 1408 01:26:57,000 --> 01:27:01,160 is successful, has a trajectory, you're not worried about the 1409 01:27:01,160 --> 01:27:04,040 day-to-day. There's a sense of calm because 1410 01:27:04,040 --> 01:27:06,240 of that. If you're in a true start up 1411 01:27:06,240 --> 01:27:10,680 where you where you don't sell, you don't eat, I would be taking 1412 01:27:10,680 --> 01:27:13,760 every possible shortcut to get the product out in time. 1413 01:27:14,560 --> 01:27:16,720 Don't be concerned about vendor lock because you might not be 1414 01:27:16,720 --> 01:27:19,720 here in six months. Then it doesn't matter. 1415 01:27:20,320 --> 01:27:22,160 But there's there's. A tipping. 1416 01:27:22,160 --> 01:27:26,160 Point like with almost anything, consider a product like segment 1417 01:27:26,160 --> 01:27:29,520 or mix panel. After a certain amount of time, 1418 01:27:29,520 --> 01:27:30,920 both of those companies will come to you. 1419 01:27:30,920 --> 01:27:33,600 And say, hey your MA. Us are pretty high we you should 1420 01:27:33,600 --> 01:27:39,000 go to enterprise because it will be cheaper well the alarm bells 1421 01:27:39,000 --> 01:27:41,160 should be ringing in your ears that well they. 1422 01:27:41,160 --> 01:27:43,480 Want to lock me under a? Multi year contract because they 1423 01:27:43,480 --> 01:27:47,880 know it's starting to get painful to pay these bills and 1424 01:27:47,880 --> 01:27:51,920 we can write our own thing. So I feel like it is a natural 1425 01:27:51,920 --> 01:27:54,720 progression no matter what. And even SAS companies that 1426 01:27:54,880 --> 01:27:58,240 cater to start-ups even know that because it's a part of 1427 01:27:58,240 --> 01:28:00,520 their sales cycle. They know that once the pain 1428 01:28:00,520 --> 01:28:04,640 starts happening, yeah, we're going to we're going to spin up 1429 01:28:04,640 --> 01:28:07,800 our own click house data database, throw analytics in 1430 01:28:07,800 --> 01:28:12,000 there and point Metabase at it. And hey, that bill just became 1431 01:28:12,000 --> 01:28:14,400 freer. So it happens. 1432 01:28:15,200 --> 01:28:17,720 Yeah, yeah. I think it's important for 1433 01:28:17,720 --> 01:28:19,640 people listening. Like if you're building 1434 01:28:19,640 --> 01:28:24,080 something new, worry less about most of this conversation, but 1435 01:28:24,080 --> 01:28:27,400 now it's gonna happen. Like I think it's good to. 1436 01:28:27,400 --> 01:28:29,320 That's the goal. Road map you you want. 1437 01:28:29,320 --> 01:28:31,240 To have these problems. Right, they're good. 1438 01:28:31,240 --> 01:28:32,760 Problems is a problem you want to have. 1439 01:28:33,040 --> 01:28:36,400 These are growing pains. They're the best kind, yes. 1440 01:28:36,800 --> 01:28:39,440 You will never get it right at the beginning and that's OK. 1441 01:28:39,840 --> 01:28:42,480 You want to get, you want to get the product in people's hands as 1442 01:28:42,480 --> 01:28:45,120 fast as possible. No matter what, you're going to 1443 01:28:45,120 --> 01:28:49,120 be vendor locked in certain ways just happens and the that's the 1444 01:28:49,560 --> 01:28:51,600 from my perspective, the hardest part is. 1445 01:28:51,600 --> 01:28:55,480 Getting your product in people's hands because I'm no salesman, I 1446 01:28:55,480 --> 01:28:58,640 just want to build things, but you sometimes have to put that 1447 01:28:59,240 --> 01:29:04,640 that sales cap on and and honestly for people who are. 1448 01:29:05,280 --> 01:29:08,000 Developers who want to do founder mode and and do 1449 01:29:08,000 --> 01:29:12,360 something like kill switch, it's hard to, it's hard to have that 1450 01:29:12,360 --> 01:29:16,000 point be understated at all. Just because you build something 1451 01:29:16,000 --> 01:29:19,720 doesn't mean people will come. Well, there are competitors out 1452 01:29:19,720 --> 01:29:22,400 there that are 10 times worse. Than your product may. 1453 01:29:22,400 --> 01:29:25,000 Have 100 times of sales because they have a marketing team 1454 01:29:25,000 --> 01:29:27,320 behind them. It's nothing against you. 1455 01:29:27,440 --> 01:29:30,520 It's that you want to build, you want to be in the code, you want 1456 01:29:30,520 --> 01:29:32,840 to do deployment, you want to do all the fun stuff that comes 1457 01:29:32,840 --> 01:29:37,720 with tech challenges. But marketing and sales and SEO 1458 01:29:37,720 --> 01:29:41,840 and ads and all that is a completely different realm that 1459 01:29:41,840 --> 01:29:44,800 people are really good at. And sometimes you might need to 1460 01:29:44,800 --> 01:29:48,040 bring someone on either a part of your team or pay the money 1461 01:29:48,200 --> 01:29:52,560 because as the adage is, it takes money to make money. 100% 1462 01:29:52,760 --> 01:29:57,520 demand driven advertising is a big, big deal. 1463 01:29:58,920 --> 01:30:01,000 Take true classic T-shirts as an example. 1464 01:30:01,560 --> 01:30:02,800 They're. Selling a pack of T. 1465 01:30:02,800 --> 01:30:05,680 Shirts for what, 30 bucks? Something like that? 1466 01:30:05,680 --> 01:30:08,720 40 bucks. At one time they were spending 1467 01:30:09,640 --> 01:30:13,800 $70 in order to make that sell. Why would you ever? 1468 01:30:13,800 --> 01:30:16,280 Why would you ever do that? Well, because they need to. 1469 01:30:16,280 --> 01:30:18,880 Get their pipeline. Running and then it started to 1470 01:30:18,880 --> 01:30:21,120 go down. Now they're only at $10.00 or $5 1471 01:30:21,120 --> 01:30:24,720 or whatever it is, but it's a multi $100 million company 1472 01:30:24,720 --> 01:30:28,840 selling T-shirts and they almost exclusively had that success 1473 01:30:28,840 --> 01:30:33,760 because of Facebook advertising. They're a behemoth now, but they 1474 01:30:33,760 --> 01:30:36,480 had to spend the money to get their name out there. 1475 01:30:36,480 --> 01:30:38,320 And then there's a tipping point that happens. 1476 01:30:38,720 --> 01:30:41,800 So it's really hard. I see a lot of people who will 1477 01:30:41,800 --> 01:30:45,520 build in public and launch their product and then six months or 1478 01:30:45,520 --> 01:30:48,680 12 months go by and they have their oh shucks, we're going to 1479 01:30:48,680 --> 01:30:51,200 spend it down probably because they have 4 1/2 users. 1480 01:30:51,200 --> 01:30:54,120 And it's like, well, it's because you cannot rely just on 1481 01:30:54,120 --> 01:30:58,640 a personal brand unless if you are someone like DHH or 1482 01:30:58,640 --> 01:31:02,160 something, right? Like a hard lesson for me that I 1483 01:31:02,160 --> 01:31:05,000 just recently. Learned too is like, I do have 1484 01:31:05,000 --> 01:31:07,760 an audience, right? I have a extra mentor, but my 1485 01:31:07,760 --> 01:31:11,200 audience is not the people buying my product. 1486 01:31:11,640 --> 01:31:15,960 So I need a new audience, you know, and that was hard. 1487 01:31:16,160 --> 01:31:19,320 You know, you're you always hope it will get you a little further 1488 01:31:19,320 --> 01:31:21,320 like what you have, but it doesn't. 1489 01:31:21,320 --> 01:31:23,680 So you have to pivot and you have to move and it can be 1490 01:31:23,680 --> 01:31:25,880 demoralizing, but you still got to keep going. 1491 01:31:25,880 --> 01:31:29,840 Of course. See, for you this is unsolicited 1492 01:31:29,840 --> 01:31:30,960 device. Don't hold me to this. 1493 01:31:30,960 --> 01:31:35,800 But there are so many like wealth conferences or real 1494 01:31:35,800 --> 01:31:38,920 estate conferences that happen all the time and and people want 1495 01:31:38,920 --> 01:31:41,320 to go and learn about real estate or how to how to do 1496 01:31:41,320 --> 01:31:44,240 wealth. I could easily see you giving a 1497 01:31:44,240 --> 01:31:46,600 talk on a panel or something about your product. 1498 01:31:46,600 --> 01:31:50,280 Because if you're interested in wealth, you need to protect 1499 01:31:50,280 --> 01:31:52,000 yourself. If the worst thing in the world 1500 01:31:52,000 --> 01:31:55,680 happens to you, everyone there would immediately be like, oh, 1501 01:31:55,680 --> 01:31:58,080 that makes sense. My family needs to have access 1502 01:31:58,080 --> 01:32:00,280 to all of my stuff. If I get hit by a bus, it's 1503 01:32:00,280 --> 01:32:02,720 amazing. So that has nothing to do with 1504 01:32:02,720 --> 01:32:04,960 tech, nothing to do with technology. 1505 01:32:05,200 --> 01:32:07,800 It's, it's just breaking out into a different sector of 1506 01:32:07,800 --> 01:32:11,320 people who are actively thinking about I need to do wealth stuff, 1507 01:32:11,880 --> 01:32:13,240 right? And that's another one. 1508 01:32:13,240 --> 01:32:18,040 Like I was actually just. Writing like a a case study blog 1509 01:32:18,040 --> 01:32:23,320 post for someone that wants to post By the way, getting 1510 01:32:23,320 --> 01:32:27,000 backlinks on other websites is really good for you as well. 1511 01:32:27,240 --> 01:32:30,160 So I'll, I'll take the time and I'll, I'll do this stuff. 1512 01:32:30,520 --> 01:32:35,400 But one of the things that kind of like brought forth to me is 1513 01:32:35,400 --> 01:32:37,520 like my. Competitors. 1514 01:32:37,520 --> 01:32:39,480 Like there's a couple competitors out there that are 1515 01:32:39,480 --> 01:32:43,640 doing like very similar things to me, but they're not, they're 1516 01:32:43,640 --> 01:32:45,440 not my problem. My problem is like the 1517 01:32:45,440 --> 01:32:50,080 procrastination of my customers, like, 'cause no one wants to 1518 01:32:50,080 --> 01:32:52,880 think like, oh, I could get hit by a bus tomorrow, I could die. 1519 01:32:52,880 --> 01:32:56,920 I should have this set up now. Everyone's, everyone's like, oh, 1520 01:32:56,920 --> 01:32:59,280 I'll worry about this in like 6 months. 1521 01:32:59,640 --> 01:33:01,680 So they're not immediate customers, right? 1522 01:33:01,680 --> 01:33:05,400 You need the, you need to like get that urgency, that fire 1523 01:33:05,400 --> 01:33:08,080 under them. And that's the difficult part. 1524 01:33:10,880 --> 01:33:14,400 I just had a flood of. Ideas of like stuff. 1525 01:33:14,400 --> 01:33:17,200 That you could do for the procrastination stuff, because 1526 01:33:17,200 --> 01:33:19,280 yet. Yeah, like, that's the same 1527 01:33:19,280 --> 01:33:21,600 thing with life insurance. Oh, I'll take care of it next 1528 01:33:21,600 --> 01:33:23,720 year. No, you won't. 1529 01:33:23,800 --> 01:33:25,120 Not if the worst thing in the world happens. 1530 01:33:25,400 --> 01:33:29,000 Yeah, exactly. Yeah, I should actually just 1531 01:33:29,000 --> 01:33:30,400 reach. Out to people selling life 1532 01:33:30,400 --> 01:33:35,680 insurance and give them affiliate links yeah or people 1533 01:33:35,680 --> 01:33:38,360 who specialize in. Doing trusts or wills? 1534 01:33:39,200 --> 01:33:40,960 Yeah, same thing. Yep. 1535 01:33:41,320 --> 01:33:44,520 I was like I'm trying to like figure out my angle but I want 1536 01:33:44,520 --> 01:33:47,280 to like like. LegalZoom or other platforms 1537 01:33:47,280 --> 01:33:51,800 like that, like those are, those are my, that's my clientele 1538 01:33:51,840 --> 01:33:53,800 right there. Everyone that's thinking about 1539 01:33:53,800 --> 01:33:56,360 that. Absolutely, yeah. 1540 01:33:58,480 --> 01:34:00,600 Yeah, got a break out of the. Comfort the See, that's the. 1541 01:34:00,600 --> 01:34:03,360 Problem as a developer. You want to stay in your comfort 1542 01:34:03,360 --> 01:34:04,920 zone. It's talking with other 1543 01:34:04,920 --> 01:34:09,240 developers. It's building products and now, 1544 01:34:09,320 --> 01:34:13,680 you know, now I need an audience of non developers and no one 1545 01:34:13,680 --> 01:34:15,880 wants to do that. I have to talk about my product 1546 01:34:15,880 --> 01:34:18,480 not from a technical perspective, from like a use 1547 01:34:18,480 --> 01:34:22,480 case perspective, right? Yeah, it's hard. 1548 01:34:24,440 --> 01:34:26,560 But fun. It is fun, I mean. 1549 01:34:26,600 --> 01:34:29,240 Like I every problem. That I get to solve. 1550 01:34:29,240 --> 01:34:30,920 I think is fun, right? You just have to. 1551 01:34:31,880 --> 01:34:36,080 It's a new kind of problem, yeah. 1552 01:34:37,560 --> 01:34:39,440 All right, well, we've been talking for a. 1553 01:34:39,440 --> 01:34:41,280 While we can. Probably wrap this up. 1554 01:34:41,520 --> 01:34:46,080 Do you have any unsolicited or solicited advice to give the 1555 01:34:46,080 --> 01:34:50,800 viewers? Oh, let's see. 1556 01:34:52,160 --> 01:34:58,000 You will never know what to do at first if you never give 1557 01:34:58,000 --> 01:34:59,800 yourself the opportunity to fail. 1558 01:35:01,000 --> 01:35:03,880 Yeah, some people ignore their gut because they're too scared 1559 01:35:03,880 --> 01:35:07,200 to make a choice. Your gut is a powerful thing. 1560 01:35:07,920 --> 01:35:11,360 Listen to it. You may be wrong, and that's all 1561 01:35:11,360 --> 01:35:13,760 right for. For founder mode. 1562 01:35:13,760 --> 01:35:17,120 Mentality. You got to own the entire 1563 01:35:17,120 --> 01:35:21,200 problem start to finish, and ignoring it is not going to get 1564 01:35:21,200 --> 01:35:25,920 you where you want to go. Yeah, but I don't think it gets 1565 01:35:25,920 --> 01:35:26,800 much. Better than that. 1566 01:35:29,000 --> 01:35:34,040 One of the funny like not to continue this even longer, but 1567 01:35:35,680 --> 01:35:39,160 as technical founders, if you're building something and you don't 1568 01:35:39,160 --> 01:35:41,360 want to do something, you'll procrastinate with building more 1569 01:35:41,360 --> 01:35:43,480 features. You have to stop doing that. 1570 01:35:45,560 --> 01:35:48,280 I I have the same problem. Like sometimes I'll just want to 1571 01:35:48,280 --> 01:35:50,720 relax and be like, I'll work on that feature I've been wanting 1572 01:35:50,720 --> 01:35:55,680 to build, but really I should be reading Reddit threads and 1573 01:35:55,680 --> 01:36:01,160 posting on there and being like, hey, this is what I do and I got 1574 01:36:02,320 --> 01:36:04,880 I'm learning. Like Reddit is actually a great 1575 01:36:04,880 --> 01:36:07,880 platform to advertise. You just have to do it really 1576 01:36:07,880 --> 01:36:11,480 smoothly. Otherwise moderators are really 1577 01:36:11,480 --> 01:36:14,520 quick to ban you. I've been learning this lately. 1578 01:36:16,800 --> 01:36:20,080 Can't just sell sometimes instead of sometimes instead of 1579 01:36:20,080 --> 01:36:21,120 adding. Features. 1580 01:36:21,280 --> 01:36:23,760 Go back and work on the features and make the ones that you 1581 01:36:23,760 --> 01:36:25,840 already have better or curate them. 1582 01:36:26,320 --> 01:36:29,160 If the feature has too many options, remove them or add more 1583 01:36:29,160 --> 01:36:32,120 options. But just continuously adding 1584 01:36:32,120 --> 01:36:36,280 features and not pounding the pavement and making sales 1585 01:36:36,280 --> 01:36:40,680 calling people it. No amount of features is going 1586 01:36:40,680 --> 01:36:43,040 to help sell a product that nobody knows about. 1587 01:36:43,880 --> 01:36:47,120 Yeah 100%. I love it all. 1588 01:36:47,160 --> 01:36:49,240 Right. Well, thank you so much, Thomas. 1589 01:36:49,240 --> 01:36:52,640 We will let you get back to your current fire and I hope it's 1590 01:36:52,640 --> 01:36:58,240 completely extinguished today and you can enjoy your weekend. 1591 01:36:59,800 --> 01:37:03,720 I hope so too. I don't particularly. 1592 01:37:03,720 --> 01:37:06,240 Like fires. But I love the challenge and I 1593 01:37:06,240 --> 01:37:08,280 find them fun. So it is what it is. 1594 01:37:08,560 --> 01:37:11,280 It's a it's a good challenge and it's like one of those things 1595 01:37:11,280 --> 01:37:14,120 you don't think. About until it happens well, now 1596 01:37:14,120 --> 01:37:18,240 hopefully more people will think about it yeah rate limit I do I 1597 01:37:18,240 --> 01:37:22,400 do think you're. Like a very unique situation. 1598 01:37:22,400 --> 01:37:24,920 For like hitting rate limits like that too, right? 1599 01:37:24,920 --> 01:37:29,000 Because everyone's essentially going through the same proxy or 1600 01:37:29,000 --> 01:37:33,600 same network, so it's the worst shape of traffic you could. 1601 01:37:33,600 --> 01:37:36,080 Possibly have which is a. It's a fun challenge. 1602 01:37:36,440 --> 01:37:37,600 Yeah. Yeah. 1603 01:37:37,600 --> 01:37:39,480 Interesting. All right, Well, thanks a lot. 1604 01:37:39,480 --> 01:37:41,800 This was this. Was a lot of fun and I'll have 1605 01:37:41,800 --> 01:37:44,200 to have you back on. I think we can we have even more 1606 01:37:44,200 --> 01:37:48,480 to talk about. So absolutely I can't wait. 1607 01:37:48,480 --> 01:37:50,320 Thanks a lot. Yeah, everyone have a good 1608 01:37:50,320 --> 01:37:54,040 weekend testing.