1 00:00:00,040 --> 00:00:01,960 Quick note, this episode isn't sponsored. 2 00:00:02,880 --> 00:00:06,560 I'm building a new kind of IDE called Rex because existing ones 3 00:00:06,560 --> 00:00:09,040 make it hard to work across multiple projects in parallel. 4 00:00:09,800 --> 00:00:11,760 I'm sharing it to get feedback from listeners. 5 00:00:11,920 --> 00:00:13,440 I'd really love to hear your thoughts. 6 00:00:14,000 --> 00:00:16,800 The link is in the description. And now let's move on with 7 00:00:16,800 --> 00:00:18,600 today's super interesting episode. 8 00:00:20,400 --> 00:00:25,960 It is 11.0 PM on a Tuesday. You are sitting on your couch, 9 00:00:25,960 --> 00:00:28,840 you're 3 episodes deep into a show you have definitely seen 10 00:00:28,840 --> 00:00:31,200 before, and suddenly it hits you. 11 00:00:31,520 --> 00:00:34,360 You need chips. And it is never just chips, is 12 00:00:34,360 --> 00:00:36,400 it? It's a very specific annoying 13 00:00:36,400 --> 00:00:37,600 craving. Exactly. 14 00:00:37,600 --> 00:00:40,800 It's not just salty potato. It is specifically the jalapeno 15 00:00:40,800 --> 00:00:43,720 cheddar kettle cooked chips. The ones in the purple bag. 16 00:00:43,720 --> 00:00:46,320 You know the ones, of course. So you pull out your phone, you 17 00:00:46,320 --> 00:00:49,720 open an app, maybe it's go puff, maybe it's gorillas or Gateer. 18 00:00:50,080 --> 00:00:52,960 You tap a button and the app says 15 minutes. 19 00:00:53,120 --> 00:00:55,400 Which, if you actually stop to think about the logistics 20 00:00:55,400 --> 00:00:57,360 involved, is. It's absolutely absurd. 21 00:00:57,400 --> 00:00:59,920 It feels like magic, but you and I know better. 22 00:00:59,920 --> 00:01:03,400 We know that underneath that magic is a terrifying, complex 23 00:01:03,400 --> 00:01:05,840 web of logistics, databases and code. 24 00:01:05,880 --> 00:01:07,880 Right, it is not just a driver driving fast. 25 00:01:07,880 --> 00:01:11,200 It is a system that has to know exactly which dusty corner of 26 00:01:11,200 --> 00:01:14,720 which micro warehouse has that specific bag of chips, and it 27 00:01:14,720 --> 00:01:17,720 has to promise it to you before someone three blocks away grabs 28 00:01:17,720 --> 00:01:18,040 it. And. 29 00:01:18,040 --> 00:01:21,760 It has to do all of that math. All of those checks in what, 30 00:01:21,760 --> 00:01:24,520 milliseconds? Maybe 200 milliseconds. 31 00:01:25,320 --> 00:01:28,240 If it takes longer than that, you have probably already closed 32 00:01:28,240 --> 00:01:30,000 the app or switched to a competitor. 33 00:01:30,040 --> 00:01:34,280 Exactly, and that high stakes, high speed environment makes 34 00:01:34,280 --> 00:01:37,600 this specific problem a goldmine for engineering interviews. 35 00:01:38,400 --> 00:01:41,680 So that is our mission today. OK, We aren't just analyzing how 36 00:01:41,680 --> 00:01:43,440 these apps work from the outside. 37 00:01:43,720 --> 00:01:46,760 We are going to build 1. We are entering the simulation. 38 00:01:46,760 --> 00:01:48,800 This is the system Design mock interview. 39 00:01:49,040 --> 00:01:52,160 We are going to walk through a classic senior level prompt 40 00:01:52,760 --> 00:01:56,760 design a local delivery service. And we've pulled together well a 41 00:01:56,760 --> 00:01:58,120 whole stack of resources for this. 42 00:01:58,120 --> 00:02:01,280 We're looking at system design guides, some pretty dense 43 00:02:01,280 --> 00:02:03,920 research on database concurrency, even some real 44 00:02:03,920 --> 00:02:06,080 world logistics papers. We're going to need them. 45 00:02:06,200 --> 00:02:08,479 This is one of those problems that looks easy on the surface. 46 00:02:08,479 --> 00:02:11,240 I just want a bag of chips, but turns into a nightmare as soon 47 00:02:11,240 --> 00:02:13,640 as you look at the scale. Oh yeah, the goal is simple. 48 00:02:13,880 --> 00:02:16,720 By the end of this hour, you, the listener should be able to 49 00:02:16,720 --> 00:02:19,560 walk into a whiteboard, interview at a top tech company 50 00:02:19,720 --> 00:02:23,000 and explain how to handle inventory scaling, heavy read 51 00:02:23,000 --> 00:02:26,440 traffic, and the absolute nightmare that is the double 52 00:02:26,440 --> 00:02:28,520 booking problem. And hopefully get hired. 53 00:02:28,520 --> 00:02:29,960 Hopefully. So. 54 00:02:29,960 --> 00:02:32,760 Let's set the scene. We walk into the interview room. 55 00:02:32,920 --> 00:02:35,680 The air conditioning is humming. The whiteboard is fresh. 56 00:02:36,000 --> 00:02:38,000 The marker smells like chemicals. 57 00:02:38,320 --> 00:02:42,280 And the interviewer looks at us and says design a system for 58 00:02:42,280 --> 00:02:45,680 instant delivery and the panic sets in. 59 00:02:46,040 --> 00:02:48,720 Where do we even start? You start by taking a deep 60 00:02:48,720 --> 00:02:51,400 breath and not building everything. 61 00:02:51,400 --> 00:02:54,680 What do you mean by that? The biggest mistake candidates 62 00:02:54,680 --> 00:02:57,120 make, and I see this all the time when I mock interview 63 00:02:57,120 --> 00:02:59,520 people, is that they try to boil the ocean. 64 00:02:59,920 --> 00:03:01,480 OK. They try to design the user 65 00:03:01,480 --> 00:03:04,760 registration flow, the credit card processing, the driver 66 00:03:04,760 --> 00:03:07,560 routing algorithm, the notification system, the push 67 00:03:07,560 --> 00:03:09,320 notifications, and then. Run out of time? 68 00:03:09,320 --> 00:03:10,560 They run out of time in 10 minutes. 69 00:03:10,560 --> 00:03:12,960 In a 45 minute interview. You have to be ruthless. 70 00:03:13,440 --> 00:03:15,000 You have to draw a box around the problem. 71 00:03:15,000 --> 00:03:17,440 We call this scoping. You need to clarify the 72 00:03:17,440 --> 00:03:20,280 requirements and cut anything that isn't the core challenge. 73 00:03:20,280 --> 00:03:23,240 OK, let's practice that. If we are building a Go Puff 74 00:03:23,240 --> 00:03:26,720 clone, what stays in the box and what gets thrown out? 75 00:03:26,920 --> 00:03:29,320 Well, let's look at the problem. What makes this hard? 76 00:03:29,800 --> 00:03:31,840 Is it signing up a user? No. 77 00:03:31,840 --> 00:03:37,360 No, that's a standard CRUD app. Create, read, update, delete. 78 00:03:38,080 --> 00:03:39,320 It's You know it's a solve problem. 79 00:03:39,480 --> 00:03:42,480 It's boring for an interview. OK, so user profiles are out. 80 00:03:42,480 --> 00:03:44,440 What about payments? That seems pretty important. 81 00:03:44,600 --> 00:03:47,080 It's critical for the business, but for a system design 82 00:03:47,080 --> 00:03:49,400 interview, it's boring. Boring. 83 00:03:49,800 --> 00:03:53,280 How can payments be boring? From a design perspective, yes. 84 00:03:53,680 --> 00:03:57,640 You aren't building Stripe or PayPal, you are just integrating 85 00:03:57,640 --> 00:03:58,920 with them. You draw a box on the 86 00:03:58,920 --> 00:04:01,680 whiteboard, you label it payment gateway and you move on. 87 00:04:02,400 --> 00:04:03,680 So you just assume it. Works. 88 00:04:03,680 --> 00:04:05,840 You assume it works. Do not waste 5 minutes 89 00:04:05,840 --> 00:04:08,600 discussing credit card validation or PCI compliance 90 00:04:09,000 --> 00:04:11,200 unless the interviewer specifically asks for. 91 00:04:11,200 --> 00:04:13,320 It got it. So we are laser focused. 92 00:04:13,320 --> 00:04:15,960 What about the drivers? I feel like the routing getting 93 00:04:15,960 --> 00:04:18,519 the driver from point A to point B efficiently is the hardest 94 00:04:18,519 --> 00:04:20,040 part. Don't we need to solve that? 95 00:04:20,040 --> 00:04:22,880 It is a hard problem. It is the traveling salesman 96 00:04:22,880 --> 00:04:25,480 problem. It is a massive mathematical 97 00:04:25,480 --> 00:04:28,520 challenge involving graph theory and optimization. 98 00:04:28,720 --> 00:04:30,640 Right, And that is exactly why you throw it out. 99 00:04:30,880 --> 00:04:33,000 Laughs. You just ignore it because it's 100 00:04:33,000 --> 00:04:34,560 too hard. For this specific interview 101 00:04:34,560 --> 00:04:37,240 slot, yes. If you get bogged down trying to 102 00:04:37,240 --> 00:04:40,200 calculate the optimal route for a driver to hit 5 houses in a 103 00:04:40,200 --> 00:04:42,800 specific order, you will never get to the database 104 00:04:42,800 --> 00:04:45,040 architecture. So you just hand wave it away. 105 00:04:45,040 --> 00:04:47,560 You just tell the interviewer. I'm assuming we have a separate 106 00:04:47,560 --> 00:04:50,800 downstream service that handles routing logistics You delegate. 107 00:04:51,200 --> 00:04:53,920 OK, fair enough. We are delegating the hard math 108 00:04:53,920 --> 00:04:56,280 to another team. So what is left? 109 00:04:56,440 --> 00:04:59,680 What are the core puzzle we are actually solving here today? 110 00:04:59,720 --> 00:05:02,480 We care about two things, availability and ordering. 111 00:05:02,640 --> 00:05:04,720 OK. Can the customer see if the item 112 00:05:04,720 --> 00:05:08,640 is in stock right now based on where they are standing and can 113 00:05:08,640 --> 00:05:12,240 they lock that inventory and buy it without the system crashing? 114 00:05:12,440 --> 00:05:15,160 Availability in ordering it sounds simple. 115 00:05:15,160 --> 00:05:17,000 It sounds simple until you look at the numbers. 116 00:05:17,080 --> 00:05:19,680 Let's talk scale. Yeah, How big is this 117 00:05:19,680 --> 00:05:22,960 hypothetical app? Are we building this for my 118 00:05:22,960 --> 00:05:26,800 neighborhood or for the world? To impress the interviewer, you 119 00:05:26,800 --> 00:05:29,680 always want to aim high. If you design a system that 120 00:05:29,680 --> 00:05:32,080 works for 100 people, you are designing a toy. 121 00:05:32,120 --> 00:05:35,400 If you design one that works for 100 million, you are operating 122 00:05:35,400 --> 00:05:37,840 at a staff engineer level. So let's go staff level. 123 00:05:37,840 --> 00:05:39,120 What are our parameters? Let's. 124 00:05:39,120 --> 00:05:43,240 Assume we are operating globally, we have maybe 10,000 125 00:05:43,240 --> 00:05:48,280 distribution centers or DC's. Whoa wait, 10,000 Amazon has 126 00:05:48,280 --> 00:05:50,360 what, a few 100 fulfillment centers? 127 00:05:50,520 --> 00:05:54,160 10,000 sounds insane. It does, but remember these 128 00:05:54,160 --> 00:05:56,920 aren't those massive million square foot warehouses with 129 00:05:56,920 --> 00:06:00,120 robots and conveyor belt. These are micro warehouses, dark 130 00:06:00,120 --> 00:06:01,320 stores. Exactly. 131 00:06:01,400 --> 00:06:04,080 A converted garage in Brooklyn or a basement in London. 132 00:06:04,440 --> 00:06:06,400 Small footprint but high density. 133 00:06:06,440 --> 00:06:07,400 They are everywhere. OK. 134 00:06:07,400 --> 00:06:09,480 And the items. Let's say each one holds maybe 135 00:06:09,480 --> 00:06:13,320 5000 to 10,000 unique items, but across the whole network our 136 00:06:13,320 --> 00:06:16,160 total catalog is maybe 100,000 different items. 137 00:06:16,160 --> 00:06:17,880 OK. And the traffic, How many people 138 00:06:17,880 --> 00:06:20,480 are buying chips? Let's push for 10 million orders 139 00:06:20,480 --> 00:06:23,320 per day. 10 million orders. Let me do some napkin math here. 140 00:06:23,480 --> 00:06:28,160 We have 10 million orders. There are 86,400 seconds in a 141 00:06:28,160 --> 00:06:30,960 day. So that is roughly what, 115 142 00:06:30,960 --> 00:06:32,840 orders per second? On average. 143 00:06:33,320 --> 00:06:35,240 But averages are a trap. Why? 144 00:06:35,920 --> 00:06:38,600 Because nobody orders chips at 4:00 AM on a Tuesday. 145 00:06:38,920 --> 00:06:43,160 The traffic is almost 0, but everyone orders lunch at noon 146 00:06:44,000 --> 00:06:46,480 and everyone orders snacks during the Super Bowl halftime 147 00:06:46,480 --> 00:06:49,120 show or the season finale of a big show. 148 00:06:49,160 --> 00:06:52,600 So the peak traffic isn't 115 orders per second. 149 00:06:52,640 --> 00:06:56,600 No, your peak traffic might be 5000 or 10,000 orders per second 150 00:06:56,600 --> 00:06:59,680 and you have to design for the peak, not the average. 151 00:06:59,760 --> 00:07:02,960 If you design for the average, your system crashes every single 152 00:07:02,960 --> 00:07:04,440 lunch hour. Every single time. 153 00:07:04,440 --> 00:07:07,120 And that is just the orders, that is the rights to the 154 00:07:07,120 --> 00:07:08,040 database. Right. 155 00:07:08,080 --> 00:07:11,080 Think about how you use the app. You don't just open it and buy. 156 00:07:11,560 --> 00:07:13,200 You scroll. You search for ice cream. 157 00:07:13,280 --> 00:07:15,280 You check candy. You click on item, look at the 158 00:07:15,280 --> 00:07:16,800 picture, check the price, go back. 159 00:07:16,960 --> 00:07:20,400 You are generating dozens, maybe hundreds of reads for every 160 00:07:20,400 --> 00:07:22,320 single write. O if we have 10 million orders, 161 00:07:22,320 --> 00:07:24,760 we might have a billion read requests per day. 162 00:07:24,800 --> 00:07:28,560 Exactly O we are looking at a system that is incredibly read 163 00:07:28,560 --> 00:07:33,040 heavy but has these terrifying bursty spikes of write traffic 164 00:07:33,280 --> 00:07:36,400 that absolutely cannot fail. That is the constraints profile, 165 00:07:36,720 --> 00:07:41,000 high read volume, bursty write volume, and this is critical low 166 00:07:41,000 --> 00:07:41,880 latency. Right. 167 00:07:41,880 --> 00:07:44,920 If I tap chips, I need to know if they are in stock now. 168 00:07:45,120 --> 00:07:49,680 And not just low latency accuracy, we need strong 169 00:07:49,680 --> 00:07:52,200 consistency. Can you define that for us? 170 00:07:52,200 --> 00:07:54,000 What does that mean in this context? 171 00:07:54,320 --> 00:07:58,440 It means if the app says in stock, it must be physically on 172 00:07:58,440 --> 00:08:00,840 the shelf. We cannot sell the same physical 173 00:08:00,840 --> 00:08:02,600 bag of chips to two different people. 174 00:08:02,680 --> 00:08:05,880 We cannot double book inventory. OK, so we have our constraints. 175 00:08:06,080 --> 00:08:09,400 Fast reads, strictly consistent rights, massive scale. 176 00:08:09,560 --> 00:08:12,160 Now we have to actually draw the shapes on whiteboard. 177 00:08:12,160 --> 00:08:14,960 We need to define our data model the nouns and verbs of the 178 00:08:14,960 --> 00:08:16,560 system. This is where I see a lot of 179 00:08:16,560 --> 00:08:19,240 people trip up immediately. They grab the marker and draw a 180 00:08:19,240 --> 00:08:22,960 box called items and in that box they put quantity. 181 00:08:23,120 --> 00:08:25,840 That seems logical. I mean, I have an item, it has a 182 00:08:25,840 --> 00:08:27,560 quantity. Why is that wrong it seems? 183 00:08:27,560 --> 00:08:29,040 Logical. For a grocery list on your 184 00:08:29,040 --> 00:08:32,400 fridge, it is catastrophic. For a distributed system, you 185 00:08:32,400 --> 00:08:34,840 have to distinguish between an item and inventory. 186 00:08:34,960 --> 00:08:36,640 OK, pause. Aren't they the same thing? 187 00:08:37,080 --> 00:08:40,240 If I have a bag of Cheetos, that is the item and it is also the 188 00:08:40,240 --> 00:08:42,600 inventory. Think of it like object oriented 189 00:08:42,600 --> 00:08:46,680 programming, or even simpler, think of it like a menu versus a 190 00:08:46,680 --> 00:08:48,960 meal. Go on an item is the definition. 191 00:08:49,000 --> 00:08:52,480 It is the abstract concept of Flaming Hot Cheetos. 8 oz bag. 192 00:08:53,280 --> 00:08:57,200 It has a name, a description, a photo of the bag, a nutritional 193 00:08:57,200 --> 00:09:00,560 label and a UPC code. And that information is static. 194 00:09:00,640 --> 00:09:03,240 It's the same everywhere. It is the same whether you're in 195 00:09:03,240 --> 00:09:07,160 New York, London or Tokyo. A bag of Cheetos is the bag of 196 00:09:07,160 --> 00:09:09,640 Cheetos. That information is the class or 197 00:09:09,640 --> 00:09:12,640 the definition. It lives in a global product 198 00:09:12,640 --> 00:09:14,640 catalog service. And it rarely changes. 199 00:09:14,800 --> 00:09:16,480 It rarely changes. Maybe once a year they update 200 00:09:16,480 --> 00:09:18,720 the packaging photo. You can cache that data 201 00:09:18,720 --> 00:09:22,160 aggressively and put it on ACDN content delivery network. 202 00:09:22,480 --> 00:09:25,120 You almost never have to query the master database for the 203 00:09:25,120 --> 00:09:27,560 picture of the chips. So what is inventory? 204 00:09:27,760 --> 00:09:30,240 Inventory is the instance. It is the physical reality of 205 00:09:30,240 --> 00:09:32,880 that item at a specific place. It is a row in a database that 206 00:09:32,880 --> 00:09:36,760 says a distribution center #four O2 in Seattle. 207 00:09:37,080 --> 00:09:41,280 On shelf B, there are 42 bags of item hashtag 999. 208 00:09:41,400 --> 00:09:44,760 I see is global inventory is hyperlocal. 209 00:09:44,960 --> 00:09:47,760 Exactly. And inventory is volatile. 210 00:09:47,920 --> 00:09:50,800 It changes every second. Every time someone buys a bag, 211 00:09:50,960 --> 00:09:53,400 that number ticks down. Every time a delivery truck 212 00:09:53,400 --> 00:09:55,720 arrives to restock, that number ticks up. 213 00:09:55,840 --> 00:09:58,320 This distinction matters because of how we store it. 214 00:09:58,480 --> 00:09:59,280 Right. Precisely. 215 00:09:59,640 --> 00:10:01,680 You don't want to store the product description, the text, 216 00:10:01,680 --> 00:10:04,360 the photo URL in the inventory table. 217 00:10:04,520 --> 00:10:07,280 That is a waste of space. The inventory table should be 218 00:10:07,280 --> 00:10:09,240 lean and mean. So it's a high performance 219 00:10:09,240 --> 00:10:10,960 table. It should basically just be 3 220 00:10:10,960 --> 00:10:14,560 columns, Distribution Center died, item died and quantity. 221 00:10:14,680 --> 00:10:16,720 That makes sense. It is a mapping table, so we 222 00:10:16,720 --> 00:10:19,360 have the user, the distribution center, the item, and the 223 00:10:19,360 --> 00:10:22,560 inventory. Now how do these talk to each 224 00:10:22,560 --> 00:10:24,000 other? What does the API look like? 225 00:10:24,160 --> 00:10:25,920 We need to keep it simple for the interview. 226 00:10:26,160 --> 00:10:29,400 We really only need 2 main endpoints to satisfy our scope. 227 00:10:29,480 --> 00:10:31,960 First we need get T availability. 228 00:10:32,240 --> 00:10:33,560 And what does that take as input? 229 00:10:33,760 --> 00:10:36,320 Just the item ID. No, and this is the trap. 230 00:10:36,520 --> 00:10:38,400 If I just ask the server, do you have chips? 231 00:10:38,920 --> 00:10:40,480 The server has to ask where are you? 232 00:10:40,720 --> 00:10:42,000 Right. It doesn't help me if there are 233 00:10:42,000 --> 00:10:44,160 chips in Chicago if I am in Miami. 234 00:10:44,160 --> 00:10:47,320 So the input must include the user's location, latitude and 235 00:10:47,320 --> 00:10:51,600 longitude. OK, input lat, long output. 236 00:10:51,720 --> 00:10:54,520 Yeah, a list of items and there's specific quantities 237 00:10:54,520 --> 00:10:56,400 available to that specific location. 238 00:10:56,400 --> 00:10:58,000 Correct. And the second endpoint. 239 00:10:58,160 --> 00:11:01,280 POST order. This is where the magic happens. 240 00:11:01,720 --> 00:11:05,480 The input is the user ID, the location again, and the list of 241 00:11:05,480 --> 00:11:08,040 items they want. The output is essentially 242 00:11:08,320 --> 00:11:11,320 success or fail. OK, we have the box defined. 243 00:11:11,360 --> 00:11:13,640 We have the data model. Now let's start building the 244 00:11:13,640 --> 00:11:15,160 flow. I open the app. 245 00:11:15,480 --> 00:11:16,880 I want to see if I can get my chips. 246 00:11:17,040 --> 00:11:18,760 What happens? First, the first step is the 247 00:11:18,760 --> 00:11:21,080 read path. The user opens the app and the 248 00:11:21,080 --> 00:11:24,080 request hits our API gateway. Now, before we can check 249 00:11:24,080 --> 00:11:26,200 inventory, we have to figure out where to look. 250 00:11:26,280 --> 00:11:27,920 We need to solve the nearby problem. 251 00:11:28,000 --> 00:11:30,560 Right, which warehouses are close enough to deliver to me in 252 00:11:30,560 --> 00:11:33,240 under an hour? Exactly, if we have 10,000 253 00:11:33,240 --> 00:11:34,880 warehouses, we can't check all of them. 254 00:11:35,120 --> 00:11:38,760 That would be insanely slow. Imagine querying 10,000 database 255 00:11:38,760 --> 00:11:40,600 tables just to load the homepage. 256 00:11:40,600 --> 00:11:43,760 The spinner would spin forever, so we need a dedicated nearby 257 00:11:43,760 --> 00:11:46,600 service. This service does one thing, it 258 00:11:46,600 --> 00:11:49,840 takes a lot long and returns a list of distribution center IDs 259 00:11:49,840 --> 00:11:51,880 that are relevant. How does that actually work? 260 00:11:51,880 --> 00:11:53,840 Do we just draw a circle around the user? 261 00:11:54,000 --> 00:11:57,040 Essentially, yes. In a simple interview answer you 262 00:11:57,040 --> 00:12:00,320 would use geospatial math. You store the warehouses in a 263 00:12:00,320 --> 00:12:04,840 database that supports spatial queries, like Postgres with the 264 00:12:04,840 --> 00:12:07,920 POST GIS extension. I've heard of POST GIS that lets 265 00:12:07,920 --> 00:12:09,920 you treat location like a data type, right? 266 00:12:10,080 --> 00:12:13,360 Exactly, instead of just storing numbers you store a point and 267 00:12:13,360 --> 00:12:17,920 then you can run a query like select from warehouses where STD 268 00:12:18,080 --> 00:12:21,520 within location user location 5 miles. 269 00:12:21,640 --> 00:12:23,960 Wait, STD within that sounds fancy. 270 00:12:23,960 --> 00:12:26,120 What is that doing? It stands for a spatial type 271 00:12:26,120 --> 00:12:29,320 Distance within. Under the hood, the database 272 00:12:29,320 --> 00:12:31,720 isn't calculating the distance to every single warehouse. 273 00:12:31,720 --> 00:12:34,440 That would be way too slow. It uses a spatial index. 274 00:12:34,720 --> 00:12:37,040 Break that down for me. What is a spatial index? 275 00:12:37,040 --> 00:12:38,960 OK, think of a regular index like a phone book. 276 00:12:39,000 --> 00:12:41,600 It's sorted alphabetically so you can find a name fast. 277 00:12:41,800 --> 00:12:43,560 A spatial index is kind of like a grid. 278 00:12:43,720 --> 00:12:46,000 Imagine laying a big grid over the map of the world. 279 00:12:46,000 --> 00:12:49,080 OK, each square of the grid has a list of all the warehouses 280 00:12:49,080 --> 00:12:51,200 inside it. When your request comes in, we 281 00:12:51,200 --> 00:12:52,680 figure out which square you are in. 282 00:12:53,000 --> 00:12:55,640 Then we only have to look at the warehouses in that square and 283 00:12:55,640 --> 00:12:59,600 maybe the 8 squares touching it. So we ignore the warehouses in 284 00:12:59,600 --> 00:13:02,560 the other 99% of the world. Exactly, it makes the look up 285 00:13:02,560 --> 00:13:05,160 incredibly fast. Common implementations are 286 00:13:05,160 --> 00:13:08,040 things like quad trees or geohashes. 287 00:13:08,160 --> 00:13:10,520 OK, so we use a grid system to find the candidates. 288 00:13:10,600 --> 00:13:15,120 We get a list of maybe what 3 or 4 warehouses that are within 5 289 00:13:15,120 --> 00:13:17,600 miles. Right now in the interview you 290 00:13:17,600 --> 00:13:21,080 might mention a constraint check here the prompt usually says one 291 00:13:21,080 --> 00:13:23,920 hour delivery. Does 5 miles equal 1 hour? 292 00:13:23,920 --> 00:13:25,920 Because where I live, definitely not. 293 00:13:25,920 --> 00:13:29,480 Not in Manhattan at 5:00 PM 5 miles in Manhattan is a day 294 00:13:29,480 --> 00:13:32,640 trip. 5 miles in rural Kansas is 5 minutes. 295 00:13:32,640 --> 00:13:35,600 So simple distance isn't enough. And this is where you get bonus 296 00:13:35,600 --> 00:13:37,440 points. You mentioned the Haverseen 297 00:13:37,440 --> 00:13:39,120 formula. The Haverseen formula. 298 00:13:39,160 --> 00:13:41,560 I feel like I learned that in trigonometry and immediately 299 00:13:41,560 --> 00:13:44,040 deleted it from my brain. It is the formula for 300 00:13:44,040 --> 00:13:46,600 calculating the distance between two points on a sphere. 301 00:13:46,960 --> 00:13:50,200 Because the Earth is curved, a straight line on a map isn't 302 00:13:50,200 --> 00:13:53,360 accurate over long distances. Haverseen gives you the as the 303 00:13:53,360 --> 00:13:55,800 crow flies distance. But crows don't have to wait at 304 00:13:55,800 --> 00:13:57,480 traffic lights. Exactly. 305 00:13:57,480 --> 00:14:00,160 So a senior or staff engineer would say this. 306 00:14:00,720 --> 00:14:04,760 For the initial filter I will use Haverseen or a post GIS 307 00:14:04,760 --> 00:14:08,960 query because it is fast. But to be truly accurate, we 308 00:14:09,120 --> 00:14:12,120 would take those top three candidates and call an external 309 00:14:12,120 --> 00:14:15,760 distance matrix API like Google Maps or a proprietary routing 310 00:14:15,760 --> 00:14:18,480 engine to get the real time driving ET. 311 00:14:18,520 --> 00:14:19,720 AI see. So it's a funnel. 312 00:14:19,720 --> 00:14:21,360 Start with thousands of warehouses. 313 00:14:21,560 --> 00:14:23,520 Use the grid to get it down to 50. 314 00:14:23,680 --> 00:14:25,480 Use Haverseen to get it down to five. 315 00:14:25,640 --> 00:14:27,560 Use the traffic API to get the final list. 316 00:14:27,560 --> 00:14:28,840 Perfect. That shows you understand 317 00:14:28,840 --> 00:14:30,960 trade-offs between speed and accuracy. 318 00:14:31,160 --> 00:14:33,600 OK, so the nearby service has done its job. 319 00:14:33,880 --> 00:14:37,560 It returns Warehouse A, Warehouse B and Warehouse C What 320 00:14:37,560 --> 00:14:39,760 happens next? Now we have the union problem. 321 00:14:39,880 --> 00:14:42,800 Warehouse A might have the chip. Warehouse B has the soda. 322 00:14:42,920 --> 00:14:45,640 Warehouse C has the ice cream. But the user needs to see a 323 00:14:45,640 --> 00:14:47,880 unified menu. They don't care which warehouse 324 00:14:47,880 --> 00:14:50,600 it comes from, they just want to know can I get this stuff? 325 00:14:50,800 --> 00:14:53,400 Right, so the availability service has to query all three 326 00:14:53,400 --> 00:14:55,360 of them. It queries the inventory tables 327 00:14:55,360 --> 00:14:58,840 for those specific DCI DS and aggregates the results. 328 00:14:59,040 --> 00:15:00,960 We call this Effective availability. 329 00:15:01,320 --> 00:15:02,880 It is the sum of what is reachable. 330 00:15:03,080 --> 00:15:04,560 But here is where it gets tricky. 331 00:15:05,280 --> 00:15:08,720 We talked about scale earlier, 10 million orders a day means 332 00:15:08,720 --> 00:15:11,320 maybe 100 million or 500 million page views. 333 00:15:11,320 --> 00:15:14,680 Or a billion. Querying the main SQL database 334 00:15:14,680 --> 00:15:17,760 for every single page load sounds like a recipe for 335 00:15:17,760 --> 00:15:18,720 disaster. It is. 336 00:15:18,760 --> 00:15:22,080 It is a massive bottleneck. Relational databases like 337 00:15:22,080 --> 00:15:25,080 Postgres are amazing at consistency, but they are 338 00:15:25,080 --> 00:15:28,040 expensive to scale for massive read throughput. 339 00:15:28,400 --> 00:15:31,480 If every person browsing the app hits the hard drive of the 340 00:15:31,480 --> 00:15:34,720 database, the server will melt. So we need to offload that 341 00:15:34,720 --> 00:15:35,960 traffic. We need caching. 342 00:15:35,960 --> 00:15:38,960 Caching is our best friend here. We need a layer between the 343 00:15:38,960 --> 00:15:42,200 service and the database that stores frequently accessed data 344 00:15:42,200 --> 00:15:44,760 in memory. Redis or Memcached are the 345 00:15:44,760 --> 00:15:47,160 industry standards for this. Walk us through the strategy 346 00:15:47,160 --> 00:15:49,040 here. What exactly are we cashing? 347 00:15:49,040 --> 00:15:51,520 We represent the inventory as a key value pair. 348 00:15:51,800 --> 00:15:54,880 The key would be a combination of the distribution center I and 349 00:15:54,880 --> 00:15:57,760 the item mid. So DC123 item cheat OS and the 350 00:15:57,880 --> 00:16:00,000 value the quantity 50. OK, so when I. 351 00:16:00,040 --> 00:16:01,600 Open the app. What is the flow? 352 00:16:01,600 --> 00:16:03,160 We use a pattern called cash aside. 353 00:16:03,400 --> 00:16:08,480 Step one the app asks the Redis cash do we have chips at DC 123. 354 00:16:08,600 --> 00:16:12,760 Step 2. If the cash says yes 50, we 355 00:16:12,760 --> 00:16:15,240 return that immediately. This takes microseconds. 356 00:16:15,240 --> 00:16:17,400 We don't touch the database. It's a cache hit, right? 357 00:16:17,760 --> 00:16:20,480 Step three. If the cache says I don't know 358 00:16:20,880 --> 00:16:24,600 which we call a cache miss, then and only then do we ask the 359 00:16:24,600 --> 00:16:26,480 database. And once we get the answer from 360 00:16:26,480 --> 00:16:28,840 the database, we. Write it into the cache so the 361 00:16:28,840 --> 00:16:31,320 next person doesn't have to ask the database right? 362 00:16:31,320 --> 00:16:33,800 We populate the cache for future requests. 363 00:16:33,880 --> 00:16:36,440 That saves the database from millions of hits. 364 00:16:37,320 --> 00:16:40,200 But I see a risk here. What if the cash says there are 365 00:16:40,200 --> 00:16:43,080 10 bags of chips, but someone just bought five of them a split 366 00:16:43,080 --> 00:16:45,840 second ago? The database knows the truth, 367 00:16:45,840 --> 00:16:47,840 but the cash is old. It's stale. 368 00:16:47,920 --> 00:16:51,760 That is the stale data problem. There is a famous quote in 369 00:16:51,760 --> 00:16:54,000 computer science. There are only two hard things 370 00:16:54,000 --> 00:16:57,120 in distributed systems, naming things and cache and validation. 371 00:16:57,320 --> 00:16:59,840 So how do we handle it? Do we show the user the wrong 372 00:16:59,840 --> 00:17:02,560 number? For browsing for the read path, 373 00:17:02,720 --> 00:17:04,200 we accept a little bit of staleness. 374 00:17:04,200 --> 00:17:06,839 It is a trade off. It is better to show a slightly 375 00:17:06,839 --> 00:17:10,160 outdated number instantly than to make the user wait 3 seconds 376 00:17:10,160 --> 00:17:12,240 for the perfect number. So we just set a time limit on 377 00:17:12,240 --> 00:17:15,800 the data. Yes, ATTL or time to live. 378 00:17:16,079 --> 00:17:19,040 We might set the cache to expire every minute, so at worst the 379 00:17:19,040 --> 00:17:22,720 data is 60 seconds old. But wait, if I try to buy it and 380 00:17:22,720 --> 00:17:26,000 it isn't there, that is bad. That is a bad user experience. 381 00:17:26,200 --> 00:17:28,840 It is, but it is better than the system being down. 382 00:17:29,360 --> 00:17:34,080 However, we can be smarter. When a purchase happens on the 383 00:17:34,080 --> 00:17:36,760 right path, we can actively fix the cash. 384 00:17:36,800 --> 00:17:39,600 We blast the cash. We invalidate it as soon as the 385 00:17:39,600 --> 00:17:41,400 database is updated with an order. 386 00:17:41,520 --> 00:17:43,600 We delete that specific key in Redis. 387 00:17:43,800 --> 00:17:47,200 The next time someone asks, the system is forced to go to the 388 00:17:47,200 --> 00:17:49,720 database and get the new true number. 389 00:17:49,720 --> 00:17:51,320 OK, so we have handled the browsing. 390 00:17:51,600 --> 00:17:53,760 Millions of people can look at pictures of chips without 391 00:17:53,760 --> 00:17:56,680 crashing the system because we're serving 99% of requests 392 00:17:56,680 --> 00:17:59,000 from Redis, right? But now comes the moment of 393 00:17:59,000 --> 00:18:01,520 truth, the Buy button. The right path. 394 00:18:02,120 --> 00:18:04,000 This is where the interview is won or lost. 395 00:18:04,240 --> 00:18:06,880 I like to call this the Taylor Swift ticket effect, or maybe 396 00:18:06,880 --> 00:18:08,840 the PS-5 launch. You have one item left. 397 00:18:08,840 --> 00:18:10,440 You have two users, Alice and Bob. 398 00:18:10,440 --> 00:18:12,240 They are neighbors. They're both hungry. 399 00:18:12,280 --> 00:18:14,240 It is the classic race condition. 400 00:18:14,440 --> 00:18:16,600 Let's play this out. Alice and Bob both have the app 401 00:18:16,600 --> 00:18:18,480 open. They both refresh their screens 402 00:18:18,480 --> 00:18:22,720 at 11.0000 PM. The system tells both of them 403 00:18:22,920 --> 00:18:24,520 one left. So far so good. 404 00:18:24,760 --> 00:18:29,880 Alice has quick fingers. She taps Buy at 11.000000, point 405 00:18:29,880 --> 00:18:33,720 01. Bob taps Buy at 11, 1.00.01 and 406 00:18:33,720 --> 00:18:36,160 1 millisecond. Yeah, and a naive system. 407 00:18:36,200 --> 00:18:38,680 A system designed by a junior engineer who hasn't dealt with 408 00:18:38,680 --> 00:18:40,680 concurrency. Here's what happens. 409 00:18:40,680 --> 00:18:43,040 Tell me. Alice's request hits the server. 410 00:18:43,040 --> 00:18:44,960 The server reads the database. It sees quantity. 411 00:18:45,200 --> 00:18:47,360 One thinks great one is greater than 0. 412 00:18:47,360 --> 00:18:50,080 Proceed. Alice's process enters the 413 00:18:50,080 --> 00:18:52,480 checkout phase. Maybe it takes 10 milliseconds 414 00:18:52,480 --> 00:18:54,760 to format the order object. And inside those 10 415 00:18:54,760 --> 00:18:56,640 milliseconds, Bob's request arrives. 416 00:18:56,640 --> 00:18:58,320 Exactly. Alice hasn't finished yet. 417 00:18:58,320 --> 00:19:00,920 She hasn't decremented the number, so the database still 418 00:19:00,920 --> 00:19:03,360 says quantity 1. Bob's server thread looks at it 419 00:19:03,360 --> 00:19:05,560 and says great, one is greater than 0. 420 00:19:05,800 --> 00:19:08,720 Proceed. Alice's thread finishes it 421 00:19:08,720 --> 00:19:10,760 subtracts 1. The quantity is now 0. 422 00:19:10,760 --> 00:19:13,360 She gets a confirmation screen chips are on the way. 2 423 00:19:13,360 --> 00:19:15,320 milliseconds later, Bob's thread finishes. 424 00:19:15,320 --> 00:19:18,440 It subtracts 1 from 0. The quantity is now negus 1. 425 00:19:18,560 --> 00:19:20,240 And Bob also gets a confirmation screen. 426 00:19:20,400 --> 00:19:23,880 He does, and now the warehouse manager is staring at an empty 427 00:19:23,880 --> 00:19:26,880 shelf with two orders to fill. And you have a customer service 428 00:19:26,880 --> 00:19:28,480 nightmare. This is the double booking. 429 00:19:28,480 --> 00:19:30,520 Problem. And to solve it, we need 430 00:19:30,520 --> 00:19:32,560 locking. Yeah, we need to stop Bob from 431 00:19:32,560 --> 00:19:34,960 even looking at the shelf until Alice is done. 432 00:19:34,960 --> 00:19:37,640 So we need a bouncer at the door of the database row. 433 00:19:37,640 --> 00:19:40,440 Effectively, yes. In an interview you need to 434 00:19:40,440 --> 00:19:42,720 discuss the two ways to implement this bouncer, 435 00:19:43,160 --> 00:19:45,800 optimistic locking and pessimistic locking. 436 00:19:46,120 --> 00:19:48,800 Let's start with optimistic. That sounds hopeful. 437 00:19:48,880 --> 00:19:52,000 It is hopeful, optimistic. Locking assumes that conflict is 438 00:19:52,000 --> 00:19:54,040 rare. It assumes Alice and Bob aren't 439 00:19:54,040 --> 00:19:55,600 usually fighting over the same bag. 440 00:19:55,600 --> 00:19:57,320 So it doesn't actually lock anything it. 441 00:19:57,320 --> 00:19:59,840 Doesn't actually lock the database row physically, instead 442 00:19:59,840 --> 00:20:02,400 it uses a version number. Like a time stamp similar. 443 00:20:02,560 --> 00:20:05,240 Yeah, imagine every row has a column called version. 444 00:20:05,760 --> 00:20:10,560 Right now it is at version 1. Alice reads the row, she notes 445 00:20:10,600 --> 00:20:13,000 OK, I see one bag and it is version one. 446 00:20:13,120 --> 00:20:15,640 OK, she does her processing. Then she sends a command to the 447 00:20:15,640 --> 00:20:21,400 database update quantity TO0B UT only do this if the version is 448 00:20:21,400 --> 00:20:23,840 still 1. Ah, I see it is a conditional 449 00:20:23,840 --> 00:20:26,320 update, compare and swap. Exactly. 450 00:20:26,560 --> 00:20:29,720 If she succeeds, the database changes the version to two. 451 00:20:30,840 --> 00:20:33,520 Now poor Bob comes in. He also saw version one 452 00:20:33,520 --> 00:20:36,640 originally. He tries to save update quantity 453 00:20:36,640 --> 00:20:40,320 to 0 if version is 1. The database says no. 454 00:20:40,440 --> 00:20:43,640 The database looks at the row and says sorry buddy, the 455 00:20:43,640 --> 00:20:45,800 current version is 2. Your condition failed. 456 00:20:45,840 --> 00:20:48,360 And Bob gets an error message. Sorry, this item is no longer 457 00:20:48,360 --> 00:20:49,240 available. Correct. 458 00:20:49,560 --> 00:20:52,520 The database remains accurate, no negative inventory. 459 00:20:52,640 --> 00:20:54,960 That sounds perfect. Why wouldn't we just use that? 460 00:20:54,960 --> 00:20:57,360 It sounds really fast. Because of the Taylor Swift 461 00:20:57,360 --> 00:20:59,760 problem. Imagine it is not just Alice and 462 00:20:59,760 --> 00:21:02,560 Bob. Imagine it is 1000 people trying 463 00:21:02,560 --> 00:21:05,240 to buy that last ticket. OK, with optimistic locking, 464 00:21:05,280 --> 00:21:08,880 1000 people try 1 succeeds, 999 fail. 465 00:21:09,440 --> 00:21:12,520 But the database still had to process 1000 requests, check the 466 00:21:12,520 --> 00:21:15,640 versions and send back errors. It creates a huge amount of 467 00:21:15,640 --> 00:21:18,440 churn and wasted CPU cycles. It's noisy. 468 00:21:18,560 --> 00:21:22,000 So for high contention items like the PS-5 launch or lunch 469 00:21:22,520 --> 00:21:25,480 rush, optimistic locking might actually overload the database 470 00:21:25,480 --> 00:21:26,400 with failures. Right. 471 00:21:26,400 --> 00:21:29,320 So for high contention systems we prefer pessimistic locking. 472 00:21:29,320 --> 00:21:32,400 This is a safe approach. This is the select for update 473 00:21:32,400 --> 00:21:35,000 command in Sequel. This locks the door as soon as 474 00:21:35,000 --> 00:21:37,720 you walk in. Yes, when Alice's transaction 475 00:21:37,720 --> 00:21:42,520 starts, she tells the database I am reading this row and I intend 476 00:21:42,520 --> 00:21:44,840 to change it. Do not let anyone else touch 477 00:21:44,840 --> 00:21:48,080 this row until I am done. So Bob is just stuck waiting 478 00:21:48,080 --> 00:21:50,400 outside. His request just hangs. 479 00:21:50,560 --> 00:21:53,280 Bob's database transaction will literally pause. 480 00:21:53,280 --> 00:21:56,040 It blocks. It waits until Alice commits her 481 00:21:56,040 --> 00:21:58,760 transaction. Once she finishes, Bob is 482 00:21:58,760 --> 00:22:01,400 allowed in. He reads the row, he sees 483 00:22:01,400 --> 00:22:05,120 quantity 0 and his logic immediately tells him out of 484 00:22:05,120 --> 00:22:07,200 stock. That sounds safer, but doesn't 485 00:22:07,200 --> 00:22:10,360 it slow everything down if Bob is waiting and Charlie is 486 00:22:10,360 --> 00:22:11,720 waiting behind Bob? It does. 487 00:22:11,720 --> 00:22:14,320 It reduces concurrency. It turns parallel traffic into 488 00:22:14,320 --> 00:22:17,400 serial traffic 1 by 1. But for an inventory system 489 00:22:17,400 --> 00:22:20,840 where accuracy is paramount and the business cost of overselling 490 00:22:20,840 --> 00:22:24,120 is high, pessimistic locking is usually the preferred answer in 491 00:22:24,120 --> 00:22:26,280 these interviews. So the guarantee is worth the 492 00:22:26,280 --> 00:22:28,600 performance hit. For this specific problem, yes, 493 00:22:28,840 --> 00:22:31,640 it guarantees that operations happen one at a time in order. 494 00:22:32,000 --> 00:22:33,720 Now I have to ask about something else. 495 00:22:33,720 --> 00:22:36,240 A lot of people online suggest using Redis for locking. 496 00:22:36,520 --> 00:22:39,120 Since Redis is so fast, why not put the lock there? 497 00:22:39,200 --> 00:22:45,040 Bronze Bronze the Redis lock. You don't like it, I can tell. 498 00:22:45,040 --> 00:22:48,200 I hate it for this specific use case and interviewers often hate 499 00:22:48,200 --> 00:22:52,160 it too. Here is why a lot of developers 500 00:22:52,160 --> 00:22:56,280 love Redis because it is fast. So they say hey let's put a lock 501 00:22:56,280 --> 00:22:58,720 in the cache. I will set a key in Redis that 502 00:22:58,720 --> 00:23:02,200 says item locked do my work and then delete the key. 503 00:23:02,440 --> 00:23:04,920 That sounds faster though. Redis is in memory. 504 00:23:04,920 --> 00:23:07,640 It is faster, but it is archintentionally dangerous. 505 00:23:08,320 --> 00:23:09,760 You're creating a distributed lock. 506 00:23:10,240 --> 00:23:11,880 Now you have two sources of truth. 507 00:23:12,040 --> 00:23:14,800 You have the lock in Redis and the data in Postgres. 508 00:23:14,800 --> 00:23:16,400 What happens if they get out of sync? 509 00:23:16,400 --> 00:23:18,520 Disaster. What happens if your server 510 00:23:18,520 --> 00:23:21,320 crashes after running to the database but before releasing 511 00:23:21,320 --> 00:23:23,600 the Redis lock? The lock stays there forever. 512 00:23:23,600 --> 00:23:27,480 Now nobody can buy the item. Or what if Redis crashes? 513 00:23:27,480 --> 00:23:30,880 Or what if there is network lag? The lock might expire before the 514 00:23:30,880 --> 00:23:33,160 database right is finished, letting someone else in. 515 00:23:33,520 --> 00:23:36,640 You lose the ACD properties that this SQL database gives you for 516 00:23:36,640 --> 00:23:38,920 free. So the lesson is keep the source 517 00:23:38,920 --> 00:23:40,880 of truth in one place. Exactly. 518 00:23:41,240 --> 00:23:43,960 Don't reinvent the wheel. Use the database's built in 519 00:23:43,960 --> 00:23:46,440 locking mechanisms. It is what they are there for. 520 00:23:46,440 --> 00:23:48,720 It's battle tested. OK, so we have chosen 521 00:23:48,720 --> 00:23:50,520 pessimistic locking. We are safe. 522 00:23:50,520 --> 00:23:54,080 No double bookings, but we mentioned earlier that we have 523 00:23:54,080 --> 00:23:59,320 10 million orders a day. Can one single Postgres database 524 00:23:59,320 --> 00:24:01,640 handle that? Absolutely not. 525 00:24:02,080 --> 00:24:04,320 A single monolithic database will choke. 526 00:24:04,600 --> 00:24:07,280 We need to scale the right path. This is where we get into 527 00:24:07,280 --> 00:24:10,400 partitioning and sharding. Yes, we need to split the data 528 00:24:10,400 --> 00:24:12,760 up. And for a delivery app, the 529 00:24:12,760 --> 00:24:15,200 strategy is remarkably obvious. Is it? 530 00:24:15,200 --> 00:24:18,000 Think about it, Does a user in New York ever order from a 531 00:24:18,000 --> 00:24:21,400 warehouse in London? No, that would be a very stale 532 00:24:21,400 --> 00:24:23,800 bag of chips. The user in Texas ever order 533 00:24:23,800 --> 00:24:27,200 from a warehouse in Tokyo? No, the data is naturally 534 00:24:27,200 --> 00:24:30,720 isolated by geography, so we use Geo sharding. 535 00:24:30,960 --> 00:24:33,240 We split the database into smaller pieces based on the 536 00:24:33,240 --> 00:24:35,080 region ID or distribution center. 537 00:24:35,080 --> 00:24:38,120 So Shard A handles all transactions for New York, Shard 538 00:24:38,120 --> 00:24:40,080 B handles California. Exactly. 539 00:24:40,080 --> 00:24:41,920 This allows us to scale linearly. 540 00:24:41,920 --> 00:24:44,560 If we get more users in Texas, we add a Texas Shard. 541 00:24:44,920 --> 00:24:47,560 The logic remains the same, but the load is distributed across 542 00:24:47,560 --> 00:24:50,360 different physical servers. New York traffic never touches 543 00:24:50,360 --> 00:24:52,960 the California database. That is elegant, but what about 544 00:24:52,960 --> 00:24:54,520 the read path? Do we Shard that too? 545 00:24:54,760 --> 00:24:57,880 We can, but typically for reads we use read replicas. 546 00:24:57,920 --> 00:25:01,240 OK, what are those? We have one master database for 547 00:25:01,240 --> 00:25:04,760 each Shard that handles the rights, the locking, the 548 00:25:04,760 --> 00:25:07,520 ordering. Then we have multiple slave 549 00:25:07,520 --> 00:25:09,440 copies that just mirror the data. 550 00:25:09,840 --> 00:25:12,120 The availability service, the one checking of chips are in 551 00:25:12,120 --> 00:25:14,840 stock reads from the copies. But there is a delay, right? 552 00:25:14,960 --> 00:25:16,760 The copy isn't updated instantly. 553 00:25:16,800 --> 00:25:19,920 That is called replica lag. It might take a few 554 00:25:19,920 --> 00:25:23,480 milliseconds, maybe even a full second in some cases, for the 555 00:25:23,480 --> 00:25:26,920 master to update the slave. So we are back to the stale data 556 00:25:26,920 --> 00:25:29,120 problem. I buy the chips on the master 557 00:25:29,120 --> 00:25:32,400 but the slave still says in stock for a split second. 558 00:25:32,440 --> 00:25:34,080 Yes. And again, for availability 559 00:25:34,080 --> 00:25:36,440 checks, we accept that eventual consistency. 560 00:25:37,040 --> 00:25:38,680 But here is a pro tip for the interview. 561 00:25:38,920 --> 00:25:40,960 You can use something called sticky sessions. 562 00:25:40,960 --> 00:25:43,560 Sticky sessions. Imagine you just bought an item. 563 00:25:44,000 --> 00:25:48,160 For the next minute, the system pins your user ID to the master 564 00:25:48,160 --> 00:25:50,440 database. So for that specific user, we 565 00:25:50,440 --> 00:25:52,760 skip the replica and go straight to the source of truth. 566 00:25:53,040 --> 00:25:54,160 Exactly. Yeah. 567 00:25:54,200 --> 00:25:57,440 That way if you refresh the page immediately after buying, you 568 00:25:57,440 --> 00:25:59,880 see the accurate data because you are looking at the master. 569 00:26:00,280 --> 00:26:02,760 Everyone else browsing can still use the replicas. 570 00:26:02,760 --> 00:26:05,680 That is a clever optimization. It keeps the user experience 571 00:26:05,680 --> 00:26:08,360 smooth without over loading the master for everyone. 572 00:26:08,520 --> 00:26:11,120 It balances the load. It shows you're thinking about 573 00:26:11,120 --> 00:26:13,920 the user's perception. So let's zoom out and look at 574 00:26:13,920 --> 00:26:15,720 the whole architecture we have built. 575 00:26:15,800 --> 00:26:17,880 It is quite a beast. It is. 576 00:26:18,200 --> 00:26:20,280 Let's trace a user journey from start to finish. 577 00:26:20,280 --> 00:26:23,920 Do it #1 user login, they open the app. 578 00:26:24,680 --> 00:26:27,600 The nearby service uses a geospatial index to find the 579 00:26:27,600 --> 00:26:29,040 three closest. Warehouses. 580 00:26:29,040 --> 00:26:30,560 Step 2. Browsing. 581 00:26:30,800 --> 00:26:33,560 The availability service takes those three warehouse ID's. 582 00:26:33,920 --> 00:26:37,280 It checks the Redis cache first. If data is missing, it hits the 583 00:26:37,280 --> 00:26:39,040 read replicas of the database shards. 584 00:26:39,160 --> 00:26:42,000 It sums up the inventory and shows chips available. 585 00:26:42,200 --> 00:26:45,280 Step 3. The moment of truth buying. 586 00:26:45,360 --> 00:26:48,640 The user taps buy, The order service takes over. 587 00:26:49,200 --> 00:26:51,480 It identifies the correct database Shard based on 588 00:26:51,480 --> 00:26:53,760 location. It opens a transaction. 589 00:26:54,360 --> 00:26:57,640 It acquires A pessimistic row lock on the inventory items. 590 00:26:58,040 --> 00:27:01,320 It checks stock. If good, it decrements stock, 591 00:27:01,320 --> 00:27:03,320 creates an order record, and commits. 592 00:27:03,720 --> 00:27:07,240 And finally clean up. The system fires an event to 593 00:27:07,240 --> 00:27:10,320 invalidate the Redis cache for those items, so the next 594 00:27:10,320 --> 00:27:12,160 browsing user sees the new numbers. 595 00:27:12,200 --> 00:27:15,960 That is a robust system. It handles the scale, it handles 596 00:27:15,960 --> 00:27:18,120 the concurrency, and it keeps the data safe. 597 00:27:18,120 --> 00:27:20,280 It is a passing answer. This passing that felt pretty 598 00:27:20,280 --> 00:27:22,840 comprehensive. Well, in an interview they grade 599 00:27:22,840 --> 00:27:25,160 you on levels. What we just described is a 600 00:27:25,160 --> 00:27:28,480 solid senior engineer answer. What does a junior answer look 601 00:27:28,480 --> 00:27:30,560 like? A junior or mid level engineer 602 00:27:30,560 --> 00:27:33,480 will usually get the API right. They will say we need a database 603 00:27:33,480 --> 00:27:36,400 table for items, but they might forget about the race 604 00:27:36,400 --> 00:27:39,160 conditions, the double booking. They missed the concurrency. 605 00:27:39,160 --> 00:27:41,640 They missed the concurrency, or they might suggest a naive 606 00:27:41,640 --> 00:27:43,880 distance calculation that kills the CPU. 607 00:27:44,200 --> 00:27:46,560 They focus on features, not failure modes. 608 00:27:46,680 --> 00:27:49,160 And what about the staff or principal engineer? 609 00:27:49,320 --> 00:27:51,160 How do you blow the interviewer away? 610 00:27:51,320 --> 00:27:54,280 The staff engineer looks at the edge cases and the business 611 00:27:54,280 --> 00:27:56,240 logic. They talk about the union of 612 00:27:56,240 --> 00:28:00,360 inventory, how to handle partial orders, where the chips come 613 00:28:00,360 --> 00:28:03,280 from warehouse A and the soda comes from warehouse B. 614 00:28:03,280 --> 00:28:05,960 Oh, that's a whole new problem. Do we split the shipment? 615 00:28:06,160 --> 00:28:08,840 Do we delay it? That gets into the logistics 616 00:28:08,840 --> 00:28:11,760 complexity. They might also discuss advanced 617 00:28:11,760 --> 00:28:14,840 locking strategies based on item popularity. 618 00:28:15,160 --> 00:28:17,560 What do you mean? They might say, maybe we use 619 00:28:17,560 --> 00:28:20,160 optimistic locking for toothpaste because no one fights 620 00:28:20,160 --> 00:28:24,800 over toothpaste, but pessimistic locking for the hot new video 621 00:28:24,800 --> 00:28:27,120 game, they nuance the architecture. 622 00:28:27,120 --> 00:28:29,480 They show they're not just applying a single rule to 623 00:28:29,480 --> 00:28:30,440 everything. Right. 624 00:28:30,760 --> 00:28:32,840 And they might also bring up the academic research. 625 00:28:33,200 --> 00:28:36,200 A staff engineer might mention the Calvin protocol, for 626 00:28:36,200 --> 00:28:38,480 instance. Calvin, I remember that from one 627 00:28:38,480 --> 00:28:41,320 of the papers we looked at. Can you summarize that for us? 628 00:28:41,400 --> 00:28:44,160 Calvin is a deterministic ordering system for distributed 629 00:28:44,160 --> 00:28:46,600 transactions. It's a different way to think 630 00:28:46,600 --> 00:28:49,800 about concurrency. It basically eliminates the need 631 00:28:49,800 --> 00:28:53,920 for complex commit protocols like 2 phase commit by agreeing 632 00:28:53,920 --> 00:28:57,080 on the order of transactions before they are even executed. 633 00:28:57,280 --> 00:29:00,600 O it's a scheduler. A very smart 1A staff engineer 634 00:29:00,600 --> 00:29:04,840 might say if we really wanted to scale globally without some of 635 00:29:04,840 --> 00:29:07,560 the headaches of traditional sharding, we could look at a 636 00:29:07,560 --> 00:29:09,600 deterministic scheduler like Calvin. 637 00:29:10,480 --> 00:29:13,200 It shows they know the cutting edge research, even if they 638 00:29:13,200 --> 00:29:15,480 stick to Postgres for the practical interview answer. 639 00:29:15,560 --> 00:29:17,400 That is the extra credit knowledge. 640 00:29:17,400 --> 00:29:19,880 It shows you aren't just a coder, you are a student of the 641 00:29:19,880 --> 00:29:21,040 craft. Exactly. 642 00:29:21,040 --> 00:29:23,880 Before we wrap up, I want to pivot to a thought that came up 643 00:29:23,880 --> 00:29:27,240 in our research. We have spent this whole time 644 00:29:27,240 --> 00:29:29,360 designing a system to be perfectly accurate. 645 00:29:29,360 --> 00:29:32,080 We don't want to oversell a single bag of chips. 646 00:29:32,120 --> 00:29:33,640 Right. Strong consistency. 647 00:29:33,640 --> 00:29:37,560 We've been hammering that .0 But does the business actually care? 648 00:29:37,640 --> 00:29:39,880 That is the final provocative question. 649 00:29:40,440 --> 00:29:44,000 In the real world, business logic often overrides system 650 00:29:44,000 --> 00:29:45,280 design. What do you mean by that? 651 00:29:45,400 --> 00:29:47,680 Think about it. What is the cost of a database 652 00:29:47,680 --> 00:29:49,640 lock? It slows down the system. 653 00:29:49,640 --> 00:29:51,680 It limits how many orders you can take per second. 654 00:29:52,080 --> 00:29:54,920 It adds friction. Now what is the cost of 655 00:29:54,920 --> 00:29:57,840 overselling one bag of ships? Well, you have to e-mail the 656 00:29:57,840 --> 00:30:01,760 customer, Say sorry, refund them their $3, and maybe give them a 657 00:30:01,760 --> 00:30:05,280 $5 coupon for their next. Order exactly for a bag of 658 00:30:05,280 --> 00:30:07,120 chips. It might actually be more 659 00:30:07,120 --> 00:30:11,120 profitable to run a loose optimistic system, process 660 00:30:11,120 --> 00:30:14,880 orders as fast as possible, and just apologize to the point 1% 661 00:30:14,880 --> 00:30:17,800 of people you oversell to. The cost of the apology might be 662 00:30:17,800 --> 00:30:21,160 lower than the cost of the walk. So we build a Ferrari, but maybe 663 00:30:21,160 --> 00:30:22,920 the business just needed a delivery van. 664 00:30:23,120 --> 00:30:25,760 It depends on the item. For Taylor Swift tickets you 665 00:30:25,760 --> 00:30:27,840 need the Ferrari. You cannot oversell a seat. 666 00:30:28,120 --> 00:30:30,560 The stadium has a physical limit for chips. 667 00:30:31,040 --> 00:30:32,320 The delivery again might be better. 668 00:30:32,840 --> 00:30:35,280 True seniority is knowing when to break the rules we just 669 00:30:35,280 --> 00:30:37,280 designed. That is a great perspective. 670 00:30:37,280 --> 00:30:40,000 It isn't just about code, it is about the product and the 671 00:30:40,000 --> 00:30:41,400 business context. Always. 672 00:30:41,720 --> 00:30:44,960 Well, we have built the back end of a logistics giant in under an 673 00:30:44,960 --> 00:30:47,560 hour. We have covered scope, data 674 00:30:47,560 --> 00:30:49,600 modelling, caching, locking and sharding. 675 00:30:49,800 --> 00:30:52,320 It is a lot to take in, but if you structure your answer like 676 00:30:52,320 --> 00:30:56,720 this, scope, model, read, write, scale, you will ace that 677 00:30:56,720 --> 00:30:58,360 interview. Good luck to everyone out there 678 00:30:58,360 --> 00:30:59,680 preparing. Go build it. 679 00:30:59,880 --> 00:31:01,920 Thanks for listening to the deep dive. 680 00:31:02,160 --> 00:31:02,920 See you next time.