1 00:00:00,040 --> 00:00:03,200 You know, I was looking through the stack of papers and blog 2 00:00:03,200 --> 00:00:08,119 posts you sent over for this dive, and it it hit me that 3 00:00:08,119 --> 00:00:10,920 we've reached a really strange inflection point in AI. 4 00:00:11,160 --> 00:00:12,400 Oh yeah. Yeah. 5 00:00:12,440 --> 00:00:15,640 I mean, we've spent the last 10, maybe 15 years completely 6 00:00:15,640 --> 00:00:18,840 obsessed with one thing, accuracy. 7 00:00:18,920 --> 00:00:20,800 Right, just get the loss function down. 8 00:00:20,800 --> 00:00:22,400 Exactly. It's been this race to the 9 00:00:22,400 --> 00:00:25,640 bottom for loss. Get the accuracy up, get the AUC 10 00:00:25,640 --> 00:00:28,720 up, and you know who cares what happens inside the layers? 11 00:00:28,760 --> 00:00:30,360 Right. It was kind of the if it works, 12 00:00:30,400 --> 00:00:31,920 don't touch it phase of engineering. 13 00:00:31,920 --> 00:00:33,400 Totally. As long as the predictions were 14 00:00:33,400 --> 00:00:35,720 good on the test set, nobody asked too many questions. 15 00:00:36,400 --> 00:00:39,080 But now, based on everything we're seeing in the industry, 16 00:00:39,440 --> 00:00:43,080 the bill is coming due. We have these these massive 17 00:00:43,400 --> 00:00:46,880 models, gradient boosted trees, deep neural networks with 18 00:00:46,880 --> 00:00:48,920 millions of parameters. And they are making decisions 19 00:00:48,920 --> 00:00:50,880 that actually matter. They really matter. 20 00:00:51,160 --> 00:00:52,920 We aren't just predicting ad clicks anymore. 21 00:00:52,920 --> 00:00:56,520 We're talking about loan approvals, medical diagnosis, 22 00:00:56,680 --> 00:00:59,280 autonomous driving. And that is where the black box 23 00:00:59,280 --> 00:01:03,040 problem goes from being, you know, an academic curiosity to a 24 00:01:03,040 --> 00:01:06,600 massive liability because eventually a stakeholder or a 25 00:01:06,600 --> 00:01:09,320 regulator or a doctor is going to look at a prediction and ask 26 00:01:09,440 --> 00:01:10,240 why. Why? 27 00:01:10,280 --> 00:01:11,800 Why did the model deny this loan? 28 00:01:11,960 --> 00:01:14,240 Why did it flag this patient for surgery? 29 00:01:14,280 --> 00:01:17,640 And usually the engineer just stares at the floor or they show 30 00:01:17,640 --> 00:01:20,520 a confusion matrix which doesn't answer the question at all. 31 00:01:20,520 --> 00:01:21,840 Not even close. I don't know. 32 00:01:22,120 --> 00:01:24,720 It isn't an acceptable answer when you're dealing with 33 00:01:24,720 --> 00:01:28,080 someone's life savings or you know their health. 34 00:01:28,440 --> 00:01:31,000 Precisely. Transparency isn't just a nice 35 00:01:31,000 --> 00:01:33,240 to have feature anymore, it's a necessity. 36 00:01:33,600 --> 00:01:35,920 We need to move from. I know that it works to. 37 00:01:36,040 --> 00:01:38,600 I know how it works. And that is the mission for 38 00:01:38,600 --> 00:01:41,120 today. We're going to demystify a tool 39 00:01:41,360 --> 00:01:44,840 that has, I think you could argue, become the industry 40 00:01:44,840 --> 00:01:46,880 standard for cracking open that black box. 41 00:01:46,880 --> 00:01:49,920 We're talking about SH. A shapely additive explanations. 42 00:01:49,920 --> 00:01:52,480 It is a mouthful, but the concept behind it is just 43 00:01:52,520 --> 00:01:54,840 fascinating. What I love about this topic is 44 00:01:54,840 --> 00:01:58,160 that it connects modern cutting edge machine learning back to 45 00:01:58,680 --> 00:02:01,840 what Nobel Prize winning game theory from the 1950s. 46 00:02:01,840 --> 00:02:05,600 It really is the bridge between the rigorous math of fairness 47 00:02:05,600 --> 00:02:09,960 and the the messy reality of data engineering. 48 00:02:10,240 --> 00:02:12,880 So here is the plan. We're going to break this down 49 00:02:12,880 --> 00:02:15,360 into a few distinct parts. We're going to start with the 50 00:02:15,360 --> 00:02:18,400 intuition, the robo runners analogy, which I think is the 51 00:02:18,400 --> 00:02:20,160 best way to visualize the math. It's a great one. 52 00:02:20,160 --> 00:02:20,920 Math. Then we'll get into the 53 00:02:20,920 --> 00:02:23,880 mechanics of how you actually calculate this, specifically 54 00:02:23,880 --> 00:02:25,720 tackling that marginalization problem. 55 00:02:26,000 --> 00:02:27,560 We'll look at the visualization toolkit. 56 00:02:27,920 --> 00:02:31,160 And finally, we have to talk about the algorithms, tree shape 57 00:02:31,160 --> 00:02:34,080 versus kernel app, and of course the pitfalls. 58 00:02:34,320 --> 00:02:35,720 And the pitfalls are significant. 59 00:02:35,720 --> 00:02:38,760 It's not a magic wand. No, it's definitely not. 60 00:02:39,000 --> 00:02:40,560 But let's start with the origins story. 61 00:02:40,600 --> 00:02:44,720 So SHAP was introduced in a paper by Scott Lundberg and Sue 62 00:02:44,720 --> 00:02:49,240 and Lee around 2017, but the math they used wasn't new at 63 00:02:49,240 --> 00:02:50,400 all. Not at all, no. 64 00:02:50,400 --> 00:02:53,280 They basically went dumpster diving in 1950s economics. 65 00:02:53,760 --> 00:02:56,480 They realize that the problem of debugging a complex machine 66 00:02:56,480 --> 00:02:59,840 learning model is well, it's mathematically identical to a 67 00:02:59,840 --> 00:03:03,440 problem in cooperative game theory that Lloyd Shapely solved 68 00:03:03,440 --> 00:03:07,200 back in 1953. Lloyd Shapely, who later won the 69 00:03:07,200 --> 00:03:09,520 Nobel Prize for this very work. Correct. 70 00:03:09,800 --> 00:03:12,880 Shapely was interested in a very specific scenario. 71 00:03:13,320 --> 00:03:15,720 Imagine you have a team of people working together to 72 00:03:15,720 --> 00:03:18,280 achieve a goal. They cooperate and together they 73 00:03:18,280 --> 00:03:21,520 generate some value. A payout, a prize, a win. 74 00:03:21,920 --> 00:03:26,000 The core question Shapley tried to answer was how do you fairly 75 00:03:26,000 --> 00:03:29,640 distribute that payout among the team members when everyone 76 00:03:29,640 --> 00:03:32,600 contributed differently and. Fairly, here's a mathematical 77 00:03:32,600 --> 00:03:34,360 term, right? Not just a feeling. 78 00:03:34,440 --> 00:03:37,040 Exactly. It has to satisfy specific 79 00:03:37,040 --> 00:03:39,440 axioms of fairness. It's rigorous. 80 00:03:39,680 --> 00:03:43,160 So to wrap our heads around this, let's use the mental model 81 00:03:43,160 --> 00:03:45,920 from the Visoira lecture, the Robo Runners. 82 00:03:45,920 --> 00:03:49,360 This is a classic setup, so imagine a robotics team called 83 00:03:49,360 --> 00:03:52,240 the Robo Runners. There are four members on this 84 00:03:52,240 --> 00:03:54,120 team. OK, they enter competition to 85 00:03:54,120 --> 00:03:56,600 build a robot. They work together and they win 86 00:03:56,600 --> 00:03:59,680 first prize. They take home $10,000. 87 00:03:59,680 --> 00:04:02,040 Simple enough. Four people, 10 grand in the 88 00:04:02,040 --> 00:04:04,320 bank. Now comes the problem payday. 89 00:04:04,680 --> 00:04:07,280 How do you split the $10,000? I mean if I'm the manager and I 90 00:04:07,280 --> 00:04:09,040 just want to go home, I divide by 4. 91 00:04:09,280 --> 00:04:13,240 Everyone gets 2500 bucks. The naive approach, but is it 92 00:04:13,240 --> 00:04:15,000 fair? Probably not. 93 00:04:15,280 --> 00:04:17,839 Let's say one person was the lead engineer who built the 94 00:04:17,839 --> 00:04:20,800 navigation system from scratch, and another person was. 95 00:04:21,760 --> 00:04:23,440 And let's say the person who brought the snacks. 96 00:04:23,640 --> 00:04:26,720 Right. If you give the snack guy $2500, 97 00:04:26,960 --> 00:04:29,840 the lead engineer is going to riot, and rightly so. 98 00:04:30,120 --> 00:04:33,320 The equal split ignores the reality that contributions are 99 00:04:33,320 --> 00:04:36,200 unequal, so that's out. OK, so we need a metric. 100 00:04:37,000 --> 00:04:39,920 If I'm trying to be analytical about this, my next instinct is 101 00:04:39,920 --> 00:04:44,720 to try a a subtraction method. OK, I'd look at the team and I 102 00:04:44,720 --> 00:04:49,760 ask how much value did Bob, let's call the hardware guy Bob, 103 00:04:49,760 --> 00:04:51,800 add to the group? And how would you calculate 104 00:04:51,840 --> 00:04:53,360 that? I'd simulate the competition 105 00:04:53,360 --> 00:04:56,280 without Bob. I'd say OK, the full team wins 106 00:04:56,280 --> 00:04:58,880 10,000. If Bob stays home, the remaining 107 00:04:58,880 --> 00:05:02,080 three people aren't as good. Maybe they only win second place 108 00:05:02,080 --> 00:05:04,920 which is $6000. So the full team gets 10,000, 109 00:05:04,920 --> 00:05:09,960 the team without Bob gets 6000. So the logic is 10,000 -, 6000 110 00:05:09,960 --> 00:05:12,840 is 4000. Therefore Bob is worth exactly 111 00:05:12,840 --> 00:05:14,800 $4000. That's his marginal 112 00:05:14,800 --> 00:05:16,960 contribution. It feels intuitive, doesn't it? 113 00:05:16,960 --> 00:05:18,400 You're just measuring the drop off. 114 00:05:18,600 --> 00:05:20,880 But there's a massive flaw in that logic. 115 00:05:21,120 --> 00:05:23,440 I suspect it has to do with the other team members. 116 00:05:23,560 --> 00:05:26,920 It's all about interactions. That subtraction method assumes 117 00:05:26,920 --> 00:05:31,640 Bob's value is static, that he is worth $4000 regardless of who 118 00:05:31,640 --> 00:05:33,520 else is in the room. But think about it. 119 00:05:33,680 --> 00:05:37,560 Bob is the hardware expert. He builds the physical robot, 120 00:05:37,640 --> 00:05:39,560 right? Now imagine another team member, 121 00:05:39,560 --> 00:05:41,640 Alice. She's the algorithm expert. 122 00:05:41,640 --> 00:05:45,480 She writes the code. OK, if Alice is also missing, 123 00:05:45,640 --> 00:05:48,120 what is Bob's value? Well, if there's no code, the 124 00:05:48,120 --> 00:05:51,440 robot is just a paperweight. Doesn't move, so Bob's hardware 125 00:05:51,440 --> 00:05:53,400 is. Useless exactly. 126 00:05:53,760 --> 00:05:57,200 If Alice isn't there, Bob's marginal contribution might be 127 00:05:57,200 --> 00:05:59,560 0. But if Alice is there, Bob's 128 00:05:59,560 --> 00:06:02,320 contribution is huge because her code has something to control. 129 00:06:02,400 --> 00:06:04,400 I see. So the team minus Bob 130 00:06:04,680 --> 00:06:08,200 calculation fails because it ignores the context. 131 00:06:08,600 --> 00:06:11,400 It only looks at one specific scenario, the full team versus 132 00:06:11,400 --> 00:06:14,280 the full team without Bob. It doesn't account for the fact 133 00:06:14,280 --> 00:06:16,120 that Bob needs Alice to be valuable. 134 00:06:16,120 --> 00:06:18,160 Precisely. And this is the genius of the 135 00:06:18,160 --> 00:06:20,920 Shapley solution. To find the true fair 136 00:06:20,920 --> 00:06:23,400 contribution of the hardware expert, you cannot just look at 137 00:06:23,400 --> 00:06:24,920 the full team. You have to look at every 138 00:06:24,960 --> 00:06:28,080 possible subset of the team. Every possible subset. 139 00:06:28,080 --> 00:06:30,880 Wow, walk me through that. You have to calculate Bob's 140 00:06:30,880 --> 00:06:32,760 impact in every possible reality. 141 00:06:33,120 --> 00:06:35,840 You measure his contribution when he works entirely alone. 142 00:06:35,840 --> 00:06:38,000 Maybe he wins nothing. Then you measure his 143 00:06:38,000 --> 00:06:41,280 contribution when he joins a team of just the snack guy, then 144 00:06:41,280 --> 00:06:44,480 when he joins just Alice, then when he joins Alice and the 145 00:06:44,480 --> 00:06:46,600 snack guy. Whoa, so you were running a 146 00:06:46,600 --> 00:06:49,160 simulation for every single permutation of the group? 147 00:06:49,200 --> 00:06:52,400 Yes, you calculate the difference his presence makes in 148 00:06:52,400 --> 00:06:55,720 every single combination. And then, and this is the key, 149 00:06:56,000 --> 00:06:58,440 you take the weighted average of all those marginal 150 00:06:58,440 --> 00:07:01,520 contributions. That average is the Shapley 151 00:07:01,520 --> 00:07:03,400 value. And that number is 152 00:07:03,400 --> 00:07:06,280 mathematically proven to be the only fairway to distribute the 153 00:07:06,280 --> 00:07:08,240 payout. Yes, Lloyd Shapley proved that 154 00:07:08,240 --> 00:07:11,760 if you want to satisfy the axioms of efficiency, symmetry 155 00:07:11,760 --> 00:07:14,160 and linearity, this is the unique solution. 156 00:07:14,160 --> 00:07:15,840 There is no other way to do it. OK. 157 00:07:15,840 --> 00:07:18,800 So that's the intuition. We have the robo runners and we 158 00:07:18,800 --> 00:07:21,440 have a way to fairly split the $10,000. 159 00:07:22,280 --> 00:07:25,360 Now we have to make the jump to machine learning because our 160 00:07:25,360 --> 00:07:28,440 listeners aren't managing robotics teams, they're building 161 00:07:28,440 --> 00:07:30,480 XG boost models. How does this map? 162 00:07:30,720 --> 00:07:34,960 It is a direct mapping. In this analogy, the players on 163 00:07:34,960 --> 00:07:37,280 the team are the features in your data set. 164 00:07:37,520 --> 00:07:42,840 Features meaning the columns age, BMI, income, zip code, that 165 00:07:42,840 --> 00:07:43,520 stuff. Correct. 166 00:07:43,800 --> 00:07:47,720 And the payout, the $10,000 prize is the models prediction. 167 00:07:48,440 --> 00:07:52,400 But wait, let's be precise here. Is the payout the raw prediction 168 00:07:52,400 --> 00:07:56,200 itself like the 80% probability? That's a great catch. 169 00:07:56,200 --> 00:07:59,400 No. Technically the payout we are 170 00:07:59,400 --> 00:08:02,640 distributing is the difference between the prediction for a 171 00:08:02,640 --> 00:08:06,000 specific instance and the average prediction of the data 172 00:08:06,000 --> 00:08:07,280 set. OK, let's unpack that. 173 00:08:07,520 --> 00:08:10,280 Suppose we are predicting the probability of diabetes. 174 00:08:10,720 --> 00:08:14,440 The average person in our data set has, let's say a 10% risk. 175 00:08:14,480 --> 00:08:15,920 That's our baseline, right? That's. 176 00:08:15,920 --> 00:08:18,240 What the model would predict if it knew nothing about a person? 177 00:08:18,440 --> 00:08:22,120 We feed my data into the model and the model spits out an 80% 178 00:08:22,120 --> 00:08:23,960 risk. So I'm at 80%. 179 00:08:23,960 --> 00:08:27,720 The baseline is 10%, the payout. The thing we need to explain is 180 00:08:27,720 --> 00:08:29,920 that 70% gap. Exactly. 181 00:08:29,920 --> 00:08:33,559 The question is how do we get from 10% to 80%? 182 00:08:33,919 --> 00:08:37,480 We have to distribute that 70 percentage points of credit or 183 00:08:37,480 --> 00:08:40,400 blame. Among your features, how much 184 00:08:40,400 --> 00:08:42,760 did your age contribute? How much did your BMI 185 00:08:42,760 --> 00:08:45,000 contribute? And just like the robot team, 186 00:08:45,200 --> 00:08:48,960 these features interact. My high BMI might not be a huge 187 00:08:48,960 --> 00:08:52,080 risk factor on its own, but combined with my age it might 188 00:08:52,080 --> 00:08:54,960 just explode the risk. Recisely high blood pressure 189 00:08:54,960 --> 00:08:57,400 routers more if you are older than if you are younger. 190 00:08:58,120 --> 00:09:01,280 The model, assuming it's a good model, captures these complex 191 00:09:01,280 --> 00:09:05,680 non linear interactions and SHA untangles them by running those 192 00:09:05,680 --> 00:09:09,120 subset calculations. It simulates the model with just 193 00:09:09,200 --> 00:09:13,800 age, then agent BMI, then agent income to find the true marginal 194 00:09:13,800 --> 00:09:17,120 contribution of each factor. That is such a cool way to frame 195 00:09:17,120 --> 00:09:19,040 it, but I want to highlight something you mentioned in the 196 00:09:19,040 --> 00:09:22,280 notes, the difference between local and global explanations 197 00:09:22,680 --> 00:09:24,480 because I see people get this mixed up all the time. 198 00:09:24,480 --> 00:09:26,240 They do. It's a critical distinction. 199 00:09:26,440 --> 00:09:28,960 Global explanations tell you how the model works in general. 200 00:09:29,280 --> 00:09:31,920 For example, a feature importance plot in a random 201 00:09:31,920 --> 00:09:34,880 forest might tell you overall income is the most important 202 00:09:34,880 --> 00:09:37,120 predictor. Which is like saying smoking is 203 00:09:37,120 --> 00:09:40,000 generally bad for your health. Right, it's a statistical 204 00:09:40,000 --> 00:09:42,520 statement about the population, but that doesn't help you 205 00:09:42,520 --> 00:09:44,680 understand a specific prediction. 206 00:09:44,880 --> 00:09:47,880 If you have a specific applicant who is denied a loan, telling 207 00:09:47,880 --> 00:09:51,000 them income is generally important doesn't explain their 208 00:09:51,000 --> 00:09:53,560 denial. Maybe their income was fine. 209 00:09:53,560 --> 00:09:55,520 Right. Maybe for this specific person 210 00:09:55,640 --> 00:09:58,000 it was their debt to income ratio that sank them. 211 00:09:58,080 --> 00:10:01,040 Exactly. SHAP provides local 212 00:10:01,040 --> 00:10:03,680 explanations. It explains this specific 213 00:10:03,680 --> 00:10:06,480 instance. Why was the 65 year old 214 00:10:06,720 --> 00:10:10,480 classified as diabetic? It allows you to debug the model 215 00:10:10,520 --> 00:10:12,960 at the row level. Which is incredibly powerful. 216 00:10:12,960 --> 00:10:16,400 But yeah, and here is where the rubber meets the road for me. 217 00:10:16,480 --> 00:10:18,200 I have a practical engineering question. 218 00:10:18,800 --> 00:10:21,480 In the robo runners example, we said we calculate these subsets 219 00:10:21,480 --> 00:10:25,000 by removing a person. I can physically tell Bob, hey, 220 00:10:25,200 --> 00:10:27,400 go wait in the hall, we're going to build the robot without you. 221 00:10:28,000 --> 00:10:30,800 But in a machine learning model, specifically something like a 222 00:10:30,800 --> 00:10:33,200 neural net, I have a matrix of data. 223 00:10:33,400 --> 00:10:36,000 I have a fixed input layer. It expects 50 numbers. 224 00:10:36,360 --> 00:10:39,440 I can't just delete the age column for one row and expect 225 00:10:39,440 --> 00:10:41,480 the model to run. The math will fail. 226 00:10:41,560 --> 00:10:45,560 The model will crash. So how do we remove a feature to 227 00:10:45,560 --> 00:10:48,120 calculate these subsets? You've hit on the central 228 00:10:48,120 --> 00:10:50,800 engineering challenge of SHA and you are right. 229 00:10:50,800 --> 00:10:55,000 Most models cannot handle missing data natively, so we 230 00:10:55,000 --> 00:10:56,720 have to fake it. We use a technique called 231 00:10:56,720 --> 00:10:58,400 marginalization. Marginalization. 232 00:10:58,800 --> 00:11:02,200 OK, unpack that. How do we mask a feature without 233 00:11:02,200 --> 00:11:05,600 breaking the model? We simulate the feature being 234 00:11:05,600 --> 00:11:09,200 missing by replacing its real value with a random value from a 235 00:11:09,200 --> 00:11:11,480 background data set, usually the training data. 236 00:11:11,480 --> 00:11:13,840 A random value, so we're just injecting noise. 237 00:11:13,840 --> 00:11:16,080 In a way, yeah. Think of it as Monte Carlo 238 00:11:16,080 --> 00:11:18,160 integration. Let's say we want to measure the 239 00:11:18,160 --> 00:11:21,280 importance of your age. We want to simulate a team 240 00:11:21,440 --> 00:11:25,080 without your age, so we keep your BMI, your blood pressure, 241 00:11:25,080 --> 00:11:26,760 and your income exactly as they are. 242 00:11:26,760 --> 00:11:28,000 Those are the fixed team members. 243 00:11:28,320 --> 00:11:30,360 But for your age, we swap it out. 244 00:11:30,400 --> 00:11:32,240 Swap it with what? We might grab the age of a 245 00:11:32,240 --> 00:11:33,840 random person from the training data. 246 00:11:34,080 --> 00:11:36,440 Let's say we pull a record and it's a 25 year old. 247 00:11:36,440 --> 00:11:38,040 We plug that in and run the prediction. 248 00:11:38,040 --> 00:11:40,720 Then we pull another record, a 50 year old, run the prediction, 249 00:11:40,880 --> 00:11:44,800 then an 80 year old, and so on. So we are creating these 250 00:11:44,800 --> 00:11:49,440 synthetic Frankenstein data points where everything is me 251 00:11:49,760 --> 00:11:52,880 except my age is just morphing into different people from the 252 00:11:52,880 --> 00:11:54,240 database. Exactly. 253 00:11:54,600 --> 00:11:57,520 We are integrating over the distribution of that feature. 254 00:11:57,680 --> 00:12:00,520 We are asking the model. If we didn't know your age and 255 00:12:00,520 --> 00:12:03,920 just assumed you had a typical age from the data set, what 256 00:12:03,920 --> 00:12:05,680 would the prediction be? I see. 257 00:12:05,680 --> 00:12:08,440 So if I swap my age with a bunch of random people's ages and the 258 00:12:08,440 --> 00:12:10,600 prediction stays basically the same. 259 00:12:10,960 --> 00:12:14,400 Then your specific age didn't matter, the model wasn't relying 260 00:12:14,400 --> 00:12:15,880 on it. It's like a light switch that 261 00:12:15,880 --> 00:12:18,000 isn't connected to the bulb. You can flip it up and down 262 00:12:18,000 --> 00:12:20,840 255080 and the light which is the prediction. 263 00:12:21,040 --> 00:12:23,320 It doesn't change. But if the prediction swings 264 00:12:23,320 --> 00:12:28,000 wildly, if putting in 25 makes me healthy and putting in 80 265 00:12:28,000 --> 00:12:30,320 makes me diabetic. Then your specific age was 266 00:12:30,320 --> 00:12:32,680 providing a lot of information. It has a high marginal 267 00:12:32,680 --> 00:12:35,840 contribution, a high SHP value. That makes sense. 268 00:12:35,840 --> 00:12:39,200 So mathematically, SHAP is really just comparing the 269 00:12:39,200 --> 00:12:41,840 prediction with your actual feature value against the 270 00:12:41,840 --> 00:12:45,640 expected prediction with random feature values averaged over 271 00:12:45,640 --> 00:12:49,040 many many combinations. That is the essence of it and 272 00:12:49,080 --> 00:12:52,280 the output. The SHAP value has a direction. 273 00:12:52,960 --> 00:12:56,480 A positive SHA value means this feature pushed the prediction 274 00:12:56,480 --> 00:12:59,920 higher towards the positive class, like yes, diabetic and. 275 00:12:59,920 --> 00:13:01,840 A negative value means it pushed it lower. 276 00:13:01,840 --> 00:13:03,520 Exactly. Which brings us to 277 00:13:03,520 --> 00:13:06,120 visualization. Because a list of numbers is 278 00:13:06,120 --> 00:13:08,280 great. But engineers, we need to see 279 00:13:08,280 --> 00:13:10,440 this stuff. The toolkit for SAP is actually 280 00:13:10,440 --> 00:13:11,760 really beautiful. It is. 281 00:13:11,760 --> 00:13:13,480 It's one of the reasons it became so popular. 282 00:13:13,800 --> 00:13:15,600 Let's talk about the force plot first. 283 00:13:15,720 --> 00:13:17,520 This is standard for tabular data. 284 00:13:17,560 --> 00:13:19,840 I love the force plot, it looks like a tug of war. 285 00:13:20,160 --> 00:13:22,920 It is a tug of war. Imagine a horizontal bar. 286 00:13:23,400 --> 00:13:25,560 In the middle you have the base value. 287 00:13:25,560 --> 00:13:28,520 That's the average prediction of the data set, let's say 10%. 288 00:13:28,520 --> 00:13:31,240 OK, a starting point. Then you have red bars pushing 289 00:13:31,240 --> 00:13:34,320 the prediction to the right, increasing the risk, and blue 290 00:13:34,320 --> 00:13:37,080 bars pushing it to the left, decreasing the risk. 291 00:13:37,120 --> 00:13:41,240 So for our diabetes example, I might see a big fat red bar 292 00:13:41,240 --> 00:13:46,000 labeled BMI 30 pushing the score way up, and maybe a small blue 293 00:13:46,000 --> 00:13:49,520 bar labeled exercise daily pushing it back down a little 294 00:13:49,520 --> 00:13:50,440 bit. Correct. 295 00:13:50,760 --> 00:13:54,240 And where those opposing forces meet that equilibrium point is 296 00:13:54,240 --> 00:13:56,560 the final prediction. It lets you tell a story. 297 00:13:56,800 --> 00:13:59,320 You can say to a customer, look, your credit score is high 298 00:13:59,320 --> 00:14:01,120 because your payment history is perfect. 299 00:14:01,360 --> 00:14:04,200 That's this big red bar, despite the fact that your account is 300 00:14:04,200 --> 00:14:05,320 new. This little blue bar. 301 00:14:05,600 --> 00:14:08,280 That's the local view, but what if I want to see the global 302 00:14:08,280 --> 00:14:10,360 patterns? The summary plot is usually what 303 00:14:10,360 --> 00:14:13,080 people put in their slide decks. The summary plot or bees worm 304 00:14:13,080 --> 00:14:16,760 plot is fantastic for that. Imagine taking those force plots 305 00:14:16,760 --> 00:14:19,440 for every single person in your database and stacking them on 306 00:14:19,440 --> 00:14:20,240 top of each other. So. 307 00:14:20,320 --> 00:14:22,160 You get a cloud of points. A swarm? 308 00:14:22,160 --> 00:14:24,200 Yeah. On the Y axis you have your 309 00:14:24,200 --> 00:14:31,000 features, Age, income, BMI. On the X axis is the SHAP value, 310 00:14:31,000 --> 00:14:33,320 the impact on the model. OK, so if you look at the row 311 00:14:33,320 --> 00:14:36,160 for age, you might see a big cluster of red dots which 312 00:14:36,160 --> 00:14:39,120 represent high values of age way over on the right side of the 313 00:14:39,120 --> 00:14:41,000 plot. Which tells you that being older 314 00:14:41,080 --> 00:14:42,760 generally increases the prediction. 315 00:14:42,920 --> 00:14:45,880 Right, but you might also see that for income, the dots are 316 00:14:45,880 --> 00:14:48,960 spread out all over the place, meaning it's a messy, complex 317 00:14:48,960 --> 00:14:51,240 feature with no simple relationship. 318 00:14:51,600 --> 00:14:54,000 The visualization that really blew my mind in the lecture 319 00:14:54,000 --> 00:14:57,520 though was for images. We aren't just stuck with 320 00:14:57,520 --> 00:15:01,600 sreadsheets, HA works on pixels. This is where it gets really 321 00:15:01,600 --> 00:15:03,840 intuitive for image classification. 322 00:15:04,080 --> 00:15:07,840 HA roduces A Ixel heat map. O If I have a picture of an 323 00:15:07,840 --> 00:15:12,000 American egret, a white bird, and the model correctly 324 00:15:12,000 --> 00:15:16,320 identifies it, SHAP can tell me which pixels convince the model. 325 00:15:16,320 --> 00:15:18,440 Exactly. It'll turn the pixels on the 326 00:15:18,440 --> 00:15:20,040 beak and the wings red. It's saying. 327 00:15:20,240 --> 00:15:23,000 I think this is an egret because of this long pointy thing in 328 00:15:23,000 --> 00:15:26,120 these feathers in the. Background The sky, the water. 329 00:15:26,120 --> 00:15:29,480 The water should be neutral. Ideally, yes, the pixels for the 330 00:15:29,480 --> 00:15:32,120 sky should be Gray. The model should be saying I'm 331 00:15:32,120 --> 00:15:34,160 ignoring the sky, it's not relevant. 332 00:15:34,280 --> 00:15:37,680 But this brings us to the famous cautionary tale, the Husky 333 00:15:37,680 --> 00:15:40,400 versus wolf story. This is a classic in the 334 00:15:40,400 --> 00:15:42,600 interpretability field. It's legendary. 335 00:15:42,880 --> 00:15:45,760 So researchers trained a model to distinguish between wolves 336 00:15:45,760 --> 00:15:48,000 and Huskies. It had incredible accuracy. 337 00:15:48,000 --> 00:15:49,280 I mean, everyone was celebrating. 338 00:15:49,280 --> 00:15:51,160 High fives all around. We solved it. 339 00:15:51,400 --> 00:15:53,000 Then they ran an explanation tool. 340 00:15:53,040 --> 00:15:55,680 It was actually Lyme in the original paper, but S Chip does 341 00:15:55,680 --> 00:15:58,760 the same thing to see why it was classifying images as wolves. 342 00:15:58,760 --> 00:16:01,000 And what did they find? Was it looking at the ears? 343 00:16:01,520 --> 00:16:03,560 The snout? The shape of the head. 344 00:16:03,680 --> 00:16:05,880 It was looking at the snow. The snow. 345 00:16:05,880 --> 00:16:07,760 The snow. It turned out that in the 346 00:16:07,760 --> 00:16:11,440 training data, almost every picture of a wolf was taken in 347 00:16:11,440 --> 00:16:13,600 the wild in the winter. So there was snow in the 348 00:16:13,600 --> 00:16:14,840 background. Oh no. 349 00:16:14,840 --> 00:16:18,920 And almost every picture of a Husky was a pet in a backyard on 350 00:16:18,920 --> 00:16:21,920 grass. So the model didn't learn wolf, 351 00:16:22,080 --> 00:16:24,480 it learned snow detector. Exactly. 352 00:16:24,480 --> 00:16:27,160 If there is snow, it's a wolf. If there is grass, it's a Husky. 353 00:16:27,520 --> 00:16:30,280 It took the lazy way out. That is terrifying. 354 00:16:30,320 --> 00:16:33,480 I mean, it's funny with dogs, but if that's a medical imaging 355 00:16:33,480 --> 00:16:36,240 model detecting tumors. It's a disaster, and there have 356 00:16:36,240 --> 00:16:39,760 been real cases where models learn to detect tumors based on 357 00:16:39,760 --> 00:16:43,000 the ruler placed next to the tumor in the training images or 358 00:16:43,000 --> 00:16:45,120 the hospital tag in the corner of the X-ray. 359 00:16:45,280 --> 00:16:48,280 Because the sickest patients always had the ruler or the tag 360 00:16:48,280 --> 00:16:49,520 next to them. Right. 361 00:16:49,800 --> 00:16:53,160 Without a tool like SHAP to visualize those heat maps, you 362 00:16:53,160 --> 00:16:55,400 would never know. You would deploy a model that 363 00:16:55,400 --> 00:16:59,480 has 99% accuracy on your test set, but fails completely in the 364 00:16:59,480 --> 00:17:01,520 real world because there's no ruler. 365 00:17:02,040 --> 00:17:05,520 SHAP is a debugging tool. It tells you if your models 366 00:17:05,520 --> 00:17:09,079 right for the wrong reasons. That's the perfect way to put 367 00:17:09,079 --> 00:17:10,880 it. OK, so we have the theory, the 368 00:17:10,880 --> 00:17:13,319 mechanics and the visuals. Now we have to talk about the 369 00:17:13,319 --> 00:17:15,800 elephant in the room. Computational cost. 370 00:17:15,839 --> 00:17:19,640 Oh yes, the big one. We said earlier that to get the 371 00:17:19,640 --> 00:17:23,480 real true shapely value you have to check every possible subset. 372 00:17:23,560 --> 00:17:25,640 Every single permutation I remember enough. 373 00:17:25,640 --> 00:17:29,080 Combinatorics to know that that grows exponentially. 374 00:17:29,480 --> 00:17:32,800 If you have N features, the number of subsets is 2 to the 375 00:17:32,800 --> 00:17:35,880 power of N, correct? So if I have a data set with 376 00:17:35,880 --> 00:17:39,440 just 50 features, which is pretty small for modern AI, 377 00:17:39,760 --> 00:17:43,200 that's 2 to the 50th power calculations per row. 378 00:17:43,200 --> 00:17:46,640 Which is roughly A quadrillion. You are not running that on your 379 00:17:46,640 --> 00:17:48,000 laptop. You aren't running that on a 380 00:17:48,000 --> 00:17:50,320 supercomputer. It's just not feasible. 381 00:17:50,400 --> 00:17:52,560 So is this all just a nice theory that we can't actually 382 00:17:52,560 --> 00:17:54,280 use? No, and this is the main 383 00:17:54,280 --> 00:17:56,000 contribution of Lundberg and Lee. 384 00:17:56,520 --> 00:17:59,800 They developed approximation algorithms that get us very very 385 00:17:59,800 --> 00:18:03,960 close to the true value without calculating every single subset. 386 00:18:04,160 --> 00:18:07,160 OK, So what are the main solvers we need to know about? 387 00:18:07,480 --> 00:18:10,640 The first one, and the most general is kernel shape. 388 00:18:11,360 --> 00:18:14,640 This is a model agnostic method. It works on anything neural 389 00:18:14,640 --> 00:18:18,560 networks, SVM's, logistic regression, any black box you 390 00:18:18,560 --> 00:18:20,320 can throw at it. And how does it work 391 00:18:20,480 --> 00:18:23,000 intuitively? It uses a specific type of 392 00:18:23,000 --> 00:18:26,160 weighted linear regression, essentially fitting a simple 393 00:18:26,160 --> 00:18:29,000 explainable model locally around the prediction you're trying to 394 00:18:29,000 --> 00:18:31,360 explain to approximate the Shapley values. 395 00:18:32,160 --> 00:18:34,640 It's actually based on an earlier method, client Lyme. 396 00:18:34,800 --> 00:18:36,400 Is it fast? No. 397 00:18:36,440 --> 00:18:38,040 It's slow. Very slow. 398 00:18:38,040 --> 00:18:40,680 It requires a lot of sampling, which means running the model 399 00:18:40,680 --> 00:18:43,120 many, many times. It's really the fall back option 400 00:18:43,120 --> 00:18:45,760 if you have a weird custom model that nothing else supports. 401 00:18:45,760 --> 00:18:48,400 OK, but most of our listeners are probably using tree based 402 00:18:48,400 --> 00:18:51,080 models. Yeah, XG, Boost, light, GBM, 403 00:18:51,360 --> 00:18:53,400 random forest. That's the bread and butter of 404 00:18:53,400 --> 00:18:55,960 tabular data. For them there is tree HAPE and 405 00:18:55,960 --> 00:18:59,200 this is a total game changer. Why is tree shape different? 406 00:18:59,280 --> 00:19:02,240 Why is it so much better? Because tree shape leverages is 407 00:19:02,240 --> 00:19:05,680 the actual structure of the decision trees, it doesn't need 408 00:19:05,680 --> 00:19:07,080 to treat the model as a black box. 409 00:19:07,080 --> 00:19:10,680 It looks inside, I see a tree is just a series of splits, right? 410 00:19:10,960 --> 00:19:12,960 Is age greater than 50 yes or no? 411 00:19:13,680 --> 00:19:17,120 So tree SHAP can calculate the exact Shapley values by 412 00:19:17,120 --> 00:19:19,440 efficiently traversing the paths of the tree. 413 00:19:19,560 --> 00:19:21,680 So it doesn't need to do all that random sampling we talked 414 00:19:21,680 --> 00:19:22,520 about. Exactly. 415 00:19:22,560 --> 00:19:26,040 No sampling needed. It computes the probability of 416 00:19:26,040 --> 00:19:29,120 reaching different leaves based on the distribution of data in 417 00:19:29,120 --> 00:19:32,560 the tree nodes. It turns an exponential problem 418 00:19:32,720 --> 00:19:35,560 2 to the north into a polynomial problem. 419 00:19:35,560 --> 00:19:37,800 In English. It's fast. 420 00:19:37,800 --> 00:19:41,120 Extremely fast. If you are using XG boost or 421 00:19:41,120 --> 00:19:43,680 light GBM. Tree shape is effectively free. 422 00:19:43,800 --> 00:19:45,960 You should always use it. It makes explain ability 423 00:19:45,960 --> 00:19:49,080 feasible for massive data sets. And for the deep learning crowd, 424 00:19:49,320 --> 00:19:51,560 the neural net folks. There's deep HP. 425 00:19:51,800 --> 00:19:54,680 It hooks into the neural network layers in a way that's similar 426 00:19:54,680 --> 00:19:57,760 to how back propagation works, to propagate the important 427 00:19:57,760 --> 00:19:59,920 scores from the output all the way back to the inputs. 428 00:20:00,560 --> 00:20:02,920 It's an approximation, but it's designed to handle the 429 00:20:02,920 --> 00:20:04,960 complexity of deep networks efficiently. 430 00:20:05,160 --> 00:20:08,720 OK, so we have the tools, but we promised the listeners we'd talk 431 00:20:08,720 --> 00:20:11,200 about the gotchas. Section 5 of the deep dive 432 00:20:11,200 --> 00:20:14,680 because nothing is perfect. The biggest pitfall, and this is 433 00:20:14,680 --> 00:20:19,000 something advanced practitioners argue about constantly, is the 434 00:20:19,040 --> 00:20:22,040 independence assumption. We test on this with 435 00:20:22,040 --> 00:20:25,480 marginalization. When we swap in a random value 436 00:20:25,480 --> 00:20:29,600 for a feature, we are assuming we can do that without, you 437 00:20:29,600 --> 00:20:33,240 know, breaking reality. Right, the core SHAP algorithm 438 00:20:33,240 --> 00:20:35,720 assumes features are independent, but in the real 439 00:20:35,720 --> 00:20:38,440 world features are often highly correlated. 440 00:20:38,440 --> 00:20:40,280 Give me the five year old millionaire example. 441 00:20:40,440 --> 00:20:42,400 This is my favorite illustration of the problem. 442 00:20:42,440 --> 00:20:45,560 It's the perfect example of creating out of distribution 443 00:20:45,560 --> 00:20:48,200 data. So imagine you have a model 444 00:20:48,200 --> 00:20:51,840 predicting credit risk. You have two features, age and 445 00:20:51,880 --> 00:20:53,800 income. And in the real world, these are 446 00:20:53,800 --> 00:20:55,960 correlated. Generally you make more money as 447 00:20:55,960 --> 00:20:57,520 you get older. Up to a point, right? 448 00:20:58,000 --> 00:21:01,720 Now suppose we are calculating the SHAP value for age. 449 00:21:02,000 --> 00:21:06,600 We take a data point, Let's say ACEO who is 55 and earns 500,000 450 00:21:06,600 --> 00:21:08,840 a year. OK, to test the importance of 451 00:21:08,880 --> 00:21:13,640 age, SHAP swaps out the CE OS real age for a random age from 452 00:21:13,640 --> 00:21:15,480 the data set. And let's say the random age it 453 00:21:15,480 --> 00:21:19,520 picks is. 5 exactly. So now we are asking the model 454 00:21:19,520 --> 00:21:24,240 to make a prediction for a 5 year old earning $500,000. 455 00:21:24,520 --> 00:21:26,600 Which is a data point that doesn't exist. 456 00:21:26,600 --> 00:21:28,680 It's impossible. It's nonsensical. 457 00:21:28,680 --> 00:21:31,840 It is completely off the manifold of real data. 458 00:21:32,200 --> 00:21:34,760 The model has never seen anything like this during 459 00:21:34,760 --> 00:21:37,120 training. It has no idea what to do with 460 00:21:37,120 --> 00:21:38,440 it. So the model effectively 461 00:21:38,440 --> 00:21:40,320 hallucinates it just makes something up it. 462 00:21:40,320 --> 00:21:42,280 Extrapolates. It might spit out a really weird 463 00:21:42,280 --> 00:21:45,360 prediction just because it's confused and that garbage 464 00:21:45,360 --> 00:21:48,720 prediction gets fed into the SAP calculation. 465 00:21:48,720 --> 00:21:51,320 So you get ASAP value that looks precise. 466 00:21:51,760 --> 00:21:54,280 You know age contributed positive 0.2. 467 00:21:54,320 --> 00:21:57,640 Yeah, but it's based on a scenario that is total nonsense. 468 00:21:57,680 --> 00:21:59,520 Exactly. Garbage in, garbage out. 469 00:21:59,840 --> 00:22:02,240 If features are heavily correlated, the credit 470 00:22:02,240 --> 00:22:06,200 assignment can get very messy. The math tries to be fair, but 471 00:22:06,200 --> 00:22:08,720 if the data points used to calculate the fairness are 472 00:22:08,720 --> 00:22:11,360 themselves nonsensical, the result is compromised. 473 00:22:11,360 --> 00:22:15,360 Is there a fix for this? There are variations of SAP that 474 00:22:15,360 --> 00:22:18,360 try to account for correlation, but they are computationally 475 00:22:18,360 --> 00:22:22,640 much much heavier. The practical advice is be aware 476 00:22:22,640 --> 00:22:25,680 of it. If 2 features are 99% 477 00:22:25,680 --> 00:22:28,760 correlated, like years of education and education level, 478 00:22:29,360 --> 00:22:32,320 SHA might split the credit between them arbitrarily, or 479 00:22:32,320 --> 00:22:34,440 they might even cancel each other out with one being 480 00:22:34,440 --> 00:22:37,360 positive and one negative. So don't just blindly trust the 481 00:22:37,360 --> 00:22:39,440 plot. You have to understand the data 482 00:22:39,440 --> 00:22:42,080 structure underneath. If you see something weird, 483 00:22:42,200 --> 00:22:45,840 check your correlations. Always SHAP is a tool for the 484 00:22:45,840 --> 00:22:47,480 engineer to understand the model. 485 00:22:47,480 --> 00:22:50,040 It doesn't replace the engineer. We're coming up on time, so I 486 00:22:50,040 --> 00:22:53,440 want to zoom out. We've gone from 1950s game 487 00:22:53,440 --> 00:22:58,120 theory to robo runners to snow detecting wolves to the math of 488 00:22:58,120 --> 00:22:59,400 masking. It's a huge. 489 00:22:59,400 --> 00:23:00,800 Landscape. If a listener takes away just 490 00:23:00,800 --> 00:23:04,320 one thing, one core concept from all this, what should it be? 491 00:23:04,400 --> 00:23:08,520 That SHAP is the bridge between accuracy and interpretability. 492 00:23:08,640 --> 00:23:11,560 For a long time we thought we had to choose, you know, Do you 493 00:23:11,560 --> 00:23:14,600 want a dumb model you can understand, like a linear 494 00:23:14,600 --> 00:23:17,640 regression, or a smart model that's a black box? 495 00:23:18,920 --> 00:23:21,920 SHA proves we can have both. We can use the complex high 496 00:23:21,920 --> 00:23:24,800 accuracy models and still debug. Them, and I'd emphasize the 497 00:23:24,800 --> 00:23:27,000 debug part. Yes, absolutely. 498 00:23:27,000 --> 00:23:29,560 Don't just use this to make pretty charts for the quarterly 499 00:23:29,560 --> 00:23:32,760 business review. Use it to check if your model is 500 00:23:32,760 --> 00:23:36,360 learning the right patterns. Use it to catch the snow before 501 00:23:36,360 --> 00:23:38,840 you deploy the wolf. That is the most valuable 502 00:23:38,840 --> 00:23:41,720 application, period. But I want to leave us with a 503 00:23:41,720 --> 00:23:44,120 provocative thought, something to chew on. 504 00:23:44,520 --> 00:23:47,040 What's that? Let's say ASAP worked perfectly. 505 00:23:47,720 --> 00:23:50,320 It tells us exactly why a model denied a loan. 506 00:23:50,720 --> 00:23:53,280 It says I denied this loan because of the applicant's zip 507 00:23:53,280 --> 00:23:55,040 code. Which is a huge red flag for 508 00:23:55,040 --> 00:23:56,800 bias. That's textbook redlining. 509 00:23:56,800 --> 00:24:00,640 Right, SHAP has done its job. It explained the why, but the 510 00:24:00,640 --> 00:24:02,480 explanation itself is just information. 511 00:24:02,680 --> 00:24:06,000 It doesn't fix the problem. The math is fair in how it 512 00:24:06,000 --> 00:24:09,120 calculates the contribution. It accurately reports that the 513 00:24:09,120 --> 00:24:12,040 zip code was the cause. But the reality it reveals is 514 00:24:12,040 --> 00:24:14,960 deeply unfair. That is a profound distinction. 515 00:24:15,120 --> 00:24:18,600 SHAP is a mirror. It reflects exactly what the 516 00:24:18,600 --> 00:24:20,480 model learned from the data you gave it. 517 00:24:20,920 --> 00:24:24,120 If the data has historical bias baked into it, the model will 518 00:24:24,120 --> 00:24:27,080 learn it and SHA tool dutifully report back. 519 00:24:27,080 --> 00:24:29,560 Yes, I am biased because of feature X. 520 00:24:29,600 --> 00:24:32,560 So explainability isn't the solution to ethical AI, it's 521 00:24:32,560 --> 00:24:35,600 just the diagnostic tool. You still need the human in the 522 00:24:35,600 --> 00:24:38,800 loop to look at that SHA plot and say that's wrong. 523 00:24:38,920 --> 00:24:41,280 We need to retrain, we need to fix the data. 524 00:24:41,280 --> 00:24:43,400 Exactly. The tool reveals the flaw, The 525 00:24:43,400 --> 00:24:46,160 human still has to fix it. Well, on that note, go check 526 00:24:46,160 --> 00:24:49,680 your models, watch out for the snow, and try to avoid creating 527 00:24:49,680 --> 00:24:51,960 too many five year old millionaires in your validation 528 00:24:51,960 --> 00:24:53,400 sets. Sound advice. 529 00:24:53,400 --> 00:24:54,560 We'll catch you on the next deep dive.