You know, I was looking through the stack of papers and blog
posts you sent over for this dive, and it it hit me that
we've reached a really strange inflection point in AI.
Oh yeah. Yeah.
I mean, we've spent the last 10, maybe 15 years completely
obsessed with one thing, accuracy.
Right, just get the loss function down.
Exactly. It's been this race to the
bottom for loss. Get the accuracy up, get the AUC
up, and you know who cares what happens inside the layers?
Right. It was kind of the if it works,
don't touch it phase of engineering.
Totally. As long as the predictions were
good on the test set, nobody asked too many questions.
But now, based on everything we're seeing in the industry,
the bill is coming due. We have these these massive
models, gradient boosted trees, deep neural networks with
millions of parameters. And they are making decisions
that actually matter. They really matter.
We aren't just predicting ad clicks anymore.
We're talking about loan approvals, medical diagnosis,
autonomous driving. And that is where the black box
problem goes from being, you know, an academic curiosity to a
massive liability because eventually a stakeholder or a
regulator or a doctor is going to look at a prediction and ask
why. Why?
Why did the model deny this loan?
Why did it flag this patient for surgery?
And usually the engineer just stares at the floor or they show
a confusion matrix which doesn't answer the question at all.
Not even close. I don't know.
It isn't an acceptable answer when you're dealing with
someone's life savings or you know their health.
Precisely. Transparency isn't just a nice
to have feature anymore, it's a necessity.
We need to move from. I know that it works to.
I know how it works. And that is the mission for
today. We're going to demystify a tool
that has, I think you could argue, become the industry
standard for cracking open that black box.
We're talking about SH. A shapely additive explanations.
It is a mouthful, but the concept behind it is just
fascinating. What I love about this topic is
that it connects modern cutting edge machine learning back to
what Nobel Prize winning game theory from the 1950s.
It really is the bridge between the rigorous math of fairness
and the the messy reality of data engineering.
So here is the plan. We're going to break this down
into a few distinct parts. We're going to start with the
intuition, the robo runners analogy, which I think is the
best way to visualize the math. It's a great one.
Math. Then we'll get into the
mechanics of how you actually calculate this, specifically
tackling that marginalization problem.
We'll look at the visualization toolkit.
And finally, we have to talk about the algorithms, tree shape
versus kernel app, and of course the pitfalls.
And the pitfalls are significant.
It's not a magic wand. No, it's definitely not.
But let's start with the origins story.
So SHAP was introduced in a paper by Scott Lundberg and Sue
and Lee around 2017, but the math they used wasn't new at
all. Not at all, no.
They basically went dumpster diving in 1950s economics.
They realize that the problem of debugging a complex machine
learning model is well, it's mathematically identical to a
problem in cooperative game theory that Lloyd Shapely solved
back in 1953. Lloyd Shapely, who later won the
Nobel Prize for this very work. Correct.
Shapely was interested in a very specific scenario.
Imagine you have a team of people working together to
achieve a goal. They cooperate and together they
generate some value. A payout, a prize, a win.
The core question Shapley tried to answer was how do you fairly
distribute that payout among the team members when everyone
contributed differently and. Fairly, here's a mathematical
term, right? Not just a feeling.
Exactly. It has to satisfy specific
axioms of fairness. It's rigorous.
So to wrap our heads around this, let's use the mental model
from the Visoira lecture, the Robo Runners.
This is a classic setup, so imagine a robotics team called
the Robo Runners. There are four members on this
team. OK, they enter competition to
build a robot. They work together and they win
first prize. They take home $10,000.
Simple enough. Four people, 10 grand in the
bank. Now comes the problem payday.
How do you split the $10,000? I mean if I'm the manager and I
just want to go home, I divide by 4.
Everyone gets 2500 bucks. The naive approach, but is it
fair? Probably not.
Let's say one person was the lead engineer who built the
navigation system from scratch, and another person was.
And let's say the person who brought the snacks.
Right. If you give the snack guy $2500,
the lead engineer is going to riot, and rightly so.
The equal split ignores the reality that contributions are
unequal, so that's out. OK, so we need a metric.
If I'm trying to be analytical about this, my next instinct is
to try a a subtraction method. OK, I'd look at the team and I
ask how much value did Bob, let's call the hardware guy Bob,
add to the group? And how would you calculate
that? I'd simulate the competition
without Bob. I'd say OK, the full team wins
10,000. If Bob stays home, the remaining
three people aren't as good. Maybe they only win second place
which is $6000. So the full team gets 10,000,
the team without Bob gets 6000. So the logic is 10,000 -, 6000
is 4000. Therefore Bob is worth exactly
$4000. That's his marginal
contribution. It feels intuitive, doesn't it?
You're just measuring the drop off.
But there's a massive flaw in that logic.
I suspect it has to do with the other team members.
It's all about interactions. That subtraction method assumes
Bob's value is static, that he is worth $4000 regardless of who
else is in the room. But think about it.
Bob is the hardware expert. He builds the physical robot,
right? Now imagine another team member,
Alice. She's the algorithm expert.
She writes the code. OK, if Alice is also missing,
what is Bob's value? Well, if there's no code, the
robot is just a paperweight. Doesn't move, so Bob's hardware
is. Useless exactly.
If Alice isn't there, Bob's marginal contribution might be
0. But if Alice is there, Bob's
contribution is huge because her code has something to control.
I see. So the team minus Bob
calculation fails because it ignores the context.
It only looks at one specific scenario, the full team versus
the full team without Bob. It doesn't account for the fact
that Bob needs Alice to be valuable.
Precisely. And this is the genius of the
Shapley solution. To find the true fair
contribution of the hardware expert, you cannot just look at
the full team. You have to look at every
possible subset of the team. Every possible subset.
Wow, walk me through that. You have to calculate Bob's
impact in every possible reality.
You measure his contribution when he works entirely alone.
Maybe he wins nothing. Then you measure his
contribution when he joins a team of just the snack guy, then
when he joins just Alice, then when he joins Alice and the
snack guy. Whoa, so you were running a
simulation for every single permutation of the group?
Yes, you calculate the difference his presence makes in
every single combination. And then, and this is the key,
you take the weighted average of all those marginal
contributions. That average is the Shapley
value. And that number is
mathematically proven to be the only fairway to distribute the
payout. Yes, Lloyd Shapley proved that
if you want to satisfy the axioms of efficiency, symmetry
and linearity, this is the unique solution.
There is no other way to do it. OK.
So that's the intuition. We have the robo runners and we
have a way to fairly split the $10,000.
Now we have to make the jump to machine learning because our
listeners aren't managing robotics teams, they're building
XG boost models. How does this map?
It is a direct mapping. In this analogy, the players on
the team are the features in your data set.
Features meaning the columns age, BMI, income, zip code, that
stuff. Correct.
And the payout, the $10,000 prize is the models prediction.
But wait, let's be precise here. Is the payout the raw prediction
itself like the 80% probability? That's a great catch.
No. Technically the payout we are
distributing is the difference between the prediction for a
specific instance and the average prediction of the data
set. OK, let's unpack that.
Suppose we are predicting the probability of diabetes.
The average person in our data set has, let's say a 10% risk.
That's our baseline, right? That's.
What the model would predict if it knew nothing about a person?
We feed my data into the model and the model spits out an 80%
risk. So I'm at 80%.
The baseline is 10%, the payout. The thing we need to explain is
that 70% gap. Exactly.
The question is how do we get from 10% to 80%?
We have to distribute that 70 percentage points of credit or
blame. Among your features, how much
did your age contribute? How much did your BMI
contribute? And just like the robot team,
these features interact. My high BMI might not be a huge
risk factor on its own, but combined with my age it might
just explode the risk. Recisely high blood pressure
routers more if you are older than if you are younger.
The model, assuming it's a good model, captures these complex
non linear interactions and SHA untangles them by running those
subset calculations. It simulates the model with just
age, then agent BMI, then agent income to find the true marginal
contribution of each factor. That is such a cool way to frame
it, but I want to highlight something you mentioned in the
notes, the difference between local and global explanations
because I see people get this mixed up all the time.
They do. It's a critical distinction.
Global explanations tell you how the model works in general.
For example, a feature importance plot in a random
forest might tell you overall income is the most important
predictor. Which is like saying smoking is
generally bad for your health. Right, it's a statistical
statement about the population, but that doesn't help you
understand a specific prediction.
If you have a specific applicant who is denied a loan, telling
them income is generally important doesn't explain their
denial. Maybe their income was fine.
Right. Maybe for this specific person
it was their debt to income ratio that sank them.
Exactly. SHAP provides local
explanations. It explains this specific
instance. Why was the 65 year old
classified as diabetic? It allows you to debug the model
at the row level. Which is incredibly powerful.
But yeah, and here is where the rubber meets the road for me.
I have a practical engineering question.
In the robo runners example, we said we calculate these subsets
by removing a person. I can physically tell Bob, hey,
go wait in the hall, we're going to build the robot without you.
But in a machine learning model, specifically something like a
neural net, I have a matrix of data.
I have a fixed input layer. It expects 50 numbers.
I can't just delete the age column for one row and expect
the model to run. The math will fail.
The model will crash. So how do we remove a feature to
calculate these subsets? You've hit on the central
engineering challenge of SHA and you are right.
Most models cannot handle missing data natively, so we
have to fake it. We use a technique called
marginalization. Marginalization.
OK, unpack that. How do we mask a feature without
breaking the model? We simulate the feature being
missing by replacing its real value with a random value from a
background data set, usually the training data.
A random value, so we're just injecting noise.
In a way, yeah. Think of it as Monte Carlo
integration. Let's say we want to measure the
importance of your age. We want to simulate a team
without your age, so we keep your BMI, your blood pressure,
and your income exactly as they are.
Those are the fixed team members.
But for your age, we swap it out.
Swap it with what? We might grab the age of a
random person from the training data.
Let's say we pull a record and it's a 25 year old.
We plug that in and run the prediction.
Then we pull another record, a 50 year old, run the prediction,
then an 80 year old, and so on. So we are creating these
synthetic Frankenstein data points where everything is me
except my age is just morphing into different people from the
database. Exactly.
We are integrating over the distribution of that feature.
We are asking the model. If we didn't know your age and
just assumed you had a typical age from the data set, what
would the prediction be? I see.
So if I swap my age with a bunch of random people's ages and the
prediction stays basically the same.
Then your specific age didn't matter, the model wasn't relying
on it. It's like a light switch that
isn't connected to the bulb. You can flip it up and down
255080 and the light which is the prediction.
It doesn't change. But if the prediction swings
wildly, if putting in 25 makes me healthy and putting in 80
makes me diabetic. Then your specific age was
providing a lot of information. It has a high marginal
contribution, a high SHP value. That makes sense.
So mathematically, SHAP is really just comparing the
prediction with your actual feature value against the
expected prediction with random feature values averaged over
many many combinations. That is the essence of it and
the output. The SHAP value has a direction.
A positive SHA value means this feature pushed the prediction
higher towards the positive class, like yes, diabetic and.
A negative value means it pushed it lower.
Exactly. Which brings us to
visualization. Because a list of numbers is
great. But engineers, we need to see
this stuff. The toolkit for SAP is actually
really beautiful. It is.
It's one of the reasons it became so popular.
Let's talk about the force plot first.
This is standard for tabular data.
I love the force plot, it looks like a tug of war.
It is a tug of war. Imagine a horizontal bar.
In the middle you have the base value.
That's the average prediction of the data set, let's say 10%.
OK, a starting point. Then you have red bars pushing
the prediction to the right, increasing the risk, and blue
bars pushing it to the left, decreasing the risk.
So for our diabetes example, I might see a big fat red bar
labeled BMI 30 pushing the score way up, and maybe a small blue
bar labeled exercise daily pushing it back down a little
bit. Correct.
And where those opposing forces meet that equilibrium point is
the final prediction. It lets you tell a story.
You can say to a customer, look, your credit score is high
because your payment history is perfect.
That's this big red bar, despite the fact that your account is
new. This little blue bar.
That's the local view, but what if I want to see the global
patterns? The summary plot is usually what
people put in their slide decks. The summary plot or bees worm
plot is fantastic for that. Imagine taking those force plots
for every single person in your database and stacking them on
top of each other. So.
You get a cloud of points. A swarm?
Yeah. On the Y axis you have your
features, Age, income, BMI. On the X axis is the SHAP value,
the impact on the model. OK, so if you look at the row
for age, you might see a big cluster of red dots which
represent high values of age way over on the right side of the
plot. Which tells you that being older
generally increases the prediction.
Right, but you might also see that for income, the dots are
spread out all over the place, meaning it's a messy, complex
feature with no simple relationship.
The visualization that really blew my mind in the lecture
though was for images. We aren't just stuck with
sreadsheets, HA works on pixels. This is where it gets really
intuitive for image classification.
HA roduces A Ixel heat map. O If I have a picture of an
American egret, a white bird, and the model correctly
identifies it, SHAP can tell me which pixels convince the model.
Exactly. It'll turn the pixels on the
beak and the wings red. It's saying.
I think this is an egret because of this long pointy thing in
these feathers in the. Background The sky, the water.
The water should be neutral. Ideally, yes, the pixels for the
sky should be Gray. The model should be saying I'm
ignoring the sky, it's not relevant.
But this brings us to the famous cautionary tale, the Husky
versus wolf story. This is a classic in the
interpretability field. It's legendary.
So researchers trained a model to distinguish between wolves
and Huskies. It had incredible accuracy.
I mean, everyone was celebrating.
High fives all around. We solved it.
Then they ran an explanation tool.
It was actually Lyme in the original paper, but S Chip does
the same thing to see why it was classifying images as wolves.
And what did they find? Was it looking at the ears?
The snout? The shape of the head.
It was looking at the snow. The snow.
It turned out that in the
training data, almost every picture of a wolf was taken in
the wild in the winter. So there was snow in the
background. Oh no.
And almost every picture of a Husky was a pet in a backyard on
grass. So the model didn't learn wolf,
it learned snow detector. Exactly.
If there is snow, it's a wolf. If there is grass, it's a Husky.
It took the lazy way out. That is terrifying.
I mean, it's funny with dogs, but if that's a medical imaging
model detecting tumors. It's a disaster, and there have
been real cases where models learn to detect tumors based on
the ruler placed next to the tumor in the training images or
the hospital tag in the corner of the X-ray.
Because the sickest patients always had the ruler or the tag
next to them. Right.
Without a tool like SHAP to visualize those heat maps, you
would never know. You would deploy a model that
has 99% accuracy on your test set, but fails completely in the
real world because there's no ruler.
SHAP is a debugging tool. It tells you if your models
right for the wrong reasons. That's the perfect way to put
it. OK, so we have the theory, the
mechanics and the visuals. Now we have to talk about the
elephant in the room. Computational cost.
Oh yes, the big one. We said earlier that to get the
real true shapely value you have to check every possible subset.
Every single permutation I remember enough.
Combinatorics to know that that grows exponentially.
If you have N features, the number of subsets is 2 to the
power of N, correct? So if I have a data set with
just 50 features, which is pretty small for modern AI,
that's 2 to the 50th power calculations per row.
Which is roughly A quadrillion. You are not running that on your
laptop. You aren't running that on a
supercomputer. It's just not feasible.
So is this all just a nice theory that we can't actually
use? No, and this is the main
contribution of Lundberg and Lee.
They developed approximation algorithms that get us very very
close to the true value without calculating every single subset.
OK, So what are the main solvers we need to know about?
The first one, and the most general is kernel shape.
This is a model agnostic method. It works on anything neural
networks, SVM's, logistic regression, any black box you
can throw at it. And how does it work
intuitively? It uses a specific type of
weighted linear regression, essentially fitting a simple
explainable model locally around the prediction you're trying to
explain to approximate the Shapley values.
It's actually based on an earlier method, client Lyme.
Is it fast? No.
It's slow. Very slow.
It requires a lot of sampling, which means running the model
many, many times. It's really the fall back option
if you have a weird custom model that nothing else supports.
OK, but most of our listeners are probably using tree based
models. Yeah, XG, Boost, light, GBM,
random forest. That's the bread and butter of
tabular data. For them there is tree HAPE and
this is a total game changer. Why is tree shape different?
Why is it so much better? Because tree shape leverages is
the actual structure of the decision trees, it doesn't need
to treat the model as a black box.
It looks inside, I see a tree is just a series of splits, right?
Is age greater than 50 yes or no?
So tree SHAP can calculate the exact Shapley values by
efficiently traversing the paths of the tree.
So it doesn't need to do all that random sampling we talked
about. Exactly.
No sampling needed. It computes the probability of
reaching different leaves based on the distribution of data in
the tree nodes. It turns an exponential problem
2 to the north into a polynomial problem.
In English. It's fast.
Extremely fast. If you are using XG boost or
light GBM. Tree shape is effectively free.
You should always use it. It makes explain ability
feasible for massive data sets. And for the deep learning crowd,
the neural net folks. There's deep HP.
It hooks into the neural network layers in a way that's similar
to how back propagation works, to propagate the important
scores from the output all the way back to the inputs.
It's an approximation, but it's designed to handle the
complexity of deep networks efficiently.
OK, so we have the tools, but we promised the listeners we'd talk
about the gotchas. Section 5 of the deep dive
because nothing is perfect. The biggest pitfall, and this is
something advanced practitioners argue about constantly, is the
independence assumption. We test on this with
marginalization. When we swap in a random value
for a feature, we are assuming we can do that without, you
know, breaking reality. Right, the core SHAP algorithm
assumes features are independent, but in the real
world features are often highly correlated.
Give me the five year old millionaire example.
This is my favorite illustration of the problem.
It's the perfect example of creating out of distribution
data. So imagine you have a model
predicting credit risk. You have two features, age and
income. And in the real world, these are
correlated. Generally you make more money as
you get older. Up to a point, right?
Now suppose we are calculating the SHAP value for age.
We take a data point, Let's say ACEO who is 55 and earns 500,000
a year. OK, to test the importance of
age, SHAP swaps out the CE OS real age for a random age from
the data set. And let's say the random age it
picks is. 5 exactly. So now we are asking the model
to make a prediction for a 5 year old earning $500,000.
Which is a data point that doesn't exist.
It's impossible. It's nonsensical.
It is completely off the manifold of real data.
The model has never seen anything like this during
training. It has no idea what to do with
it. So the model effectively
hallucinates it just makes something up it.
Extrapolates. It might spit out a really weird
prediction just because it's confused and that garbage
prediction gets fed into the SAP calculation.
So you get ASAP value that looks precise.
You know age contributed positive 0.2.
Yeah, but it's based on a scenario that is total nonsense.
Exactly. Garbage in, garbage out.
If features are heavily correlated, the credit
assignment can get very messy. The math tries to be fair, but
if the data points used to calculate the fairness are
themselves nonsensical, the result is compromised.
Is there a fix for this? There are variations of SAP that
try to account for correlation, but they are computationally
much much heavier. The practical advice is be aware
of it. If 2 features are 99%
correlated, like years of education and education level,
SHA might split the credit between them arbitrarily, or
they might even cancel each other out with one being
positive and one negative. So don't just blindly trust the
plot. You have to understand the data
structure underneath. If you see something weird,
check your correlations. Always SHAP is a tool for the
engineer to understand the model.
It doesn't replace the engineer. We're coming up on time, so I
want to zoom out. We've gone from 1950s game
theory to robo runners to snow detecting wolves to the math of
masking. It's a huge.
Landscape. If a listener takes away just
one thing, one core concept from all this, what should it be?
That SHAP is the bridge between accuracy and interpretability.
For a long time we thought we had to choose, you know, Do you
want a dumb model you can understand, like a linear
regression, or a smart model that's a black box?
SHA proves we can have both. We can use the complex high
accuracy models and still debug. Them, and I'd emphasize the
debug part. Yes, absolutely.
Don't just use this to make pretty charts for the quarterly
business review. Use it to check if your model is
learning the right patterns. Use it to catch the snow before
you deploy the wolf. That is the most valuable
application, period. But I want to leave us with a
provocative thought, something to chew on.
What's that? Let's say ASAP worked perfectly.
It tells us exactly why a model denied a loan.
It says I denied this loan because of the applicant's zip
code. Which is a huge red flag for
bias. That's textbook redlining.
Right, SHAP has done its job. It explained the why, but the
explanation itself is just information.
It doesn't fix the problem. The math is fair in how it
calculates the contribution. It accurately reports that the
zip code was the cause. But the reality it reveals is
deeply unfair. That is a profound distinction.
SHAP is a mirror. It reflects exactly what the
model learned from the data you gave it.
If the data has historical bias baked into it, the model will
learn it and SHA tool dutifully report back.
Yes, I am biased because of feature X.
So explainability isn't the solution to ethical AI, it's
just the diagnostic tool. You still need the human in the
loop to look at that SHA plot and say that's wrong.
We need to retrain, we need to fix the data.
Exactly. The tool reveals the flaw, The
human still has to fix it. Well, on that note, go check
your models, watch out for the snow, and try to avoid creating
too many five year old millionaires in your validation
sets. Sound advice.
We'll catch you on the next deep dive.