1
00:00:00,040 --> 00:00:02,760
Welcome to Inference and 
Intelligence Lab, a podcast 

2
00:00:02,760 --> 00:00:05,640
about making decisions you can 
trust using statistical 

3
00:00:05,640 --> 00:00:08,320
inference, causal inference, 
machine learning, and Gen. 

4
00:00:08,320 --> 00:00:11,800
AI evaluation. 
This is the Causal Inference 

5
00:00:11,800 --> 00:00:14,800
from the Ground Up series, where
we build the fundamentals of 

6
00:00:14,800 --> 00:00:18,760
causal inference step by step 
with simple mental models, real 

7
00:00:18,760 --> 00:00:21,840
world examples, and just enough 
math to be precise. 

8
00:00:22,280 --> 00:00:25,480
Today is episode 2 and we're 
talking about the bridge to 

9
00:00:25,480 --> 00:00:28,000
truth. 
Why identification comes before 

10
00:00:28,000 --> 00:00:30,520
estimation. 
It's good to be back and you 

11
00:00:30,520 --> 00:00:32,920
know this topic. 
I think it's the single most 

12
00:00:32,920 --> 00:00:34,920
critical distinction in our 
entire field. 

13
00:00:35,080 --> 00:00:37,240
I completely agree. 
It's the difference between 

14
00:00:37,240 --> 00:00:41,680
being a a a data miner just 
digging for shiny Nuggets, 

15
00:00:41,760 --> 00:00:44,760
right, and being a causal 
architect, someone who actually 

16
00:00:44,760 --> 00:00:47,000
understands the structure of the
problem they're modeling. 

17
00:00:47,000 --> 00:00:49,360
And we need that distinction 
because there's this reflex in 

18
00:00:49,360 --> 00:00:51,520
the industry, right? 
You build a model, it's not 

19
00:00:51,520 --> 00:00:53,480
quite working. 
What's the first thing everyone 

20
00:00:53,480 --> 00:00:55,560
says? 
Oh, always get more data. 

21
00:00:55,600 --> 00:00:57,240
Just feed the beast. 
Exactly. 

22
00:00:57,240 --> 00:01:00,600
Feed the beast. 
There's this like core belief 

23
00:01:00,600 --> 00:01:03,280
that if you have enough 
petabytes, the truth has to be 

24
00:01:03,280 --> 00:01:05,720
in there somewhere. 
And that is precisely what 

25
00:01:05,720 --> 00:01:07,880
researchers call the infinite 
data trap. 

26
00:01:08,200 --> 00:01:12,160
It's this incredibly seductive 
idea that just, you know, having

27
00:01:12,160 --> 00:01:15,200
N equal to Infinity is going to 
solve all your problems. 

28
00:01:15,320 --> 00:01:19,120
But for causal inference, that's
it's not just wrong, it's a 

29
00:01:19,120 --> 00:01:21,240
fatal flaw in your thinking. 
It is. 

30
00:01:21,560 --> 00:01:24,680
I mean, for a lot of ML, sure 
more data is better performance,

31
00:01:25,040 --> 00:01:29,040
image recognition, LLMS. 
It works, so why not here? 

32
00:01:29,040 --> 00:01:31,960
Why does that logic breakdown so
spectacularly? 

33
00:01:32,120 --> 00:01:34,200
Because those are prediction 
tasks. 

34
00:01:34,480 --> 00:01:37,400
All you need is correlation. 
If A predicts B, you're done. 

35
00:01:38,080 --> 00:01:40,960
We're asking if A causes BA. 
Completely different question. 

36
00:01:40,960 --> 00:01:44,960
And if that causal effect isn't 
identifiable, which is the 

37
00:01:44,960 --> 00:01:48,600
keyword for today, then an 
infinite amount of data won't 

38
00:01:48,600 --> 00:01:50,440
help you. 
Right, so you can have a model 

39
00:01:50,440 --> 00:01:54,680
with 99.9% validation accuracy. 
Trained on a mountain of. 

40
00:01:54,680 --> 00:01:56,760
Data, and it can still give you 
a causal estimate. 

41
00:01:56,760 --> 00:01:59,320
That is, I think the term is 
confidently wrong. 

42
00:01:59,320 --> 00:02:01,160
Which is the worst kind of 
wrong, isn't it? 

43
00:02:01,320 --> 00:02:03,040
It's misinformation that looks 
like math. 

44
00:02:03,360 --> 00:02:04,600
Yeah. 
You go into a meeting and say 

45
00:02:04,600 --> 00:02:06,840
this feature drove 10 million 
revenue and you've got the 

46
00:02:06,840 --> 00:02:08,840
charts and everything, but it's 
it's just a ghost. 

47
00:02:09,080 --> 00:02:11,760
A very precise, very expensive 
ghost. 

48
00:02:12,160 --> 00:02:15,400
So OK, This is why we need to 
separate the two ideas running 

49
00:02:15,400 --> 00:02:19,440
the algorithm. 
That's estimation, but first 

50
00:02:19,920 --> 00:02:22,760
figuring out if the question can
even be answered. 

51
00:02:23,360 --> 00:02:26,200
That's identification. 
I like to think of it as 

52
00:02:26,200 --> 00:02:28,960
building a bridge. 
Identification is the blueprint 

53
00:02:29,000 --> 00:02:31,960
it's asking. 
Can a bridge even be built here,

54
00:02:31,960 --> 00:02:33,680
given the physics and the 
terrain? 

55
00:02:34,000 --> 00:02:37,200
And estimation is just pouring 
the concrete. 

56
00:02:37,200 --> 00:02:39,600
And what do most teams do? 
They skip the blueprint and 

57
00:02:39,600 --> 00:02:42,360
start pouring concrete because 
you know, that's where the fun 

58
00:02:42,360 --> 00:02:44,720
Python libraries. 
Are we do love our libraries, 

59
00:02:44,840 --> 00:02:47,960
fit, predict, deploy. 
It feels so productive. 

60
00:02:48,120 --> 00:02:51,560
It does. 
But today we are all about the 

61
00:02:51,560 --> 00:02:54,680
blueprint and we get there. 
We need to get really Crist on 

62
00:02:54,680 --> 00:02:56,520
three terms. 
They're often called the three 

63
00:02:56,520 --> 00:02:57,760
trios. 
OK, let's walk through them. 

64
00:02:57,760 --> 00:02:59,600
They sound similar, and I think 
this is where a lot of the 

65
00:02:59,600 --> 00:03:01,200
confusion comes from. 
Absolutely. 

66
00:03:01,200 --> 00:03:05,040
So term #1 the causal estimate. 
That's estimate with AD. 

67
00:03:05,160 --> 00:03:08,280
Right, this is your North Star. 
The causal estimate is the 

68
00:03:09,080 --> 00:03:12,480
theoretical what if quantity 
that you actually care about. 

69
00:03:12,720 --> 00:03:14,560
It's defined by counterfactuals,
right? 

70
00:03:14,600 --> 00:03:17,920
Parallel universes. 
Exactly what would sales be if 

71
00:03:17,920 --> 00:03:20,800
everyone got the discount versus
if no one got the discount? 

72
00:03:20,920 --> 00:03:24,040
And the problem is you can't see
both of those worlds at once. 

73
00:03:24,040 --> 00:03:26,800
I can't give a user a coupon and
not give them a coupon at the 

74
00:03:26,800 --> 00:03:29,040
same time. 
That is the fundamental problem 

75
00:03:29,040 --> 00:03:32,880
of causal inference right there.
So this quantity, this causal 

76
00:03:32,880 --> 00:03:37,520
estimate, it's hypothetical. 
We often call it Theta star in 

77
00:03:37,520 --> 00:03:40,080
the notation. 
OK, so Theta star is the wish, 

78
00:03:40,080 --> 00:03:43,360
the truth we want. 
That brings us to #2 the 

79
00:03:43,360 --> 00:03:45,720
statistical estimate still with 
AD. 

80
00:03:46,280 --> 00:03:47,840
OK, so now we're coming down to 
earth. 

81
00:03:48,240 --> 00:03:51,760
The statistical estimate is an 
actual analytical expression. 

82
00:03:52,160 --> 00:03:54,720
It asks a really specific 
question, which is can we 

83
00:03:54,720 --> 00:03:59,080
rewrite that impossible causal 
wish using only probabilities we

84
00:03:59,080 --> 00:04:00,720
can actually measure from our 
data? 

85
00:04:01,360 --> 00:04:03,200
So it's a translation. 
You're trying to take the what 

86
00:04:03,200 --> 00:04:05,440
if and express it in terms of 
what is. 

87
00:04:05,520 --> 00:04:07,640
Perfectly put. 
You're trying to find a formula 

88
00:04:07,680 --> 00:04:10,880
using things like Pyx that 
mathematically equals your Theta

89
00:04:10,880 --> 00:04:11,360
star. 
OK. 

90
00:04:11,360 --> 00:04:14,440
That makes sense. 
And then #3 the statistical 

91
00:04:14,440 --> 00:04:17,040
estimate now with AT. 
This is the number. 

92
00:04:17,399 --> 00:04:21,720
This is Theta hat the concrete 
result you get when you run your

93
00:04:21,720 --> 00:04:24,600
algorithm on your actual finite 
data set. 

94
00:04:24,680 --> 00:04:29,520
So 4.2% left or P value of .05. 
That's the estimate. 

95
00:04:29,640 --> 00:04:31,440
That's the estimate. 
That's where machine learning 

96
00:04:31,440 --> 00:04:34,800
usually lives, approximating 
that statistical quantity. 

97
00:04:34,840 --> 00:04:36,560
OK, so. 
Let's recap that flow. 

98
00:04:37,040 --> 00:04:41,120
Causal estimate is the truth. 
Statistical estimate is the 

99
00:04:41,120 --> 00:04:43,760
formula we hope gets us to the 
truth. 

100
00:04:44,080 --> 00:04:46,000
The blueprint. 
And the statistical estimate is 

101
00:04:46,000 --> 00:04:48,200
the final number we calculate. 
The brick. 

102
00:04:48,520 --> 00:04:50,480
And here's the kicker. 
This is why this whole thing 

103
00:04:50,480 --> 00:04:52,560
matters so much. 
Identification is the link 

104
00:04:52,560 --> 00:04:55,920
between step one and Step 2. 
It's the arrow in the diagrams 

105
00:04:55,920 --> 00:04:58,600
that goes from the causal 
estimate into the statistical 

106
00:04:58,600 --> 00:05:00,920
estimate. 
Yes, and what is that arrow 

107
00:05:00,920 --> 00:05:02,720
always labeled? 
Assumptions. 

108
00:05:02,840 --> 00:05:05,720
That is the bridge to truth. 
That is the only thing 

109
00:05:05,720 --> 00:05:08,960
connecting the hypothetical 
reality you care about to the 

110
00:05:08,960 --> 00:05:10,800
data you actually have. 
And you can't. 

111
00:05:11,320 --> 00:05:13,680
You can't just check those 
assumptions with a Sequel query.

112
00:05:13,840 --> 00:05:16,280
No, you can't write a unit test 
for them. 

113
00:05:16,560 --> 00:05:17,960
You have to use domain 
knowledge. 

114
00:05:18,120 --> 00:05:19,840
You have to argue about how the 
world works. 

115
00:05:19,840 --> 00:05:22,200
It makes people uncomfortable. 
Because if that bridge of 

116
00:05:22,200 --> 00:05:25,680
assumptions is broken. 
You can't cross it, and trying 

117
00:05:25,680 --> 00:05:28,400
to cross it with more data, 
yeah, just means you fall into 

118
00:05:28,400 --> 00:05:31,160
the ravine with more confidence.
Right, you can't machine 

119
00:05:31,160 --> 00:05:33,600
learning your way out of an 
identification problem. 

120
00:05:33,600 --> 00:05:35,680
Never. 
If your assumptions are wrong, 

121
00:05:35,800 --> 00:05:38,160
your statistical estimate 
doesn't equal your causal 

122
00:05:38,160 --> 00:05:40,080
estimate. 
So the number you calculate at 

123
00:05:40,080 --> 00:05:43,800
the end is just. 
It's a precise estimate of the 

124
00:05:43,800 --> 00:05:45,880
wrong thing. 
OK, let's make this real. 

125
00:05:46,400 --> 00:05:49,120
I want to see this fail in a way
that people will recognize from 

126
00:05:49,120 --> 00:05:51,720
their own jobs. 
Let's do an e-commerce example. 

127
00:05:51,720 --> 00:05:53,920
Perfect. 
So you're a data scientist. 

128
00:05:54,320 --> 00:05:58,320
Marketing comes to you and asks,
does our new promo e-mail 'cause

129
00:05:58,320 --> 00:06:00,120
people to buy? 
Things simple enough. 

130
00:06:00,120 --> 00:06:02,280
I've got my database. 2 million 
users. 

131
00:06:02,320 --> 00:06:05,760
Yeah, a column for received 
e-mail, A column for purchased. 

132
00:06:05,760 --> 00:06:09,200
What's the naive approach? 
Well, I'd calculate the purchase

133
00:06:09,200 --> 00:06:12,040
rate for the group that got the 
e-mail, Subtract the rate for 

134
00:06:12,040 --> 00:06:14,240
the group that didn't. 
Boom, 10% lift. 

135
00:06:14,320 --> 00:06:18,040
I'm a hero. 
And you have just completely 

136
00:06:18,040 --> 00:06:21,400
failed the identification step. 
You calculated A correlation and

137
00:06:21,400 --> 00:06:23,560
you called it A cause. 
Because I didn't ask the most 

138
00:06:23,560 --> 00:06:26,440
important question. 
Why did certain people get that 

139
00:06:26,440 --> 00:06:28,880
e-mail in the 1st place? 
Was it random? 

140
00:06:29,320 --> 00:06:31,880
And let's say it wasn't. 
Let's say marketing targets 

141
00:06:32,600 --> 00:06:34,440
users they think are high 
potential. 

142
00:06:34,640 --> 00:06:36,600
People who are already browsing 
the site a lot. 

143
00:06:36,600 --> 00:06:39,480
Exactly O we have a hidden 
variable, A confounder, let's 

144
00:06:39,480 --> 00:06:43,040
call it prior engagement. 
OK, highly engaged users are 

145
00:06:43,040 --> 00:06:47,400
more likely to get the e-mail 
from the algorithm, but they're 

146
00:06:47,400 --> 00:06:50,360
also more likely to buy anyway, 
e-mail or no e-mail. 

147
00:06:50,520 --> 00:06:53,920
So engagement is causing both 
the treatment and the outcome. 

148
00:06:54,040 --> 00:06:57,080
It's that classic backdoor path.
And it messes everything up. 

149
00:06:57,160 --> 00:06:59,840
The group that got the e-mail 
looks amazing, but it's not 

150
00:06:59,840 --> 00:07:01,840
because of the e-mail, it's 
because they were already your 

151
00:07:01,840 --> 00:07:04,800
best customers. 
So to identify the true effect, 

152
00:07:05,120 --> 00:07:09,360
my statistical estimate, my 
formula has to account for that 

153
00:07:09,360 --> 00:07:11,960
I need to adjust for prior 
engagement. 

154
00:07:11,960 --> 00:07:16,080
Yes, you need to compare engaged
users who got the e-mail to 

155
00:07:16,080 --> 00:07:18,280
other engaged users who for some
reason didn't. 

156
00:07:18,520 --> 00:07:20,120
But what if you don't have that 
data? 

157
00:07:20,200 --> 00:07:24,200
What if prior engagement isn't a
column in my table? 

158
00:07:24,280 --> 00:07:26,760
Then the causal effect is not 
identifiable. 

159
00:07:26,800 --> 00:07:28,800
Just full stop. 
Full stop. 

160
00:07:29,120 --> 00:07:31,920
If you're missing a Keycon 
founder, you cannot 

161
00:07:31,920 --> 00:07:34,000
mathematically untangle the two 
effects. 

162
00:07:34,680 --> 00:07:37,760
The bridge is out. 
And no amount of extra data will

163
00:07:37,760 --> 00:07:40,160
build that bridge for me. 
It'll just narrow your error 

164
00:07:40,160 --> 00:07:43,240
bars around the wrong answer. 
You'll be more and more certain 

165
00:07:43,240 --> 00:07:45,200
about a number that is totally 
meaningless. 

166
00:07:45,200 --> 00:07:48,160
That's terrifying for anyone who
builds or uses dashboards. 

167
00:07:48,280 --> 00:07:52,480
It is, and it gets worse. 
Confounding is one problem, but 

168
00:07:52,480 --> 00:07:55,640
what about another assumption 
that's almost always broken in 

169
00:07:55,640 --> 00:08:01,160
tech interference? 
Yeah, the S in SUTVA. 

170
00:08:01,640 --> 00:08:04,440
The idea that my outcome only 
depends on my treatment. 

171
00:08:04,560 --> 00:08:06,320
It assumes every user is an 
island. 

172
00:08:06,360 --> 00:08:08,760
Which works for like clinical 
trials for headache pill. 

173
00:08:08,760 --> 00:08:10,440
My aspirin doesn't cure your 
headache. 

174
00:08:10,480 --> 00:08:14,080
Right, but what about a social 
network or an e-commerce 

175
00:08:14,080 --> 00:08:15,440
platform? 
Not a chance. 

176
00:08:15,440 --> 00:08:18,160
If I get an e-mail for 50% off a
new phone, the first thing I'm 

177
00:08:18,160 --> 00:08:20,640
going to do is text you. 
About it and now you who are 

178
00:08:20,640 --> 00:08:21,880
technically in the control 
group. 

179
00:08:21,960 --> 00:08:25,360
I didn't get the e-mail. 
You buy the product because I 

180
00:08:25,360 --> 00:08:27,440
got the treatment. 
The treatment spilled over. 

181
00:08:27,560 --> 00:08:29,160
Which contaminates my control 
group. 

182
00:08:29,160 --> 00:08:31,440
They're no longer a true 
baseline of what would have 

183
00:08:31,440 --> 00:08:34,120
happened otherwise. 
Right, your estimate of the lift

184
00:08:34,200 --> 00:08:37,600
is now biased downwards. 
You're under reporting the true 

185
00:08:37,600 --> 00:08:39,760
impact because the effect went 
viral. 

186
00:08:40,000 --> 00:08:44,200
So if my identification strategy
just assumes no interference and

187
00:08:44,200 --> 00:08:47,920
I work at a social media 
company, my bridge is already 

188
00:08:47,920 --> 00:08:50,480
buckling. 
It is so OK. 

189
00:08:50,600 --> 00:08:52,080
This all sounds pretty 
paralyzing. 

190
00:08:52,600 --> 00:08:55,840
We have hidden confounders, 
network effects, this infinite 

191
00:08:55,840 --> 00:08:56,920
data trap. 
Right. 

192
00:08:56,960 --> 00:08:59,480
If the assumptions are basically
never perfect, how do we get 

193
00:08:59,480 --> 00:09:01,480
anything done? 
Do we just say we can't know 

194
00:09:01,480 --> 00:09:03,240
anything? 
No, that's the other trap, 

195
00:09:03,240 --> 00:09:06,880
causal nihilism. 
The goal isn't perfection, it's 

196
00:09:06,880 --> 00:09:09,480
robustness. 
So what's the toolkit for that? 

197
00:09:09,480 --> 00:09:13,640
How does a causal architect, as 
you put it, handle a shaky 

198
00:09:13,640 --> 00:09:15,480
bridge? 
First, you are radically 

199
00:09:15,480 --> 00:09:16,960
transparent about the 
assumptions. 

200
00:09:17,000 --> 00:09:18,760
You don't hide them, you lead 
with them. 

201
00:09:19,160 --> 00:09:21,160
This result assumes there's no 
interference. 

202
00:09:21,160 --> 00:09:24,160
Which forces the product 
managers and the business folks 

203
00:09:24,160 --> 00:09:27,520
to actually think about the 
mechanism, not just the number. 

204
00:09:27,680 --> 00:09:32,280
Exactly. 
And second, you use a tool that 

205
00:09:32,280 --> 00:09:36,240
is so under used it's almost 
criminal sensitivity analysis. 

206
00:09:36,240 --> 00:09:38,680
So how does that work? 
You're kind of stress testing 

207
00:09:38,680 --> 00:09:40,960
the assumption. 
That's a perfect way to put it. 

208
00:09:41,120 --> 00:09:43,680
You ask the math. 
OK, let's pretend there is a 

209
00:09:43,680 --> 00:09:47,520
hidden confounder we missed. 
How strong would it have to be 

210
00:09:47,520 --> 00:09:49,560
to make my result go to 0? 
Oh, that's cool. 

211
00:09:49,560 --> 00:09:51,400
So you quantify the tipping 
point. 

212
00:09:51,480 --> 00:09:55,200
Yes, it gives you what's called 
an E value if the analysis says 

213
00:09:55,760 --> 00:09:58,920
an unobserved confounder would 
need to be 10 times stronger 

214
00:09:58,920 --> 00:10:02,800
than any variable we did measure
to erase this effect well. 

215
00:10:03,040 --> 00:10:04,760
Then you can feel pretty good 
about your result. 

216
00:10:04,760 --> 00:10:08,480
You can sleep at night, but if 
it says, even a tiny unobserved 

217
00:10:08,480 --> 00:10:10,360
effect could flip this entire 
result. 

218
00:10:10,400 --> 00:10:12,920
Then you flag it. 
You say this is a low confidence

219
00:10:12,920 --> 00:10:14,960
finding and we shouldn't bet the
company on it. 

220
00:10:14,960 --> 00:10:18,000
You're not just a calculator 
spitting out T values anymore, 

221
00:10:18,120 --> 00:10:20,720
you're delivering A structural 
integrity report for the 

222
00:10:20,720 --> 00:10:23,080
insight. 
That's a fantastic way to frame 

223
00:10:23,080 --> 00:10:25,320
it. 
Identification forces you to be 

224
00:10:25,320 --> 00:10:26,960
honest about the structure of 
the world. 

225
00:10:27,120 --> 00:10:28,880
It really does. 
It's the hard work that comes 

226
00:10:28,880 --> 00:10:31,480
before the fun code. 
You can pip install a regression

227
00:10:31,480 --> 00:10:33,480
model. 
You can't pip install critical 

228
00:10:33,480 --> 00:10:36,880
thinking about your data. 
And in a world where AI can run 

229
00:10:36,880 --> 00:10:40,120
the models for us, that human 
part, the identification part, 

230
00:10:41,040 --> 00:10:43,320
that becomes the most valuable 
skill. 

231
00:10:43,320 --> 00:10:45,760
It is the premium skill. 
OK, let's wrap this up. 

232
00:10:46,200 --> 00:10:49,760
We started with the danger of 
the infinite data trap. 

233
00:10:50,080 --> 00:10:52,400
The idea that more data solves 
bias. 

234
00:10:52,600 --> 00:10:56,320
It doesn't not causal bias. 
We broke down the three trios, 

235
00:10:57,200 --> 00:11:00,160
the causal estimate, the 
statistical estimate, the 

236
00:11:00,160 --> 00:11:02,240
blueprint, and the statistical 
estimate. 

237
00:11:02,240 --> 00:11:05,680
The final number? 
The brick and the key is that 

238
00:11:05,800 --> 00:11:08,960
identification is the bridge 
built on assumptions. 

239
00:11:09,040 --> 00:11:12,680
And we saw how things like 
unobserved confounders or user 

240
00:11:12,680 --> 00:11:15,720
interference can just break that
bridge completely. 

241
00:11:15,840 --> 00:11:19,200
But the answer isn't giving up. 
It's being honest and using 

242
00:11:19,200 --> 00:11:22,120
tools like sensitivity analysis 
to see how sturdy your bridge 

243
00:11:22,120 --> 00:11:23,800
actually is. 
So the next time you're looking 

244
00:11:23,800 --> 00:11:26,920
at a dashboard and you see some 
huge lift number for a new 

245
00:11:26,920 --> 00:11:29,760
feature, don't look at the P 
value first. 

246
00:11:29,760 --> 00:11:32,520
Resist the urge. 
Ask yourself, is this 

247
00:11:32,520 --> 00:11:34,880
identified? 
What are the assumptions 

248
00:11:34,880 --> 00:11:37,880
building the bridge between that
number and the actual truth? 

249
00:11:38,360 --> 00:11:40,680
And what happens if that bridge 
collapses? 

250
00:11:40,920 --> 00:11:43,600
That's the question that 
matters, because if you don't 

251
00:11:43,600 --> 00:11:45,720
ask it, eventually the market 
will. 

252
00:11:46,160 --> 00:11:48,040
Thanks for joining them, Ladder.
We'll see you in the next deep 

253
00:11:48,040 --> 00:11:48,320
dive.