1
00:00:20,290 --> 00:00:22,890
Welcome to Thinking Deeply About
Primary Education, the podcast 

2
00:00:22,890 --> 00:00:25,130
that makes time and space to 
think about pedagogy, teaching 

3
00:00:25,130 --> 00:00:26,690
and learning professional 
development. 

4
00:00:27,010 --> 00:00:29,490
Anything of interest to time, 
poorer but enthusiasm rich 

5
00:00:29,490 --> 00:00:32,049
primary teachers. 
This week I'm joined by Jonathan

6
00:00:32,049 --> 00:00:33,290
Kerr. 
Hi, Jonathan, how are you? 

7
00:00:33,850 --> 00:00:37,450
Hi, not too bad. 
Jonathan, we're going to explore

8
00:00:37,450 --> 00:00:40,410
evidence synthesis, which is 
something I know listeners will 

9
00:00:40,410 --> 00:00:44,560
be extremely interested to hear 
about, but I thought it made 

10
00:00:44,560 --> 00:00:47,200
sense for you to sort of explain
who you are, where you're from 

11
00:00:47,200 --> 00:00:48,800
and how you got here. 
Sure. 

12
00:00:48,800 --> 00:00:51,880
Yeah. 
So, hi, I'm John Kay. 

13
00:00:52,040 --> 00:00:56,000
I am Head of Evidence, Synthesis
and International at the 

14
00:00:56,000 --> 00:01:00,840
Education Endowment Foundation. 
And a lot of my career history 

15
00:01:00,840 --> 00:01:04,840
has been at the EEF. 
So actually on Sunday it was 10 

16
00:01:04,840 --> 00:01:08,360
years that I've been at the 
organization in lots of 

17
00:01:08,360 --> 00:01:10,920
different roles. 
So sometimes supporting with 

18
00:01:10,960 --> 00:01:14,920
more of our policy work, 
sometimes on our program 

19
00:01:14,920 --> 00:01:18,640
evaluations. 
But largely most of my time has 

20
00:01:18,640 --> 00:01:21,840
been focused on evidence 
synthesis and in particular a 

21
00:01:22,720 --> 00:01:25,760
couple of key resources that EF 
is responsible for. 

22
00:01:25,760 --> 00:01:29,760
So probably the most well known 
is our teaching and learning 

23
00:01:29,760 --> 00:01:34,960
tool kits, but also we produce 
guidance reports and and other 

24
00:01:34,960 --> 00:01:39,320
evidence reviews that hopefully 
are informative for practice. 10

25
00:01:39,320 --> 00:01:42,240
years is huge. 
How close to the very first 

26
00:01:42,240 --> 00:01:46,360
cohort of members were you? 
Pretty close. 

27
00:01:46,440 --> 00:01:49,200
So I never, I never feel like 
one of the originals. 

28
00:01:49,200 --> 00:01:52,440
There's still a couple of people
from from right at the start of 

29
00:01:52,440 --> 00:01:57,760
the EF, but I think we've been 
going 13 years now. 

30
00:01:57,960 --> 00:02:02,040
So we were started in 2011. 
So, so pretty close and a lot 

31
00:02:02,120 --> 00:02:07,240
of, a lot of changes, a lot of 
different areas and activity in 

32
00:02:07,240 --> 00:02:09,639
those ten years. 
Yes, Stuart Canyon does a talk 

33
00:02:09,639 --> 00:02:11,520
and he starts off by talking 
about what's happened since 

34
00:02:11,520 --> 00:02:14,200
2010. 
And it's always shocking. 

35
00:02:14,200 --> 00:02:17,680
It makes you feel really old. 
So I mean, I think, you know, 

36
00:02:17,680 --> 00:02:21,160
there's no one better place to 
explore the idea of evidence 

37
00:02:21,160 --> 00:02:23,080
synthesis. 
And I thought I'd start by 

38
00:02:23,080 --> 00:02:27,640
asking if you could share some 
insights on how the EEF 

39
00:02:27,680 --> 00:02:31,800
evaluates the the quality and 
robustness of educational 

40
00:02:31,800 --> 00:02:35,800
research and what criteria are 
used to determine which evidence

41
00:02:35,800 --> 00:02:38,520
is most reliable and actionable 
for teachers. 

42
00:02:39,600 --> 00:02:41,640
Yeah, sure. 
So, so I'll talk. 

43
00:02:42,120 --> 00:02:46,600
Umm about the three key 
questions that the EF asks when 

44
00:02:46,600 --> 00:02:51,480
thinking about evidence And I'll
I'll go into bits that are a bit

45
00:02:51,480 --> 00:02:55,080
more nerdy and methodological, 
but I'll I'll also try and 

46
00:02:55,400 --> 00:02:58,800
remain as accessible as possible
and definitely interrupt me if I

47
00:02:59,560 --> 00:03:03,520
if I explain something in a way 
that isn't entirely accessible. 

48
00:03:05,000 --> 00:03:07,200
So. 
So the first question we ask 

49
00:03:07,200 --> 00:03:12,880
ourselves if we were to find an 
individual study is could this 

50
00:03:12,880 --> 00:03:16,240
study tell us anything about the
question that we're interested 

51
00:03:16,240 --> 00:03:18,200
in. 
So there we'd be thinking about 

52
00:03:18,480 --> 00:03:21,640
has the study been designed in a
way that could answer the 

53
00:03:21,640 --> 00:03:24,360
question. 
And what's really important 

54
00:03:24,360 --> 00:03:28,520
there is there isn't going to be
a hard and fast rule that that 

55
00:03:28,520 --> 00:03:31,120
some studies are good and some 
studies are bad, but the 

56
00:03:31,120 --> 00:03:34,120
different methodologies we might
consider better or worse at 

57
00:03:34,520 --> 00:03:39,080
answering different questions. 
So if we were trying to answer a

58
00:03:39,080 --> 00:03:43,400
question about the efficacy of a
practice, then what we'd be 

59
00:03:43,400 --> 00:03:47,360
looking for at EF is some kind 
of comparison group in that 

60
00:03:47,360 --> 00:03:50,960
study. 
And that's that's mainly because

61
00:03:51,120 --> 00:03:53,800
children are always making 
progress as they go through 

62
00:03:53,800 --> 00:03:56,360
schools. 
So quite often you'll see a 

63
00:03:56,360 --> 00:04:00,160
study where someone is given a 
new maths intervention and you 

64
00:04:00,160 --> 00:04:02,480
measure them at the beginning 
and you measure them at the end 

65
00:04:02,480 --> 00:04:06,760
and you say, wow, all of the 
kids have improved in maths to 

66
00:04:06,760 --> 00:04:09,280
which any teacher would say, 
well of course they have because

67
00:04:09,400 --> 00:04:12,880
children are always progressing 
in maths and are receiving other

68
00:04:12,880 --> 00:04:18,000
bits of tuition and and support.
And so one of the things we're 

69
00:04:18,000 --> 00:04:22,400
we're quite strong on at EF is 
when we talk about efficacy we 

70
00:04:22,400 --> 00:04:25,280
we like to see some kind of 
comparison group. 

71
00:04:26,200 --> 00:04:29,360
It doesn't always have to be a 
randomized comparison group. 

72
00:04:29,360 --> 00:04:32,560
I think EF does have this 
reputation as loving randomized 

73
00:04:32,560 --> 00:04:36,880
control trials and and we've 
certainly funded a large number.

74
00:04:37,360 --> 00:04:39,720
But it's really important that 
there is some kind of 

75
00:04:39,720 --> 00:04:43,520
counterfactual so that you can 
understand is the intervention 

76
00:04:43,520 --> 00:04:47,560
or the approach or the new thing
that's being tried making a 

77
00:04:47,560 --> 00:04:51,040
difference over and above the 
the regular practice within a 

78
00:04:51,040 --> 00:04:53,360
classroom. 
But other questions we might 

79
00:04:53,440 --> 00:04:56,240
have other methodologies that 
that we think are appropriate. 

80
00:04:56,240 --> 00:05:01,040
So we also find quite a lot of 
qualitative evidence as part of 

81
00:05:01,040 --> 00:05:04,840
our evaluations and and 
qualitative evidence is much 

82
00:05:04,840 --> 00:05:07,800
better for understanding how 
things will be received in the 

83
00:05:07,800 --> 00:05:10,440
classroom. 
So their feasibility, their 

84
00:05:10,440 --> 00:05:14,360
acceptability, whether there are
any barriers to to their 

85
00:05:14,360 --> 00:05:16,760
implementation. 
So that first question, what 

86
00:05:16,760 --> 00:05:21,240
we're trying to understand is, 
is the methodology appropriate 

87
00:05:21,240 --> 00:05:25,320
to the question that we want to 
report or that that we want to 

88
00:05:25,320 --> 00:05:27,880
answer? 
Then the second question is you 

89
00:05:27,880 --> 00:05:31,800
might have a fantastically 
appropriate methodology, but you

90
00:05:31,800 --> 00:05:34,360
might have a terrible study 
within that methodology. 

91
00:05:34,640 --> 00:05:38,920
So the second question is really
about what is the quality within

92
00:05:38,920 --> 00:05:41,880
methodology. 
And in particular, we often 

93
00:05:41,880 --> 00:05:45,200
frame it as are there risks of 
bias? 

94
00:05:45,200 --> 00:05:48,960
So are there things that might 
be part of the study that might 

95
00:05:48,960 --> 00:05:51,920
make us less likely to trust the
results of it? 

96
00:05:52,800 --> 00:05:57,400
So one risk of bias might be is 
the study independence. 

97
00:05:57,680 --> 00:06:00,560
And there's lots of evidence 
that shows that when people 

98
00:06:00,560 --> 00:06:04,840
evaluate their own work, you 
tend to see higher impacts. 

99
00:06:05,120 --> 00:06:10,360
My favorite example is a lot of 
the evidence based on painting 

100
00:06:10,360 --> 00:06:12,480
classrooms. 
So you might have heard people 

101
00:06:12,480 --> 00:06:15,680
say you know you have a a blue 
calming classroom and all of the

102
00:06:15,680 --> 00:06:19,000
kids will be well behaved. 
Quite a lot of that research was

103
00:06:19,000 --> 00:06:21,320
funded by large American paint 
companies. 

104
00:06:21,760 --> 00:06:26,120
And so regardless of whether you
think the actual study is any 

105
00:06:26,120 --> 00:06:29,160
good, there might be some 
suspicions about about the kind 

106
00:06:29,160 --> 00:06:31,440
of the level of independence 
there and whether you're only 

107
00:06:31,440 --> 00:06:33,440
hearing about the positive 
results. 

108
00:06:34,200 --> 00:06:37,400
But there are lots of other 
aspects to to methodology or or 

109
00:06:37,400 --> 00:06:42,240
risks potentially to bias. 
So one might be around the size 

110
00:06:42,240 --> 00:06:46,000
of the study, you you might get 
a less secure result if you only

111
00:06:46,000 --> 00:06:49,160
test something with one or two 
kids than if you test it in 100 

112
00:06:49,160 --> 00:06:51,160
schools. 
And particularly when you think 

113
00:06:51,160 --> 00:06:54,160
about whether something will be 
generalizable, you really want 

114
00:06:54,160 --> 00:06:58,280
to have a a large number of of 
kids involved in the study to 

115
00:06:58,280 --> 00:07:00,400
make sure that it's not just a 
A1 off. 

116
00:07:00,640 --> 00:07:02,960
And then of course there's 
there's things like how many 

117
00:07:02,960 --> 00:07:08,040
people drop out of studies like 
we talk about attrition, how 

118
00:07:08,040 --> 00:07:10,200
comparable the groups actually 
are. 

119
00:07:10,240 --> 00:07:13,240
If you did a study where I 
talked about having a comparison

120
00:07:13,240 --> 00:07:16,000
group, but if your comparison 
group looks really, really 

121
00:07:16,000 --> 00:07:19,680
different to the intervention 
group, then maybe it's actually 

122
00:07:19,680 --> 00:07:22,840
the results you're saying or 
about differences in cohorts or 

123
00:07:22,840 --> 00:07:26,600
or population rather than the 
intervention itself. 

124
00:07:27,360 --> 00:07:31,600
So when we think about the the 
quality of the study within the 

125
00:07:31,600 --> 00:07:35,320
methodology, we tend to go 
through a a list of different 

126
00:07:35,320 --> 00:07:37,240
questions. 
I won't go into all of the the 

127
00:07:37,240 --> 00:07:39,600
kind of methodological detail 
but those are those are the 

128
00:07:39,600 --> 00:07:44,560
types of things we'd be asking. 
And then the final question is 

129
00:07:44,560 --> 00:07:48,200
how relevant the study is and 
and people often get caught up 

130
00:07:48,200 --> 00:07:53,080
on relevance being did it take 
place in the same country and 

131
00:07:53,600 --> 00:07:56,560
some things may be more 
transferable than others, that 

132
00:07:56,560 --> 00:07:58,840
there are things that we've 
tested that have been really 

133
00:07:58,840 --> 00:08:02,000
positive in America that then 
haven't translated at all. 

134
00:08:02,360 --> 00:08:05,640
But then there are also lots of 
theories of education that seem 

135
00:08:05,640 --> 00:08:09,000
to be applicable across lots and
lots of education systems. 

136
00:08:09,000 --> 00:08:12,360
You know, most education systems
consider how to give high 

137
00:08:12,360 --> 00:08:16,200
quality feedback to the children
in their settings. 

138
00:08:16,200 --> 00:08:21,080
Most systems will have some form
of scaffolding the the content 

139
00:08:21,080 --> 00:08:23,600
of their learning and and and 
approaches like that. 

140
00:08:24,520 --> 00:08:27,320
But it is really important to 
think about context, 

141
00:08:27,840 --> 00:08:30,880
particularly when we interpret 
what we can say from results. 

142
00:08:31,720 --> 00:08:34,640
And that isn't just geographic, 
but also might be about things 

143
00:08:34,640 --> 00:08:38,120
like outcomes. 
So a trends that you often see 

144
00:08:38,120 --> 00:08:42,960
in studies is studies measuring 
outcomes that are really, really

145
00:08:42,960 --> 00:08:46,360
close to the intervention and 
then making really, really big 

146
00:08:46,360 --> 00:08:48,480
claims on the basis of those 
studies. 

147
00:08:48,480 --> 00:08:53,880
So a classic example might be if
I wanted to evaluate Duolingo 

148
00:08:53,880 --> 00:08:58,080
and and learning some new 
vocabulary for through an app 

149
00:08:58,080 --> 00:08:59,880
like that, that's nicely 
gamified. 

150
00:09:00,240 --> 00:09:04,960
What I could do is test that I 
learned my 10 words of the day, 

151
00:09:05,320 --> 00:09:08,240
and then at the end I test 
whether I know those 10 words 

152
00:09:09,320 --> 00:09:12,160
and I will perform better on 
remembering those 10 words than 

153
00:09:12,160 --> 00:09:15,320
someone that has received no 
instruction whatsoever on those 

154
00:09:15,320 --> 00:09:17,680
10 specific words. 
You'll almost always get a 

155
00:09:17,680 --> 00:09:19,840
really exciting, promising 
result. 

156
00:09:20,360 --> 00:09:23,840
But does that tell you anything 
about learning Spanish language 

157
00:09:23,840 --> 00:09:26,600
in general or does that just 
measure something that's so 

158
00:09:26,600 --> 00:09:30,440
close to the intervention that 
it's, it's not really giving us 

159
00:09:30,440 --> 00:09:32,320
a, a, a big claim that we can 
make. 

160
00:09:32,600 --> 00:09:36,800
And so we think really carefully
about context not just in terms 

161
00:09:36,800 --> 00:09:41,960
of country but in terms of is 
the study, is the study 

162
00:09:42,120 --> 00:09:45,160
answering the question that that
we we want to make a claim 

163
00:09:45,160 --> 00:09:46,840
about. 
So we wouldn't try and make A a 

164
00:09:47,320 --> 00:09:51,440
sweeping claim about improving 
key stage 2 results on the basis

165
00:09:51,440 --> 00:09:56,520
of a vocab very proximal measure
and and together we think about 

166
00:09:56,520 --> 00:09:58,560
those three things when we talk 
about quality. 

167
00:09:58,560 --> 00:10:02,880
So, so methodology study quality
within the methodology or risk 

168
00:10:02,880 --> 00:10:05,120
of bias and and then relevance 
as well. 

169
00:10:05,280 --> 00:10:07,800
Fantastic. 
I love the the paint example 

170
00:10:07,800 --> 00:10:11,920
because I Reg reasonably 
regularly had emails from a 

171
00:10:12,160 --> 00:10:15,280
company and I don't know how 
best to describe and maintain 

172
00:10:15,280 --> 00:10:17,760
their anonymity. 
But uh, the first thing I ask is

173
00:10:17,760 --> 00:10:21,640
can I have every study that was 
associated with this the claims 

174
00:10:21,640 --> 00:10:24,560
that are being made here and 
they don't get sent to me So. 

175
00:10:26,280 --> 00:10:30,040
And Chris such always talks 
about how lots of reading 

176
00:10:30,040 --> 00:10:33,120
research even though I think 
reading is reasonably robust and

177
00:10:33,120 --> 00:10:35,920
certainly compared with 
mathematics he always, he's 

178
00:10:35,920 --> 00:10:39,480
always looking for comparisons 
within the within the trials 

179
00:10:39,480 --> 00:10:42,960
because he finds that that's 
something that doesn't doesn't 

180
00:10:42,960 --> 00:10:46,080
feature too heavily. 
And so yes, I think, I think 

181
00:10:46,080 --> 00:10:49,280
teachers can definitely, there's
lots of things they can look for

182
00:10:49,280 --> 00:10:51,920
themselves when they're engaging
with the with the papers. 

183
00:10:51,920 --> 00:10:55,320
I mean, what do you consider to 
be a large sample size? 

184
00:10:55,320 --> 00:10:56,880
Like what do you what's your 
benchmark? 

185
00:10:57,680 --> 00:11:00,960
Yeah, that's a, it's a good 
question because I think it does

186
00:11:00,960 --> 00:11:04,160
slightly depend on the type of 
question that you're answering. 

187
00:11:04,160 --> 00:11:07,200
So we've done some things that 
we call teacher choices trial 

188
00:11:07,200 --> 00:11:11,760
where we're testing a decision a
teacher might make in the 

189
00:11:11,760 --> 00:11:15,680
classroom and we might have a 
smaller sample for that then say

190
00:11:15,680 --> 00:11:18,760
a big structural intervention 
where we're looking across the 

191
00:11:18,760 --> 00:11:25,000
system. 
So EFS pipeline for an efficacy 

192
00:11:25,000 --> 00:11:28,120
trial, which might be a kind of 
a smallish trial that EFS 

193
00:11:28,120 --> 00:11:31,600
funding, we'd probably be 
looking for 50 intervention 

194
00:11:31,600 --> 00:11:33,520
schools and 50 comparison 
schools. 

195
00:11:33,520 --> 00:11:36,680
And then as we go to the larger 
ends of our trials and look at 

196
00:11:36,680 --> 00:11:39,120
things that might be ready for 
scale across the system, we 

197
00:11:39,120 --> 00:11:43,880
might be talking about 100 
schools versus 100 control 

198
00:11:43,880 --> 00:11:47,600
schools. 
I should say though, that almost

199
00:11:47,880 --> 00:11:52,600
a lot of the power you get from 
your sample size comes from not 

200
00:11:52,600 --> 00:11:56,120
the total number of pupils, but 
from where the comparison is 

201
00:11:56,120 --> 00:11:58,360
derived. 
So you might get a similar. 

202
00:11:58,640 --> 00:12:01,360
If you did have an intervention 
that you could test at the pupil

203
00:12:01,360 --> 00:12:05,680
level rather than at the school 
level, then 100 pupils each side

204
00:12:05,680 --> 00:12:09,720
might be a perfectly valid 
comparison a thing. 

205
00:12:09,720 --> 00:12:15,320
And this goes into kind of a 
very nerdy fractal I guess, but 

206
00:12:15,520 --> 00:12:18,400
I think that I would look at to 
try and understand whether it's 

207
00:12:18,400 --> 00:12:21,640
an appropriate sample size is 
the level of uncertainty around 

208
00:12:21,640 --> 00:12:25,120
the final result. 
So people often talk about 

209
00:12:25,120 --> 00:12:27,520
statistical significance of 
results. 

210
00:12:28,440 --> 00:12:32,440
We at EF don't really like to 
use the the kind of the binary 

211
00:12:32,440 --> 00:12:34,840
cut off of is something 
significant or not. 

212
00:12:35,280 --> 00:12:39,080
But you can always see there's 
some uncertainty, statistical 

213
00:12:39,080 --> 00:12:43,040
uncertainty around any result 
and depending on how well 

214
00:12:43,040 --> 00:12:46,560
powered or how appropriate the 
sample of the study is, that 

215
00:12:46,560 --> 00:12:50,800
statistical uncertainty will be 
either reduced or or increased. 

216
00:12:50,800 --> 00:12:53,720
So similarly, if you see a 
political poll, you might have a

217
00:12:53,720 --> 00:12:56,040
margin of error. 
All studies have have that 

218
00:12:56,040 --> 00:12:58,680
similar margin of error. 
I think I remember seeing once 

219
00:12:58,680 --> 00:13:01,720
there was maybe six months or 
so, there were times, but the 

220
00:13:01,720 --> 00:13:04,440
margin of error was also 12 
months or something like that. 

221
00:13:04,440 --> 00:13:06,120
I can't remember what the exact 
example was. 

222
00:13:06,560 --> 00:13:09,240
And so it almost meant that 
actually nothing was being said.

223
00:13:09,240 --> 00:13:09,880
So I can see. 
Yeah. 

224
00:13:09,880 --> 00:13:13,440
And it's really, it's really 
hard to interpret that because 

225
00:13:13,760 --> 00:13:16,720
you know there is always going 
to be uncertainty and we 

226
00:13:16,720 --> 00:13:20,280
shouldn't, you know, it's 
educated, you can't wait for the

227
00:13:20,280 --> 00:13:23,560
perfect result that gives you 
100% certainty. 

228
00:13:23,560 --> 00:13:26,840
We're always going to be talking
about best bets and we're going 

229
00:13:26,840 --> 00:13:31,400
to be talking about risk. 
You know, I I don't think we 

230
00:13:31,400 --> 00:13:34,280
ever will get to the point where
you can say if you do this you 

231
00:13:34,280 --> 00:13:38,080
will get exactly this result. 
But we should definitely be 

232
00:13:38,080 --> 00:13:40,680
cautious. 
If we're seeing really, really 

233
00:13:41,000 --> 00:13:44,760
uncertain results that include 
really, really positive results 

234
00:13:44,760 --> 00:13:48,280
and also incredibly negative 
results, then that might change 

235
00:13:48,280 --> 00:13:50,720
our risk appetite when we're 
thinking about doing something 

236
00:13:50,720 --> 00:13:52,360
in a classroom. 
Nice. 

237
00:13:52,360 --> 00:13:55,520
I mean, as the sort of lead 
person on this, do you feel any 

238
00:13:55,520 --> 00:13:58,880
pressure with the final message 
that comes out of all this, this

239
00:13:58,920 --> 00:14:03,080
sort of analysis? 
I I should be very clear that I 

240
00:14:03,080 --> 00:14:07,240
have a team around me, so I've 
got plenty of people to blame. 

241
00:14:07,560 --> 00:14:12,040
But I I think that the important
thing that that makes me 

242
00:14:12,040 --> 00:14:17,440
confident actually is and and we
can we can talk about this when 

243
00:14:17,440 --> 00:14:21,960
we talk about synthesis more 
generally when we do evidence 

244
00:14:21,960 --> 00:14:26,440
synthesis, a key part of it is 
having a really clear pre 

245
00:14:26,440 --> 00:14:30,760
specified process. 
That means that my opinion 

246
00:14:30,760 --> 00:14:34,760
shouldn't come into it. 
And so almost what we get out in

247
00:14:34,760 --> 00:14:37,960
the tool kit, I don't really 
have a say in sometimes I'll be 

248
00:14:37,960 --> 00:14:40,920
really surprised by it, but I 
won't have a say in it. 

249
00:14:41,080 --> 00:14:45,240
And what we do is we specify a 
process for engaging studies and

250
00:14:45,240 --> 00:14:49,240
we specify analysis processes 
before we see what any of the 

251
00:14:49,240 --> 00:14:52,480
results are. 
And so sometimes I might be 

252
00:14:52,480 --> 00:14:57,080
nervous about the way a result 
might be received or sometimes I

253
00:14:57,080 --> 00:15:01,040
might be nervous about did we 
get our process completely 

254
00:15:01,040 --> 00:15:03,680
right. 
But I I try and stay out of the 

255
00:15:03,680 --> 00:15:07,960
kind of that final how the 
result is because I think it's 

256
00:15:08,240 --> 00:15:11,360
the the value of it has to be 
that it's process based rather 

257
00:15:11,360 --> 00:15:14,760
than than what I think about 
metacognition or or what I think

258
00:15:14,760 --> 00:15:16,040
about feedback. 
Nice. 

259
00:15:16,040 --> 00:15:17,760
So yeah, so you've almost got 
like this. 

260
00:15:17,760 --> 00:15:21,520
If if the system works, then, 
you know, there's less pressure 

261
00:15:21,520 --> 00:15:23,760
on yourself, which is probably 
healthy. 

262
00:15:23,840 --> 00:15:27,800
And if the system doesn't work, 
then I'm worried and then very 

263
00:15:27,800 --> 00:15:33,760
accountable as well. 
I can teach you to effectively 

264
00:15:33,760 --> 00:15:37,400
navigate and make sense of the 
what is essentially a very vast 

265
00:15:37,400 --> 00:15:41,160
body of educational research, 
and particularly when findings 

266
00:15:41,160 --> 00:15:45,200
may seem contradictory, 
counterintuitive in my 

267
00:15:45,200 --> 00:15:47,840
experience or like you said, 
context specific. 

268
00:15:48,160 --> 00:15:51,000
Yeah and and I'm going to 
reflect my bias here and 

269
00:15:51,120 --> 00:15:53,800
evidence synthesis is in my my 
job title. 

270
00:15:53,880 --> 00:15:58,280
So I do have a a bias, but but I
would when looking for research,

271
00:15:58,280 --> 00:16:02,680
I would always try and look 
first for reviews and evidence 

272
00:16:02,680 --> 00:16:05,040
synthesis rather than single 
studies. 

273
00:16:06,040 --> 00:16:07,640
And there's a few reasons for 
that. 

274
00:16:08,280 --> 00:16:10,960
One is just convenient. 
If you find a really good 

275
00:16:10,960 --> 00:16:15,200
systematic review, then rather 
than reading 200 individual 

276
00:16:15,200 --> 00:16:18,640
studies, you can read one 
summary that hopefully is able 

277
00:16:18,640 --> 00:16:20,640
to deal with those 
contradictions. 

278
00:16:22,040 --> 00:16:25,000
Secondly, it's more rigorous 
because if you look at any 

279
00:16:25,000 --> 00:16:27,960
individual study, there's always
the chance that that you have 

280
00:16:27,960 --> 00:16:31,600
that outlier. 
And there there are outliers for

281
00:16:31,600 --> 00:16:34,120
lots of reasons. 
But a real risk is the things 

282
00:16:34,120 --> 00:16:37,600
that tend to get most coverage 
or most attention are often the 

283
00:16:37,600 --> 00:16:40,640
strangest results. 
So the things that might be 

284
00:16:41,520 --> 00:16:45,840
hardest or hardest to find might
actually be the most 

285
00:16:45,840 --> 00:16:52,160
representative studies. 
And so I think that what a good 

286
00:16:52,160 --> 00:16:54,800
synthesis should do is pull 
evidence together. 

287
00:16:55,160 --> 00:16:59,000
And I'm certainly not suggesting
that every teacher individually 

288
00:16:59,000 --> 00:17:01,560
does a systematic review and 
then meta analysis. 

289
00:17:01,560 --> 00:17:06,079
But I think that's part of the 
role of organisations like EF is

290
00:17:06,880 --> 00:17:10,359
Also authors of systematic 
reviews is compiling evidence 

291
00:17:10,359 --> 00:17:15,520
into a way that can hopefully be
interpreted really easily by 

292
00:17:15,520 --> 00:17:19,200
teachers or school leaders and 
giving them an overview of the 

293
00:17:19,200 --> 00:17:23,640
evidence space. 
So, so when we do a A toolkit 

294
00:17:23,640 --> 00:17:26,599
strand for example, as I said, 
we we have this independent 

295
00:17:26,599 --> 00:17:30,920
process where what we try and do
is look for every single study 

296
00:17:31,120 --> 00:17:34,880
at the beginning we'll lay out 
some criteria for inclusion. 

297
00:17:35,280 --> 00:17:39,760
So we might say we only want 
studies with children age 3 to 

298
00:17:39,760 --> 00:17:44,880
18, but we only want studies 
that have an outcome that is an 

299
00:17:44,880 --> 00:17:47,640
attainment outcome. 
But then we're trying to find 

300
00:17:47,720 --> 00:17:50,600
every single study. 
So we're not just trying to find

301
00:17:50,600 --> 00:17:54,240
studies that match our opinion 
or studies that are done by 

302
00:17:54,240 --> 00:17:59,520
people we know or even studies 
that are in the most the closest

303
00:17:59,520 --> 00:18:01,400
context. 
So we'd be looking for studies 

304
00:18:01,400 --> 00:18:04,800
from around the world and then 
we'd be gathering them together 

305
00:18:04,840 --> 00:18:07,760
and trying to summarize the 
average impacts. 

306
00:18:07,760 --> 00:18:11,600
And hopefully that gives a sense
of where the overall evidence 

307
00:18:11,600 --> 00:18:13,800
base is. 
Of course, the important thing 

308
00:18:13,800 --> 00:18:17,880
then is to be able to understand
what sits behind the average 

309
00:18:17,880 --> 00:18:20,680
because maybe you want to know 
what is the impact just for 

310
00:18:20,680 --> 00:18:23,080
schools in England. 
So what is the impact just in 

311
00:18:23,080 --> 00:18:25,800
mathematics or just in in 
reading outcomes? 

312
00:18:26,200 --> 00:18:30,120
And so we also do a process once
we've got all of the studies 

313
00:18:30,120 --> 00:18:33,040
together, once we found the 
studies that we're interested 

314
00:18:33,040 --> 00:18:37,400
in, of doing analysis that can 
compare how different factors 

315
00:18:37,400 --> 00:18:39,840
explain the variation of those 
studies. 

316
00:18:40,120 --> 00:18:43,200
So we do a process called meta 
regression or moderator 

317
00:18:43,200 --> 00:18:47,320
analysis, and what that allows 
us to do is say all of these 

318
00:18:47,320 --> 00:18:51,040
studies have very different 
impacts and all of these studies

319
00:18:51,040 --> 00:18:53,280
have slightly different 
characteristics. 

320
00:18:53,640 --> 00:18:56,160
Now, what are the 
characteristics that seem to 

321
00:18:56,160 --> 00:18:59,200
explain the variations in 
impact? 

322
00:18:59,480 --> 00:19:02,600
And sometimes that will be a 
pedagogical factor. 

323
00:19:02,600 --> 00:19:05,240
So maybe something is more 
effective when it's delivered by

324
00:19:05,240 --> 00:19:07,960
teachers than by teaching 
assistants or digital 

325
00:19:07,960 --> 00:19:10,040
technology. 
Sometimes that'll be a 

326
00:19:10,040 --> 00:19:12,440
methodological factor that we've
got a correct for. 

327
00:19:12,440 --> 00:19:16,000
Maybe we're seeing much lower 
impacts when we're having more 

328
00:19:16,000 --> 00:19:18,080
rigorous studies and we want to 
make sure that we're 

329
00:19:18,080 --> 00:19:21,000
appropriately describing the 
security of the evidence base. 

330
00:19:21,120 --> 00:19:25,000
But sitting behind that is is a 
process that hopefully gives 

331
00:19:25,000 --> 00:19:28,960
teachers access to a much larger
evidence base and and a much 

332
00:19:28,960 --> 00:19:31,680
more rigorous and independent 
evidence base than looking for 

333
00:19:31,680 --> 00:19:34,880
an individual study. 
So if I was a a teacher trying 

334
00:19:34,880 --> 00:19:38,000
to navigate and make sense of 
the vast body of of research, 

335
00:19:38,000 --> 00:19:42,680
what I'm trying to 1st is go, is
there a review Maybe that's the 

336
00:19:42,680 --> 00:19:45,640
tool kit, maybe it's the 
guidance report, maybe it's just

337
00:19:45,640 --> 00:19:48,880
putting what topic you're 
interested in and the phrase 

338
00:19:48,880 --> 00:19:52,680
systematic review because there 
is a vast body of literature out

339
00:19:52,680 --> 00:19:54,640
there. 
Obviously that won't always be 

340
00:19:54,640 --> 00:19:56,120
the case. 
That hasn't always been a a 

341
00:19:56,120 --> 00:19:59,960
review done and then there's 
probably a smaller number of 

342
00:19:59,960 --> 00:20:02,280
single studies. 
And what I try and do is look 

343
00:20:02,280 --> 00:20:06,240
across the evidence base, not 
just take the two studies that 

344
00:20:06,240 --> 00:20:09,360
look closest to your setting 
because context isn't 

345
00:20:09,360 --> 00:20:14,480
everything, but to try and in a 
broad ranging approach, assess 

346
00:20:14,480 --> 00:20:17,520
the quality of the studies, the 
relevance of the studies and the

347
00:20:17,520 --> 00:20:20,680
underlying methodology of the 
studies to see in combination 

348
00:20:20,680 --> 00:20:23,000
what you think that message 
might be that. 

349
00:20:23,320 --> 00:20:26,120
Makes total sense. 
I mean, I've got a favorite meta

350
00:20:26,120 --> 00:20:27,640
analysis I think. 
I think it's pronounced 

351
00:20:27,640 --> 00:20:29,200
carbonyl. 
I think it was 2014. 

352
00:20:29,200 --> 00:20:31,160
They explored the use of 
manipulatives in classes. 

353
00:20:31,880 --> 00:20:34,360
How interesting A. 
Lot of those threads I used to 

354
00:20:34,360 --> 00:20:36,560
sort of like you say go down 
rabbit holes and things like 

355
00:20:36,560 --> 00:20:38,440
that there. 
I mean I think I may have seen 

356
00:20:38,440 --> 00:20:40,640
people talking about the use of 
artificial intelligence to 

357
00:20:40,640 --> 00:20:43,360
conduct systematic reviews. 
Have you seen, has this come 

358
00:20:43,360 --> 00:20:45,440
into your sort of peripheral 
vision or anything recently? 

359
00:20:46,200 --> 00:20:50,320
That is, it's very much on trend
at the moment and the big move 

360
00:20:50,320 --> 00:20:53,600
that that we're trying to do, 
particularly if the toolkit is 

361
00:20:53,880 --> 00:20:57,160
for the toolkit to be as living 
as possible. 

362
00:20:57,520 --> 00:21:01,000
So what you want with your 
reviews is not only that they 

363
00:21:01,000 --> 00:21:04,040
summarize the evidence base, but
they continue to summarize the 

364
00:21:04,040 --> 00:21:07,280
evidence base. 
So as new studies are published,

365
00:21:07,280 --> 00:21:11,080
you want to really quickly add 
them to your evidence base so 

366
00:21:11,080 --> 00:21:15,120
that you don't immediately have 
your go to resource being just 

367
00:21:15,120 --> 00:21:17,080
evidence from five or six years 
ago. 

368
00:21:17,240 --> 00:21:21,560
And So what AI is frequently 
seen as the solution to is how 

369
00:21:21,560 --> 00:21:25,480
can we find the studies quicker 
and then how can we extract 

370
00:21:25,480 --> 00:21:27,280
information from the studies 
quicker. 

371
00:21:27,560 --> 00:21:30,040
So underpinning the toolkit. 
At the moment, there's a lot of 

372
00:21:30,040 --> 00:21:32,800
people that I get to go on 
podcast and take credit for 

373
00:21:32,800 --> 00:21:35,400
their work. 
But we have, you know, 20 

374
00:21:35,400 --> 00:21:39,200
research assistants that work 
with Durham University whose 

375
00:21:39,200 --> 00:21:42,280
jobs are to read studies and 
extract information from those 

376
00:21:42,280 --> 00:21:44,320
studies so that we can collate 
them. 

377
00:21:44,880 --> 00:21:50,280
And that I'm sure may sound 
enticing, but after you've been 

378
00:21:50,280 --> 00:21:53,520
doing it day after day is not 
the most glamorous or 

379
00:21:53,520 --> 00:21:56,480
interesting job. 
And it's the type of thing that 

380
00:21:56,480 --> 00:22:00,680
we can work with AI to try and 
make more efficient so that we 

381
00:22:00,680 --> 00:22:03,560
can use those people for the the
harder and more interesting 

382
00:22:03,560 --> 00:22:05,680
challenges around communicating 
evidence. 

383
00:22:05,800 --> 00:22:09,600
And so at the moment on the 
Toolkit we always use bits of 

384
00:22:09,600 --> 00:22:12,280
machine learning. 
It's not generative AI, but we 

385
00:22:12,280 --> 00:22:15,680
use machine learning to help us 
find studies more efficiently. 

386
00:22:16,080 --> 00:22:19,360
So within the toolkit, the 
published toolkit, I think 

387
00:22:19,360 --> 00:22:23,440
that's around 2500 studies. 
And what machine learning can do

388
00:22:23,440 --> 00:22:28,440
is say we can look at all of the
information from these studies 

389
00:22:28,440 --> 00:22:31,560
and suggest studies that look 
like them, so that we can find 

390
00:22:31,560 --> 00:22:34,440
the studies that we're missing. 
So at the moment, there's a big 

391
00:22:34,440 --> 00:22:38,440
project underpinning the toolkit
that's using machine learning to

392
00:22:38,480 --> 00:22:42,120
find those studies and to screen
those studies more efficiently. 

393
00:22:42,600 --> 00:22:45,560
The big win that everybody wants
is to be able to get the 

394
00:22:45,560 --> 00:22:48,680
machines to just extract all the
information from the study so 

395
00:22:48,680 --> 00:22:52,520
that you could really quickly 
run a meta analysis or or really

396
00:22:52,520 --> 00:22:56,160
quickly run a narrative summary.
But I think we're a long way 

397
00:22:56,160 --> 00:22:58,280
there. 
And I think the thing that 

398
00:22:58,280 --> 00:23:03,760
worries me is, is the idea that 
ChatGPT or chat bot can 

399
00:23:03,760 --> 00:23:10,800
summarize the evidence 
effectively already because they

400
00:23:10,800 --> 00:23:13,800
really can't. 
And and the thing that machines 

401
00:23:13,800 --> 00:23:18,280
are really bad at is what they 
tend to do is take the findings 

402
00:23:18,280 --> 00:23:20,720
from individual studies and 
summarize them and try and 

403
00:23:20,720 --> 00:23:23,200
amalgamate them. 
But the thing that Metro 

404
00:23:23,200 --> 00:23:27,680
analysis allows you to do is get
the quantitative trends. 

405
00:23:28,000 --> 00:23:32,400
And so we were talking earlier 
about studies that may have a 

406
00:23:32,400 --> 00:23:34,880
lot of statistical uncertainty 
and might be lacking in 

407
00:23:34,880 --> 00:23:38,320
statistical power. 
Now if you have 10 studies that 

408
00:23:38,360 --> 00:23:44,000
are all underpowered, give very 
kind of notional findings, they 

409
00:23:44,000 --> 00:23:46,560
maybe they all have a very small
positive result. 

410
00:23:47,040 --> 00:23:50,320
Those 10 studies together might 
tell us something really 

411
00:23:50,320 --> 00:23:53,440
meaningful. 
And if we did a metro analysis, 

412
00:23:53,440 --> 00:23:56,640
we might get a really 
statistically secure result by 

413
00:23:56,640 --> 00:23:59,640
looking at all of the small 
effect sizes together. 

414
00:24:00,400 --> 00:24:04,000
But each individual author might
say there's no evidence of 

415
00:24:04,000 --> 00:24:07,000
impact. 
So if you give that to ChatGPT, 

416
00:24:07,000 --> 00:24:10,040
it will look at those 10 studies
and at the end it will say we're

417
00:24:10,040 --> 00:24:12,600
really confident there's no 
evidence of impact. 

418
00:24:12,960 --> 00:24:15,760
Because we found ten studies 
that all agree that there's no 

419
00:24:15,760 --> 00:24:19,120
evidence of impact. 
Because it it's not quite yet to

420
00:24:19,120 --> 00:24:21,560
the point where it can go 
through the processes of of 

421
00:24:21,560 --> 00:24:25,320
taking out the statistical data 
and then manipulating and and 

422
00:24:25,320 --> 00:24:27,840
combining it. 
I'm sure at some point it will 

423
00:24:27,840 --> 00:24:32,120
get there and I'll become as 
expensible as as anyone else but

424
00:24:32,520 --> 00:24:34,000
but but we're not quite there 
yet. 

425
00:24:34,080 --> 00:24:35,240
I totally see where you're 
coming from. 

426
00:24:35,240 --> 00:24:37,480
I mean, Elon Musk seems to think
that there's this point in the 

427
00:24:37,480 --> 00:24:42,080
near future where sort of AI 
will know more than the combined

428
00:24:42,080 --> 00:24:44,560
sum of human knowledge. 
But I think there's something 

429
00:24:44,560 --> 00:24:46,880
innately human in engaging with 
research. 

430
00:24:46,880 --> 00:24:50,440
And for me, I'm not. 
I'm not yet ready to, you know, 

431
00:24:50,440 --> 00:24:53,720
I'd rather read the paper than 
ask a computer, but it thinks 

432
00:24:54,640 --> 00:24:57,680
meta aggression sounds 
particularly complicated. 

433
00:24:57,800 --> 00:25:02,120
And I mean is that almost the 
perhaps one of the reasons we 

434
00:25:02,120 --> 00:25:07,240
need large bodies like the EEF 
because you've got the manpower 

435
00:25:07,240 --> 00:25:11,480
to effectively negotiate that 
and whereas it feels like it's 

436
00:25:11,720 --> 00:25:14,880
almost beyond the scope of 
individuals within the school 

437
00:25:14,880 --> 00:25:16,800
system. 
I've I've learned Metro 

438
00:25:16,800 --> 00:25:20,480
aggression and I I don't think 
it's it's purely the kind of the

439
00:25:20,480 --> 00:25:24,760
might of the the EEF that has 
that has enabled that. 

440
00:25:24,760 --> 00:25:30,440
But I think I think for the EF 
to be a useful role in the 

441
00:25:30,440 --> 00:25:35,440
system what we want to do is the
most burdensome, the most kind 

442
00:25:35,440 --> 00:25:38,520
of time intensive bits of 
summarizing research. 

443
00:25:38,800 --> 00:25:42,040
We want to do that in a way that
is rigorous, independent and 

444
00:25:42,040 --> 00:25:45,360
gives practitioners access to 
the findings. 

445
00:25:45,360 --> 00:25:48,240
And so I think the thing that 
I'd say on synthesis is, 

446
00:25:48,240 --> 00:25:52,480
regardless of the meta 
aggression, I think the feedback

447
00:25:52,480 --> 00:25:55,680
strands in the toolkit has 
around 200 studies in it. 

448
00:25:56,120 --> 00:26:01,160
Now I myself wouldn't want to 
sit and read all 200 of those 

449
00:26:01,160 --> 00:26:04,080
studies and I doubt any 
individual teacher would want to

450
00:26:04,080 --> 00:26:07,520
sit and read all 200 of those 
studies because there are much 

451
00:26:07,520 --> 00:26:09,800
better things that they can 
spend their time doing. 

452
00:26:10,240 --> 00:26:13,920
But by working with a team of 
research assistants, extracting 

453
00:26:13,920 --> 00:26:18,000
the data, we can do the really 
time intensive part that then 

454
00:26:18,000 --> 00:26:21,320
gives teachers access to that 
evidence base of those 200 

455
00:26:21,320 --> 00:26:24,400
studies and understanding the 
average and what sits behind the

456
00:26:24,400 --> 00:26:27,960
average and also interesting 
findings for implementation. 

457
00:26:28,360 --> 00:26:32,160
And so it's really important 
that we empower teachers to 

458
00:26:32,160 --> 00:26:34,760
engage with the research and 
challenge the research and be 

459
00:26:34,760 --> 00:26:39,000
able to zoom in on individual 
studies and think about how to 

460
00:26:39,000 --> 00:26:42,120
question information that's 
presented to them in the system.

461
00:26:42,120 --> 00:26:44,000
Because not all of it will be 
reliable. 

462
00:26:44,440 --> 00:26:47,920
But I think the aim of what the 
EF wants to do is those really 

463
00:26:47,920 --> 00:26:51,000
time intensive parts of 
interpreting the research is to 

464
00:26:51,040 --> 00:26:54,160
to try and do as much of that as
possible and and present it in 

465
00:26:54,160 --> 00:26:57,600
as accessible a way as possible.
Now you mentioned independence, 

466
00:26:57,600 --> 00:27:02,160
obviously we are in at the end 
of an electoral cycle. 

467
00:27:02,160 --> 00:27:03,960
I think if we managed to make it
through five years with our 

468
00:27:03,960 --> 00:27:09,400
general action almost, will the 
EF continue on beyond that? 

469
00:27:09,400 --> 00:27:12,360
Are you dependent on? 
No, no, we, we, I mean I think 

470
00:27:12,360 --> 00:27:15,840
it's really important that we 
are completely independent of 

471
00:27:15,840 --> 00:27:20,040
government. 
So the the EF funding model 

472
00:27:20,040 --> 00:27:23,200
unlike lots of kind of things 
that were initially funded by 

473
00:27:23,200 --> 00:27:27,360
government is that we were given
our endowment as a lump sum. 

474
00:27:28,320 --> 00:27:31,920
And what that meant that we 
could do is firstly plan for the

475
00:27:31,920 --> 00:27:34,440
long term. 
So doing the studies that we 

476
00:27:34,440 --> 00:27:38,480
fund takes a really long time, 
you know and an EF funded RCT 

477
00:27:38,960 --> 00:27:41,560
from commissioning to 
publication is, is probably 

478
00:27:41,560 --> 00:27:45,640
around three years. 
But most importantly what it 

479
00:27:45,640 --> 00:27:49,600
means is we have no government 
involvement in the decisions 

480
00:27:49,600 --> 00:27:52,440
that we make. 
So our our research agenda is 

481
00:27:52,440 --> 00:27:57,360
completely decided by us and the
government gets no say over what

482
00:27:57,360 --> 00:27:59,360
we publish. 
We send them the results. 

483
00:27:59,360 --> 00:28:03,440
We hope they're interested, but 
there is no world in which they 

484
00:28:03,440 --> 00:28:05,640
can say do this or don't do 
this. 

485
00:28:07,040 --> 00:28:12,960
So we will continue to exist. 
We have funding for for at least

486
00:28:12,960 --> 00:28:16,520
another 10 years and and the 
thing that that makes that 

487
00:28:16,520 --> 00:28:20,120
powerful and I think makes the 
EF work is the fact that we are 

488
00:28:20,400 --> 00:28:22,520
are not part of government 
machinery. 

489
00:28:22,520 --> 00:28:25,720
And we don't have to go each 
year to negotiate with the 

490
00:28:25,720 --> 00:28:29,680
Secretary of State and say you 
know give us some more money 

491
00:28:29,880 --> 00:28:32,720
because it's always those points
of leverage that you might lose 

492
00:28:32,720 --> 00:28:35,760
that independence. 
And I think there are lots of 

493
00:28:35,760 --> 00:28:38,440
things that we all agree on but 
there are still lots of areas 

494
00:28:38,440 --> 00:28:41,680
that are are really contested in
education and and evidence 

495
00:28:41,680 --> 00:28:44,080
should be a way of cutting 
through that rather than 

496
00:28:44,400 --> 00:28:46,920
something that that becomes 
ideological. 

497
00:28:46,920 --> 00:28:50,680
And I think a lot of the the 
methods that we put in place are

498
00:28:50,680 --> 00:28:52,880
also about protecting that 
independence. 

499
00:28:52,880 --> 00:28:56,960
So that systematic review 
approach that I talked through, 

500
00:28:57,320 --> 00:28:59,480
part of that is because it's the
most rigorous. 

501
00:28:59,800 --> 00:29:03,000
I think it's also really 
important that having a process 

502
00:29:03,000 --> 00:29:07,280
that's transparent and that is 
replicable means that people 

503
00:29:07,280 --> 00:29:10,120
understand where where EF 
getting its results from. 

504
00:29:10,440 --> 00:29:13,800
And you wouldn't want it if 
everything we put out seemed 

505
00:29:13,800 --> 00:29:17,440
like the opinion of our CEO or 
the opinion of me or or of 

506
00:29:17,440 --> 00:29:21,240
another team member. 
It always has to be very process

507
00:29:21,240 --> 00:29:24,160
driven and sometimes that can 
make it very technical or 

508
00:29:24,160 --> 00:29:29,080
sometimes a bit opaque, but I 
think the transparency, 

509
00:29:29,080 --> 00:29:32,680
independence and replicability 
of things that we think a lot 

510
00:29:32,680 --> 00:29:34,720
about at the EEF think that is 
really. 

511
00:29:34,720 --> 00:29:37,440
Powerful. 
I mean that 10 years, you know, 

512
00:29:37,840 --> 00:29:42,480
because I remember when I when I
went to Shanghai in 2014, they 

513
00:29:42,480 --> 00:29:44,920
talked about how they had a 10 
year plan for education. 

514
00:29:44,920 --> 00:29:48,880
And I know that the place of 
education in politics and vice 

515
00:29:48,880 --> 00:29:53,520
versa is is reasonably contested
and but in my mind it would be 

516
00:29:53,520 --> 00:29:56,480
great if there were like this 10
year agenda and then we see how 

517
00:29:56,480 --> 00:29:58,360
things are going and it almost 
feels like you've got this, 

518
00:29:58,720 --> 00:30:01,040
you're doing that and you know 
it might not be like you said 

519
00:30:01,040 --> 00:30:03,680
what the Secretary of State is, 
talented schools but actually 

520
00:30:03,680 --> 00:30:07,920
this is more powerful because it
goes beyond and political 

521
00:30:07,920 --> 00:30:09,280
influence. 
Yeah. 

522
00:30:09,280 --> 00:30:12,160
And I think, you know it's 
really easy to get caught up on 

523
00:30:12,160 --> 00:30:15,760
the the kind of policy side and 
we do have a policy team that 

524
00:30:15,840 --> 00:30:19,400
that try and inform the DfE with
our research, but actually the 

525
00:30:19,920 --> 00:30:22,280
the powerful act that's going to
make a difference in the 

526
00:30:22,280 --> 00:30:24,160
outcomes we care about the 
schools. 

527
00:30:25,120 --> 00:30:28,440
And so the thing that I think 
means that the EF can be a 

528
00:30:28,440 --> 00:30:32,320
success and the thing that means
that we're actually very 

529
00:30:32,960 --> 00:30:36,720
unlikely to get political 
interference is because it kind 

530
00:30:36,720 --> 00:30:38,280
of doesn't matter that 
conversation. 

531
00:30:38,280 --> 00:30:42,520
The conversation that matters is
is the EF and research into 

532
00:30:42,520 --> 00:30:45,880
schools, and that conversation 
can happen regardless of what's 

533
00:30:45,880 --> 00:30:47,640
happening in the Department for 
Education. 

534
00:30:47,840 --> 00:30:49,480
Absolutely. 
I'm really pleased to ask now 

535
00:30:49,520 --> 00:30:52,320
because that's like really 
useful information and really 

536
00:30:52,320 --> 00:30:58,080
important information. 
So we we've talked almost about 

537
00:30:58,840 --> 00:31:02,000
the work of the EEF, what 
teachers might want to do if 

538
00:31:02,000 --> 00:31:04,120
they want to effectively 
navigate research. 

539
00:31:05,240 --> 00:31:09,000
What common pitfalls, or perhaps
misconceptions, should teachers 

540
00:31:09,000 --> 00:31:12,800
be aware of when attempting to 
implement evidence based 

541
00:31:12,800 --> 00:31:17,200
practices in their classrooms? 
So I think that the main thing 

542
00:31:17,360 --> 00:31:21,080
is not to, not to trust us too 
much. 

543
00:31:21,560 --> 00:31:24,880
So I think that it's really 
important that evidence is a 

544
00:31:24,880 --> 00:31:28,520
best bet and it it it tells us 
something that's useful. 

545
00:31:28,520 --> 00:31:31,680
If I was doing an approach, I'd 
want to know if that approach 

546
00:31:31,680 --> 00:31:35,440
had been successful in 100 
classrooms or unsuccessful in 

547
00:31:35,440 --> 00:31:38,200
100 classrooms. 
But I don't think it's ever 

548
00:31:38,200 --> 00:31:41,400
going to be a prediction. 
And so I think that the key 

549
00:31:41,400 --> 00:31:45,760
thing in terms of implementing 
evidence based education is 

550
00:31:47,120 --> 00:31:50,760
using teachers professional 
expertise when thinking about 

551
00:31:50,760 --> 00:31:55,400
how to implement that evidence. 
And so doing a full 

552
00:31:55,400 --> 00:31:58,680
implementation process where you
think about what are the needs 

553
00:31:58,680 --> 00:32:01,960
of the children in my classroom 
and not starting with a 

554
00:32:01,960 --> 00:32:03,520
solution, it'd be really easy to
go. 

555
00:32:03,520 --> 00:32:05,040
There's a new evidence based 
thing. 

556
00:32:05,040 --> 00:32:08,320
So I'm just going to do that. 
But but that isn't really how to

557
00:32:08,320 --> 00:32:10,800
do things. 
Well, thinking about what do I 

558
00:32:10,800 --> 00:32:13,960
need, then looking at the 
evidence and hopefully looking 

559
00:32:14,160 --> 00:32:16,840
at a systematic overview of the 
evidence to find the best 

560
00:32:16,840 --> 00:32:18,920
solutions. 
And then going through an 

561
00:32:18,920 --> 00:32:22,920
implementation process that that
brings the rest of the school 

562
00:32:22,920 --> 00:32:25,520
with you. 
And I think that the hardest 

563
00:32:25,520 --> 00:32:28,000
thing actually isn't finding the
evidence. 

564
00:32:28,000 --> 00:32:30,520
The hardest thing is how you 
then mobilize the evidence 

565
00:32:30,520 --> 00:32:33,800
through the school. 
This is timely. 

566
00:32:33,800 --> 00:32:36,760
We we recently published an 
implementation guidance report 

567
00:32:36,760 --> 00:32:41,560
that that hopefully provides 
some supportive frameworks for 

568
00:32:41,560 --> 00:32:44,600
how you might implement evidence
based approaches. 

569
00:32:45,000 --> 00:32:51,320
I think the key lesson there 
across all of it is, is the main

570
00:32:51,320 --> 00:32:54,680
kind of enabling factor is the 
knowledge of the teachers of 

571
00:32:54,680 --> 00:32:58,400
their classrooms and that it's 
really easy to get into a world 

572
00:32:58,400 --> 00:33:01,120
in which evidence and 
professional expertise seem like

573
00:33:01,120 --> 00:33:03,160
they're in competition, but 
they're really not. 

574
00:33:03,560 --> 00:33:06,800
One is hopefully supporting the 
other and and hopefully that's a

575
00:33:06,800 --> 00:33:09,720
positive conversation going 
between those two factors. 

576
00:33:10,320 --> 00:33:13,240
And I think that the other kind 
of key things are not trying to 

577
00:33:13,240 --> 00:33:16,600
do too many things at once. 
We've definitely, maybe this 

578
00:33:16,600 --> 00:33:18,320
goes back to the policy 
conversation. 

579
00:33:18,320 --> 00:33:21,320
There's there's often a kind of 
impetus of in a particular 

580
00:33:21,320 --> 00:33:25,120
region or particular school, 
let's throw 100 different 

581
00:33:25,120 --> 00:33:28,160
interventions and and try and 
improve the school. 

582
00:33:28,560 --> 00:33:32,840
But doing 100 things badly and 
then stopping most of them after

583
00:33:33,160 --> 00:33:37,440
six weeks, two months a year 
isn't really the way to make 

584
00:33:37,440 --> 00:33:39,800
anything work. 
And I think the key lesson from 

585
00:33:39,800 --> 00:33:44,680
the evidence is if you look 
through even the most positive 

586
00:33:44,680 --> 00:33:48,760
strands in the toolkit there are
examples where it has a negative

587
00:33:48,760 --> 00:33:52,480
impact and and that should be 
intuitive, you know the most 

588
00:33:52,480 --> 00:33:55,400
well evidenced approach in the 
world and if you really wanted 

589
00:33:55,400 --> 00:33:57,360
it to fail, you could make it 
fail. 

590
00:33:57,360 --> 00:34:00,400
I know I could make anything 
fail completely and have really 

591
00:34:00,400 --> 00:34:04,800
negative impacts on on children 
and. 

592
00:34:04,800 --> 00:34:08,920
And so I think it's that buy in 
that that professional 

593
00:34:08,920 --> 00:34:12,920
application of the knowledge and
that targeting appropriately is 

594
00:34:12,920 --> 00:34:15,840
always going to come from the 
people in the classroom, you 

595
00:34:15,840 --> 00:34:18,159
know the sense that I take. 
From the the recent report is 

596
00:34:18,159 --> 00:34:20,080
definitely it's about you know 
how can we bring everyone along 

597
00:34:20,080 --> 00:34:23,440
together and and really make 
this impactful and it's almost 

598
00:34:23,440 --> 00:34:25,679
like you know when you've got 
the breathing space and clean 

599
00:34:25,679 --> 00:34:28,520
school then that's when these 
things go really well. 

600
00:34:28,520 --> 00:34:31,080
So it's almost a case of make 
the breathing space for your 

601
00:34:31,080 --> 00:34:33,920
teachers, you know, if you're 
and it it. 

602
00:34:33,920 --> 00:34:38,560
Takes a really long time and 
it's like revisiting things and 

603
00:34:38,960 --> 00:34:42,760
checking things have embedded so
that it's not intervention after

604
00:34:42,760 --> 00:34:47,080
intervention but it becomes a 
part of the the school processes

605
00:34:47,120 --> 00:34:50,360
Is is really challenging because
we can all get excited for the 

606
00:34:50,360 --> 00:34:54,199
first six months of something 
but it then becomes quite boring

607
00:34:54,199 --> 00:34:58,280
and not exciting and and that's 
the the tricky but I think one 

608
00:34:58,280 --> 00:34:59,520
of the things I get asked about 
quite. 

609
00:34:59,520 --> 00:35:04,200
Often is curriculum and choice 
of curriculum and my response is

610
00:35:04,200 --> 00:35:06,360
generally it doesn't really 
matter which one you have the 

611
00:35:06,360 --> 00:35:08,800
best in the world. 
Like you say, it's very easy to 

612
00:35:08,880 --> 00:35:13,080
make things feel, but what 
matters is, is that this is a 7 

613
00:35:13,080 --> 00:35:15,320
year journey and it's those kids
who are near 1. 

614
00:35:15,320 --> 00:35:18,800
Now you've got to wait to see 
how impactful this was over that

615
00:35:18,800 --> 00:35:21,960
time so I can, I can totally see
where you're where you're coming

616
00:35:21,960 --> 00:35:26,560
from. 
Now my last question I. 

617
00:35:26,560 --> 00:35:29,640
Don't know if it's fair or not. 
It's almost like asking you to 

618
00:35:29,640 --> 00:35:33,880
pick your favorite, your 
favorite children, and like what

619
00:35:33,880 --> 00:35:36,520
are the most significant 
evidence based strategies or 

620
00:35:36,520 --> 00:35:39,800
interventions that have 
consistently demonstrated 

621
00:35:39,800 --> 00:35:43,840
positive impacts on student 
learning outcomes and perhaps 

622
00:35:43,840 --> 00:35:46,840
across educational settings and 
age groups if such a thing 

623
00:35:46,840 --> 00:35:52,640
exists, I think simultaneously. 
Interesting and boring that the 

624
00:35:52,640 --> 00:35:55,640
things that have positive 
impacts are probably the things 

625
00:35:55,640 --> 00:35:57,480
you expect to have positive 
impacts. 

626
00:35:58,480 --> 00:36:02,880
So there is a large evidence 
base that shows that additional 

627
00:36:02,880 --> 00:36:06,600
small group tuition provides 
positive outcomes and it 

628
00:36:06,600 --> 00:36:10,160
probably isn't that surprising 
you get more targeted support 

629
00:36:10,200 --> 00:36:13,600
particularly if it's well 
targeted to to kids that really 

630
00:36:13,600 --> 00:36:16,720
need the support, it's likely to
have a positive impact. 

631
00:36:17,640 --> 00:36:22,040
Similarly, high quality feedback
is generally has a really 

632
00:36:22,040 --> 00:36:25,560
positive impact for examples of 
it having less positive impacts,

633
00:36:25,560 --> 00:36:28,480
but generally it has a very 
positive impact. 

634
00:36:28,480 --> 00:36:32,520
And I think one of the lessons 
in evidence synthesis is we get 

635
00:36:32,520 --> 00:36:35,720
fewer shocking results. 
Actually a lot of the results of

636
00:36:35,720 --> 00:36:38,680
the things that that you might 
just expect to be the case 

637
00:36:38,680 --> 00:36:42,240
because what we're doing is 
summarizing an evidence base 

638
00:36:42,240 --> 00:36:45,040
that builds on professional 
knowledge that the practitioners

639
00:36:45,200 --> 00:36:48,480
often know really well. 
Then the interesting bit though 

640
00:36:48,520 --> 00:36:51,680
is, is what sits behind those 
averages of those positive 

641
00:36:51,680 --> 00:36:55,240
impacts. 
So you know feedback we know is 

642
00:36:55,240 --> 00:36:58,480
really positive, but we've also 
seen examples where it's been 

643
00:36:58,600 --> 00:37:02,800
negative in unexpected ways. 
So the rise of kind of triple 

644
00:37:02,800 --> 00:37:07,800
marking policies where workload 
was being really damaged you you

645
00:37:07,800 --> 00:37:10,280
have less time for planning 
lessons, you've had less time 

646
00:37:10,280 --> 00:37:13,560
for quality teaching and there 
wasn't really an evidence base 

647
00:37:13,560 --> 00:37:15,400
for that beyond feedback is 
good. 

648
00:37:15,400 --> 00:37:18,600
And when we looked at the 
evidence base for those triple 

649
00:37:18,600 --> 00:37:21,560
marking policies, it was pretty 
much non existent. 

650
00:37:22,080 --> 00:37:26,160
And so again it's that 
combination of as research as we

651
00:37:26,160 --> 00:37:29,160
need to be good, EF needs to be 
good at looking behind the 

652
00:37:29,160 --> 00:37:33,040
average, anticipating those 
unexpected trade-offs and making

653
00:37:33,040 --> 00:37:36,800
sure that we don't just look at 
impacts, but we also look about 

654
00:37:36,800 --> 00:37:39,400
how it's received by the 
profession, the qualitative 

655
00:37:39,400 --> 00:37:41,880
staff, the barriers, the 
workload. 

656
00:37:43,680 --> 00:37:47,520
But then as a profession that we
think about how we implement 

657
00:37:47,520 --> 00:37:50,480
things in ways that are most 
efficient and aren't creating 

658
00:37:50,480 --> 00:37:54,840
those those kind of increases on
on workload and then similarly 

659
00:37:54,840 --> 00:37:57,760
small group tuition, you know 
that can be done really well or 

660
00:37:57,760 --> 00:37:59,840
really badly. 
If small group tuition is very 

661
00:37:59,840 --> 00:38:04,480
tokenistic, not connected to 
classroom learning, then it 

662
00:38:04,480 --> 00:38:07,160
won't have a big impact. 
And we've we've got a really 

663
00:38:07,160 --> 00:38:11,480
strong synthesis in the tool kit
that shows an overall positive 

664
00:38:11,480 --> 00:38:13,680
impacts. 
But we've also evaluated 

665
00:38:13,680 --> 00:38:17,400
specific programs where we've 
seen real mixed impact and some 

666
00:38:17,400 --> 00:38:21,120
are positive, some are null, 
some are even slightly negative.

667
00:38:21,400 --> 00:38:23,800
I suppose in answer to your 
question it depends what you 

668
00:38:23,800 --> 00:38:26,680
mean by consistently. 
There's nothing that is always 

669
00:38:26,680 --> 00:38:30,360
positive but there are things 
that that trend towards positive

670
00:38:30,360 --> 00:38:34,880
and a consistently consistently 
positive or more consistently 

671
00:38:34,880 --> 00:38:38,760
positive and and for those I'd 
say, and I'm very much quoting 

672
00:38:38,760 --> 00:38:42,400
from the toolkit here but but 
feedback metacognition 

673
00:38:42,400 --> 00:38:45,320
approaches, although they also 
have their pitfalls you know 

674
00:38:45,320 --> 00:38:48,920
they can be done really badly or
really well small group tuition 

675
00:38:49,080 --> 00:38:51,560
and also some of the approaches 
around reading. 

676
00:38:51,560 --> 00:38:54,680
So appropriately targeted 
support for reading 

677
00:38:54,680 --> 00:38:58,560
comprehension and combined with 
with phonological approaches, 

678
00:38:59,240 --> 00:39:01,560
supporting of early reading. 
I think it's reassuring that 

679
00:39:01,560 --> 00:39:03,040
those seem. 
To be the things that T that 

680
00:39:03,040 --> 00:39:06,000
schools are focusing their 
attention on. 

681
00:39:06,920 --> 00:39:10,280
I mean, I once tried to follow 
the thread for triple marking to

682
00:39:10,280 --> 00:39:15,120
see if I could find the origin. 
I got to a 2003, you know, State

683
00:39:15,120 --> 00:39:18,800
of the Union, Ofsted Inspector 
or Chief Inspector report. 

684
00:39:18,960 --> 00:39:20,320
How far back did you get in 
here? 

685
00:39:20,320 --> 00:39:23,680
Do you remember? 
That is a good question. 

686
00:39:23,680 --> 00:39:27,840
I mean we would have probably 
hit a blank wolf sooner than 

687
00:39:27,840 --> 00:39:30,480
that because we'd have been 
looking one. 

688
00:39:30,800 --> 00:39:34,280
When we pre specify what we 
want, we'd have been saying, 

689
00:39:34,480 --> 00:39:37,160
well what we want in order for 
us to be making any claim about 

690
00:39:37,160 --> 00:39:41,400
efficacy is some kind of 
comparison group is some kind of

691
00:39:42,160 --> 00:39:44,560
attainment outcome. 
Not just people saying they 

692
00:39:44,560 --> 00:39:47,080
think it works, but but some 
evidence of that actually 

693
00:39:47,080 --> 00:39:50,640
changing outcomes. 
And once you set your parameters

694
00:39:50,840 --> 00:39:53,560
for things that would allow you 
to make the kind of claims 

695
00:39:53,560 --> 00:39:57,960
people were making about triple 
marking, you very quickly find 

696
00:39:57,960 --> 00:40:01,080
there's nothing. 
And so I think we, we wouldn't 

697
00:40:01,080 --> 00:40:04,600
have done the deep investigation
you did to see the more kind of 

698
00:40:04,600 --> 00:40:07,520
I guess political markers that 
have got us to the point of it 

699
00:40:07,520 --> 00:40:10,320
being sad. 
But very quickly found that the 

700
00:40:10,320 --> 00:40:15,360
actual efficacy of the approach 
was was not particularly strong 

701
00:40:15,480 --> 00:40:17,120
for me it's probably one of the 
most important. 

702
00:40:17,600 --> 00:40:21,200
Parts of the conversation over 
the last ten years or so is that

703
00:40:21,200 --> 00:40:23,560
is this idea that actually there
are ways we can spend our time 

704
00:40:23,560 --> 00:40:26,640
that don't involve this but can 
actually be more impactful. 

705
00:40:27,920 --> 00:40:29,760
Yeah. 
And I think you know, we talked 

706
00:40:29,760 --> 00:40:33,080
a bit about implementation. 
I think one of the things that 

707
00:40:33,560 --> 00:40:37,080
that I'm proud of, stuff that 
the EF hopefully occasionally 

708
00:40:37,080 --> 00:40:39,000
does is support the 
implementation. 

709
00:40:39,400 --> 00:40:42,160
So it's it's less on the 
synthesis side, but we've 

710
00:40:42,680 --> 00:40:47,320
evaluated I think about 200 
different programs in the 

711
00:40:47,320 --> 00:40:51,680
English education system and 
only about one in four of those 

712
00:40:51,760 --> 00:40:55,600
has a positive impact and 
evidence that, oh, you're a bit 

713
00:40:55,600 --> 00:40:57,880
depressed by that. 
Surely EF should have a higher 

714
00:40:57,880 --> 00:40:59,600
hit rate than that. 
You should be getting lots of 

715
00:40:59,600 --> 00:41:03,240
positive results. 
But those negative results or no

716
00:41:03,240 --> 00:41:06,560
results are really important 
because they're not programs 

717
00:41:06,560 --> 00:41:09,480
that EF have dreamed up. 
Those are programs that people 

718
00:41:09,480 --> 00:41:12,360
are selling to schools and that 
people are spending their time 

719
00:41:12,360 --> 00:41:14,400
doing. 
And when you're finding that 

720
00:41:14,400 --> 00:41:18,080
they're not having a a positive 
impact over and above just 

721
00:41:18,080 --> 00:41:21,760
regular business as usual and 
people are spending money on 

722
00:41:21,760 --> 00:41:25,320
them, that really becomes a de 
implementation challenge. 

723
00:41:25,320 --> 00:41:31,120
And I won't name organizations 
for fear of legal repercussions 

724
00:41:31,120 --> 00:41:33,560
if nothing else. 
But but but some of the things 

725
00:41:33,560 --> 00:41:37,640
I'm proud to stop is where we 
see organizations cease to do as

726
00:41:37,640 --> 00:41:40,480
much business as they used to 
because they've had a negative 

727
00:41:40,480 --> 00:41:44,080
result in an EF trial because 
that is the sign that actually 

728
00:41:44,080 --> 00:41:47,440
people aren't buying a product 
that that wasn't improving 

729
00:41:47,440 --> 00:41:51,720
outcomes. 
And so I think you know business

730
00:41:51,720 --> 00:41:54,240
as usual in English schools is 
really good. 

731
00:41:54,480 --> 00:41:57,400
So we should be sceptical that 
the new program or the new 

732
00:41:57,400 --> 00:42:00,840
evidence based fad is going to 
fundamentally change outcomes 

733
00:42:00,840 --> 00:42:03,160
because what's happening already
is really good. 

734
00:42:04,160 --> 00:42:09,040
And so I think that part of the 
power of evidence is cutting 

735
00:42:09,040 --> 00:42:12,160
through those fads and and 
supporting the DE implementation

736
00:42:12,160 --> 00:42:15,040
process when we might be doing 
extra things that that aren't 

737
00:42:15,040 --> 00:42:17,600
having a positive impact. 
I think that's massive and I'm 

738
00:42:17,600 --> 00:42:18,720
really. 
Grateful you didn't mention any 

739
00:42:18,720 --> 00:42:21,320
names, because the podcast legal
team isn't as big. 

740
00:42:23,760 --> 00:42:27,840
And but yeah, you're absolutely.
Yeah, because time is important.

741
00:42:27,840 --> 00:42:30,080
Budgets and schools are 
extremely important in the 

742
00:42:30,120 --> 00:42:32,080
present. 
So you know that you're 

743
00:42:32,080 --> 00:42:35,240
contributing to the reduction. 
In time. 

744
00:42:35,240 --> 00:42:37,720
Waste, which I think is really, 
really important. 

745
00:42:37,960 --> 00:42:41,400
It's been fascinating talking to
you about. 

746
00:42:41,400 --> 00:42:44,320
Sort of. 
The work that you do, the the 

747
00:42:44,320 --> 00:42:47,200
ways the teachers can benefit 
from it, Is there anything you 

748
00:42:47,200 --> 00:42:49,120
think? 
It'd be great if every teacher 

749
00:42:49,120 --> 00:42:51,720
in the country knew this and 
that. 

750
00:42:51,760 --> 00:42:54,480
You might want to share with 
them a lot of the key things. 

751
00:42:54,480 --> 00:42:57,840
That that I've already said. 
So I think that seeing evidence 

752
00:42:57,840 --> 00:43:01,320
is something that can support 
but not remove professional 

753
00:43:01,320 --> 00:43:05,000
expertise is really important. 
I guess there's a really nerdy 

754
00:43:05,000 --> 00:43:11,480
one which is thinking about what
a good evidence review is. 

755
00:43:11,840 --> 00:43:16,360
Is is kind of my day-to-day work
at the EF and I've talked a lot 

756
00:43:16,600 --> 00:43:21,000
across the podcast on the 
process that goes behind what we

757
00:43:21,000 --> 00:43:25,440
call a systematic review. 
But one of the things that 

758
00:43:25,440 --> 00:43:29,240
really worries me is things that
call themselves evidence reviews

759
00:43:29,560 --> 00:43:32,160
but but don't follow that 
systematic process. 

760
00:43:32,200 --> 00:43:35,600
And so I think that one message 
I'd give generally to the 

761
00:43:35,600 --> 00:43:38,840
profession is when engaging with
things that call themselves 

762
00:43:38,840 --> 00:43:44,920
evidence reviews being really, 
really critical of things that 

763
00:43:44,920 --> 00:43:48,120
don't have some kind of 
systematic process. 

764
00:43:48,120 --> 00:43:52,160
Because what really worries me 
is evidence becoming a bit of a 

765
00:43:52,160 --> 00:43:55,280
buzzwords. 
And the idea that everyone 

766
00:43:55,280 --> 00:43:58,320
claims that everything is 
evidence informed and that 

767
00:43:58,320 --> 00:44:02,120
everything is, we've got great 
research underpinning that 

768
00:44:02,120 --> 00:44:06,600
whether it's a paint company or 
an organization trying to make 

769
00:44:06,600 --> 00:44:09,800
claims about education in the 
system. 

770
00:44:09,800 --> 00:44:15,240
And evidence reviews can either 
be a systematic process whereby 

771
00:44:15,240 --> 00:44:18,120
we look at all the evidence in a
non bias way and collate 

772
00:44:18,120 --> 00:44:22,400
findings or they can be an 
argument where you say here are 

773
00:44:22,400 --> 00:44:25,680
my opinions and I'm going to 
find as many sources as possible

774
00:44:25,960 --> 00:44:30,240
to support those opinions. 
And I think that as we think 

775
00:44:30,240 --> 00:44:33,400
about what it means to be 
evidence informed, it's really 

776
00:44:33,400 --> 00:44:38,720
important that and this goes for
for EF findings as as much as 

777
00:44:38,720 --> 00:44:43,080
anyone else, that the teachers 
question, when people say this 

778
00:44:43,080 --> 00:44:46,640
is based on evidence and think 
about what does that actually 

779
00:44:46,640 --> 00:44:49,760
mean and what sits behind it. 
Is there a clear methods, are 

780
00:44:49,760 --> 00:44:52,920
there clear limitations that 
have been listed or is this 

781
00:44:52,920 --> 00:44:55,520
someone trying to make an 
argument to to make me do 

782
00:44:55,520 --> 00:44:59,480
something they want us to do. 
And I think the more that we can

783
00:44:59,480 --> 00:45:02,760
all work together on on 
questioning those actors, the 

784
00:45:02,760 --> 00:45:05,680
the better the evidence informed
system will be in the long run. 

785
00:45:05,760 --> 00:45:07,640
Beautiful. 
Yeah, I'm totally with you and 

786
00:45:07,640 --> 00:45:09,800
I. 
Think nerdy is definitely our 

787
00:45:10,320 --> 00:45:13,240
our MO. 
And so I think it's a really, 

788
00:45:13,280 --> 00:45:17,120
really positive way to finish 
because I know listeners will be

789
00:45:17,120 --> 00:45:19,080
interested. 
But also, you know, that healthy

790
00:45:19,080 --> 00:45:22,040
skepticism is definitely, you 
know, feeds the system and stops

791
00:45:22,320 --> 00:45:25,480
Big Tobacco from making spelling
interventions and that kind of 

792
00:45:25,480 --> 00:45:28,200
thing. 
And I do remember my curveball. 

793
00:45:28,360 --> 00:45:30,280
And it's curve. 
Because you may not have an 

794
00:45:30,280 --> 00:45:32,800
answer, but I've been thinking 
recently about the parallels 

795
00:45:33,440 --> 00:45:40,640
between medicine and education. 
Did anyone track what the 

796
00:45:40,640 --> 00:45:44,680
experience of doctors was like? 
Because we existed in the world 

797
00:45:44,680 --> 00:45:46,680
where it seems like all doctors 
have always been evidence 

798
00:45:46,680 --> 00:45:51,240
informed, but that obviously 
happened over a not too distant 

799
00:45:51,440 --> 00:45:53,840
past. 
Is are there any studies that I 

800
00:45:53,840 --> 00:45:56,920
could go to and go, oh, here's 
the experience, so then it 

801
00:45:56,920 --> 00:45:59,480
becomes comparable with our 
research yet experiences and 

802
00:45:59,480 --> 00:46:01,440
that kind of thing. 
That's. 

803
00:46:01,440 --> 00:46:04,800
Really, that is a really 
interesting question and I don't

804
00:46:04,800 --> 00:46:09,720
know of any studies, but I 
definitely we for example with 

805
00:46:09,720 --> 00:46:11,680
the What Work? 
Center for Education, but we 

806
00:46:11,680 --> 00:46:14,720
often meet with the What Work 
Center for Health, NICE, the 

807
00:46:14,720 --> 00:46:17,120
National Institute for Clinical 
Excellence. 

808
00:46:19,160 --> 00:46:23,200
And there are a lot of 
parallels, but there are also 

809
00:46:23,200 --> 00:46:25,960
places where we shouldn't 
underestimate how education is 

810
00:46:25,960 --> 00:46:30,320
ahead of healthcare. 
So I think that what healthcare 

811
00:46:30,320 --> 00:46:34,560
has is lots and lots of strong 
funding of very rigorous 

812
00:46:34,560 --> 00:46:38,000
evidence. 
Lots of RCTs that often come out

813
00:46:38,000 --> 00:46:41,120
of a regulatory environment 
where you need that in order to 

814
00:46:41,120 --> 00:46:44,360
get your medicine onto the 
market and you know all of that 

815
00:46:44,360 --> 00:46:48,000
regulation within a marketplace 
economy means there's lots and 

816
00:46:48,000 --> 00:46:50,800
lots of research. 
But the thing that maybe 

817
00:46:50,800 --> 00:46:56,160
education is is stronger on is 
that profession LED voice when 

818
00:46:56,160 --> 00:47:01,280
it comes to evidence access. 
And you know, NICE are actually 

819
00:47:01,280 --> 00:47:04,080
very good at engaging 
stakeholders in the process. 

820
00:47:04,080 --> 00:47:06,280
They'll often have patient 
groups, they'll have doctor 

821
00:47:06,280 --> 00:47:10,280
groups. 
But what I found interesting is,

822
00:47:10,600 --> 00:47:14,320
is in some ways the actual 
conversation between profession 

823
00:47:14,320 --> 00:47:17,800
and evidence base is more 
advanced in education. 

824
00:47:17,800 --> 00:47:21,280
I don't think there's a research
Ed equivalent in in medicine. 

825
00:47:21,280 --> 00:47:24,760
There are obviously conferences,
but they're often funded in by 

826
00:47:24,760 --> 00:47:29,160
big conglomerates or are very 
linked to to academic 

827
00:47:29,160 --> 00:47:31,720
institutions rather than the 
profession itself. 

828
00:47:31,720 --> 00:47:36,640
So I think, you know maybe 
there's a book in it for you to 

829
00:47:36,640 --> 00:47:40,160
to to do the comparison. 
I can't think of a point of of 

830
00:47:40,160 --> 00:47:46,200
reference at the moment or a 
study that has tracked it, but 

831
00:47:46,200 --> 00:47:48,440
but there's there's definitely 
something interesting there 

832
00:47:48,440 --> 00:47:50,880
about the way that different 
evidence systems adapt. 

833
00:47:50,880 --> 00:47:53,760
And I think that you know in 
education we can learn a lot 

834
00:47:53,760 --> 00:47:55,760
from medicine. 
I'm definitely not one of those 

835
00:47:55,760 --> 00:47:59,320
people that says we just take 
the whole thing and assume it 

836
00:47:59,320 --> 00:48:01,800
applies to education because 
because they're clearly such 

837
00:48:01,800 --> 00:48:04,920
different contexts and we should
be really proud of some of the 

838
00:48:04,920 --> 00:48:07,680
things we're doing in education 
that are actually ahead of the 

839
00:48:07,680 --> 00:48:09,760
curve when it comes to an 
evidence LED movement. 

840
00:48:09,920 --> 00:48:11,760
Nice. 
I mean that is definitely enough

841
00:48:11,760 --> 00:48:15,000
to get to get going and. 
Like I say, find a rabbit hole 

842
00:48:15,000 --> 00:48:18,200
and then put off the work I 
should be doing and looking for 

843
00:48:18,200 --> 00:48:20,480
interesting things to to read 
about. 

844
00:48:20,480 --> 00:48:23,000
And I mean it has been an 
absolute pleasure and I know 

845
00:48:23,000 --> 00:48:26,880
that listeners will really value
the chance to hear about your 

846
00:48:26,880 --> 00:48:28,800
work. 
And all set to do say thank you 

847
00:48:28,800 --> 00:48:30,880
very much for joining me. 
Thank you. 

848
00:48:30,880 --> 00:48:31,560
It's been. 
Lovely. 

849
00:48:32,360 --> 00:48:35,400
And to everyone at home until. 
Next time, thanks for listening.

