I really am beginning to dislike the term AI tutors.
My problem is that we say human in the loop.
There's a human slowing down that feedback loop because one
of the advantages I see that AI tutors could have is that
closing this feedback loop to pupils.
We have a problem with our education system where you just
go through the school years and what are the gaps?
The gaps just get wider and wider and wider.
And if there is a hope, or if there's a way that we, each
pupil, can have access to a pedagogically rich AI tutor that
supports them in that consolidation, well, surely
that's something we should explore.
I think that what makes it so difficult is that we just don't
know the proportion of errors that teachers make.
Welcome to Thinking Deeply about AI for Schools.
I'm James Rabin, and joining me, as always, is Neil Almond.
We've got a fascinating topic to dive into today.
Absolutely. And for those just joining us,
this is the podcast where we explored the rapidly evolving
intersection of artificial intelligence and education.
Every month we're going to pose the big questions, examine
what's changing in the AI and education landscape, and just
try to make sense of it all and what that means for education,
the pupils, and the future of learning.
And. We're not here to give you all
of the answers because honestly, we're just trying to figure this
out together. Instead, we want to create a
space where we can think critically, question
assumptions, and explore those possibilities.
Each episode we explore one topic, 1 research article or one
AI update and ask the ultimate question, what does this mean
for education? So this week we're going to go
into this world that's very moving very quickly from product
demos to policy conversations, and that is about AI tutors.
And the key thing here is, are AI tutors generally in
breakthroughs and learning or are we moving faster than this
evidence? So in this episode, which I can
envisage already that will become several episodes, we'll
explore what AI tutors are, why there is so much excitement
about them and whether actually this is a hype or whether they
there is research out there and examples that this is concrete.
But we'll also consider some of those deeper concerns around
buyer safeguarding, cheating, the cognitive offloading and
what this happens to 1 to 1 support.
And because this is not just a story about shiny new tools, it
is really a story about what we think tutoring is, what good
learning looks like, and whether AI can genuinely support people
to think more deeply, rather than just helping them get to
that answer faster. Now, one of the things that I've
been listening to lots of podcasts and my journeys up and
down the M5 and M6, I think I did the whole of the M5 the
other day, is this concept of what an AIT 2 is.
I wouldn't throw in this span this term around.
And I think it's important to distinguish what it could be
because with all the research we're discussing and we'll
share, bits and pieces have come out recently around Harford and
others. One of the things you may see is
like it's a almost a GPT on ChatGPT where you're using an
open source product and you're asking it questions.
So that's one idea. You've got the open large
language models such as Gemini or ChatGPT using pupils for
homework, for help, not much education to sign behind them.
The other end, you kind of got these old intelligent tutoring
systems that very subject specific, structured and signed
to adapt to questions the student use.
But also you've almost got this middle ground as well where
they're purpose built. So things like Google who are
looking at this learn LM or what Khan Academy have done as well.
And even some of these models are being used with a human in
the loop coined the phrase of that almost.
So when we say AI tutor, you've got to have quite an open mind
of what that looks like in practicality or especially when
you're deploying them or using them as well because they are a
massive range. It's quite a fast concept behind
it. So it could be quite a carefully
designed Socratic learning tool that's particular for maths.
So it's got the whole knowledge graph behind it.
Or sometimes it could be people thinking, oh, it's just a
channel jackpot that people are using online.
Or it could be something that's actually just very structured,
very controlled inputs where a pupil actually doesn't chat to
it. It they answer, they ask, it
asks the students questions, they answer them and it will
then adapt in the background. The AI is almost in the
background adapting to it as well.
So we're not in this episode. We're kind of not going into
those, but just be aware that's what AI tutoring is.
And we'll probably say that at some point.
And I know Neil Stone and podcasts done an article on this
before around what makes a good AI tutor.
And I think that distinguish between what an AI tutor is and
that technical kind of element of it is is important.
I really am beginning to dislike the term AI tutors for the
reasons that you have said. And you will see, yeah, a
research paper or one of the tech AI Bros on Twitter or
LinkedIn, whatever it is and it will say, you know, AI tutor
fundamentally changes everything.
And then you sort of you have to dig a little bit deeper, as you
say, just to kind of work out or what are we talking about when
we're talking about an AI tutor here, which is difficult because
I think there are some that are going to be pretty useful.
And I think there are some that are or some uses of the term of
AI tutor when it is more just the right kids, like we trust
you to use ChatGPT as a tutor going to be less helpful.
So I am all for really kind of distinguishing these different
types of tutors that use technology to support them,
whether that be sort of, you know, complete generative AI
chat bot free. You rely on the the pupil to do
the right thing versus ones that have far more sort of guard
rails that won't let you deviate the conversation much through to
ones, as you say, where perhaps the learner does have a little
bit more control about. This isn't like Google learn LM
where you write, you know, you say the kind of things that
you're interested in and it will kind of create the problems
based on those things. Whole other conversation to have
whether that is the right thing to do for kids or what not.
But I think we really sort of need more specific terms to help
us sort of really clarify what we mean when we're saying AI
tutors because they say for every, we're going to talk about
one bit of research now. But for every bit of research
that I see around AI tutors and them being effective, I can
point to you with another research article that says these
AI AI tutors are not effective, but you should never give them
to to children. And then you sort of have to go
in to say, you have to look at the difference between what we
mean when we say AI tutors to really sort of get that, dare I
say nuance of what is what's happening in these studies.
Yeah. And one of those studies you
were talking about there was the Google Learn LM study with Ed as
well and it was an RPT. So you know, the quality of this
is important as well. And I think when you're looking
at this research, we're we're professionals in education, but
we don't not necessarily research specialists as well.
And so the quality that we look at this, these research and how
they measure are important are really, really key elements
behind this. And I know charter college and
the EEF do put guidance out there around actually what what
quality research looks like. But the Google one's interesting
because it looked at chat based maths tutoring.
Now, mass tutoring I think is a really interesting one to pick
on, mainly because the subject matter is hierarchical.
You need to know X before you can go to Y and move it down.
So if you don't have understanding of basic place
value and number and then multiplication and division,
actually some of the fraction work that you will get to down
the line and algebra will actually there will be gaps in
the knowledge if you don't have that.
So it's a really interesting way of looking at it.
But essentially the the research, and I'll cipher some
of a bit, some of it often put some of the key points out
there, but feel free to go out and have a look at yourself.
But it's this kind of idea where pupils receive short, short
sessions with the learn LM and it said there were 5.5
percentage points more likely to solve novel problems than pupils
who worked with human tutors alone.
Now with this model, what was really important and with this
research is that every message that was sent out by the AI was
then checked by a human. So therefore there were reported
very low factual errors, nought .1% of messages.
So it was in terms of quite reliable and not all very
reliable you can say, and we'll probably come on to that debate
later. We've already talked about it
before, we've started recording, but it makes it one and more
practical and policy related studies.
Now those in the UK, this would be really interesting because it
was 165 students across 5 secondary schools between years
9:00 and 10:00. And they want to replicate this
and want to do this again this year I think in the US as wow.
And it meant that was a really that they approved these tutors,
these human tutors approved 76.4% of the messages with
basically no edits. That's a huge point of view
behind it. But I think my problem with
almost this is that we say human in the loop, but it's a human
slowing down that feedback loop. Because one of the advantages I
see that AI tutors could have is that closing this feedback loop
to pupils. And if you are adding that human
in, actually are you more efficient?
And it's a quandary I kind of question all the time.
I don't know. Thinking that maybe right now it
probably is a sort of a trade off potentially worth having.
That's the sort of gut feeling that I have based on a, you
know, this sort of technology or this, you know, generative
artificial intelligence and tutoring.
It is certainly new. And I think it's, you know, the
right thing that a human safeguard right now is at that
point where they can intervene if need be.
Obviously, there's no point in closing that feedback loop
quicker if the the feedback is wrong, That's you could then
potentially sort of multiply the errors rather than fix any
errors without human oversight. I also think it's again, we
obviously weren't involved in the study at all.
We might have to get Craig on to talk about it.
Maybe he'll he'll want to do that after we've been on his
podcast a few times. Maybe he'll come come.
It's an open invitation, creative you're listening,
you're free to come on. But I would imagine like it's I
can't imagine that the feedback that the language, large
language model, whatever they might be using is going to, you
know, provide so much recent feedback that, you know, a human
and to be part of the to work with Edie, you know, you need to
be able to teach mathematics up to, you know, secondary GCSE
comfortably. So we're talking about not as
humans, but experts in their subject domain as well.
I can't imagine they would take too long to cipher through those
messages and flag them whether they're right or wrong or, you
know, having to correct them in any way, shape or form.
So I think right now it's it's the right thing to do.
Will we need it all the time? Probably not, but I think it
might be one of those things going back to what you said.
And when you look at the history of of tutors, I'm going to call
it tutors with through technology because it's always
maths, right? You go back to like the old
intelligent tutoring systems from like the 80s.
Like I think about like Alex is 1 like famous one that's been
out there for quite some time. I think like McGraw Gill or
something own it now, but it's always maths.
Why? Because it's hierarchical.
It's very simple. If they get a, they can go to BI
think. We then sort of look at more
different subjects like humanities, English literature,
where sort of we've got that more sort of, you know, it's
wider rather than deeper in terms of that knowledge
structure. I would think you would need far
more humans overlooking what that chat bot was providing just
because there's so much more margin for error I would have
thought based on the training data and the way it could
potentially could go. It comes back to this point that
it needs to know the curriculum you're teaching and it needs to
know where it goes. So Math Academy is a popular one
in the US at the moment, and they've recently released a book
that I found APDF on LinkedIn about which is all about the
cognitive science. They've gone through it all and
they've got their hierarchical document, but it's all about
this adaptive diagnosis where that child is that hierarchical
knowledge. Is it it's against the Common
Core in the US? That's fine, which is great.
So it knows where it is. But I think there's another
question going completely off tangent with any of these AI
tutors is the fact that let's say I was in year 4 as a year 4
as a student. This may more work more in
secondary context actually. So let's say I'm a year a
student and I've been given a tutor like this as well as can
high quality teaching in class. But this tutor does this
diagnostic. And most of these platforms do
have some kind of diagnostic and any intervention.
Same thing we do that within schools, we do very much a human
aid approach of it. That's, that's great.
That's what we need to know where that student is.
Is there a danger then that if it finds that I've got content
knowledge from the year 4 year, five year 6 curriculum, well, is
it going to take me all the way back there, fill in all those
gaps and build it up? Or is it going to work alongside
and support what I'm being taught in class and then almost
consolidate that as well as fill the gap?
So I don't, it's like there's a balance here as well.
There's not only just with what knowledge do I need to fill the
gap so I can work with it. And mass is an easy one with
this because hierarchical, but there's also a how is it
approached from a child's perspective?
Because one of the advantages of technology and AI and where
we're seeing a lot with pupils talking to these AI toys or
things online, it's the fact that they don't want to talk to
us, to their teachers. They actually get this feedback
back from something that's inanimate, but they feel like
it's a human. It's a lot softer, it's a lot
kinder. It's kind of, they're happy to
listen to that before they go back into class and try it.
So I think there's a there's a moral side of this of how does
it feel like from there pupil and student perspective.
And I wonder, and I haven't seen it yet, too much evidence from
what are pupils viewpoints on this as well?
No. It's been a really interesting
bit of data together about how they feel about it and what they
think is best. And because you're getting from
the couple of experiences that I know, you know, it's it's really
mixed. And, you know, adults are as you
know, yeah, there are some who are, you know, vastly pro, some
who are vastly against it and vast majority of us in the
middle. And I, from what I hear, it's
the same with pupils and that use around AI as well.
So it'd be really interesting to see whether, you know, are we
there are, you know, the concept of schooling, we're forcing
children to be in somewhere that they don't want to be anyway for
the vast majority of the time. But we know it's good for them.
But does that change when their experience of that would be, you
know, right, You will have your own sort of personal
workstations and get the headphones on.
And your maths lesson is just, you know, wherever it is that
you may be in whatever intelligence tutoring system, AI
tutoring tool that rears its head and that schools are able
to to afford or deploy in their, you know, infrastructure.
Yeah, no, it's, it's a weird 1. And I think, you know, we've
lots have been said about sort of belonging and all of that
stuff. And you know, if the, I, I think
I maybe struggle with school a little bit more than I did if my
experience of it was, you know, come in and log in and just do
what is what you need to do. And with a teacher teaching
assistant wandering around and just being that extra bit of
support for me if I needed it, I don't know.
But no getting there, getting Jordan's thoughts about all of
this would be really interesting.
And I think that you mentioned that another interesting point
there as well around sort of, you know, how best would we use
these in terms of, is it just a, a consolidation tool or do you
give teachers the, the pupils the choice to go, Hey, you know,
if you're smashing it and you're in year 4 and you want to move
on to the Year 5 stuff, you know, crack on because then, you
know, you're potentially, you know, widening at a gap within
your own sort of microcosm of your own classroom, which if
anyone's, you know, taught any maths at primary school upper
towards the upper levels. And you know, the lowest, you
know, you've still got people who struggle to number bond to
five and those who are ready to do some interest, more
interesting, complex stuff. And if you're if you're just
exasperating that, that would make any sort of poll class
teaching that you're doing harder.
But again, you know, this conversation is not it's not
new. We've been having this sort of
conversation of, you know, do you give your quote UN quote
rapid graspers, which is a term I absolutely lower than detest.
Do you give your what you're doing?
If you're rapid graspers, they clearly do it know it.
So what you're doing to move them on quickly.
And then, you know, you get to year 6 and you wonder, you know,
it's how did, how did the gap go from, you know, a couple of
months in reception all the way then through to several, several
years by year 6? Because we've been widening
those gaps as part of our teaching because we had to, you
know, get those rapid graspers through more interesting stuff
because that's what teachers are some members of SLT wanted in
some schools. So I think I'd be more
comfortable with it as a. Consolidation tool.
I quite like the idea of it just being like 2 weeks behind, like
what you're doing in your like lessons coupled with a bit more
sort of like spaced retrieval from some fancy algorithm.
Or I could even tell the system what we've done and what I want
them to do. And the system could use all its
knowledge that it knows that peoples have questions right and
wrong and whatnot. And it can do that to sort of
give them their own sort of variation or their own sort of
real specific space practice quiz based to them and then go
through that content that they all had.
You know, I think for me, that's a better place to start with
rather than just, you know, here's a, here's a system.
Go wild, kids. Yeah, In some ways, this whole
idea of spatial treatment, I've always thought that technology
is perfectly suited to it, putting up the right points for
pupils at the right time. Like as teachers we try and put
this through like a do now activity in the in the classroom
before we go to the main bit of learning.
We try and do it, but I don't think it's a necessarily of a
clever way of doing it. We're manually doing it and
every child in our class will need to practice things more
times than their peers next to them or less to make sure it's
consolidated and it has encoded into a long term memory.
So I think technology's got a really interesting way of being
able to do that really, really well.
And I think the AI tutors, if they consolidate the learning
that's happening in the classroom, having that space
retrieval aspects into it. So they really mastered it.
It would ensure that it would honor that mastery element that
we talk about so often. They say you need to grasp it
before we can move on and make sure those prerequisites are
secure. But I feel like it goes into a
bigger conversation that we won't get into because we'll be
here all day and it's not a job need of us want probably is.
We have a problem with our education system where you just
go through the school years and what are the gaps?
The gaps just get wider and wider and wider.
And if there is a hope or if there's a way that we, each
pupil can have access to a pedagogically rich AI tutor that
supports them in that consolidation, well surely
that's something we should explore because it seems like
the benefits of it could, could be huge.
We don't know. And then it goes on to the other
thing. I think you you said earlier and
it was a came from, I think it was a new news article as well,
where a dad at North created AAI chat bot.
It was a couple of years ago before the AI tutors were a
thing, really a news ChatGPT and the and the GPT custom GPT in
there, but actually personalized tutoring to that child.
So whatever their interests were, they kind of then applied
the maths or their English, whatever they were doing to that
context. I don't know if I'm comfortable
with that. I don't know if I'm comfortable
with this idea of personalizing learning to that context of that
child. I, I see the benefits of it, but
I don't, it feel makes me uncomfortable and I'm not sure
why. Do you see what I mean or?
Yeah, it's, I think it's longer term.
I think it's just that message of a you can't do this unless
it's you're interested in it, in which case that's blatantly not
true because eventually we want them to transfer this knowledge
to things that they're certainly not interested in.
Like, you know, you hear all the time, don't you, about, you
know, the, the, the ever updated list of things that we should be
teaching kids throughout their, their schooling and, you know,
mortgages and personal finance and all of that stuff.
You know, no 14 year old is interested in tax returns and,
and mortgages, but we've certainly given the maths to be
able to sort of apply it to that.
So the idea that we would then give them information that it
could be about football, rugby or whatever.
And then, you know, that's some issues right there as well,
instead of potentially, you know, reinforcing those gender
stereotypes as well, which, you know, is not needed.
They'll then be able to, you know, after only sort of working
with rugby and football or, you know, whatever it might be that
they can then magically transfer this to mortgages and tax
returns or, you know, whatever it is, basic percentages when
they're out shopping or whatever.
I don't think necessarily is going to be helpful for them.
So I think there's like a transfer issue.
I think there's sort of a long term sort of motivation issue in
the terms that, you know, it signals.
Well, don't worry, you know, you don't have to worry about things
that aren't interesting to you right now.
But you know, you know, you introducing something
interesting in mathematics, you know, whatever it might be,
might give them a new interest or spur some interest that
wasn't there before. And so I think the idea and it's
also, you know, 30 kids, right? You know, yes, we can probably
do this a little bit quicker now because of ChatGPT and and
whatnot. But you know, 30 different kids
with 30 different interests. That's a lot of yeah, that's a
lot of worksheets or a lot of questions that.
And obviously, if you're doing 1 to 30 teaching in your in, you
know, in class, I'll say, you know, in an analogue style, you
know, you're not going to be able to to do it like that.
And then you just go into, you know, you think as a teacher,
you're thinking about the wrong things as well.
You're not thinking about how am I going to teach us in a way
that is going to means that the vast majority of the class doing
it, you're thinking, oh, what was Sandy's favorite thing that
I need to be putting this what that I should wrap this question
in? And so I think you know, big
issues when it comes when it comes to that for me I think as
well, I think those are my 3 reasons why I sort of don't like
it. Long term motivation.
I think there's a transfer issue and in terms of I think it just
increases teachers workload with importantly sort of very little
evidence that it actually improves, improves anything.
I think there will be use in terms of applying contacts in
certain situations. I don't think it's a complete.
Let's not personalize anything, but I do wonder whether the when
these tools get more important and more capable down the line,
let's say, and you've got your own quick and you've got your
own pedagogical model. It's Socrative and it's it is
way of working. There could be things like
financial literacy. It could be one of those painful
lessons on a Friday afternoon, the graveyard shift, where
actually let's put it into context of a project that
they're creating a business or something.
Actually, they want to learn those skills and they need to
learn those skills to be able to apply it in the right way.
And I think that could be an interesting aspect on it,
Something you were saying around the role of the teacher and
where these AI tutors come in place.
And Harvard released ARCT as well, and they came out with
some interesting things that compared.
They compared a custom AI tutor to active in class learning.
And they found that they, through the AI tutor, they
learned significantly more in less time.
And then students felt more engaged, They felt more
motivated. And it was like pedagogically
following the best practices. But I think what it for me, what
I took away from that report more than anything, because it's
about 194 undergraduate physics and this one sort of science is
the design of how you use these tutors are important because
what it comes back to is if you're consolidating learning at
home and supporting it, it's that what do you do in the
classroom? Does your shift in the classroom
that yes, we will really important guided in practice
that we're going to do regarding instruction and we're going to
do some really purposeful practice.
We'll do some low stakes quizzing behind it.
Do a lots of check for understanding that you know that
you are confident you're clear with this in this application.
But some of that extra consolidation work or something
else can be done outside the class.
And I think that whole shift of what happens in school, what
happens outside the school and raise the question of homework
again is a really interesting debate that I think education is
going to have very soon because there is every school across the
country and across the world that they will give homework
out. At the moment that will be just
consolidating bits and pieces on paper or using online platforms
like Sparks or what what not. And you can just click the right
answer, get AI to do it for you anyway because you copy and
paste it and they don't learn anything.
So for me, it raises this whole AI tutor thing, almost raises
the question on what is important that we should be
keeping in the classroom and what are the important things?
What are some of the things that we could offload to AI to ensure
that our classroom time is the most powerful and rich and those
relationships are built and we really have an idea of where
those children are and we can guide them to the next steps?
I don't know the answer to it. I just think it's going to be an
interesting debate that AI is going to create for us.
Yeah. I'm not sure how much I agree
with that in the sense of whether AI would be able to sort
of completely take over aspects of that.
And that means that we can focus on different things in the
classroom because I just don't think we know enough about
whether AI will ever get there to that point.
I mean, the Harvard is a useful is an interesting study, but as
you mentioned, I think it was university physics students, you
know, I don't it's it's interesting.
Of course it is. But when you go to, you know,
people are on that physics course because they want to
learn about physics and they are, they are paying.
It's a mistake. So, you know, a fair and it's
Harvard, so a fair whack to to, to be there.
And so there is a motivation that that doesn't exist, I think
for the 14 year old and 15 year old who again, is going to
school, because the, the law says that, that, that you, you
need to and have to be there. I also think, you know, if
you're studying physics at university, you have a a whole
host of prior knowledge of which you can use to help you with
your physics. And you have obviously
demonstrated through various sort of entrance examinations,
tests, whatever it might be in America.
So, you know, GPA S and all that sort of fun stuff that you are
capable of studying physics at a very prestigious university at,
you know, this level, which again, goes back to what we were
talking about in, you know, that's not the case where you
can sort of guarantee that everyone is going to be far
closer in terms of their understanding and knowledge
rather than that. And I think, yeah, you could
absolutely. I think, you know, and for
humans, say humans for adults to learn something when you have
that motivation to do so, I could imagine that I would
probably learn a little bit more faster and certainly report
higher engagement motivation about it.
I just don't know whether that's the same for teenagers and
children. Often when I talk about AI or
think about AI and have conversations, I think I always
go back to where shouldn't AI be and where shouldn't we use
technology? And where does the human bit
come in place? And I'm just trying to take
something that I don't think as adults, often we're in that
tricky space of remembering how learning a new thing is
difficult. So let's take something like
physical like golf for example. OK I've got a golf set here and
I am absolutely rubbish because all I'm doing is watching a
YouTube video. I don't know how it feels like
what a good swing does. I didn't want 1 intro.
And yes I can get a camera. Who does AI tutor?
It literally is like an AI tutor.
It will see where I am, put the pinpoints on it saying these are
where I think it's going to work.
This is what you need to work at.
But I need that human, I need that someone to really guide me
and point me in the right direction.
And I think that's important as well.
And it's with all of this, it's I've got a passion to want to
learn on it, but I don't know what I'm doing.
I'm wasting almost wasting my time trying to explore it.
I need AI, need a professional guide or professional helper to
help me get to that initial stage where I can then use
almost a hybrid AIG Turing almost to consolidate what the
teacher saying or the students like.
Well, I'm getting told, and I think that could be replica in
education in some ways. This idea of space retrieval,
this idea of some pupils will need to consolidate and practice
things a lot more than others. I think that's where we've got a
real hope in this use. And when the DfE announced
they're looking into AI tutors, they explicitly aimed at
disadvantaged pupils. And it's from 2 points of view,
one, to close a cap. But two, I think there's an
equity thing here as well because those who have devices
can access free AI online quite easily, quite quickly, whether
it's to do their homework or whether to actually use it
purposefully. I think it's an interesting
scope there as well. I have succumbed and I give open
AI £20 a month for my hard earned money because I was
getting really annoyed with my rate limits being, you know, put
to 0 within sort of, you know, 5-10 minutes of a, of a
conversation. And you know, with all my 20 LB
it's I very rarely ever have that problem.
It's the same with with Claude. And so absolutely, you know,
currently if you are getting your, if you're fortunate enough
to be able to give, have a spare 20 quids to give to Anthropic or
open AI, you know, and you're sitting your 1415 year olds down
and getting them to do things that they are getting more out
of it than the and they probably have again, yeah, not new
conversations around advantage versus disadvantage.
They probably have a set space for them to do it.
They probably have a decent computer that would that be a
laptop. So, you know, at least a sort of
13 inch screen. And for the disadvantaged
pupils, they probably have to try and manage on, you know,
some sort of mobile device, whether that might be some sort
of tablet or a mobile phone, which is just, you know, not
necessarily the the optimum way to go about it.
So yeah, that exists. And I think it's really
interesting to see how the UK government are going to approach
this AI powered 1 to 1 tutoring and whether the well, again, you
know, is the expectation going to be that that's that they do
it in in school, whether that be after school lunchtime or are
they hoping that it would, you know, they'll, you know, these
pupils will magically want to decide.
So now it will potentially do their homework if they didn't do
so because it's AI powered. I think it's going to be really
interesting to see what how they go about this and how they sort
of measure success and and all that stuff.
It's yeah, looking forward to it.
Yeah. And I think with anything with
this, in this AI world, what you, what it is trifecta of
policy needs to be supportive of it.
You need actually companies who can build this, but also you
need the, you need the teachers with the practicalities of the
implementation, but also the cognitive science to build it as
well, as well as the curriculum. So there's a quite a few
different things that need to work together in order to create
an AI tutor or however we use it to be purposeful.
One thing I think we could go on for hours, but I think one thing
I think we should leave on today and then we can probably come
back to this, come back and actually, yeah, see if we can
get Craig on to talk about the Ed report in a little bit more
detail and and see what their plans are for it.
And we talked about it before and you brought it up, but I
think with all of these reports, there's something around
accuracy of AI tutors that comes up time and time and time again.
And you say that, OK, it may be nought .41% or whatever it is
that an AI tutor will get something wrong.
And therefore that that could equate to about a million
questions a year, a day or so if it's for every student.
But are we, is our expectation of AI to be 100% correct or any
technology to be 100% correct 100% of the time too ambitious?
Because I know as teachers, there are lessons that we've all
taught. We've walked away from them and
it's like, why did I do that or completely different?
And I didn't plan to do that. And it just went off on a
tangent. And we've all done that, whether
it's through an ECT or whether we're experienced teachers.
So are we holding AI to this pedestal where actually it will
never reach to our expectations And it's probably a better way
to think about, OK, how can we integrate this into what we're
doing already? Yeah, definitely.
I think that is the approach is how we can integrate into what's
what we're already doing. I do think we, I think that what
makes it so difficult is that we just don't know the proportion
of errors that teachers make on a regular basis.
And thinking of the top head, like the only way you could do
that is if you hired probably thousands of observers to go
into a representative number of schools across the age spectrum,
making sure that you were primarily looking at across a
whole, you know, all the different types of lessons for a
large enough portion of time. And they were just noting down
errors as they sort of noticed them happen.
And maybe then you would get some idea of the proportion of
errors that happen, you know, daily on a, you know, every sort
of day current situation. But then, you know, you've got
problems like, you know, how I'm being observed.
So I'm going to make sure that my lessons are really up to
scratch this week and stuff like that.
So there could still be some issues with that, but it's the
only way that I could think about that you could actually do
it. I would be surprised.
And again, there is no it's just a vibe because humans are
fallible to all sorts of erroneous mistakes and errors.
But I would be amazed if more, if a tutor, if a teacher didn't
make more errors than this sort of, you know, 0.14
hallucinations or whatever that was, if every student in England
used them for two lessons. So this is from that Ed tutor.
So if every student used this EED tutor for two lessons, that
would mean about sort of a million errors across that
sample. And I hate it when, when you
sort of measure things that way as well.
And you know, when there's 700,000 days missed a week or
something like that. Yeah, it's a rubbish way to
report things. But anyway, I digress.
Yeah, I, I think we do. I also think, think about EED as
well. You know, that is a sort of a
one to one situation. So if one child is making that
gets an error wrong in that message, that's just that one
child that's been affected. If a teacher makes an error in a
in a lesson, that's a 1 to 30 error.
And so unless the teacher, the teacher made it realise that
they've made a mistake, in which case they might try and try and
self correct. We've certainly all been there
where we tried to self correct and the people still remember
the wrong thing because that's the way they are because we go,
oh, that was a mistake. I think we also got to think
about like the kinds of areas that we're talking about here I
go into when I certainly did go into many lessons, like, you
know, I would still see work that wasn't necessarily, you
know, text on the board or whatever, that wasn't
necessarily like punctuated correctly.
You know, obviously the pupils aren't noticing that, but they
are still reading texts that isn't punctuated correctly.
So is that an error that we're sort of happy with to say that,
you know, that's a potentially, you know, saying that it could
impact pupils learning because they're reading it and they're
not taking in that orthographic system because they realise, oh,
teacher didn't start that proper noun with a capital letter that
time. And I haven't realised it's a
proper noun, but I'm just going to read on anyway and go, oh,
OK, fine. I, you know, don't need to use
their capital letter for that word because teacher didn't have
to a teacher forgot to, even though it's not a sort of, you
know, outwardly facing whilst I'm doing some teaching sort of
mistake. So, yeah, I think we do hold
technology because we're so used to it sort of maybe we, we sort
of put it on that sort of pedestal of, you know, it's all
it's either right or wrong. And I think that's a big issue
that we have, you know, with it, we do just take it as gospel.
You know, I'm guilty if it's anything you get a confident
message back from ChatGPT or insert whichever one of these
chat bots. And so you do think it's right
because we're so used to thinking that, you know,
technology is sort of somehow superior.
And you know, when you think about a calculator, calculator
will always give you the right answer unless you unless it's a
user error through the input. And I think the kind of
technology that we've had is been that sort of system that
this new technology that does Saffler from this sort of
feature of hallucinations is kind of quite jarring to our
mental models of how we suspect this technology to behave.
And so we aren't necessarily good at sort of updating our
model around that. This could be right, could be
wrong. But that said, it's always
important to remember that yes, we talk about keep the human in
the loop, keep the expert in the loop, absolutely.
But humans are fallible, they make errors.
And I would imagine up and down the country on any day, assuming
100% attendance across the every school in the country, I would
suggest that within two lessons there are probably more than a
million errors, either that being like direct or indirect
being made. And I'd be surprised and shocked
if it wasn't. And I'd call on any researcher
who is listening to this try to find a way to work it out
because I think it would be really interesting.
But even I think it would change the conversation around how we
think about these. Because if we can get the
cheater, that's better than the cube.
Even if you know, let's say for example, it's 0.2, well now you
know, this combination through EED is better.
And I think that would change the conversation a lot.
Yeah, and even in, you wouldn't even need to see what teachers
have written. It's just their language.
Exactly. Across the across the country,
it will be different in terms of how different nouns, phrases,
colloquialisms that are being used as well.
Actually that's not as easy as black and white, right or wrong
as well. So I think that is really
interesting. So I think while you were
talking, I was thinking, we always, we've said to ourselves,
we're healthy skeptics around AI.
Often I think that's the phrase we've used.
But I was like, OK, what are the problems?
What are the hopeful things then?
Because we can come down on this, say, oh, yeah, we don't
think it's going to work. But I think there are some hopes
in this. And I think this AI isn't the
next big thing that's going to be like interactive whiteboards.
I think this could change education or be embedded in.
And I think we've got this, we've got it on a pedestal at
the moment. It's going to change everything.
But I think actually we should think about how it's embedded
in. But the three things I thought
of that I'm hopeful for in some ways is how we can close
feedback loop, whether that's actually just for students
getting feedback so they can Draw Something again or get
immediate feedback on an answer. And we don't build those
misconceptions. There are even there are even
tools out there that are coaching teachers just on that
kind of idea as well. So closing that feedback loop.
I think the other thing with this is scalability.
There's a massive jump to start off with in trying to integrate
these, whether you may need more humans in the loop, but over
time these will be scalable. And I think that's interesting.
And the third thing for me is actually how you can get really
accurate diagnosis of where every child is and where the
gaps are or where the commonalities are for those
pupils. I think that could be really
interesting insights that will help a teacher because I think
learning teaching is difficult because the learning isn't
always visible. And the more data points that we
get for high quality to identify what are those misconceptions
that come in or where those gaps of knowledge that we can address
something really concrete that could be amazing.
And then we address that through space retrieval as well.
That could really help us as teachers, just having that raw
data, being able to action it as well, both in in class and our
teaching, but also through whether it's extra time on the
AI tutor or other things as well.
I think else be important. Are there any hopes from you
where you think AI tutors could be really purposeful?
Just building on that last one around after your diagnosis,
just thinking about how that hand over meeting that you get
sort of certainly in sort of in primary or for the secondary
teaching. If it's a, you know, newest
teaching year 8 or year 9. Like how useful that would be to
have sort of knowledge graph of each child's personal learning
journey. And you know, hopefully and you
know, undoubtedly you would then be able to talk to the system
and it would just tell you class 60s, seventies, 80s, you know,
whatever it is, you know, main sort of knowledge gaps and stuff
like that. So you don't have to do too much
of the sieving through the data yourself.
And I think that's a really sort of interesting bit around that.
I think in terms for me, I think what I'm sort of hopeful about
is that certainly I think the EED study is really powerful
because it just kind of it keeps the human there.
And I think that's something to be really interested in, I think
for, you know, for teachers who potentially might be fearful
that the role of the teacher might change.
I think this is kind of evidence that points towards that, you
know, teacher expertise within your subjects is still important
right now. And this study only worked only
got the results that it did because it had those those
experts there that kept them going, which I think is really,
really useful signal to the profession that keep on keep on
getting better at teaching and keep on improving your subject
knowledge because that's going to, you know, only help pupils
learn more and potentially faster too.
The other thing that I think is potentially quite interesting
for me, I think there is a bit of an, again, we need to see how
it all plays out and stuff, But again, linked to that sort of
scalability of just the cost of it all and if you can get the
conditions of it right. If you are ensuring that you are
providing space for pupils who potentially can't do this at
home or won't do it at home and you're providing space in the
school day, well, that's sort of after school, a bit of a lunch
club, whatever it might be. I know, you know, there's pros
and cons to all of that, but I think we are sort of seeing
that, you know, with every iteration of AI modelling, it
gets cheaper to use and it is cheaper than human capital, you
know, tutors £4045.00 an hour, this, you know, it's £20 a
month. You know, there's an economic
argument as well. I think that this potentially,
again, all other conditions being right, which is a big if
for sure. But I do think, you know,
looking forward, you know, there's an economic argument
that it's a, well, it's a, you know, potentially a useful spend
of people, premium, for example. So I think in terms of your
scales. So it's kind of more just adding
I guess to to yours than having my own.
No, that's good. I'll put you on the spot anyway.
I just as you were talking, I was thinking about it anyway.
But thank you for everyone listening to thinking Deeply
about AI for Schools. I'm James Rabin, and joining me,
as always, is Neil Almond. If you've enjoyed the
conversation, scribe to Neil's newsletter at 4 schools dot AI.
That's four schools dot AI for your weekly dose of AI news
around education around the wider world.
And so often a really good point for you to take and steal.
Thank you. And next time we will be, I'm
not sure what we're going to do next time, Neil, but there are
hundreds of episodes that we could do.
And I think we'll explore some more AI tutors and explore
whether we can get some guests involved as well.
Thank you as always, goodbye. Always a pleasure.