1
00:00:03,280 --> 00:00:05,880
Hello and welcome to the 
Behavioral Design Podcast. 

2
00:00:05,960 --> 00:00:08,680
This season we're diving into 
the intersection of behavioral 

3
00:00:08,680 --> 00:00:11,080
science and AI. 
We want to make sense of the 

4
00:00:11,080 --> 00:00:14,080
state of AI, from understanding 
how humans interact with 

5
00:00:14,080 --> 00:00:17,520
intelligent systems to using AI 
to do behavioral design itself. 

6
00:00:17,720 --> 00:00:21,280
I'm Aline Holsworth, a health 
tech advisor specializing in AI 

7
00:00:21,280 --> 00:00:23,880
and product design. 
Over the past 15 years, I've 

8
00:00:23,880 --> 00:00:26,720
been crafting human centered 
products with behavioral science

9
00:00:26,720 --> 00:00:29,160
at the core. 
At Apple, I LED Behavioral 

10
00:00:29,160 --> 00:00:32,119
Science for Health AI, designing
and launching AI powered 

11
00:00:32,119 --> 00:00:34,440
features to help users reach 
their health goals. 

12
00:00:35,160 --> 00:00:37,160
And I'm Samuel Sultzer, your 
second Co host. 

13
00:00:37,400 --> 00:00:40,600
I'm a behavioral strategist 
specializing in hybrid formation

14
00:00:40,600 --> 00:00:43,200
and designing products that 
drive long term baby change. 

15
00:00:43,640 --> 00:00:46,880
I work with leading tech 
organizations integrating AI to 

16
00:00:46,880 --> 00:00:48,240
scale behavioral design for 
good. 

17
00:00:48,800 --> 00:00:52,200
And I'm also the founder of Baby
Bites, a dedicated community on 

18
00:00:52,440 --> 00:00:56,240
behavioral science and AI. 
Quick word on Nuance Behavior 

19
00:00:56,240 --> 00:00:59,640
where we help organizations 
build impactful digital products

20
00:00:59,640 --> 00:01:02,640
using behavioral design. 
We only take on a few clients at

21
00:01:02,640 --> 00:01:05,200
a time to ensure the highest 
level of quality for our 

22
00:01:05,200 --> 00:01:07,400
tailored evidence based 
solutions. 

23
00:01:08,280 --> 00:01:11,080
If you'd like to become one of 
our special projects, e-mail us 

24
00:01:11,120 --> 00:01:14,760
at hello@nuancebehavior.com or 
we could call directly on our 

25
00:01:14,760 --> 00:01:33,160
website, nuancebehavior.com. 
So Eileen, I'm interested to 

26
00:01:33,160 --> 00:01:38,360
know, have you ever considered 
buying AAI hardware or like 

27
00:01:38,360 --> 00:01:42,840
wearable device? 
Have I ever considered buying 

28
00:01:42,840 --> 00:01:46,880
one? 
I mean, AI is already in all of 

29
00:01:46,880 --> 00:01:49,720
our devices. 
So I feel like that's a trick 

30
00:01:49,720 --> 00:01:51,720
question. 
Maybe that's true. 

31
00:01:52,360 --> 00:01:54,360
I know what you're getting at, 
yes. 

32
00:01:54,360 --> 00:01:58,240
The clip that goes into your 
shirt pocket that listens to all

33
00:01:58,240 --> 00:02:01,360
of your conversations, you're 
not talking about like, FaceTime

34
00:02:01,360 --> 00:02:04,200
or like, you know, your sort of 
standard encryption, No. 

35
00:02:04,520 --> 00:02:05,240
That's right. 
Yeah. 

36
00:02:05,360 --> 00:02:09,720
Like the humane pin that you 
describe or the rabbit, R1 I 

37
00:02:09,720 --> 00:02:10,880
don't know if you saw that at 
all. 

38
00:02:10,960 --> 00:02:15,160
Orange red thing. 
Google Glasses. 

39
00:02:15,160 --> 00:02:16,320
No juice? 
Yeah, Yeah. 

40
00:02:18,040 --> 00:02:21,040
No, no, I have not ever 
considered purchasing one of 

41
00:02:21,040 --> 00:02:22,600
these. 
OK. 

42
00:02:23,800 --> 00:02:27,240
Well, I think they're 
interesting because they have 

43
00:02:27,600 --> 00:02:30,760
created quite a bit of hype and 
interest especially in the last 

44
00:02:31,440 --> 00:02:36,560
year or like 1 1/2 year or so. 
And the one you mentioned, 

45
00:02:36,560 --> 00:02:40,400
Humane Pin and also the one I 
referenced to Rabbit R1. 

46
00:02:40,400 --> 00:02:44,200
Both of them have followed this 
kind of like curve of being like

47
00:02:44,200 --> 00:02:47,960
really, really hyped and people 
getting excited about especially

48
00:02:47,960 --> 00:02:50,680
the R1. 
But then once I actually get to 

49
00:02:50,680 --> 00:02:54,760
use it, they've like used 
crashed and burned like it just 

50
00:02:55,040 --> 00:02:57,800
hasn't been received very well. 
Lots of. 

51
00:02:57,800 --> 00:03:00,840
Excitement from the early 
adopters and then a big, big 

52
00:03:00,840 --> 00:03:03,120
dip. 
Yeah, not even early adopters, 

53
00:03:03,120 --> 00:03:06,000
like people from pre-orders 
before they actually get this 

54
00:03:06,000 --> 00:03:07,200
thing. 
And then once they get the 

55
00:03:07,200 --> 00:03:10,320
thing, they realize like this is
crap basically. 

56
00:03:10,800 --> 00:03:14,400
And so in terms of ingredients, 
I feel like one thing that they 

57
00:03:14,400 --> 00:03:17,440
have in common is that they have
100% hype and syrups and 

58
00:03:17,440 --> 00:03:19,640
behavioral science in terms of 
being used. 

59
00:03:20,520 --> 00:03:25,040
And I think that's a little bit 
of maybe the theme, of course, 

60
00:03:25,040 --> 00:03:26,800
in this season that we're going 
to kind of look at payable 

61
00:03:26,800 --> 00:03:30,880
science and AI and how they 
overlap or how we can see and 

62
00:03:30,880 --> 00:03:35,400
understand each from different 
perspective better. 

63
00:03:35,760 --> 00:03:40,040
Because I think that's one thing
that's like a big loss is this 

64
00:03:40,040 --> 00:03:42,640
kind of idea that use because 
you can assign a very 

65
00:03:42,640 --> 00:03:46,200
intelligent system, like some 
form of trained language model, 

66
00:03:46,680 --> 00:03:49,920
doesn't really mean that you per
extension know how people are 

67
00:03:49,920 --> 00:03:52,360
going to use it to interact. 
With right, right. 

68
00:03:52,760 --> 00:03:56,880
I mean, obviously I am biased as
a behavioral scientist, but it 

69
00:03:57,280 --> 00:04:02,400
completely boggles my mind that 
you could even consider making a

70
00:04:02,400 --> 00:04:06,000
piece of technology primarily 
for humans to interact with 

71
00:04:06,240 --> 00:04:08,880
without considering how humans 
will interact with it. 

72
00:04:09,960 --> 00:04:15,400
It just like, wait a minute, 
it's, it's, it's I don't know 

73
00:04:15,400 --> 00:04:18,640
how much of my bias is playing 
into that, but yeah, like, I 

74
00:04:18,640 --> 00:04:21,880
would love for someone to 
explain to me why behavioral 

75
00:04:21,880 --> 00:04:27,080
science is not absolutely 
essential to success. 

76
00:04:27,280 --> 00:04:29,800
This situation. 
When I worked at a large 

77
00:04:29,800 --> 00:04:33,840
consumer electronics company a 
few years ago, we were very 

78
00:04:33,840 --> 00:04:38,160
interested in using behavioral 
science and AI together and 

79
00:04:38,160 --> 00:04:40,680
really tried to intentionally 
look at how they could work 

80
00:04:40,680 --> 00:04:42,320
together to make each other 
better. 

81
00:04:42,320 --> 00:04:46,360
So AI helping behavioral 
science, but also behavioral 

82
00:04:46,360 --> 00:04:48,960
science helping AI and and 
really kind of thinking of the 

83
00:04:48,960 --> 00:04:53,240
two as a two way St. 
So if you think about how 

84
00:04:53,240 --> 00:04:56,680
behavioral science could 
contribute to AI, how an 

85
00:04:56,680 --> 00:04:59,880
understanding of human behavior 
can help both generate better 

86
00:04:59,880 --> 00:05:03,520
predictive models, maybe 
eliminate bias and so on. 

87
00:05:03,920 --> 00:05:07,320
But then on the other side of 
things, how AI can make 

88
00:05:07,320 --> 00:05:11,360
behavioral science better. 
So as a research method or 

89
00:05:11,360 --> 00:05:13,240
intervention technology, for 
example. 

90
00:05:13,560 --> 00:05:16,320
And and we talk a lot about that
in other episodes. 

91
00:05:16,320 --> 00:05:20,360
But today, when we talk to Amy, 
we're really gonna focus on the 

92
00:05:20,360 --> 00:05:24,720
1st St. how behavioral science 
can contribute to this AI tool. 

93
00:05:25,160 --> 00:05:28,600
And of course, the ultimate goal
it always, at least in our 

94
00:05:28,640 --> 00:05:32,280
opinion, is how do we change 
behavior for good using this 

95
00:05:32,280 --> 00:05:35,040
combination of AI and behavioral
science. 

96
00:05:35,040 --> 00:05:38,400
And I feel like that really came
through in our conversation with

97
00:05:38,400 --> 00:05:40,920
Amy as well. 
Yeah. 

98
00:05:40,920 --> 00:05:43,200
And I'm interested because so 
you, for example, you led 

99
00:05:43,200 --> 00:05:49,280
behavioral science for healthy 
AI team at Apple and having have

100
00:05:49,280 --> 00:05:53,040
this kind of exposure for you, 
is there in commonality? 

101
00:05:53,040 --> 00:05:56,360
Like do you see anything that 
this AI side of things and also 

102
00:05:56,360 --> 00:05:58,920
the behavioral science side, do 
you have anything in common? 

103
00:05:59,320 --> 00:06:01,400
Lots of things. 
Many, many things. 

104
00:06:01,480 --> 00:06:06,680
One sort of overarching theme 
that I noticed as I was working 

105
00:06:06,680 --> 00:06:11,440
in the the weeds of AI and 
behavioral science was just how 

106
00:06:12,120 --> 00:06:17,400
both of these fields are often 
packaged as these miracle cures.

107
00:06:17,400 --> 00:06:22,360
And our behavioral science 
listeners will relate to this, 

108
00:06:22,360 --> 00:06:25,440
I'm sure. 
And it's so frustrating, right? 

109
00:06:25,440 --> 00:06:28,520
This idea that you can just 
throw in a bunch of nudges and 

110
00:06:28,520 --> 00:06:31,640
all your behavioral problems 
will be solved or even your non 

111
00:06:31,640 --> 00:06:34,320
behavioral problems, like 
everything will be fine. 

112
00:06:34,960 --> 00:06:39,200
Well, the the the exact same 
situation machine learning 

113
00:06:39,200 --> 00:06:40,840
engineers and scientists are 
faced with. 

114
00:06:40,920 --> 00:06:43,400
Well, there's this perception 
that you can just automate 

115
00:06:43,440 --> 00:06:46,160
everything with AI and you know,
we don't need to work anymore. 

116
00:06:46,160 --> 00:06:49,960
And you know, of course, like 
the reality is that neither of 

117
00:06:49,960 --> 00:06:53,640
these things are magic. 
They both require a lot of work 

118
00:06:53,640 --> 00:06:57,840
and thoughts and intentionality.
But yeah, we often did laugh 

119
00:06:57,840 --> 00:07:02,520
about this similarity and just 
how wrong the perception is and 

120
00:07:02,520 --> 00:07:04,800
how this hype is countered by 
the reality. 

121
00:07:06,280 --> 00:07:10,320
Yeah, I mean, it is almost 
hilarious where you kind of see 

122
00:07:10,320 --> 00:07:13,840
in the same way as maybe people 
naively at some point with like 

123
00:07:13,840 --> 00:07:16,280
thinking about throwing use 
notice at problems to solve 

124
00:07:16,280 --> 00:07:18,000
them. 
It's also a little bit like the 

125
00:07:18,000 --> 00:07:22,360
last couple of years you've seen
this kind of like panicking of 

126
00:07:22,360 --> 00:07:25,280
like, OK, we have to make our 
products AI somehow. 

127
00:07:25,280 --> 00:07:28,440
Let's throw in some form of AI 
function and that's going to 

128
00:07:28,440 --> 00:07:30,040
make the user experience better 
somehow. 

129
00:07:30,160 --> 00:07:32,960
We're not really sure why or 
how, but we're just going to 

130
00:07:32,960 --> 00:07:35,520
throw AI everybody else. 
Is doing it so. 

131
00:07:35,840 --> 00:07:39,240
We must Exactly. 
No, but I think that's super 

132
00:07:39,280 --> 00:07:41,000
great tech and I think it's so 
true. 

133
00:07:41,600 --> 00:07:46,400
And yeah, we have a great guest 
in this episode that we explore 

134
00:07:46,400 --> 00:07:49,880
some of this and much more with.
And yeah, so let me introduce 

135
00:07:50,160 --> 00:07:53,000
Amy Buker. 
She's the chief here, officer at

136
00:07:53,000 --> 00:07:55,640
Lerio. 
And if her name sounds familiar,

137
00:07:55,640 --> 00:07:58,120
it could be for many good 
reasons. 

138
00:07:58,120 --> 00:08:02,680
It could be that you have read 
your book Engaged, which is a 

139
00:08:02,680 --> 00:08:06,360
really fantastic book. 
Find one either getting starting

140
00:08:06,360 --> 00:08:09,200
behavioral science or applied 
behavioral science or just wants

141
00:08:09,200 --> 00:08:11,920
to get some insights for how it 
can be done in practice. 

142
00:08:12,440 --> 00:08:17,040
You might have seen her in some 
form of conference speaking or 

143
00:08:17,560 --> 00:08:20,960
you've listened to the podcast 
before, because we had her in a 

144
00:08:20,960 --> 00:08:25,080
fantastic conversation in a 
previous season and she's one of

145
00:08:25,080 --> 00:08:28,040
the cases where actually there's
so many people to speak to. 

146
00:08:28,040 --> 00:08:30,600
And so we tried to invite new 
people to the podcast. 

147
00:08:30,600 --> 00:08:34,679
But with Amy, we want to have 
her back not only because she's 

148
00:08:34,679 --> 00:08:37,760
a fantastic representative of 
the field and practitioner, but 

149
00:08:37,760 --> 00:08:41,760
also because since we last spoke
to her, she has really been 

150
00:08:41,919 --> 00:08:45,080
thrown into this intersection of
AI and behavioral science with 

151
00:08:45,080 --> 00:08:48,280
her work, Illyrio. 
And she's been focusing on 

152
00:08:48,440 --> 00:08:54,400
specifically working with and 
understanding how AI systems can

153
00:08:54,400 --> 00:08:57,760
be used in tandem with 
behavioral science to 

154
00:08:57,760 --> 00:08:59,880
personalize health 
interventions. 

155
00:09:00,120 --> 00:09:04,600
So this includes looking at how 
to support patients going 

156
00:09:04,600 --> 00:09:08,280
through some form of chronic 
disease management, but many 

157
00:09:08,360 --> 00:09:10,280
different types of context and 
scenarios. 

158
00:09:10,640 --> 00:09:14,520
And her work specifically using 
precision nudging, I think it's 

159
00:09:14,520 --> 00:09:17,760
even references hyper precision 
looks like kind of how to make 

160
00:09:17,760 --> 00:09:21,080
these interventions both 
effective, but also scalable, 

161
00:09:21,200 --> 00:09:26,720
personalized and equitable. 
So super excited to bring her 

162
00:09:26,720 --> 00:09:31,200
back for another conversation. 
And yeah, what do we cover? 

163
00:09:31,360 --> 00:09:34,920
So as you might expect, we talk 
all about how Amy does just 

164
00:09:34,920 --> 00:09:36,960
this, applying AI to 
personalized health 

165
00:09:36,960 --> 00:09:39,920
interventions and specifically 
how she uses reinforcement 

166
00:09:39,920 --> 00:09:42,240
learning to nudge users toward 
healthier behaviors. 

167
00:09:42,800 --> 00:09:45,800
Amy shares her findings from a 
recent scoping review where she 

168
00:09:45,800 --> 00:09:48,800
looked at the landscape of AI 
being used in digital behavior 

169
00:09:48,800 --> 00:09:51,480
change interventions. 
And for me, I don't know about 

170
00:09:51,480 --> 00:09:54,920
you, Sam, but one take away was 
just that behavioral scientists 

171
00:09:54,920 --> 00:09:58,560
might need a little bit of help 
understanding what's AI and what

172
00:09:58,560 --> 00:10:03,280
isn't. 
We cover why behavioral science 

173
00:10:03,280 --> 00:10:07,440
is so, so essential for getting 
it right in AI and especially 

174
00:10:07,440 --> 00:10:10,440
when it comes to health 
behavior, and so, so much more, 

175
00:10:10,440 --> 00:10:14,360
including Amy's most 
controversial opinion in AI and 

176
00:10:14,360 --> 00:10:17,720
what it would take for cat memes
to be personalized to her 

177
00:10:17,720 --> 00:10:24,040
satisfaction. 
Happens to Murgatroyd. 

178
00:10:31,760 --> 00:10:34,840
I'm happy to say welcome back, 
Amy. 

179
00:10:34,920 --> 00:10:38,200
Good to have you on the podcast.
It's great to be here. 

180
00:10:38,200 --> 00:10:39,840
I'm so excited to talk to you 
both again. 

181
00:10:41,560 --> 00:10:43,840
Yeah. 
And I guess the first question I

182
00:10:43,840 --> 00:10:48,400
want to dive into is AI. 
Think still rather recently 

183
00:10:48,400 --> 00:10:52,480
shared scoping review that you 
made with a few of your 

184
00:10:52,480 --> 00:10:55,720
colleagues looking at something 
really relevant for the system. 

185
00:10:55,720 --> 00:10:58,520
Given that we're kind of like 
trying to understand this kind 

186
00:10:58,520 --> 00:11:01,320
of intersection between 
behavioral science and AI making

187
00:11:01,320 --> 00:11:06,360
sense of, OK, what is AIBS in 
terms of like behavioral science

188
00:11:06,360 --> 00:11:10,120
and what is AIBS in terms of 
bullshit and kind of seeing 

189
00:11:10,120 --> 00:11:11,880
between the lines of like which 
is which. 

190
00:11:12,320 --> 00:11:14,760
And I think this scoping review 
was quite interesting in that it

191
00:11:14,760 --> 00:11:20,200
looked at how AI and ML is used 
in digital, maybe change 

192
00:11:20,200 --> 00:11:23,400
interventions. 
And so, so Daveen, maybe you can

193
00:11:23,400 --> 00:11:26,480
give us kind of like the summary
of what you found, like what was

194
00:11:26,480 --> 00:11:29,520
the main takeaways and then we 
can maybe dive into do some 

195
00:11:29,520 --> 00:11:32,440
further findings as well. 
Sure and I love that you 

196
00:11:32,440 --> 00:11:36,280
characterized it as what is BS 
and AI cuz we actually had the 

197
00:11:36,280 --> 00:11:40,000
same 2 conceptions. 
Like from a a sort of legitimate

198
00:11:40,000 --> 00:11:43,160
scientific work perspective. 
I really wanted to know how is 

199
00:11:43,160 --> 00:11:44,400
this happening in the real 
world? 

200
00:11:44,400 --> 00:11:47,200
Like what is the landscape out 
there of the different AI tools 

201
00:11:47,200 --> 00:11:50,040
that are being used to change 
real world behaviors? 

202
00:11:50,240 --> 00:11:52,440
And I was going to add through 
digital means that AI pretty 

203
00:11:52,440 --> 00:11:55,640
much is through digital means. 
But the other, the other thing, 

204
00:11:55,640 --> 00:11:59,600
the sort of more personal reason
is we have noticed that there's 

205
00:11:59,600 --> 00:12:02,480
a lot of companies or groups who
talk about doing AI and then 

206
00:12:02,480 --> 00:12:04,440
when you look under the hood 
and, and carefully read the 

207
00:12:04,440 --> 00:12:07,160
description of what they're 
doing, it's not really AI at 

208
00:12:07,160 --> 00:12:08,480
all. 
It might be something like a 

209
00:12:08,480 --> 00:12:11,280
hard coded algorithm or a thing 
that I found a lot in the 

210
00:12:11,280 --> 00:12:14,480
scoping review were 
interventions, products that 

211
00:12:14,480 --> 00:12:17,040
called themselves AI. 
But we're using a Wizard of Oz 

212
00:12:17,040 --> 00:12:19,960
setup for the testing. 
So it was basically mocked up. 

213
00:12:19,960 --> 00:12:23,440
There was a live person behind 
the screen making it appear that

214
00:12:23,440 --> 00:12:25,560
there was an artificial 
intelligence interacting with 

215
00:12:25,560 --> 00:12:27,320
somebody, and so we ruled those 
out as well. 

216
00:12:27,320 --> 00:12:30,120
That might be an early prototype
for an AI, but in and of itself 

217
00:12:30,120 --> 00:12:32,680
it's not. 
Normally my team, my behavioral 

218
00:12:32,680 --> 00:12:35,920
science team will do these sort 
of encapsulated projects that we

219
00:12:35,920 --> 00:12:38,520
can run. 
It's a side passion project 

220
00:12:38,520 --> 00:12:40,920
without involving colleagues 
elsewhere in the company because

221
00:12:40,920 --> 00:12:44,000
it can be difficult to get time.
I mean, you know, it's it's hard

222
00:12:44,000 --> 00:12:46,720
to get time from other teams. 
But with this paper, we knew 

223
00:12:46,720 --> 00:12:49,240
from the beginning we had to 
co-author with at least one of 

224
00:12:49,240 --> 00:12:51,600
our AI scientists. 
So one of our co-authors is our 

225
00:12:51,600 --> 00:12:53,400
chief AI scientist, Chris 
Simons. 

226
00:12:54,080 --> 00:12:56,840
And the reason why is we needed 
help as well figuring out like 

227
00:12:56,840 --> 00:12:59,280
what are the legitimate search 
terms for AI? 

228
00:12:59,280 --> 00:13:02,000
And then once we have done our 
lit review and identified some 

229
00:13:02,000 --> 00:13:05,520
papers, we really needed Chris 
to read some of them in detail 

230
00:13:05,520 --> 00:13:09,240
and tell us, does this meet your
definition as a career computer 

231
00:13:09,240 --> 00:13:12,800
scientist who specializes in ML 
of AI? 

232
00:13:12,800 --> 00:13:17,400
And so that was unusual for us 
to absolutely require an AI 

233
00:13:17,400 --> 00:13:19,320
colleague to collaborate on the 
paper. 

234
00:13:20,080 --> 00:13:21,920
When once we built our search 
terms, we ended up finding a 

235
00:13:21,920 --> 00:13:25,200
total of 3000 exactly, which I 
think looks fake, but it was 

236
00:13:25,200 --> 00:13:29,280
3000 exactly papers that met 
those high level inclusion 

237
00:13:29,280 --> 00:13:31,000
criteria. 
And by the time we went through 

238
00:13:31,000 --> 00:13:34,800
all of our filtering and review,
we got it down to 32 papers that

239
00:13:34,800 --> 00:13:38,240
covered only 23 different 
interventions or products. 

240
00:13:39,360 --> 00:13:40,920
And you know, of course, these 
are the ones that have gone 

241
00:13:40,920 --> 00:13:42,600
through the trouble of 
publishing in the peer reviewed 

242
00:13:42,600 --> 00:13:44,520
literature, which means 
typically you have to have some 

243
00:13:44,520 --> 00:13:46,040
kind of results. 
Like there's the barrier to 

244
00:13:46,040 --> 00:13:48,160
being included on this list was 
somewhat high. 

245
00:13:48,160 --> 00:13:51,360
And I know there are companies 
out there that are doing real AI

246
00:13:51,360 --> 00:13:52,760
to change behavior in the real 
world. 

247
00:13:52,760 --> 00:13:54,840
But in the scientific 
literature, we only found these 

248
00:13:54,840 --> 00:13:57,240
23. 
And it was interesting to me as 

249
00:13:57,240 --> 00:14:01,080
someone who spent her career in 
industry, only six of them were 

250
00:14:01,080 --> 00:14:03,960
commercial products. 
The others were all developed 

251
00:14:03,960 --> 00:14:07,600
for research purposes and may or
may not ever be commercialized 

252
00:14:07,600 --> 00:14:09,680
and made available to the 
general public, which to me 

253
00:14:09,680 --> 00:14:13,400
seems like a little bit, well, 
it's a pain point with the 

254
00:14:13,400 --> 00:14:15,840
academia industry crossover that
I knew. 

255
00:14:15,840 --> 00:14:18,040
But it's, it's really poignant 
to the example of it because 

256
00:14:18,040 --> 00:14:19,240
some of these products were 
great. 

257
00:14:20,120 --> 00:14:23,160
But yeah, we we wanted to break 
down what types of AI are being 

258
00:14:23,160 --> 00:14:26,200
used and who's doing it. 
And one of the things that 

259
00:14:26,200 --> 00:14:31,120
really surprised me was that all
but two had proprietary AI where

260
00:14:31,120 --> 00:14:33,200
they developed that technology 
themselves. 

261
00:14:33,200 --> 00:14:35,920
And Lyrio is among them. 
Our, one of our papers actually 

262
00:14:35,920 --> 00:14:38,160
met our inclusion criteria. 
We have developed our own. 

263
00:14:38,600 --> 00:14:40,600
And I, I know that there's good 
reasons to do that. 

264
00:14:40,600 --> 00:14:42,720
You know, you want to own the 
IP, you really want to create 

265
00:14:42,720 --> 00:14:45,600
something that's competitively 
differentiated and and owning it

266
00:14:45,600 --> 00:14:48,240
makes a lot of sense. 
But it's also really limiting 

267
00:14:48,240 --> 00:14:50,280
because first of all, the 
barrier to doing that is much 

268
00:14:50,280 --> 00:14:53,600
higher than using products like 
ChatGPT and integrating that 

269
00:14:53,600 --> 00:14:56,200
into your product. 
There was one that came up in 

270
00:14:56,200 --> 00:14:58,400
our review that had cobbled 
together some products from 

271
00:14:58,400 --> 00:15:00,600
Amazon and Microsoft, but you 
know, these commercially 

272
00:15:00,600 --> 00:15:03,880
available and they probably got 
that stood up much more quickly 

273
00:15:03,880 --> 00:15:05,600
than somebody who grew up their 
own technology. 

274
00:15:05,600 --> 00:15:07,880
And the other thing is, when 
everybody develops their own 

275
00:15:07,880 --> 00:15:10,400
technology, it limits the 
ability to compare a bit because

276
00:15:10,400 --> 00:15:13,440
you don't truly know the 
intimate differences between 

277
00:15:13,440 --> 00:15:15,960
what one company develops and 
what another company develops to

278
00:15:15,960 --> 00:15:19,160
understand how does this work 
and how do we draw conclusions 

279
00:15:19,160 --> 00:15:21,000
about the effect of having our 
behavior. 

280
00:15:22,160 --> 00:15:23,520
Yeah. 
And I think that speaks to kind 

281
00:15:23,520 --> 00:15:26,200
of this interesting thing that 
happened where I feel like in 

282
00:15:26,200 --> 00:15:29,080
the last three years especially,
there's been like this booming 

283
00:15:29,080 --> 00:15:32,480
quote, UN quote, like AI tools. 
And it feels like all these 

284
00:15:32,480 --> 00:15:36,200
tools are in some ways unique. 
But if again, looking under the 

285
00:15:36,200 --> 00:15:41,600
hood, it goes back to maybe you 
see like 3 or 4 foundational 

286
00:15:41,720 --> 00:15:44,680
large language models that are 
used like either opening eyes, 

287
00:15:44,680 --> 00:15:48,840
either entropics, maybe some 
like llama model, or, you know, 

288
00:15:48,840 --> 00:15:51,480
a lot of them are basically 
using basically the same things 

289
00:15:51,480 --> 00:15:52,920
on the hood. 
And then they use basically 

290
00:15:52,920 --> 00:15:56,560
create a product out of 
interface, like how to kind of 

291
00:15:56,560 --> 00:15:59,640
make it easier for people to 
maybe directly make use of those

292
00:15:59,640 --> 00:16:02,840
models and so on. 
And yeah, is that something 

293
00:16:02,840 --> 00:16:04,800
similar that you came across as 
well or maybe? 

294
00:16:04,800 --> 00:16:07,320
Yeah. 
Well, actually on on large 

295
00:16:07,320 --> 00:16:10,360
language models specifically, we
found that they're basically not

296
00:16:10,360 --> 00:16:13,640
being used in patient or 
consumer facing behavior change 

297
00:16:13,640 --> 00:16:16,040
technology. 
And I wasn't surprised by that. 

298
00:16:16,040 --> 00:16:18,640
I was actually pleased by that. 
I don't think this is a 

299
00:16:18,640 --> 00:16:21,760
controversial take, but I don't 
think that large language models

300
00:16:21,760 --> 00:16:24,760
are ready for prime time in some
scenarios like providing medical

301
00:16:24,760 --> 00:16:29,040
information to patients. 
And actually, Lyrio was just a 

302
00:16:29,040 --> 00:16:31,600
sub awardee on a grant from ARPA
H, where we're going to be 

303
00:16:31,600 --> 00:16:33,320
looking at exactly that kind of 
thing. 

304
00:16:33,320 --> 00:16:36,080
We're going to be helping to 
develop some quality tools for 

305
00:16:36,080 --> 00:16:38,840
mental health chat bots that 
would help detect if the 

306
00:16:38,840 --> 00:16:41,360
information it produces is 
credible and complete. 

307
00:16:42,240 --> 00:16:43,920
That's the sort of thing that's 
missing right now. 

308
00:16:43,920 --> 00:16:46,360
And there's research on the use 
of large language models in 

309
00:16:46,360 --> 00:16:49,800
healthcare that from my read, 
the place where I've wound up is

310
00:16:50,120 --> 00:16:52,440
it can be appropriate for some 
clinician facing scenarios 

311
00:16:52,440 --> 00:16:54,960
because someone who's a 
clinician has gone through that 

312
00:16:54,960 --> 00:16:58,400
education and professional 
experience can put the bullshit 

313
00:16:58,400 --> 00:17:00,120
meter on it. 
They can read something that's 

314
00:17:00,120 --> 00:17:03,440
in a patient portal developed by
LLM and they can edit that and 

315
00:17:03,440 --> 00:17:06,319
make it accurate. 
There's a paper that I've cited 

316
00:17:06,319 --> 00:17:08,319
a few Times Now where they had a
Gen. 

317
00:17:08,359 --> 00:17:11,240
AI create first draft responses 
to patients in the patient 

318
00:17:11,240 --> 00:17:13,720
portal and then the provider had
to review review them before 

319
00:17:13,720 --> 00:17:16,880
they sent it out. 
One that really struck me was 

320
00:17:16,920 --> 00:17:20,760
the ChatGPT or the the Gen. 
AI said, yeah, patient, you're 

321
00:17:20,760 --> 00:17:23,400
taking 20 milligrams of this 
drug, but you take it twice a 

322
00:17:23,400 --> 00:17:25,880
day in the 10 milligram dose. 
It's fine if you get the 20 

323
00:17:25,880 --> 00:17:27,760
milligram pill in your 
subscription to save money and 

324
00:17:27,760 --> 00:17:30,360
just split it in half. 
And when the clinician read it 

325
00:17:30,360 --> 00:17:34,080
right away, they said, no, 
there's no score line on that 28

326
00:17:34,080 --> 00:17:36,040
milligram tablet. 
You're not gonna be able to 

327
00:17:36,040 --> 00:17:39,240
accurately divide it in half. 
That's like a human quality 

328
00:17:39,240 --> 00:17:42,680
information that would be very 
difficult to get out of the LLM.

329
00:17:42,680 --> 00:17:45,040
And the clinician has the 
expertise to instantly recognize

330
00:17:45,040 --> 00:17:46,960
that it's missing. 
We're on the patient side. 

331
00:17:46,960 --> 00:17:50,280
If they'd received that original
message, you know, at best they 

332
00:17:50,280 --> 00:17:52,400
would have learned through hard 
experience picking that pill up 

333
00:17:52,400 --> 00:17:54,560
at the drugstore and seeing they
couldn't break it in half. 

334
00:17:54,800 --> 00:17:57,960
So I, I, I think LLMS are not 
really ready for prime time for 

335
00:17:57,960 --> 00:18:00,000
that scenario. 
And that was kind of reflected 

336
00:18:00,000 --> 00:18:02,720
in the evidence we found in the 
paper where there were several 

337
00:18:02,720 --> 00:18:04,640
different interventions that 
were using natural language 

338
00:18:04,640 --> 00:18:07,560
understanding. 
So they were deriving intent and

339
00:18:07,560 --> 00:18:11,080
meaning from free text entered 
by the user, but whatever they 

340
00:18:11,080 --> 00:18:14,040
were responding back with had 
been pre written by an expert. 

341
00:18:15,000 --> 00:18:18,000
That's an interesting and I 
guess one of my reflections from

342
00:18:18,000 --> 00:18:20,880
also reading this and I think 
you have it in the conclusion in

343
00:18:20,880 --> 00:18:24,400
terms of one of the significant 
opportunities for people in 

344
00:18:24,400 --> 00:18:30,440
Marvel Science is to become more
like able at knowing and 

345
00:18:30,440 --> 00:18:33,400
understanding terminology in AI.
And we can touch upon that 

346
00:18:33,480 --> 00:18:36,840
already a bit, you know, and I 
guess I want to hear like a 

347
00:18:36,840 --> 00:18:39,200
little bit from your journey 
working with Leary, for example,

348
00:18:39,200 --> 00:18:41,280
like you've had a chance for the
last couple of years to really 

349
00:18:41,280 --> 00:18:44,440
work closely with like you 
mentioned colleagues that are 

350
00:18:44,840 --> 00:18:48,680
like coming from the maybe more 
like hardcore machine learning 

351
00:18:48,680 --> 00:18:51,800
backgrounds or similar. 
Like how do you think about kind

352
00:18:51,800 --> 00:18:54,360
of like developing that 
terminology and how how's that 

353
00:18:54,360 --> 00:18:56,800
been for you? 
It's been hard. 

354
00:18:56,840 --> 00:19:00,120
The Dunning Kruger effect is 
really all I'm living it. 

355
00:19:00,240 --> 00:19:03,400
I know more about AI than I've 
ever known, and I'm also so 

356
00:19:03,400 --> 00:19:05,600
aware of the limits of my own 
knowledge. 

357
00:19:06,160 --> 00:19:08,680
But I think what I'm getting 
good at is actually what I like 

358
00:19:08,680 --> 00:19:10,760
to see on the B side side 
sometimes too. 

359
00:19:10,800 --> 00:19:12,800
And I've been really passionate 
about making the tools of 

360
00:19:12,800 --> 00:19:15,480
behavioral science available 
widely to people of different 

361
00:19:15,480 --> 00:19:17,360
backgrounds and different 
professions. 

362
00:19:17,760 --> 00:19:20,440
But I do also believe there's a 
point at which you might be 

363
00:19:20,440 --> 00:19:23,480
working on a problem and you 
say, oh, I need someone who 

364
00:19:23,480 --> 00:19:26,920
really has, you know, trained in
this and reach out for that 

365
00:19:26,920 --> 00:19:28,720
expertise. 
And I, I think I'm in a similar 

366
00:19:28,720 --> 00:19:31,800
place with AI where now I can 
talk about it intelligently and 

367
00:19:31,800 --> 00:19:34,680
correctly to a point. 
And I've gotten pretty good at 

368
00:19:34,680 --> 00:19:39,360
recognizing what that point is, 
which I think is probably about 

369
00:19:39,360 --> 00:19:41,720
as good as it's going to get. 
I'm never going to replicate, 

370
00:19:42,240 --> 00:19:45,240
you know, Chris, who I mentioned
our chief AI scientist has his 

371
00:19:45,240 --> 00:19:47,840
PhD in computer science and, you
know, worked at Oak Ridge 

372
00:19:47,840 --> 00:19:49,640
National Labs leading teams over
there. 

373
00:19:49,640 --> 00:19:51,400
Like, I'm not going to learn 
what he knows. 

374
00:19:52,600 --> 00:19:55,600
But the the other good thing 
about it is one of the things 

375
00:19:55,600 --> 00:20:01,160
he's taught me is that AI is. 
Some of these models that we're 

376
00:20:01,160 --> 00:20:04,160
using, the supervised training 
models, like they benefit from 

377
00:20:04,160 --> 00:20:07,320
subject matter expertise in the 
domain where they're being used.

378
00:20:08,160 --> 00:20:11,040
When he frames it that way, we 
become essential partners. 

379
00:20:11,040 --> 00:20:14,080
And that's kind of a really cool
thing about working with the AI 

380
00:20:14,080 --> 00:20:16,440
is that we can't independently 
operate. 

381
00:20:16,440 --> 00:20:19,560
We have to work together. 
And so the questioning is going 

382
00:20:19,560 --> 00:20:21,480
back and forth, which makes it 
more comfortable for me to say, 

383
00:20:21,480 --> 00:20:22,960
Chris, is this written 
correctly? 

384
00:20:22,960 --> 00:20:25,320
Is this really what I'm supposed
to be saying? 

385
00:20:25,320 --> 00:20:26,760
Does this mean what I think it's
meaning? 

386
00:20:26,760 --> 00:20:28,640
And I learned a lot through that
kind of thing, too. 

387
00:20:29,480 --> 00:20:31,640
And what would you say are some 
of the most common 

388
00:20:31,640 --> 00:20:34,640
misconceptions about AI, at 
least in terms of from the 

389
00:20:34,640 --> 00:20:36,920
perspective of the behavioral 
science community? 

390
00:20:36,920 --> 00:20:40,760
You mentioned hard coded rules 
and really algorithms that don't

391
00:20:40,760 --> 00:20:43,720
adapt as being AI but but what 
else is there? 

392
00:20:44,760 --> 00:20:46,600
Yeah. 
Well, IA big one I'm seeing 

393
00:20:46,600 --> 00:20:49,280
right now that drives me up the 
wall is I think people are 

394
00:20:49,280 --> 00:20:52,560
increasingly in conflating 
generative AI with AI as an 

395
00:20:52,560 --> 00:20:55,080
overall category. 
They say AI and they just mean 

396
00:20:55,080 --> 00:20:55,920
generative AI. 
Adjust me. 

397
00:20:56,360 --> 00:20:57,560
Yep. 
Yeah. 

398
00:20:57,560 --> 00:21:00,000
And that's just such a small 
subset of what AI can do. 

399
00:21:00,000 --> 00:21:03,160
And it's not the type of AII 
work with our product uses 

400
00:21:03,240 --> 00:21:06,320
reinforcement learning. 
And I am actually a huge fan of 

401
00:21:06,320 --> 00:21:08,640
reinforcement learning. 
I think it can be very powerful 

402
00:21:09,120 --> 00:21:11,560
in behavioral science. 
And so I guess it's not a 

403
00:21:11,560 --> 00:21:14,960
misconception so much as a lack 
of awareness of additional AI 

404
00:21:14,960 --> 00:21:19,160
tools and approaches that could 
go well beyond or augment what 

405
00:21:19,160 --> 00:21:21,280
Jen AI could do. 
And again, I also think Jen AI 

406
00:21:21,280 --> 00:21:23,840
is not ready for prime time with
that patient or consumer facing 

407
00:21:23,840 --> 00:21:26,240
content. 
So to hyper focus on that I 

408
00:21:26,240 --> 00:21:28,600
think is really, really limiting
for a behavioral scientist who 

409
00:21:28,600 --> 00:21:31,160
wants to change behavior. 
Yeah. 

410
00:21:31,160 --> 00:21:33,960
And, and it definitely seems 
like this is a phenomenon that's

411
00:21:34,160 --> 00:21:36,760
really only come about in the 
past couple of years. 

412
00:21:36,880 --> 00:21:40,200
I remember when Gen. 
AI was really pretty niche, 

413
00:21:40,240 --> 00:21:41,960
right? 
You only heard about natural 

414
00:21:41,960 --> 00:21:46,800
language processing in these 
like much more narrow use cases.

415
00:21:46,800 --> 00:21:49,560
And now it's everyone is using 
the LLMS. 

416
00:21:50,680 --> 00:21:52,520
I mean ChatGPT was a game 
changer. 

417
00:21:52,520 --> 00:21:57,760
Putting a public facing, easy to
use interface on an LLM just 

418
00:21:57,760 --> 00:22:00,160
totally changed the world. 
I mean, that's an important 

419
00:22:00,160 --> 00:22:03,040
lesson, right? 
Do you think that lesson can be 

420
00:22:03,040 --> 00:22:05,600
transferred to other use cases 
of AI? 

421
00:22:06,640 --> 00:22:09,360
Probably, actually. 
So reinforcement learning that I

422
00:22:09,360 --> 00:22:11,960
work with, we call each 
algorithm an agent. 

423
00:22:12,000 --> 00:22:14,040
And I'm clarifying that because 
sometimes we'll talk about the 

424
00:22:14,040 --> 00:22:17,080
agent decisions and people are 
like, wait, who's this agent? 

425
00:22:17,400 --> 00:22:19,760
It's an algorithm. 
When we set up our 

426
00:22:19,760 --> 00:22:23,440
interventions, they are multi 
agent systems and each agent is 

427
00:22:23,440 --> 00:22:25,320
responsible for a specific 
choice. 

428
00:22:25,320 --> 00:22:28,080
But those choices ladder up. 
So with Lyrio and Precision 

429
00:22:28,080 --> 00:22:31,120
Nudging, we're sending 
personalized messaging and we 

430
00:22:31,120 --> 00:22:33,600
might have an agent that decides
what channel we use for a 

431
00:22:33,600 --> 00:22:34,840
person. 
Is this person getting an 

432
00:22:34,840 --> 00:22:37,120
e-mail, a text message, a push 
notification? 

433
00:22:37,520 --> 00:22:41,000
Another agent might actually 
decide on the headline that the 

434
00:22:41,000 --> 00:22:43,360
e-mail gets. 
And a third agent decides on 

435
00:22:43,360 --> 00:22:44,800
what's the body copy of this 
e-mail? 

436
00:22:44,800 --> 00:22:47,440
What are we actually saying to 
this person? 

437
00:22:47,440 --> 00:22:50,160
And then yet another agent might
decide when we send it. 

438
00:22:50,760 --> 00:22:55,440
And each of those agents there, 
we we consider what we do, a 

439
00:22:55,440 --> 00:22:58,560
type of objective driven AI. 
That's a term that Yan Likun, 

440
00:22:58,560 --> 00:23:01,760
who's the head of AI at Meta, 
has coined in the last year or 

441
00:23:01,760 --> 00:23:03,840
so. 
And reinforcement learning is a 

442
00:23:03,840 --> 00:23:05,800
perfect symbol of that because 
what you do when you train those

443
00:23:05,800 --> 00:23:07,920
agents, when you set up those 
algorithms is you give them 

444
00:23:07,920 --> 00:23:10,520
objectives. 
So the objectives in our case 

445
00:23:10,520 --> 00:23:13,280
are almost always we want the 
person who receives this message

446
00:23:13,280 --> 00:23:16,680
to complete the target behavior.
And I always use the example of 

447
00:23:16,680 --> 00:23:19,080
mammogram. 
That's one of our most mature 

448
00:23:19,080 --> 00:23:22,000
interventions where we have 
pretty strong evidence. 

449
00:23:22,000 --> 00:23:24,480
And it's a very straightforward 
behavior because what we're 

450
00:23:24,480 --> 00:23:27,640
asking people to do is open our 
message, schedule your 

451
00:23:27,640 --> 00:23:29,560
mammogram, attend that 
mammogram. 

452
00:23:30,000 --> 00:23:32,920
And what we can basically do is 
tell the various agents that are

453
00:23:32,920 --> 00:23:35,440
making decisions about how we 
communicate with somebody that 

454
00:23:35,640 --> 00:23:37,800
it will be rewarded. 
And again, that's conceptual. 

455
00:23:37,800 --> 00:23:40,880
It's not a money thing. 
At the highest level, if someone

456
00:23:40,880 --> 00:23:43,320
has that behavioral achievement,
they've actually finished the 

457
00:23:43,320 --> 00:23:47,360
mammogram, but as a lower reward
if they schedule it and a lower 

458
00:23:47,360 --> 00:23:50,320
reward reward yet if they 
interact with the message. 

459
00:23:50,320 --> 00:23:52,640
And in that way, we're sort of 
able to prioritize the 

460
00:23:52,640 --> 00:23:55,880
meaningful behaviors. 
That's something where you could

461
00:23:55,880 --> 00:23:58,080
have an interface that people 
interact with and you could 

462
00:23:58,080 --> 00:24:00,640
start to play around with, you 
know, what do we make the most 

463
00:24:00,640 --> 00:24:02,080
important here? 
And especially when you get into

464
00:24:02,080 --> 00:24:04,320
more complex behavioral 
patterns, you know, mammograms, 

465
00:24:04,320 --> 00:24:06,480
easy to talk about because it is
so straightforward. 

466
00:24:06,480 --> 00:24:10,520
But in reality, if someone has a
chronic condition or even is 

467
00:24:10,520 --> 00:24:13,040
healthy but is orchestrating all
their healthcare, like Wellness 

468
00:24:13,040 --> 00:24:16,040
visits and vaccines and cancer 
screenings, that can get pretty 

469
00:24:16,040 --> 00:24:17,920
complex. 
And that's that again, is where,

470
00:24:17,960 --> 00:24:20,480
you know, I think there's some 
room to pull levers and twist 

471
00:24:20,480 --> 00:24:22,920
knobs, so to speak, and figure 
out what is the right way to 

472
00:24:22,920 --> 00:24:24,760
orchestrate. 
But again, if you set those 

473
00:24:24,760 --> 00:24:26,720
objectives, reinforcement 
learning is going to start to 

474
00:24:26,720 --> 00:24:29,240
figure out what is the best way 
to get this person to get to 

475
00:24:29,240 --> 00:24:32,000
that objective under the 
scenario that it's implemented 

476
00:24:32,000 --> 00:24:32,240
in. 
Yeah. 

477
00:24:32,240 --> 00:24:34,000
And. 
I think it's interesting in 

478
00:24:34,000 --> 00:24:37,360
terms of like there's been 
increasingly use of the term 

479
00:24:37,360 --> 00:24:42,640
agent, but I do think that most 
uses where I see agents being 

480
00:24:43,000 --> 00:24:47,000
referenced, it's actually not 
with that reinforcement part to 

481
00:24:47,000 --> 00:24:48,240
it. 
It's often times a little bit 

482
00:24:48,240 --> 00:24:51,760
simpler in some ways. 
It's often times we have some 

483
00:24:51,760 --> 00:24:53,480
form of setup that automates 
that. 

484
00:24:53,800 --> 00:24:58,400
You know, if this thing is 
triggered, then that activates 

485
00:24:58,400 --> 00:25:03,400
this agent that maybe uses some 
form of let's say open AIS large

486
00:25:03,400 --> 00:25:06,680
language model to execute on 
some form of like quite simple 

487
00:25:06,680 --> 00:25:08,000
thing. 
And it's pretty good. 

488
00:25:08,200 --> 00:25:11,080
You can do some cool things just
by doing some of that stuff, but

489
00:25:11,080 --> 00:25:13,160
it's still not used enough time 
to reinforcement learning. 

490
00:25:13,160 --> 00:25:15,400
So I think, I think that's 
actually still like a pretty 

491
00:25:15,400 --> 00:25:20,760
good indicator of maybe a little
more advanced use cases where 

492
00:25:20,760 --> 00:25:23,160
often times there's people 
involved that are also coming 

493
00:25:23,160 --> 00:25:25,160
from lately the machine learning
background or have a more 

494
00:25:25,600 --> 00:25:29,000
technical backgrounds as well. 
Whereas they they can be like a 

495
00:25:29,000 --> 00:25:31,480
lot of Bros on YouTube. 
This is like, hey I I designed 

496
00:25:31,480 --> 00:25:35,240
an AI agent or something. 
And that that actually is where 

497
00:25:35,240 --> 00:25:37,600
I mentioned in the paper that we
did the scoping review. 

498
00:25:37,600 --> 00:25:40,480
Only two of those interventions 
used publicly available and I 

499
00:25:40,480 --> 00:25:42,720
can't remember the name of it, 
but the one that didn't use the 

500
00:25:42,720 --> 00:25:46,200
Microsoft and the Amazon used it
was some kind of web interface 

501
00:25:46,200 --> 00:25:48,720
that allows you to basically say
if this, then this, you know, 

502
00:25:48,720 --> 00:25:51,040
deploy this. 
And so that was how that one 

503
00:25:51,040 --> 00:25:53,320
worked. 
Maybe quick thing to tie up 

504
00:25:53,640 --> 00:25:54,720
before we move to the next 
question. 

505
00:25:54,720 --> 00:25:59,760
What are them the Pareto 
principle terminology terms that

506
00:25:59,760 --> 00:26:01,800
people should know about within 
Tai? 

507
00:26:02,200 --> 00:26:05,200
Is there some like really 
important ones that people in 

508
00:26:05,200 --> 00:26:07,960
Babel science should you know or
be able to better understand? 

509
00:26:09,360 --> 00:26:11,280
I think we've talked about some 
of them already. 

510
00:26:11,280 --> 00:26:15,600
So things like generative AI and
understanding that's a subset of

511
00:26:15,600 --> 00:26:19,200
AII think the term objective 
driven AI is very useful. 

512
00:26:19,200 --> 00:26:22,360
I'm not one to jump on a 
buzzword that an executive at a 

513
00:26:22,360 --> 00:26:25,760
buzzy company like Meta has has 
coined, but in this case I think

514
00:26:25,760 --> 00:26:29,480
it really does a nice job both 
describing what certain types of

515
00:26:29,520 --> 00:26:33,040
AI are built for and also 
helping to distinguish it a bit 

516
00:26:33,200 --> 00:26:35,800
from generative AI. 
Although there, you know, those 

517
00:26:35,800 --> 00:26:37,440
could overlap a bit in the Venn 
diagram. 

518
00:26:37,440 --> 00:26:40,880
I can imagine scenarios. 
We included a glossary in that 

519
00:26:40,880 --> 00:26:44,560
paper that Chris actually wrote,
but then I edited to make sure 

520
00:26:44,560 --> 00:26:46,720
that it was understandable to 
someone like me. 

521
00:26:47,080 --> 00:26:49,920
And we're just about to actually
make a blog post version of that

522
00:26:49,920 --> 00:26:52,120
on our website. 
That because we think that's 

523
00:26:52,120 --> 00:26:54,840
really important, that these 
terms are at least of passing 

524
00:26:54,840 --> 00:26:56,720
familiarity to people. 
And I'll tell you, I can't use 

525
00:26:56,720 --> 00:27:00,280
all of those terms comfortably 
in a conversation myself, but 

526
00:27:00,280 --> 00:27:03,520
there's, you know, the ones that
I use, the ones that sort of 

527
00:27:03,520 --> 00:27:05,400
affect our product. 
So I understand things like 

528
00:27:05,400 --> 00:27:07,480
supervised learning and 
unsupervised learning. 

529
00:27:07,480 --> 00:27:09,240
And you know, where that 
supervised learning is really 

530
00:27:09,240 --> 00:27:12,040
where our behavioral models come
to bear because what we're 

531
00:27:12,040 --> 00:27:15,520
essentially doing is looking at 
what the agents want, want to do

532
00:27:15,520 --> 00:27:18,360
and course correcting them based
on our expert knowledge and the 

533
00:27:18,360 --> 00:27:20,360
logic models that we set up with
our content. 

534
00:27:20,880 --> 00:27:23,040
So for me, understanding that 
kind of thing is important. 

535
00:27:23,040 --> 00:27:25,480
But I think for the average 
person, just understanding that 

536
00:27:25,480 --> 00:27:28,280
the models need to be trained, 
that there's different ways to 

537
00:27:28,280 --> 00:27:32,360
train them, and maybe even that 
the quality and origin of the 

538
00:27:32,360 --> 00:27:35,160
training data is really 
impactful on what the final 

539
00:27:35,160 --> 00:27:36,800
product looks like. 
I don't think you need to 

540
00:27:36,800 --> 00:27:39,960
understand all of the guts of 
how that works, but to know 

541
00:27:39,960 --> 00:27:41,720
that's a piece of it is really 
critical. 

542
00:27:42,160 --> 00:27:45,000
And you're working with 
contextual bandits, which I like

543
00:27:45,000 --> 00:27:49,160
to think of as kind of like 
superhuman experimenters, right?

544
00:27:49,160 --> 00:27:52,800
They're like sending little bits
out into the fray and seeing 

545
00:27:52,800 --> 00:27:55,160
what comes back and then 
choosing the winner and then 

546
00:27:55,160 --> 00:27:58,920
moving on to the next experiment
and just optimizing in that way.

547
00:27:59,400 --> 00:28:02,400
Well, and you'll like to The 
early papers, the ones that my 

548
00:28:02,400 --> 00:28:05,400
colleagues cite all the time 
about contextual bandits are 

549
00:28:05,600 --> 00:28:07,120
based on the video game 
StarCraft. 

550
00:28:07,560 --> 00:28:09,080
So when you say like shooting 
things out like. 

551
00:28:09,080 --> 00:28:14,400
Yes exactly, almost literally. 
That's so funny yeah some of the

552
00:28:14,520 --> 00:28:18,160
AIML terms just crack me up 
every time I see them. 

553
00:28:18,160 --> 00:28:22,840
The multi armed bandits and just
like I hard to take seriously 

554
00:28:22,840 --> 00:28:25,760
but. 
QQ Learning is one that I hear 

555
00:28:25,760 --> 00:28:28,560
sometimes and I think it might 
actually be in our glossary, but

556
00:28:28,560 --> 00:28:30,960
I don't know exactly how it 
works, but I always think of 

557
00:28:30,960 --> 00:28:35,800
Star Trek. 
Hard to not go to sci-fi really 

558
00:28:35,800 --> 00:28:38,280
when you're when you're talking 
about AI in general, right? 

559
00:28:39,560 --> 00:28:43,600
Totally. 
So I would love to really get 

560
00:28:43,600 --> 00:28:48,080
into the nitty gritty of what 
you're doing at Lyrio with the 

561
00:28:48,120 --> 00:28:50,560
large behavior model with 
precision nudging. 

562
00:28:50,680 --> 00:28:52,960
Like I want to understand how 
does it all work? 

563
00:28:52,960 --> 00:28:54,880
What are you doing with your 
bandits? 

564
00:28:54,880 --> 00:28:58,520
What for, you know, the the 
goodness of science and other 

565
00:28:58,520 --> 00:29:01,480
people trying to solve for 
health problems out there. 

566
00:29:01,480 --> 00:29:04,560
Like what can you share about 
how how this works, where 

567
00:29:04,560 --> 00:29:07,920
behavioral science fits in, and 
how AI and behavioral science 

568
00:29:07,920 --> 00:29:10,560
are really like, really working 
together to make your 

569
00:29:10,560 --> 00:29:13,520
interventions amazing? 
Sure. 

570
00:29:13,560 --> 00:29:16,840
And I think I can share quite a 
bit actually, when we begin 

571
00:29:16,840 --> 00:29:19,480
designing a new intervention, 
it's my team, the behavioral 

572
00:29:19,480 --> 00:29:21,400
science team that is first on 
call. 

573
00:29:21,440 --> 00:29:24,800
So we almost always design 
interventions because we have a 

574
00:29:24,800 --> 00:29:27,440
client who has a use case that 
we're helping them to solve for 

575
00:29:27,440 --> 00:29:29,120
it. 
So there's an external partner 

576
00:29:29,120 --> 00:29:32,000
in the loop, and what that 
provides for us is the 

577
00:29:32,000 --> 00:29:34,280
opportunity to research the 
behavior in context. 

578
00:29:34,680 --> 00:29:37,440
So we work with the client to 
understand their objectives. 

579
00:29:37,440 --> 00:29:40,160
And a lot of times these are 
things, no surprises here. 

580
00:29:40,160 --> 00:29:41,920
They want to close the gap in 
care. 

581
00:29:41,920 --> 00:29:44,200
They want to improve their 
reimbursements from insurance 

582
00:29:44,200 --> 00:29:46,600
companies because they've raised
their quality or they're meeting

583
00:29:46,600 --> 00:29:49,720
a certain metric better. 
Sometimes it's a population 

584
00:29:49,720 --> 00:29:51,640
health goal. 
We want to get X percent of our 

585
00:29:51,640 --> 00:29:54,440
population below this health 
metric. 

586
00:29:54,840 --> 00:29:57,640
We have an example right now 
with a client, Cone Health, who 

587
00:29:57,640 --> 00:30:00,600
is awesome, awesome, awesome, 
and has been really wonderful 

588
00:30:00,600 --> 00:30:02,960
about allowing us to talk about 
the work that we do with them. 

589
00:30:03,560 --> 00:30:06,080
They're helping us to, well, we 
actually have launched it 

590
00:30:06,080 --> 00:30:08,160
already. 
We have an intervention for 

591
00:30:08,160 --> 00:30:11,040
hypertension management. 
And one of Cone Health's goals, 

592
00:30:11,040 --> 00:30:13,040
because they're really committed
to HealthEquity, is not only to 

593
00:30:13,040 --> 00:30:16,200
get, you know, meet meet the 
sort of target metrics that are 

594
00:30:16,240 --> 00:30:18,400
nationally available for 
hypertension control, but also 

595
00:30:18,400 --> 00:30:21,360
to make sure that they're black 
patients also meet that metric. 

596
00:30:21,360 --> 00:30:24,760
Because typically if you look at
a patient population, if it 

597
00:30:24,760 --> 00:30:28,320
meets that metric, on average, 
it tends to be white patients 

598
00:30:28,320 --> 00:30:30,200
who have met it. 
And then black patients 

599
00:30:30,200 --> 00:30:32,680
specifically will lag behind. 
So Cone is committed to making 

600
00:30:32,680 --> 00:30:35,440
sure that that gap doesn't exist
in their population, which is 

601
00:30:35,440 --> 00:30:37,240
awesome. 
But one of the first things we 

602
00:30:37,240 --> 00:30:39,760
did with them was understand 
that was specifically a goal of 

603
00:30:39,760 --> 00:30:42,280
theirs because then that allows 
us to design the rest of the 

604
00:30:42,280 --> 00:30:43,920
intervention appropriately for 
that. 

605
00:30:44,800 --> 00:30:46,200
We identify the target 
behaviors. 

606
00:30:46,200 --> 00:30:49,240
We do our, you know, beautiful 
literature review to understand 

607
00:30:49,240 --> 00:30:51,640
the determinants as seen in 
other research. 

608
00:30:51,640 --> 00:30:53,800
And then this is where having 
that partner is also really 

609
00:30:53,800 --> 00:30:55,560
helpful. 
We will do research in context 

610
00:30:55,560 --> 00:30:57,960
with them as well. 
So we talked to, you know, 

611
00:30:57,960 --> 00:31:00,040
particularly like their 
clinicians actually to 

612
00:31:00,040 --> 00:31:03,760
understand how hypertension 
management happens and you know,

613
00:31:03,760 --> 00:31:06,200
what are the sort of contextual 
barriers that we need to 

614
00:31:06,200 --> 00:31:09,720
understand and design for. 
We put all of that into a logic 

615
00:31:09,720 --> 00:31:11,360
model. 
This is all still my team. 

616
00:31:11,400 --> 00:31:14,680
So we have a document that we 
create that we call our strategy

617
00:31:14,680 --> 00:31:17,960
matrix and we actually outline 
what are the determinants that 

618
00:31:17,960 --> 00:31:20,120
we've prioritized to addressing 
this intervention? 

619
00:31:20,120 --> 00:31:22,080
What are the mechanisms of 
action through which we could 

620
00:31:22,080 --> 00:31:24,680
influence those? 
What are the specific behavior 

621
00:31:24,680 --> 00:31:28,200
change techniques that we are 
going to operationalize in our 

622
00:31:28,200 --> 00:31:31,760
messaging to use that MOA and 
address the barrier? 

623
00:31:32,360 --> 00:31:34,760
And then the content and visual 
design teams are part of 

624
00:31:34,760 --> 00:31:37,760
behavioral science at Lyrio and 
they work on operationalizing 

625
00:31:37,760 --> 00:31:41,280
all of that into assets. 
We create lots of little content

626
00:31:41,280 --> 00:31:45,640
components, so we sort of 
atomize what goes into a message

627
00:31:45,640 --> 00:31:48,320
so that we have more things that
we can select from to 

628
00:31:48,320 --> 00:31:51,600
personalize. 
And we also create visuals that 

629
00:31:51,600 --> 00:31:53,640
operationalize the behavior 
change techniques, which I think

630
00:31:53,640 --> 00:31:56,200
is pretty unique for us. 
And we're starting to accumulate

631
00:31:56,200 --> 00:31:58,480
some data. 
That approach is helpful in both

632
00:31:58,480 --> 00:32:00,400
engaging and converting people 
to the behavior. 

633
00:32:00,400 --> 00:32:03,360
So that the idea of the 
hypothesis was if we can kind of

634
00:32:03,600 --> 00:32:06,600
infuse the BCT into somebody's 
brain through multiple channels,

635
00:32:06,600 --> 00:32:09,400
let's do that. 
We build out all those assets 

636
00:32:09,400 --> 00:32:12,200
and around here is where we 
really start interacting with 

637
00:32:12,440 --> 00:32:15,120
the AI team because some of the 
things that we need to ensure 

638
00:32:15,120 --> 00:32:18,400
that they do, they have to train
the agent or the agents on the 

639
00:32:18,400 --> 00:32:20,840
content that we've developed. 
And we're using our logic model 

640
00:32:20,840 --> 00:32:22,240
to do that. 
So that's where part of our 

641
00:32:22,240 --> 00:32:26,200
expert knowledge is infused. 
One of the steps of training the

642
00:32:26,200 --> 00:32:28,760
agent is doing a semantic 
conversion of the text. 

643
00:32:28,760 --> 00:32:33,600
So if you've seen like encoder 
decoder models, Bert is Google's

644
00:32:33,600 --> 00:32:35,760
tool for some of this. 
And I believe we actually do use

645
00:32:35,760 --> 00:32:37,400
Bert at one point in our 
process. 

646
00:32:38,080 --> 00:32:40,360
That's a semantic conversion. 
So it looks at, are these words 

647
00:32:40,360 --> 00:32:42,640
similar or different in a multi 
dimensional space? 

648
00:32:42,640 --> 00:32:45,240
How would they cluster together?
And this is actually a place 

649
00:32:45,240 --> 00:32:49,160
where our expert knowledge makes
a big difference because there 

650
00:32:49,160 --> 00:32:51,800
are pieces of content that might
seem very similar from a 

651
00:32:51,800 --> 00:32:55,120
semantic level if you don't 
really know the meaning of the 

652
00:32:55,120 --> 00:32:57,400
words, but you know that the 
words are kind of similar. 

653
00:32:57,880 --> 00:33:00,280
But if you're a human being and 
especially behavioral scientist 

654
00:33:00,280 --> 00:33:02,360
and you read them, you're like, 
those are very different. 

655
00:33:02,800 --> 00:33:06,160
And so we can basically use our 
logic models to force a 

656
00:33:06,160 --> 00:33:09,360
separation between items that 
are different in terms of their 

657
00:33:09,360 --> 00:33:12,400
meaning, where the model might 
have otherwise put them close 

658
00:33:12,400 --> 00:33:14,320
together because of their words 
being similar. 

659
00:33:14,320 --> 00:33:16,680
So there's that piece of it. 
And then the other piece of it 

660
00:33:16,680 --> 00:33:18,560
is helping to set the reward 
function. 

661
00:33:18,560 --> 00:33:21,200
So I again, I use that really 
simple example of a mammogram 

662
00:33:21,200 --> 00:33:23,000
where the highest reward 
function is achieving the 

663
00:33:23,000 --> 00:33:24,480
mammogram, actually getting it 
done. 

664
00:33:25,080 --> 00:33:27,520
But with some of these more 
complex behaviors we have to 

665
00:33:27,520 --> 00:33:29,720
think about, you know, how do we
prioritize them? 

666
00:33:29,720 --> 00:33:32,760
What is really the objective 
from a behavioral perspective 

667
00:33:33,160 --> 00:33:35,240
and how do we build that into 
the algorithm? 

668
00:33:35,240 --> 00:33:36,960
So that's what it's maximizing 
for. 

669
00:33:37,440 --> 00:33:41,120
It would be really easy to abuse
this sort of approach, and I 

670
00:33:41,120 --> 00:33:43,560
would suspect that Facebook 
probably uses some of this 

671
00:33:43,560 --> 00:33:46,040
stuff, for example, because of 
the way that their news feed is 

672
00:33:46,040 --> 00:33:47,680
created. 
And sort of putting the next 

673
00:33:47,680 --> 00:33:49,960
thing in front of you that keeps
you on the screen like that is 

674
00:33:49,960 --> 00:33:51,840
an objective that you could 
train an agent for. 

675
00:33:52,360 --> 00:33:54,680
In our case, we specifically 
don't want to do that. 

676
00:33:54,680 --> 00:33:59,120
I would rather you not even read
the messages if I can somehow 

677
00:33:59,120 --> 00:34:01,440
otherwise get you to do the 
behavior. 

678
00:34:01,440 --> 00:34:03,600
So we just want to be really 
careful that whatever we're 

679
00:34:03,600 --> 00:34:06,680
rewarding the agent for is the 
thing that's really most helpful

680
00:34:06,680 --> 00:34:08,679
to our objective. 
And sometimes that's also 

681
00:34:08,679 --> 00:34:10,400
thinking from a behavioral 
scientist perspective. 

682
00:34:10,400 --> 00:34:13,840
If you have somebody who has a 
lot of healthcare needs, it's 

683
00:34:13,840 --> 00:34:16,800
not always the most sort of 
medically impactful thing you 

684
00:34:16,800 --> 00:34:20,120
want to go after first. 
Because people who have a lot on

685
00:34:20,120 --> 00:34:22,480
their plates in terms of 
changing their behaviors and 

686
00:34:22,480 --> 00:34:24,960
addressing their health might be
better served by trying 

687
00:34:24,960 --> 00:34:27,280
something easy first, you know, 
building self efficacy and 

688
00:34:27,280 --> 00:34:30,800
notching that small initial win.
They may be better served by, 

689
00:34:30,880 --> 00:34:33,520
you know, doing something that 
is easier for them for some 

690
00:34:33,520 --> 00:34:35,400
reason. 
Maybe they live a lot closer to 

691
00:34:35,400 --> 00:34:38,360
the facility where certain 
healthcare appointment happens 

692
00:34:38,360 --> 00:34:40,719
than the one where the like, the
big appointment happens. 

693
00:34:41,400 --> 00:34:43,239
We need to think about those 
things too. 

694
00:34:43,239 --> 00:34:45,440
And that's where AI can be 
really helpful at parsing out 

695
00:34:45,440 --> 00:34:48,840
like complex data about a person
and, you know, sort of 

696
00:34:48,840 --> 00:34:52,199
identifying what is most likely 
to work for this person, which 

697
00:34:52,239 --> 00:34:54,760
could be a bridge to the LLBM, 
the large behavior model. 

698
00:34:54,760 --> 00:34:57,560
But I don't want to just talk 
for like 4 minutes you guys say.

699
00:34:57,760 --> 00:35:02,160
Well, this actually brings up a 
maybe semi interesting question 

700
00:35:02,160 --> 00:35:06,160
for me is what if you put a 
negative incentive on reading of

701
00:35:06,160 --> 00:35:09,560
the e-mail or you know, time 
spent on the e-mail as opposed 

702
00:35:09,560 --> 00:35:13,360
to just a small incentive, 
really disincentivize that 

703
00:35:13,440 --> 00:35:17,320
outcome and then really, you 
know, outweigh the attendance at

704
00:35:17,320 --> 00:35:19,880
the mammogram for example? 
Yeah, it. 

705
00:35:19,960 --> 00:35:22,880
Happen. 
So I'm going to give you a 

706
00:35:22,880 --> 00:35:25,080
really, I'm sorry, this is a 
disappointing response. 

707
00:35:25,080 --> 00:35:27,240
I don't think that was actually 
possible because one of the 

708
00:35:27,240 --> 00:35:29,440
things that we're actually 
learning and and this just, it 

709
00:35:29,440 --> 00:35:31,960
doesn't matter to us that much 
because we've never put a huge 

710
00:35:31,960 --> 00:35:33,720
reward on interaction with the 
product. 

711
00:35:34,240 --> 00:35:36,680
But e-mail metrics in particular
are getting really hard to 

712
00:35:36,680 --> 00:35:39,680
measure because some of the 
changes that like Apple and 

713
00:35:39,680 --> 00:35:41,360
Google have made to their e-mail
clients. 

714
00:35:41,360 --> 00:35:43,720
So all of those engagement 
metrics are artificially 

715
00:35:43,720 --> 00:35:46,200
inflated by some of the design 
changes they've made recently. 

716
00:35:46,680 --> 00:35:48,600
And so we're actually even 
talking about them less and 

717
00:35:48,600 --> 00:35:50,080
less. 
But what we have been doing that

718
00:35:50,080 --> 00:35:54,320
I think is in the area of your 
question, people will reply to 

719
00:35:54,320 --> 00:35:57,320
us if we send text messages, if 
they received the intervention 

720
00:35:57,320 --> 00:36:00,680
through a text message, a 
surprising amount of people were

721
00:36:00,680 --> 00:36:03,200
fly back and this is not with a 
system command, it's with 

722
00:36:03,200 --> 00:36:08,000
substantive information. 
And so we have actually we have 

723
00:36:08,000 --> 00:36:11,280
in market now, but it's going to
be it's an MVP that's going to 

724
00:36:11,320 --> 00:36:15,920
become I think really powerful. 
We are basically analyzing with 

725
00:36:15,920 --> 00:36:17,880
some natural language 
understanding with an intent 

726
00:36:17,880 --> 00:36:19,880
model, what people are replying 
to us. 

727
00:36:19,880 --> 00:36:22,840
And we are using that to 
sometimes negatively reward the 

728
00:36:22,840 --> 00:36:26,240
agent because sometimes people 
are responding very negatively. 

729
00:36:26,240 --> 00:36:29,960
You know, they're they are upset
that they're being contacted or 

730
00:36:30,040 --> 00:36:32,480
this is at the campaign I'm 
thinking of as a vaccination 

731
00:36:32,480 --> 00:36:34,720
intervention. 
And vaccines are always a little

732
00:36:34,720 --> 00:36:36,440
bit of a hot button for some 
people. 

733
00:36:36,440 --> 00:36:38,600
And so there's some people who 
are replying like with 

734
00:36:38,600 --> 00:36:42,600
conspiracy theories and 
political commentary and we're 

735
00:36:42,600 --> 00:36:45,720
basically negatively rewarding 
the agent in some of those cases

736
00:36:45,720 --> 00:36:48,800
to a small amount. 
And we're also taking actions in

737
00:36:48,800 --> 00:36:51,760
those cases as well. 
So we we can send replies that 

738
00:36:51,760 --> 00:36:54,040
are appropriate to what the 
person said if like they ask a 

739
00:36:54,040 --> 00:36:56,160
question. 
In the case of somebody who's 

740
00:36:56,160 --> 00:36:59,680
really angry, what we do is we 
suppress the messaging because 

741
00:37:00,240 --> 00:37:02,200
they don't wanna hear from us. 
You're not gonna get through to 

742
00:37:02,200 --> 00:37:03,840
that person. 
No, no. 

743
00:37:03,840 --> 00:37:06,320
And it's like, why are we 
spending the 10th of a set to 

744
00:37:06,720 --> 00:37:09,120
piss this person off even more? 
Let's not do that. 

745
00:37:09,560 --> 00:37:12,240
Why won't you get the vaccine? 
I really want to implant this. 

746
00:37:15,480 --> 00:37:17,240
Yeah. 
So I'm really excited about that

747
00:37:17,240 --> 00:37:20,240
because I think that our 
bidirectional capabilities are 

748
00:37:20,240 --> 00:37:22,560
going to become a really 
critical part of what we do. 

749
00:37:22,560 --> 00:37:24,920
And that becomes a way for us to
also gather information about a 

750
00:37:24,920 --> 00:37:28,080
person's experience and more 
quickly identify what barriers 

751
00:37:28,080 --> 00:37:30,800
they're actually experiencing 
and better train those agents. 

752
00:37:30,800 --> 00:37:32,840
Like what is the right type of 
messaging for this person for 

753
00:37:32,840 --> 00:37:35,800
this behavior? 
Can we talk a little bit about 

754
00:37:35,880 --> 00:37:37,880
barriers? 
So you mentioned you do a lit 

755
00:37:37,880 --> 00:37:41,360
review and I know you have some 
a lot of theory that's really 

756
00:37:41,560 --> 00:37:44,440
oozing through your models. 
We talked a lot about Com B. 

757
00:37:44,560 --> 00:37:49,080
How does that work with the sort
of top down theory driven 

758
00:37:49,600 --> 00:37:53,320
systematization versus the 
bottom up like talking to health

759
00:37:53,320 --> 00:37:56,000
systems and clinicians and 
trying to understand what did 

760
00:37:56,000 --> 00:37:59,000
patients actually say? 
How did those two come together?

761
00:37:59,640 --> 00:38:01,600
Yeah, that's a challenge. 
It's not a challenge. 

762
00:38:01,600 --> 00:38:04,040
It's, it's a fun part of what we
have to pay attention to. 

763
00:38:04,600 --> 00:38:06,760
A lot of it is just early 
documentation. 

764
00:38:07,040 --> 00:38:09,760
You know, we, we put all the lit
review things and I, I mentioned

765
00:38:09,760 --> 00:38:11,320
we have a document called 
Strategy Matrix. 

766
00:38:11,320 --> 00:38:14,720
We use Excel to build it. 
It's very glamorous fancy. 

767
00:38:14,840 --> 00:38:18,640
But the initial version of that 
is fairly large because of 

768
00:38:18,640 --> 00:38:21,080
course all of the lit review 
papers are using different 

769
00:38:21,080 --> 00:38:23,760
terminology or maybe you're 
slightly, you know, defining a 

770
00:38:23,760 --> 00:38:26,400
barrier differently. 
So we do some organization and 

771
00:38:26,400 --> 00:38:28,240
right sizing of what we find in 
the literature. 

772
00:38:28,720 --> 00:38:31,880
We're recording what we found 
from the client research and 

773
00:38:31,880 --> 00:38:34,480
we're trying to basically bucket
it together with what we see in 

774
00:38:34,480 --> 00:38:36,120
the literature. 
And what we typically wind up 

775
00:38:36,120 --> 00:38:39,560
with is the clients experiencing
things that were already known 

776
00:38:39,560 --> 00:38:42,520
to research for the most part, 
but there's usually a handful of

777
00:38:42,520 --> 00:38:43,960
things that are really specific 
to them. 

778
00:38:44,600 --> 00:38:47,520
What we tend to do is we include
them in our design process. 

779
00:38:47,520 --> 00:38:50,440
And then when we develop our 
content, we always, even if 

780
00:38:50,440 --> 00:38:52,320
somebody comes to us and they 
want to use an existing 

781
00:38:52,320 --> 00:38:54,880
intervention, we go through a 
tailoring process and then what 

782
00:38:54,880 --> 00:38:57,680
we call content activation to 
make it live in their 

783
00:38:57,680 --> 00:39:00,040
environment. 
And that's an opportunity. 

784
00:39:00,240 --> 00:39:01,920
You know, we have system 
variables, we have the 

785
00:39:01,920 --> 00:39:05,520
opportunity to tweak and change 
some of the messaging and we can

786
00:39:05,560 --> 00:39:07,760
infuse some of those client 
specific things in at that 

787
00:39:07,760 --> 00:39:09,280
point. 
And what we typically try to do 

788
00:39:09,280 --> 00:39:12,360
is just make sure that we do it 
in a way that is consistent with

789
00:39:12,360 --> 00:39:15,800
the logic model we've built. 
So what that might look like is 

790
00:39:15,800 --> 00:39:18,400
if we already know that 
convenience is a barrier for 

791
00:39:18,400 --> 00:39:21,280
somebody, we've already 
identified, you know, the BCT in

792
00:39:21,280 --> 00:39:23,840
the messaging, but we're working
with a retail pharmacy that 

793
00:39:24,120 --> 00:39:26,560
really it's important to them 
that the patients know that 

794
00:39:26,560 --> 00:39:27,840
they're open more than 9:00 to 
5:00. 

795
00:39:28,560 --> 00:39:31,120
We can build that into the 
existing message that already 

796
00:39:31,120 --> 00:39:33,440
uses that sort of approach, but 
now it becomes very customer 

797
00:39:33,440 --> 00:39:35,360
specific. 
And if it's something that's 

798
00:39:35,360 --> 00:39:38,000
really unique to that client, we
can just build that out as their

799
00:39:38,000 --> 00:39:41,600
own, you know, bundle of content
that is activated for them, but 

800
00:39:41,600 --> 00:39:43,640
doesn't become part of our core 
library. 

801
00:39:44,280 --> 00:39:47,120
And our CMS, I won't get into 
it, but our CMS allows us to 

802
00:39:47,120 --> 00:39:49,960
sort of this is the master 
content library, the overall 

803
00:39:49,960 --> 00:39:52,240
intervention, but then here's 
the iterations of it for various

804
00:39:52,240 --> 00:39:56,440
clients. 
When you're going into a new 

805
00:39:56,440 --> 00:39:59,280
intervention, are you starting 
at 0? 

806
00:39:59,280 --> 00:40:01,880
Say we know nothing about this 
patient. 

807
00:40:01,880 --> 00:40:05,760
We are going to send every 
patient, you know, some randomly

808
00:40:05,760 --> 00:40:08,680
selected message from the 
library, and then we'll learn to

809
00:40:08,680 --> 00:40:11,360
get to, you know, their optimal 
message. 

810
00:40:11,640 --> 00:40:15,520
Or can you already start from 
somewhere and say, well, I know 

811
00:40:15,520 --> 00:40:17,640
that this patient lives, you 
know? 

812
00:40:18,000 --> 00:40:22,840
20 miles from the clinic. 
So that's likely to be a barrier

813
00:40:22,840 --> 00:40:24,720
to actually attending the 
appointment. 

814
00:40:25,160 --> 00:40:29,000
Which of those? 
Both, both and sometimes the 

815
00:40:29,040 --> 00:40:31,840
other places as well. 
So we have some clients that we 

816
00:40:31,840 --> 00:40:35,200
work with and it's a net new use
case, new intervention and for 

817
00:40:35,200 --> 00:40:38,320
whatever reason they are unable 
or unwilling to share very much 

818
00:40:38,320 --> 00:40:40,160
data with us. 
And in that case, it is 

819
00:40:40,160 --> 00:40:43,640
essentially, let's send 
something randomly and start to 

820
00:40:43,640 --> 00:40:46,080
learn from scratch about these 
individuals. 

821
00:40:46,720 --> 00:40:50,480
And I know that Chris, my 
counterpart on the AI side is 

822
00:40:50,480 --> 00:40:52,000
probably would probably jump in 
and correct me. 

823
00:40:52,000 --> 00:40:53,520
It'll be like, it's not really 
from scratch yet. 

824
00:40:53,520 --> 00:40:59,040
But for my, let's just call it 
from scratch sometimes if it's a

825
00:40:59,040 --> 00:41:01,400
new intervention and it has 
similarities to other 

826
00:41:01,400 --> 00:41:03,800
interventions, we're able to 
leverage the training of the 

827
00:41:03,800 --> 00:41:05,280
Asians for those other 
interventions. 

828
00:41:05,280 --> 00:41:07,520
So a good example here again is 
vaccination. 

829
00:41:07,880 --> 00:41:10,040
We have nudged all different 
types of vaccines. 

830
00:41:10,040 --> 00:41:15,880
We've done COVID, flu, RSV, 
pneumo, shingles, and there are.

831
00:41:15,920 --> 00:41:18,520
We've done the research to know 
that the barriers for receiving 

832
00:41:18,520 --> 00:41:21,160
those vaccines are not 
completely overlapping, but 

833
00:41:21,160 --> 00:41:23,880
they're a lot overlapping. 
Covic is the one that's kind of 

834
00:41:23,880 --> 00:41:26,440
different, but the others are a 
lot overlapping. 

835
00:41:26,960 --> 00:41:30,360
And so we have been able to 
basically use agents trained for

836
00:41:30,360 --> 00:41:32,920
other vaccinations to get so 
that kind of jump start. 

837
00:41:33,200 --> 00:41:35,240
And then you specifically 
mentioned home address. 

838
00:41:35,240 --> 00:41:37,280
And I know you didn't do this on
purpose, but that's a piece of 

839
00:41:37,280 --> 00:41:41,400
information that's particularly 
useful to us because on my team,

840
00:41:41,880 --> 00:41:44,680
Sarah D dot, my VP of Behavioral
Design, has her PhD in health 

841
00:41:44,680 --> 00:41:47,040
geography. 
So she intimately understands 

842
00:41:47,800 --> 00:41:49,720
space and place from a 
behavioral perspective. 

843
00:41:49,720 --> 00:41:53,920
And then on the AI side, we have
a team member, Rajiv Vatsavai, 

844
00:41:54,040 --> 00:41:57,440
who is also faculty at NC State,
and his expertise is geospatial 

845
00:41:57,440 --> 00:41:59,840
transformation. 
So between the two of them, we 

846
00:41:59,840 --> 00:42:02,160
actually can get a ton of 
mileage out of somebody's 

847
00:42:02,160 --> 00:42:04,120
address and specifically 
identifying things like 

848
00:42:04,120 --> 00:42:06,120
proximity to where the 
healthcare behavior is 

849
00:42:06,120 --> 00:42:09,360
delivered, but also things like,
you know, are they near green 

850
00:42:09,360 --> 00:42:12,560
space? 
Are there food deserts depending

851
00:42:12,560 --> 00:42:14,040
on what's relevant for that 
intervention. 

852
00:42:14,040 --> 00:42:17,480
So we can sometimes do 
transformations of variables 

853
00:42:17,480 --> 00:42:21,160
that we receive, or that we have
access to derive much more 

854
00:42:21,160 --> 00:42:22,600
meaningful information from 
them. 

855
00:42:23,880 --> 00:42:25,960
Wow, awesome. 
Those are definitely specialties

856
00:42:25,960 --> 00:42:28,280
that I did not. 
I'm very glad that they exist, 

857
00:42:28,280 --> 00:42:31,880
but I had no idea. 
I know it's kind of wild too, 

858
00:42:31,880 --> 00:42:36,320
that we didn't deliberately set 
out to assemble this super team 

859
00:42:36,320 --> 00:42:37,880
of geography people. 
But. 

860
00:42:38,320 --> 00:42:39,400
We did it. 
Really cool. 

861
00:42:39,400 --> 00:42:42,480
Yeah, yeah. 
That's cool. 

862
00:42:42,680 --> 00:42:47,520
And I guess I wanted to in some 
ways turn the tables a little 

863
00:42:47,520 --> 00:42:50,040
bit of what we've explored here 
in terms of we've talked about 

864
00:42:50,040 --> 00:42:53,880
kind of from a behavioral 
science point of view, what we 

865
00:42:53,880 --> 00:42:56,520
can do with AI and the benefit 
that we can have in our 

866
00:42:56,520 --> 00:43:00,360
interventions and so on. 
But maybe as a hypothetical, 

867
00:43:00,760 --> 00:43:03,080
because I think a lot of 
behavioral scientists is kind of

868
00:43:03,080 --> 00:43:07,640
exploring like, what is my value
in this new evolving landscape? 

869
00:43:08,280 --> 00:43:11,560
And you know, if you then look 
at the kind of Euro illyria and 

870
00:43:11,560 --> 00:43:14,560
the thing you described, you 
know, what would be the cost of 

871
00:43:14,560 --> 00:43:16,440
removing behavioral science from
that equation? 

872
00:43:16,440 --> 00:43:19,720
Like how do you see you come in 
as a behavioral scientist and 

873
00:43:19,720 --> 00:43:23,760
adding value and what would be 
there if you were not there? 

874
00:43:25,920 --> 00:43:28,960
Obviously I have a bias. 
I don't think I would work 

875
00:43:29,120 --> 00:43:30,920
without us. 
No, I really do. 

876
00:43:30,920 --> 00:43:34,840
Actually, I, I think it would be
pretty bad and I'll give you a 

877
00:43:34,840 --> 00:43:37,840
couple of reasons why. 
So the first is we have actually

878
00:43:37,840 --> 00:43:41,320
done tests where we do our 
behavioral science messaging 

879
00:43:41,320 --> 00:43:43,880
without the AI. 
So, and there's been, you know, 

880
00:43:43,880 --> 00:43:46,200
various reasons that we've done 
that, whether it's deliberate 

881
00:43:46,200 --> 00:43:48,760
or, you know, we were tweaking 
something. 

882
00:43:48,760 --> 00:43:53,280
And but the behavioral science 
messaging by itself has a huge 

883
00:43:53,280 --> 00:43:57,240
impact compared to control. 
So, you know, I, I think that 

884
00:43:57,240 --> 00:44:00,080
having that behavioral lens on 
the content that we send out to 

885
00:44:00,080 --> 00:44:02,480
people, and I actually believe 
that the logic model behind it 

886
00:44:02,480 --> 00:44:05,240
is also really important. 
I've seen other studies, other 

887
00:44:05,240 --> 00:44:08,400
interventions where people 
create intervention outreach 

888
00:44:08,400 --> 00:44:11,240
that's very strong, you know it,
and it's what I might call a 

889
00:44:11,240 --> 00:44:12,840
kitchen sink behavioral science 
approach. 

890
00:44:12,840 --> 00:44:16,320
Like how much can we put into 
this messaging to, you know, 

891
00:44:16,320 --> 00:44:19,320
just convey all the different 
tools and reasons why somebody 

892
00:44:19,360 --> 00:44:21,680
might be able to do this thing 
for us? 

893
00:44:21,680 --> 00:44:24,560
Any given message really only 
has one BCT in it, one behavior 

894
00:44:24,560 --> 00:44:27,320
change technique. 
But I think the fact that we get

895
00:44:27,320 --> 00:44:30,000
very specific and spend a lot of
time, you know, just really 

896
00:44:30,000 --> 00:44:31,480
making sure that comes through 
extreme. 

897
00:44:31,480 --> 00:44:34,960
We go through a quality check 
process where we have blind 

898
00:44:34,960 --> 00:44:38,080
raters review all the items and 
determine, does this adequately 

899
00:44:38,080 --> 00:44:40,400
operationalize the BCT and does 
it do it in a way that's 

900
00:44:40,400 --> 00:44:43,480
distinguished from other BC TS? 
I think all of that means that 

901
00:44:43,480 --> 00:44:45,440
the content by itself is just a 
huge value add. 

902
00:44:45,480 --> 00:44:48,840
But then the other pieces, as I 
mentioned, there is no universal

903
00:44:48,840 --> 00:44:50,840
learner. 
That's a thing that Chris says 

904
00:44:50,840 --> 00:44:54,000
all the time. 
You really have to train your AI

905
00:44:54,440 --> 00:44:56,080
against subject matter 
expertise. 

906
00:44:56,080 --> 00:44:59,280
And in our case, we are doing 
that against, you know, behavior

907
00:44:59,280 --> 00:45:01,040
science and healthcare 
expertise. 

908
00:45:01,520 --> 00:45:04,600
We use an example in our sales 
presentations that I think is, 

909
00:45:04,720 --> 00:45:07,360
you know, it's a little bit 
intended to be provocative, but 

910
00:45:07,360 --> 00:45:10,280
we actually have seen healthcare
organizations who hire former 

911
00:45:10,280 --> 00:45:13,000
executives from companies like 
Amazon or Disney, you know, 

912
00:45:13,000 --> 00:45:15,520
these companies that are amazing
at engaging people and they 

913
00:45:15,520 --> 00:45:17,880
bring them in. 
And I, I so far have not seen a 

914
00:45:17,880 --> 00:45:20,320
major success story in improving
patient engagement in a 

915
00:45:20,320 --> 00:45:22,520
healthcare setting with that 
expertise. 

916
00:45:22,520 --> 00:45:25,200
And I, I think it speaks to the 
fact that you really do have to 

917
00:45:25,280 --> 00:45:29,480
infuse your approach with a deep
understanding of well, how human

918
00:45:29,480 --> 00:45:32,480
behavior works, but also how 
does human healthcare behavior 

919
00:45:32,480 --> 00:45:37,360
work in a particular system that
provides barriers, limitations, 

920
00:45:37,640 --> 00:45:40,840
enables some behaviors. 
So I, I, I don't think that the 

921
00:45:40,840 --> 00:45:45,120
product would work without 
infusing that type of expertise.

922
00:45:45,120 --> 00:45:47,520
We, we might be able to get it 
to do something else. 

923
00:45:47,520 --> 00:45:50,640
It we might be able to get it to
do something that's still like 

924
00:45:50,640 --> 00:45:52,800
better than random, but I just 
don't think that. 

925
00:45:53,400 --> 00:45:56,000
I think the delta between what 
we can do today and what we 

926
00:45:56,000 --> 00:45:58,120
would do without VISI is huge 
and negative. 

927
00:46:00,680 --> 00:46:03,960
And if you think about it, 
that's where most companies are,

928
00:46:04,160 --> 00:46:06,840
right? 
Lacking the behavioral science. 

929
00:46:07,840 --> 00:46:11,400
Well, or or the AI, yeah, I 
mean, it's funny and and again, 

930
00:46:11,400 --> 00:46:14,200
like we're we're a little bit 
jerks about this. 

931
00:46:14,200 --> 00:46:16,760
But you know, Chris and I have 
been to conferences together. 

932
00:46:16,760 --> 00:46:19,120
We've locked the exhibit floor 
and we'll see these companies 

933
00:46:19,120 --> 00:46:21,360
that are like AI decide and 
between the two of us, we're 

934
00:46:21,360 --> 00:46:24,880
like, come on. 
Yeah, that's not real AI or 

935
00:46:24,880 --> 00:46:26,080
that's not real D side. 
I know. 

936
00:46:26,120 --> 00:46:27,920
Exactly which ones you're 
talking about. 

937
00:46:28,720 --> 00:46:30,640
Yes, yes. 
And that was, again, it gets 

938
00:46:30,640 --> 00:46:33,080
back to why we wrote that 
scoping review and sort of the, 

939
00:46:33,080 --> 00:46:38,400
you know, the dual meaning of BS
there, because we wanted to 

940
00:46:38,840 --> 00:46:41,600
really know for sure what is out
there in terms of AI. 

941
00:46:42,160 --> 00:46:44,440
Obviously everybody's gonna 
market in the way that they 

942
00:46:44,440 --> 00:46:46,480
think is gonna be most appealing
to their customers and things 

943
00:46:46,480 --> 00:46:48,200
like that. 
But some of them are really 

944
00:46:48,200 --> 00:46:49,440
doing AI. 
We wanted to know who they. 

945
00:46:49,440 --> 00:46:55,480
Were all right. 
It is time for our quick fire 

946
00:46:55,480 --> 00:46:57,480
round. 
Are you ready? 

947
00:46:59,080 --> 00:47:00,400
I'll tell you what it is. 
Don't worry. 

948
00:47:00,920 --> 00:47:03,040
So this is a game that we've 
developed. 

949
00:47:03,040 --> 00:47:08,360
It's called to AI or Not to AI. 
And so it's your job to 

950
00:47:08,360 --> 00:47:12,240
determine whether the activity 
that we present to you is well 

951
00:47:12,240 --> 00:47:15,880
suited to AI or not. 
All right, let's begin. 

952
00:47:17,000 --> 00:47:21,960
So first one, negotiate a job 
offer to AI or not to AI? 

953
00:47:23,320 --> 00:47:26,920
I think you can use AI in this. 
I wouldn't directly interface 

954
00:47:26,920 --> 00:47:30,280
the hiring folks with the AI, 
but I, I could use it to like 

955
00:47:30,280 --> 00:47:32,320
revise, you know? 
Hey, this is what I'm thinking 

956
00:47:32,320 --> 00:47:34,040
of replying to this e-mail. 
Can you revise it? 

957
00:47:34,040 --> 00:47:36,720
Can you make it more succinct, 
more professional or getting 

958
00:47:36,720 --> 00:47:38,600
stats? 
Can you tell me what a good 

959
00:47:38,600 --> 00:47:40,880
salary might be for this role 
based in the city? 

960
00:47:41,080 --> 00:47:44,920
So I say AI has a role in this. 
So really as a as a partner, not

961
00:47:44,920 --> 00:47:46,920
a replacement. 
Correct. 

962
00:47:47,600 --> 00:47:50,800
How about file taxes? 
That makes me nervous. 

963
00:47:50,800 --> 00:47:53,040
I think that the risk of being 
wrong is too high. 

964
00:47:53,040 --> 00:47:57,000
I say no AI. 
OK recommend a mystery book. 

965
00:47:57,240 --> 00:48:00,840
Absolutely yes. 
Write a mystery book. 

966
00:48:01,240 --> 00:48:03,560
No. 
It we've done a lot of work 

967
00:48:03,560 --> 00:48:06,640
looking at the quality of 
content and I think the short 

968
00:48:06,640 --> 00:48:08,920
form content can be pretty good,
but anytime we ask it to do 

969
00:48:08,920 --> 00:48:12,360
something long form or repeated,
it's boring. 

970
00:48:13,480 --> 00:48:15,800
The Sign. 
Behavioral interventions for a 

971
00:48:15,800 --> 00:48:19,640
mental health app. 
No, I will not design A 

972
00:48:19,640 --> 00:48:21,600
behavioral intervention for a 
mental health app without a 

973
00:48:21,600 --> 00:48:26,600
clinical partner. 
About Taylor Cat memes to one 

974
00:48:26,600 --> 00:48:32,200
specific preferences 100% how 
personalized will it get? 

975
00:48:32,920 --> 00:48:36,120
Well, so I have a cat named 
Jackie Daytona after What We Do 

976
00:48:36,120 --> 00:48:37,960
in the Shadows. 
So Jackie Daytona, human 

977
00:48:37,960 --> 00:48:39,480
bartender. 
That's so funny, by the way. 

978
00:48:39,920 --> 00:48:41,840
I love that. 
Thank you. 

979
00:48:42,640 --> 00:48:45,080
But I've tried really hard to 
get like Dolly mid journey to 

980
00:48:45,080 --> 00:48:47,640
give me Jackie Daytona, human 
bartender, but he's really a 

981
00:48:47,640 --> 00:48:50,400
brown tabby cat type things and 
they're very they're not where I

982
00:48:50,400 --> 00:48:52,240
want them to be. 
We're going to keep working on 

983
00:48:52,240 --> 00:48:52,960
it all. 
Right. 

984
00:48:54,040 --> 00:48:55,560
Maybe some promise in the 
future. 

985
00:48:56,160 --> 00:48:58,280
Yeah, I'm now just thinking 
about what we're doing in the 

986
00:48:58,280 --> 00:49:01,240
shadows and I completely lost 
track of of thinking about this 

987
00:49:01,400 --> 00:49:06,240
quick fire around, but OK, next 
one device and investment plan. 

988
00:49:07,240 --> 00:49:09,400
Yeah, I think it could. 
But I this is again where I 

989
00:49:09,400 --> 00:49:12,160
would want to review it and then
go execute it. 

990
00:49:12,160 --> 00:49:14,760
I wouldn't want AI to devise the
investment plan and then execute

991
00:49:14,760 --> 00:49:17,640
it. 
And last one, identify patients 

992
00:49:17,640 --> 00:49:22,200
at risk for diabetes. 
Well, yes, we we actually do a 

993
00:49:22,200 --> 00:49:24,240
little bit of that. 
So yeah, I think that's a great 

994
00:49:24,240 --> 00:49:25,080
use case. 
Awesome. 

995
00:49:25,680 --> 00:49:29,800
All right, it's time to ask you 
the question that we end with 

996
00:49:29,800 --> 00:49:33,480
all of our guests this season. 
Amy, what is your most 

997
00:49:33,520 --> 00:49:39,760
controversial opinion about AI? 
I think AI is not that good yet.

998
00:49:39,840 --> 00:49:44,080
And I also think that we are 
unnecessarily afraid of it in 

999
00:49:44,080 --> 00:49:47,200
the wrong places. 
So I see a lot of people who are

1000
00:49:47,200 --> 00:49:49,880
like, Oh my gosh, AI is going to
replace us. 

1001
00:49:49,960 --> 00:49:52,640
And I just don't think it's good
enough to do that yet. 

1002
00:49:52,640 --> 00:49:56,680
And like I say, I'm on LinkedIn 
a lot, mostly because I'm told 

1003
00:49:56,680 --> 00:49:57,840
that's a thing I should be 
doing. 

1004
00:50:00,080 --> 00:50:02,760
Obviously, notice a lot of 
people write their posts with 

1005
00:50:02,840 --> 00:50:05,360
ChatGPT. 
And so first of all, I can tell,

1006
00:50:05,640 --> 00:50:08,080
you can tell if somebody 
actually wrote their post this 

1007
00:50:08,080 --> 00:50:11,560
way, because like I said, 
ChatGPT might look great in 

1008
00:50:11,560 --> 00:50:15,080
isolation, but it has its own 
tics and mannerisms that show up

1009
00:50:15,680 --> 00:50:17,520
in it's writing across different
pieces. 

1010
00:50:17,520 --> 00:50:19,880
And like once you know it's 
voice, it's kind of easy to see 

1011
00:50:19,880 --> 00:50:22,120
it. 
And so I think that people 

1012
00:50:22,120 --> 00:50:25,200
overlook the fact that, you 
know, I can tell the ChatGPT 

1013
00:50:25,200 --> 00:50:28,200
wrote that it doesn't mean it's 
bad, but it's no replacement for

1014
00:50:28,200 --> 00:50:30,680
your own voice. 
And then even things like 

1015
00:50:30,680 --> 00:50:33,400
LinkedIn fired a lot of their 
content writers and now they do 

1016
00:50:33,400 --> 00:50:35,840
that thing where they ask people
to contribute to articles. 

1017
00:50:35,840 --> 00:50:38,040
I'm assuming there's some kind 
of AI behind that in terms of 

1018
00:50:38,040 --> 00:50:41,120
generating like which prompts 
which people see and how they 

1019
00:50:41,120 --> 00:50:43,400
frame those to you. 
And I'll tell you, I don't 

1020
00:50:43,400 --> 00:50:46,480
actually don't read those social
articles the way that I 

1021
00:50:46,480 --> 00:50:48,800
sometimes used to read the ones 
that LinkedIn staff writers 

1022
00:50:48,800 --> 00:50:52,240
wrote because there's just like 
a a difference in kind of 

1023
00:50:52,240 --> 00:50:55,480
quality and organization that to
me as a reader leaves me 

1024
00:50:55,480 --> 00:50:58,880
unsatisfied. 
So I just don't think that AI 

1025
00:50:58,880 --> 00:51:01,040
has nearly reached its 
capabilities. 

1026
00:51:01,040 --> 00:51:04,640
And I think that the human in 
the loop is likely to be 

1027
00:51:04,640 --> 00:51:09,360
important forever, forever. 
Maybe not in every case, maybe 

1028
00:51:09,360 --> 00:51:11,040
not in every case. 
I mean, we certainly have seen 

1029
00:51:11,040 --> 00:51:13,880
things where AI has been able 
to, you know, manage supply 

1030
00:51:13,880 --> 00:51:16,040
chains and computational sorts 
of things. 

1031
00:51:16,760 --> 00:51:19,920
But I just don't think AI is 
going to replace humans in 

1032
00:51:20,400 --> 00:51:22,040
nearly the way that some people 
think it will. 

1033
00:51:24,080 --> 00:51:27,000
Yeah, I guess time to the first 
thing is that I'm talking about 

1034
00:51:27,000 --> 00:51:28,120
memes as well. 
We're talking about cat memes 

1035
00:51:28,120 --> 00:51:30,680
before, but a non cat meme that 
I've seen quite a bit with 

1036
00:51:31,040 --> 00:51:35,120
imagery is basically first 
there's this person who says 

1037
00:51:35,120 --> 00:51:39,120
like I haven't seen a lot AI 
generated images lately. 

1038
00:51:39,600 --> 00:51:42,520
And then you see them like with 
like a Oh my God face. 

1039
00:51:42,520 --> 00:51:46,160
Like I haven't seen an AI 
generated images lately. 

1040
00:51:46,880 --> 00:51:49,880
Yeah, Yeah, it is. 
It is getting better. 

1041
00:51:49,880 --> 00:51:51,160
A. 
Little bit to your point, like 

1042
00:51:51,160 --> 00:51:54,440
maybe there's some subset where 
it's like really well done where

1043
00:51:54,440 --> 00:51:56,320
you don't notice it, but a lot 
of it you notice. 

1044
00:51:56,320 --> 00:51:59,720
And especially, I know also you 
get recruitment stuff and I I 

1045
00:51:59,720 --> 00:52:02,640
get people applying for things. 
And I understand why people use 

1046
00:52:02,640 --> 00:52:05,320
AI for writing applications and 
various things. 

1047
00:52:05,320 --> 00:52:08,600
But it's also like I can see AI 
was used here. 

1048
00:52:08,640 --> 00:52:11,120
Like, yeah. 
So we just hired several roles 

1049
00:52:11,120 --> 00:52:15,800
on my team and one of them was a
writing role, a copywriter role,

1050
00:52:16,120 --> 00:52:19,960
and that was the one where we 
saw a lot of ChatGPT generated 

1051
00:52:19,960 --> 00:52:21,920
application materials. 
We didn't see it for the 

1052
00:52:21,920 --> 00:52:24,120
behavioral science roles for or 
we didn't notice it for the 

1053
00:52:24,120 --> 00:52:26,160
behavioral science roles. 
I'll say that we might have seen

1054
00:52:26,160 --> 00:52:29,760
it, but one of the ways that it 
was most evident was people 

1055
00:52:29,760 --> 00:52:31,600
would put their work histories. 
Obviously that's what you do on 

1056
00:52:31,600 --> 00:52:33,800
a resume and they would have 
things in there. 

1057
00:52:33,800 --> 00:52:37,000
That was like 2012 to 2013 
worked on precision nudging 

1058
00:52:37,000 --> 00:52:39,400
messaging. 
And it's like, I don't think you

1059
00:52:39,400 --> 00:52:43,040
did. 
And I also don't think a human 

1060
00:52:43,040 --> 00:52:46,760
being would make that mistake, 
you know, Weird. 

1061
00:52:47,800 --> 00:52:49,320
Yeah. 
And And the other thing, again, 

1062
00:52:49,320 --> 00:52:52,520
because we're reviewing lots of 
resumes, you start to see 

1063
00:52:52,520 --> 00:52:54,440
patterns across them. 
Yeah. 

1064
00:52:54,880 --> 00:52:57,240
I even think this was not AI, 
but when I was a grad student 

1065
00:52:57,240 --> 00:52:59,760
and I was teaching, you know, 
we'd always get people who 

1066
00:52:59,760 --> 00:53:02,640
plagiarized on their papers. 
And almost always the way that I

1067
00:53:02,640 --> 00:53:06,080
caught them was they weren't the
only one plagiarizing from the 

1068
00:53:06,080 --> 00:53:08,520
same source. 
Like I vividly remember one 

1069
00:53:08,520 --> 00:53:10,920
night reading a paper view, like
did I just read this paper or 

1070
00:53:10,920 --> 00:53:13,120
looked and there were actually 3
copies of essentially the same 

1071
00:53:13,120 --> 00:53:16,040
paper in my pile. 
So I think that's a lot of where

1072
00:53:16,040 --> 00:53:18,920
AI gets caught is when multiple 
people use it for similar 

1073
00:53:18,920 --> 00:53:21,320
purposes and it isn't different 
enough. 

1074
00:53:21,640 --> 00:53:24,000
Yeah, it feels different to 
them, but then to the one who's 

1075
00:53:24,000 --> 00:53:27,640
receiving all of the AI 
generated text, Yeah. 

1076
00:53:29,640 --> 00:53:33,280
Yeah, I did use AII, gave it 
like my LinkedIn bio, which I've

1077
00:53:33,280 --> 00:53:35,240
never liked and had it edited 
for me. 

1078
00:53:35,240 --> 00:53:39,560
And I, I what's on there now was
edited for me by ChatGPT, but I 

1079
00:53:39,560 --> 00:53:41,160
wrote the source. 
Yeah. 

1080
00:53:41,200 --> 00:53:44,560
Well, I think like that's in the
end, I think it will often goes 

1081
00:53:44,560 --> 00:53:46,520
back to like the importance of 
the understanding of the 

1082
00:53:46,520 --> 00:53:48,800
context. 
And I think that's really is 

1083
00:53:48,800 --> 00:53:52,160
true for AI as as it's been for 
us as payroll scientists before.

1084
00:53:52,640 --> 00:53:58,440
And yeah, I guess to wrap up, 
it's such a gift to have you on 

1085
00:53:58,440 --> 00:54:02,320
the podcast and maybe it like as
a little bit of a thank you as 

1086
00:54:02,320 --> 00:54:04,560
always. 
I, I was going to message you 

1087
00:54:04,560 --> 00:54:08,880
this, but I had kind of randomly
this experience going to an 

1088
00:54:08,880 --> 00:54:11,520
office for a client. 
I was going to start working 

1089
00:54:11,520 --> 00:54:14,840
with a kind of digital health 
company and the product manager,

1090
00:54:15,200 --> 00:54:17,040
he welcomed me. 
And then in the, in his office, 

1091
00:54:17,040 --> 00:54:19,440
he's like, I've, I've read this 
payable science book. 

1092
00:54:19,640 --> 00:54:22,760
Do you know about it? 
And he showed me your book and I

1093
00:54:22,760 --> 00:54:26,080
was like, oh, thank God he's 
showing me that book because 

1094
00:54:26,080 --> 00:54:28,000
there's so many books that you 
could have shown me and. 

1095
00:54:28,040 --> 00:54:29,640
I'll be like rolling my. 
You would have just been like, 

1096
00:54:29,960 --> 00:54:32,120
oh. 
Yeah, but I was like, OK, we're 

1097
00:54:32,120 --> 00:54:33,200
going to be in good place. 
Let's get. 

1098
00:54:33,200 --> 00:54:35,840
Started. 
Yeah, I guess thank you for all 

1099
00:54:35,840 --> 00:54:38,480
the work you've been doing and 
recently now also sharing the 

1100
00:54:38,480 --> 00:54:41,080
work with AI and so on as well. 
And thanks for coming on today. 

1101
00:54:41,080 --> 00:54:44,040
It's really fun to shut. 
Yeah, and I, I have to thank 

1102
00:54:44,040 --> 00:54:45,680
both of you too. 
I can't tell you how many people

1103
00:54:45,680 --> 00:54:48,040
I refer to have it weekly. 
I just think it's one of the 

1104
00:54:48,240 --> 00:54:50,720
absolute best resources out 
there and especially for people 

1105
00:54:50,720 --> 00:54:53,720
who want to learn more, it's so 
accessible. 

1106
00:54:53,720 --> 00:54:56,440
And you know, one of the few 
things that I got where I am 

1107
00:54:56,480 --> 00:54:59,120
always clicking links. 
So thank you for the service 

1108
00:54:59,120 --> 00:55:01,960
that you both do and I'm honored
to be able to be a small part of

1109
00:55:01,960 --> 00:55:05,080
it. 
And that's a wrap. 

1110
00:55:05,400 --> 00:55:08,080
You've been listening to the 
Behavioral Design Podcast, 

1111
00:55:08,360 --> 00:55:10,920
brought to you by Habit Weekly 
and Nuanced Behavior. 

1112
00:55:11,280 --> 00:55:14,160
Sam and Elaine tell me this 
season is packed with incredible

1113
00:55:14,160 --> 00:55:18,320
insights about behavioral design
and AI, so be sure to subscribe 

1114
00:55:18,320 --> 00:55:20,040
and share the podcast with your 
friends. 

1115
00:55:20,280 --> 00:55:22,440
Though you might want to keep it
away from your enemies. 

1116
00:55:23,800 --> 00:55:26,440
In case you haven't noticed, I'm
an AI voice. 

1117
00:55:27,600 --> 00:55:30,400
Yep, pretty crazy. 
Quite the improvement since last

1118
00:55:30,400 --> 00:55:32,400
season's AI outro, don't you 
think? 

1119
00:55:33,760 --> 00:55:36,320
If you'd like to collaborate 
with us at Nuance Behavior, 

1120
00:55:36,480 --> 00:55:39,240
where we use behavioral design 
to craft digital products with 

1121
00:55:39,240 --> 00:55:43,560
Nuance, e-mail us at 
hello@nuancebehavior.com or book

1122
00:55:43,560 --> 00:55:46,800
a call directly on our website, 
nuancebehavior.com. 

1123
00:55:48,240 --> 00:55:51,640
A special thanks to the amazing 
Dave Pizarro for our show music,

1124
00:55:51,880 --> 00:55:54,760
and to Mei Chen Yap and April 
English for their help in 

1125
00:55:54,760 --> 00:55:56,680
producing and publishing this 
episode. 

1126
00:55:57,240 --> 00:56:00,000
Thanks again for tuning in. 
We'll be back soon with another 

1127
00:56:00,000 --> 00:56:03,200
exciting conversation where 
behavioral design and AI 

1128
00:56:03,200 --> 00:56:10,280
intersect. 
Happens to Mugatroid. 

1129
00:56:27,240 --> 00:57:04,120
Oh. 
It's been hard. 

1130
00:57:04,600 --> 00:57:06,320
The Dunning Kruger effect is 
real, y'all. 

1131
00:57:07,240 --> 00:57:08,000
I'm living it.