Hello and welcome to the Behavioral Design Podcast.
This season we're diving into the intersection of behavioral
science and AI. We want to make sense of the
state of AI, from understanding how humans interact with
intelligent systems to using AI to do behavioral design itself.
I'm Aline Holsworth, a health tech advisor specializing in AI
and product design. Over the past 15 years, I've
been crafting human centered products with behavioral science
at the core. At Apple, I LED Behavioral
Science for Health AI, designing and launching AI powered
features to help users reach their health goals.
And I'm Samuel Sultzer, your second Co host.
I'm a behavioral strategist specializing in hybrid formation
and designing products that drive long term baby change.
I work with leading tech organizations integrating AI to
scale behavioral design for good.
And I'm also the founder of Baby Bites, a dedicated community on
behavioral science and AI. Quick word on Nuance Behavior
where we help organizations build impactful digital products
using behavioral design. We only take on a few clients at
a time to ensure the highest level of quality for our
tailored evidence based solutions.
If you'd like to become one of our special projects, e-mail us
at hello@nuancebehavior.com or we could call directly on our
website, nuancebehavior.com. So Eileen, I'm interested to
know, have you ever considered buying AAI hardware or like
wearable device? Have I ever considered buying
one? I mean, AI is already in all of
our devices. So I feel like that's a trick
question. Maybe that's true.
I know what you're getting at, yes.
The clip that goes into your shirt pocket that listens to all
of your conversations, you're not talking about like, FaceTime
or like, you know, your sort of standard encryption, No.
That's right. Yeah.
Like the humane pin that you describe or the rabbit, R1 I
don't know if you saw that at all.
Orange red thing. Google Glasses.
No juice? Yeah, Yeah.
No, no, I have not ever considered purchasing one of
these. OK.
Well, I think they're interesting because they have
created quite a bit of hype and interest especially in the last
year or like 1 1/2 year or so. And the one you mentioned,
Humane Pin and also the one I referenced to Rabbit R1.
Both of them have followed this kind of like curve of being like
really, really hyped and people getting excited about especially
the R1. But then once I actually get to
use it, they've like used crashed and burned like it just
hasn't been received very well. Lots of.
Excitement from the early adopters and then a big, big
dip. Yeah, not even early adopters,
like people from pre-orders before they actually get this
thing. And then once they get the
thing, they realize like this is crap basically.
And so in terms of ingredients, I feel like one thing that they
have in common is that they have 100% hype and syrups and
behavioral science in terms of being used.
And I think that's a little bit of maybe the theme, of course,
in this season that we're going to kind of look at payable
science and AI and how they overlap or how we can see and
understand each from different perspective better.
Because I think that's one thing that's like a big loss is this
kind of idea that use because you can assign a very
intelligent system, like some form of trained language model,
doesn't really mean that you per extension know how people are
going to use it to interact. With right, right.
I mean, obviously I am biased as a behavioral scientist, but it
completely boggles my mind that you could even consider making a
piece of technology primarily for humans to interact with
without considering how humans will interact with it.
It just like, wait a minute, it's, it's, it's I don't know
how much of my bias is playing into that, but yeah, like, I
would love for someone to explain to me why behavioral
science is not absolutely essential to success.
This situation. When I worked at a large
consumer electronics company a few years ago, we were very
interested in using behavioral science and AI together and
really tried to intentionally look at how they could work
together to make each other better.
So AI helping behavioral science, but also behavioral
science helping AI and and really kind of thinking of the
two as a two way St. So if you think about how
behavioral science could contribute to AI, how an
understanding of human behavior can help both generate better
predictive models, maybe eliminate bias and so on.
But then on the other side of things, how AI can make
behavioral science better. So as a research method or
intervention technology, for example.
And and we talk a lot about that in other episodes.
But today, when we talk to Amy, we're really gonna focus on the
1st St. how behavioral science can contribute to this AI tool.
And of course, the ultimate goal it always, at least in our
opinion, is how do we change behavior for good using this
combination of AI and behavioral science.
And I feel like that really came through in our conversation with
Amy as well. Yeah.
And I'm interested because so you, for example, you led
behavioral science for healthy AI team at Apple and having have
this kind of exposure for you, is there in commonality?
Like do you see anything that this AI side of things and also
the behavioral science side, do you have anything in common?
Lots of things. Many, many things.
One sort of overarching theme that I noticed as I was working
in the the weeds of AI and behavioral science was just how
both of these fields are often packaged as these miracle cures.
And our behavioral science listeners will relate to this,
I'm sure. And it's so frustrating, right?
This idea that you can just throw in a bunch of nudges and
all your behavioral problems will be solved or even your non
behavioral problems, like everything will be fine.
Well, the the the exact same situation machine learning
engineers and scientists are faced with.
Well, there's this perception that you can just automate
everything with AI and you know, we don't need to work anymore.
And you know, of course, like the reality is that neither of
these things are magic. They both require a lot of work
and thoughts and intentionality. But yeah, we often did laugh
about this similarity and just how wrong the perception is and
how this hype is countered by the reality.
Yeah, I mean, it is almost hilarious where you kind of see
in the same way as maybe people naively at some point with like
thinking about throwing use notice at problems to solve
them. It's also a little bit like the
last couple of years you've seen this kind of like panicking of
like, OK, we have to make our products AI somehow.
Let's throw in some form of AI function and that's going to
make the user experience better somehow.
We're not really sure why or how, but we're just going to
throw AI everybody else. Is doing it so.
We must Exactly. No, but I think that's super
great tech and I think it's so true.
And yeah, we have a great guest in this episode that we explore
some of this and much more with. And yeah, so let me introduce
Amy Buker. She's the chief here, officer at
Lerio. And if her name sounds familiar,
it could be for many good reasons.
It could be that you have read your book Engaged, which is a
really fantastic book. Find one either getting starting
behavioral science or applied behavioral science or just wants
to get some insights for how it can be done in practice.
You might have seen her in some form of conference speaking or
you've listened to the podcast before, because we had her in a
fantastic conversation in a previous season and she's one of
the cases where actually there's so many people to speak to.
And so we tried to invite new people to the podcast.
But with Amy, we want to have her back not only because she's
a fantastic representative of the field and practitioner, but
also because since we last spoke to her, she has really been
thrown into this intersection of AI and behavioral science with
her work, Illyrio. And she's been focusing on
specifically working with and understanding how AI systems can
be used in tandem with behavioral science to
personalize health interventions.
So this includes looking at how to support patients going
through some form of chronic disease management, but many
different types of context and scenarios.
And her work specifically using precision nudging, I think it's
even references hyper precision looks like kind of how to make
these interventions both effective, but also scalable,
personalized and equitable. So super excited to bring her
back for another conversation. And yeah, what do we cover?
So as you might expect, we talk all about how Amy does just
this, applying AI to personalized health
interventions and specifically how she uses reinforcement
learning to nudge users toward healthier behaviors.
Amy shares her findings from a recent scoping review where she
looked at the landscape of AI being used in digital behavior
change interventions. And for me, I don't know about
you, Sam, but one take away was just that behavioral scientists
might need a little bit of help understanding what's AI and what
isn't. We cover why behavioral science
is so, so essential for getting it right in AI and especially
when it comes to health behavior, and so, so much more,
including Amy's most controversial opinion in AI and
what it would take for cat memes to be personalized to her
satisfaction. Happens to Murgatroyd.
I'm happy to say welcome back, Amy.
Good to have you on the podcast. It's great to be here.
I'm so excited to talk to you both again.
Yeah. And I guess the first question I
want to dive into is AI. Think still rather recently
shared scoping review that you made with a few of your
colleagues looking at something really relevant for the system.
Given that we're kind of like trying to understand this kind
of intersection between behavioral science and AI making
sense of, OK, what is AIBS in terms of like behavioral science
and what is AIBS in terms of bullshit and kind of seeing
between the lines of like which is which.
And I think this scoping review was quite interesting in that it
looked at how AI and ML is used in digital, maybe change
interventions. And so, so Daveen, maybe you can
give us kind of like the summary of what you found, like what was
the main takeaways and then we can maybe dive into do some
further findings as well. Sure and I love that you
characterized it as what is BS and AI cuz we actually had the
same 2 conceptions. Like from a a sort of legitimate
scientific work perspective. I really wanted to know how is
this happening in the real world?
Like what is the landscape out there of the different AI tools
that are being used to change real world behaviors?
And I was going to add through digital means that AI pretty
much is through digital means. But the other, the other thing,
the sort of more personal reason is we have noticed that there's
a lot of companies or groups who talk about doing AI and then
when you look under the hood and, and carefully read the
description of what they're doing, it's not really AI at
all. It might be something like a
hard coded algorithm or a thing that I found a lot in the
scoping review were interventions, products that
called themselves AI. But we're using a Wizard of Oz
setup for the testing. So it was basically mocked up.
There was a live person behind the screen making it appear that
there was an artificial intelligence interacting with
somebody, and so we ruled those out as well.
That might be an early prototype for an AI, but in and of itself
it's not. Normally my team, my behavioral
science team will do these sort of encapsulated projects that we
can run. It's a side passion project
without involving colleagues elsewhere in the company because
it can be difficult to get time. I mean, you know, it's it's hard
to get time from other teams. But with this paper, we knew
from the beginning we had to co-author with at least one of
our AI scientists. So one of our co-authors is our
chief AI scientist, Chris Simons.
And the reason why is we needed help as well figuring out like
what are the legitimate search terms for AI?
And then once we have done our lit review and identified some
papers, we really needed Chris to read some of them in detail
and tell us, does this meet your definition as a career computer
scientist who specializes in ML of AI?
And so that was unusual for us to absolutely require an AI
colleague to collaborate on the paper.
When once we built our search terms, we ended up finding a
total of 3000 exactly, which I think looks fake, but it was
3000 exactly papers that met those high level inclusion
criteria. And by the time we went through
all of our filtering and review, we got it down to 32 papers that
covered only 23 different interventions or products.
And you know, of course, these are the ones that have gone
through the trouble of publishing in the peer reviewed
literature, which means typically you have to have some
kind of results. Like there's the barrier to
being included on this list was somewhat high.
And I know there are companies out there that are doing real AI
to change behavior in the real world.
But in the scientific literature, we only found these
23. And it was interesting to me as
someone who spent her career in industry, only six of them were
commercial products. The others were all developed
for research purposes and may or may not ever be commercialized
and made available to the general public, which to me
seems like a little bit, well, it's a pain point with the
academia industry crossover that I knew.
But it's, it's really poignant to the example of it because
some of these products were great.
But yeah, we we wanted to break down what types of AI are being
used and who's doing it. And one of the things that
really surprised me was that all but two had proprietary AI where
they developed that technology themselves.
And Lyrio is among them. Our, one of our papers actually
met our inclusion criteria. We have developed our own.
And I, I know that there's good reasons to do that.
You know, you want to own the IP, you really want to create
something that's competitively differentiated and and owning it
makes a lot of sense. But it's also really limiting
because first of all, the barrier to doing that is much
higher than using products like ChatGPT and integrating that
into your product. There was one that came up in
our review that had cobbled together some products from
Amazon and Microsoft, but you know, these commercially
available and they probably got that stood up much more quickly
than somebody who grew up their own technology.
And the other thing is, when everybody develops their own
technology, it limits the ability to compare a bit because
you don't truly know the intimate differences between
what one company develops and what another company develops to
understand how does this work and how do we draw conclusions
about the effect of having our behavior.
Yeah. And I think that speaks to kind
of this interesting thing that happened where I feel like in
the last three years especially, there's been like this booming
quote, UN quote, like AI tools. And it feels like all these
tools are in some ways unique. But if again, looking under the
hood, it goes back to maybe you see like 3 or 4 foundational
large language models that are used like either opening eyes,
either entropics, maybe some like llama model, or, you know,
a lot of them are basically using basically the same things
on the hood. And then they use basically
create a product out of interface, like how to kind of
make it easier for people to maybe directly make use of those
models and so on. And yeah, is that something
similar that you came across as well or maybe?
Yeah. Well, actually on on large
language models specifically, we found that they're basically not
being used in patient or consumer facing behavior change
technology. And I wasn't surprised by that.
I was actually pleased by that. I don't think this is a
controversial take, but I don't think that large language models
are ready for prime time in some scenarios like providing medical
information to patients. And actually, Lyrio was just a
sub awardee on a grant from ARPA H, where we're going to be
looking at exactly that kind of thing.
We're going to be helping to develop some quality tools for
mental health chat bots that would help detect if the
information it produces is credible and complete.
That's the sort of thing that's missing right now.
And there's research on the use of large language models in
healthcare that from my read, the place where I've wound up is
it can be appropriate for some clinician facing scenarios
because someone who's a clinician has gone through that
education and professional experience can put the bullshit
meter on it. They can read something that's
in a patient portal developed by LLM and they can edit that and
make it accurate. There's a paper that I've cited
a few Times Now where they had a Gen.
AI create first draft responses to patients in the patient
portal and then the provider had to review review them before
they sent it out. One that really struck me was
the ChatGPT or the the Gen. AI said, yeah, patient, you're
taking 20 milligrams of this drug, but you take it twice a
day in the 10 milligram dose. It's fine if you get the 20
milligram pill in your subscription to save money and
just split it in half. And when the clinician read it
right away, they said, no, there's no score line on that 28
milligram tablet. You're not gonna be able to
accurately divide it in half. That's like a human quality
information that would be very difficult to get out of the LLM.
And the clinician has the expertise to instantly recognize
that it's missing. We're on the patient side.
If they'd received that original message, you know, at best they
would have learned through hard experience picking that pill up
at the drugstore and seeing they couldn't break it in half.
So I, I, I think LLMS are not really ready for prime time for
that scenario. And that was kind of reflected
in the evidence we found in the paper where there were several
different interventions that were using natural language
understanding. So they were deriving intent and
meaning from free text entered by the user, but whatever they
were responding back with had been pre written by an expert.
That's an interesting and I guess one of my reflections from
also reading this and I think you have it in the conclusion in
terms of one of the significant opportunities for people in
Marvel Science is to become more like able at knowing and
understanding terminology in AI. And we can touch upon that
already a bit, you know, and I guess I want to hear like a
little bit from your journey working with Leary, for example,
like you've had a chance for the last couple of years to really
work closely with like you mentioned colleagues that are
like coming from the maybe more like hardcore machine learning
backgrounds or similar. Like how do you think about kind
of like developing that terminology and how how's that
been for you? It's been hard.
The Dunning Kruger effect is really all I'm living it.
I know more about AI than I've ever known, and I'm also so
aware of the limits of my own knowledge.
But I think what I'm getting good at is actually what I like
to see on the B side side sometimes too.
And I've been really passionate about making the tools of
behavioral science available widely to people of different
backgrounds and different professions.
But I do also believe there's a point at which you might be
working on a problem and you say, oh, I need someone who
really has, you know, trained in this and reach out for that
expertise. And I, I think I'm in a similar
place with AI where now I can talk about it intelligently and
correctly to a point. And I've gotten pretty good at
recognizing what that point is, which I think is probably about
as good as it's going to get. I'm never going to replicate,
you know, Chris, who I mentioned our chief AI scientist has his
PhD in computer science and, you know, worked at Oak Ridge
National Labs leading teams over there.
Like, I'm not going to learn what he knows.
But the the other good thing about it is one of the things
he's taught me is that AI is. Some of these models that we're
using, the supervised training models, like they benefit from
subject matter expertise in the domain where they're being used.
When he frames it that way, we become essential partners.
And that's kind of a really cool thing about working with the AI
is that we can't independently operate.
We have to work together. And so the questioning is going
back and forth, which makes it more comfortable for me to say,
Chris, is this written correctly?
Is this really what I'm supposed to be saying?
Does this mean what I think it's meaning?
And I learned a lot through that kind of thing, too.
And what would you say are some of the most common
misconceptions about AI, at least in terms of from the
perspective of the behavioral science community?
You mentioned hard coded rules and really algorithms that don't
adapt as being AI but but what else is there?
Yeah. Well, IA big one I'm seeing
right now that drives me up the wall is I think people are
increasingly in conflating generative AI with AI as an
overall category. They say AI and they just mean
generative AI. Adjust me.
Yep. Yeah.
And that's just such a small subset of what AI can do.
And it's not the type of AII work with our product uses
reinforcement learning. And I am actually a huge fan of
reinforcement learning. I think it can be very powerful
in behavioral science. And so I guess it's not a
misconception so much as a lack of awareness of additional AI
tools and approaches that could go well beyond or augment what
Jen AI could do. And again, I also think Jen AI
is not ready for prime time with that patient or consumer facing
content. So to hyper focus on that I
think is really, really limiting for a behavioral scientist who
wants to change behavior. Yeah.
And, and it definitely seems like this is a phenomenon that's
really only come about in the past couple of years.
I remember when Gen. AI was really pretty niche,
right? You only heard about natural
language processing in these like much more narrow use cases.
And now it's everyone is using the LLMS.
I mean ChatGPT was a game changer.
Putting a public facing, easy to use interface on an LLM just
totally changed the world. I mean, that's an important
lesson, right? Do you think that lesson can be
transferred to other use cases of AI?
Probably, actually. So reinforcement learning that I
work with, we call each algorithm an agent.
And I'm clarifying that because sometimes we'll talk about the
agent decisions and people are like, wait, who's this agent?
It's an algorithm. When we set up our
interventions, they are multi agent systems and each agent is
responsible for a specific choice.
But those choices ladder up. So with Lyrio and Precision
Nudging, we're sending personalized messaging and we
might have an agent that decides what channel we use for a
person. Is this person getting an
e-mail, a text message, a push notification?
Another agent might actually decide on the headline that the
e-mail gets. And a third agent decides on
what's the body copy of this e-mail?
What are we actually saying to this person?
And then yet another agent might decide when we send it.
And each of those agents there, we we consider what we do, a
type of objective driven AI. That's a term that Yan Likun,
who's the head of AI at Meta, has coined in the last year or
so. And reinforcement learning is a
perfect symbol of that because what you do when you train those
agents, when you set up those algorithms is you give them
objectives. So the objectives in our case
are almost always we want the person who receives this message
to complete the target behavior. And I always use the example of
mammogram. That's one of our most mature
interventions where we have pretty strong evidence.
And it's a very straightforward behavior because what we're
asking people to do is open our message, schedule your
mammogram, attend that mammogram.
And what we can basically do is tell the various agents that are
making decisions about how we communicate with somebody that
it will be rewarded. And again, that's conceptual.
It's not a money thing. At the highest level, if someone
has that behavioral achievement, they've actually finished the
mammogram, but as a lower reward if they schedule it and a lower
reward reward yet if they interact with the message.
And in that way, we're sort of able to prioritize the
meaningful behaviors. That's something where you could
have an interface that people interact with and you could
start to play around with, you know, what do we make the most
important here? And especially when you get into
more complex behavioral patterns, you know, mammograms,
easy to talk about because it is so straightforward.
But in reality, if someone has a chronic condition or even is
healthy but is orchestrating all their healthcare, like Wellness
visits and vaccines and cancer screenings, that can get pretty
complex. And that's that again, is where,
you know, I think there's some room to pull levers and twist
knobs, so to speak, and figure out what is the right way to
orchestrate. But again, if you set those
objectives, reinforcement learning is going to start to
figure out what is the best way to get this person to get to
that objective under the scenario that it's implemented
in. Yeah.
And. I think it's interesting in
terms of like there's been increasingly use of the term
agent, but I do think that most uses where I see agents being
referenced, it's actually not with that reinforcement part to
it. It's often times a little bit
simpler in some ways. It's often times we have some
form of setup that automates that.
You know, if this thing is triggered, then that activates
this agent that maybe uses some form of let's say open AIS large
language model to execute on some form of like quite simple
thing. And it's pretty good.
You can do some cool things just by doing some of that stuff, but
it's still not used enough time to reinforcement learning.
So I think, I think that's actually still like a pretty
good indicator of maybe a little more advanced use cases where
often times there's people involved that are also coming
from lately the machine learning background or have a more
technical backgrounds as well. Whereas they they can be like a
lot of Bros on YouTube. This is like, hey I I designed
an AI agent or something. And that that actually is where
I mentioned in the paper that we did the scoping review.
Only two of those interventions used publicly available and I
can't remember the name of it, but the one that didn't use the
Microsoft and the Amazon used it was some kind of web interface
that allows you to basically say if this, then this, you know,
deploy this. And so that was how that one
worked. Maybe quick thing to tie up
before we move to the next question.
What are them the Pareto principle terminology terms that
people should know about within Tai?
Is there some like really important ones that people in
Babel science should you know or be able to better understand?
I think we've talked about some of them already.
So things like generative AI and understanding that's a subset of
AII think the term objective driven AI is very useful.
I'm not one to jump on a buzzword that an executive at a
buzzy company like Meta has has coined, but in this case I think
it really does a nice job both describing what certain types of
AI are built for and also helping to distinguish it a bit
from generative AI. Although there, you know, those
could overlap a bit in the Venn diagram.
I can imagine scenarios. We included a glossary in that
paper that Chris actually wrote, but then I edited to make sure
that it was understandable to someone like me.
And we're just about to actually make a blog post version of that
on our website. That because we think that's
really important, that these terms are at least of passing
familiarity to people. And I'll tell you, I can't use
all of those terms comfortably in a conversation myself, but
there's, you know, the ones that I use, the ones that sort of
affect our product. So I understand things like
supervised learning and unsupervised learning.
And you know, where that supervised learning is really
where our behavioral models come to bear because what we're
essentially doing is looking at what the agents want, want to do
and course correcting them based on our expert knowledge and the
logic models that we set up with our content.
So for me, understanding that kind of thing is important.
But I think for the average person, just understanding that
the models need to be trained, that there's different ways to
train them, and maybe even that the quality and origin of the
training data is really impactful on what the final
product looks like. I don't think you need to
understand all of the guts of how that works, but to know
that's a piece of it is really critical.
And you're working with contextual bandits, which I like
to think of as kind of like superhuman experimenters, right?
They're like sending little bits out into the fray and seeing
what comes back and then choosing the winner and then
moving on to the next experiment and just optimizing in that way.
Well, and you'll like to The early papers, the ones that my
colleagues cite all the time about contextual bandits are
based on the video game StarCraft.
So when you say like shooting things out like.
Yes exactly, almost literally. That's so funny yeah some of the
AIML terms just crack me up every time I see them.
The multi armed bandits and just like I hard to take seriously
but. QQ Learning is one that I hear
sometimes and I think it might actually be in our glossary, but
I don't know exactly how it works, but I always think of
Star Trek. Hard to not go to sci-fi really
when you're when you're talking about AI in general, right?
Totally. So I would love to really get
into the nitty gritty of what you're doing at Lyrio with the
large behavior model with precision nudging.
Like I want to understand how does it all work?
What are you doing with your bandits?
What for, you know, the the goodness of science and other
people trying to solve for health problems out there.
Like what can you share about how how this works, where
behavioral science fits in, and how AI and behavioral science
are really like, really working together to make your
interventions amazing? Sure.
And I think I can share quite a bit actually, when we begin
designing a new intervention, it's my team, the behavioral
science team that is first on call.
So we almost always design interventions because we have a
client who has a use case that we're helping them to solve for
it. So there's an external partner
in the loop, and what that provides for us is the
opportunity to research the behavior in context.
So we work with the client to understand their objectives.
And a lot of times these are things, no surprises here.
They want to close the gap in care.
They want to improve their reimbursements from insurance
companies because they've raised their quality or they're meeting
a certain metric better. Sometimes it's a population
health goal. We want to get X percent of our
population below this health metric.
We have an example right now with a client, Cone Health, who
is awesome, awesome, awesome, and has been really wonderful
about allowing us to talk about the work that we do with them.
They're helping us to, well, we actually have launched it
already. We have an intervention for
hypertension management. And one of Cone Health's goals,
because they're really committed to HealthEquity, is not only to
get, you know, meet meet the sort of target metrics that are
nationally available for hypertension control, but also
to make sure that they're black patients also meet that metric.
Because typically if you look at a patient population, if it
meets that metric, on average, it tends to be white patients
who have met it. And then black patients
specifically will lag behind. So Cone is committed to making
sure that that gap doesn't exist in their population, which is
awesome. But one of the first things we
did with them was understand that was specifically a goal of
theirs because then that allows us to design the rest of the
intervention appropriately for that.
We identify the target behaviors.
We do our, you know, beautiful literature review to understand
the determinants as seen in other research.
And then this is where having that partner is also really
helpful. We will do research in context
with them as well. So we talked to, you know,
particularly like their clinicians actually to
understand how hypertension management happens and you know,
what are the sort of contextual barriers that we need to
understand and design for. We put all of that into a logic
model. This is all still my team.
So we have a document that we create that we call our strategy
matrix and we actually outline what are the determinants that
we've prioritized to addressing this intervention?
What are the mechanisms of action through which we could
influence those? What are the specific behavior
change techniques that we are going to operationalize in our
messaging to use that MOA and address the barrier?
And then the content and visual design teams are part of
behavioral science at Lyrio and they work on operationalizing
all of that into assets. We create lots of little content
components, so we sort of atomize what goes into a message
so that we have more things that we can select from to
personalize. And we also create visuals that
operationalize the behavior change techniques, which I think
is pretty unique for us. And we're starting to accumulate
some data. That approach is helpful in both
engaging and converting people to the behavior.
So that the idea of the hypothesis was if we can kind of
infuse the BCT into somebody's brain through multiple channels,
let's do that. We build out all those assets
and around here is where we really start interacting with
the AI team because some of the things that we need to ensure
that they do, they have to train the agent or the agents on the
content that we've developed. And we're using our logic model
to do that. So that's where part of our
expert knowledge is infused. One of the steps of training the
agent is doing a semantic conversion of the text.
So if you've seen like encoder decoder models, Bert is Google's
tool for some of this. And I believe we actually do use
Bert at one point in our process.
That's a semantic conversion. So it looks at, are these words
similar or different in a multi dimensional space?
How would they cluster together? And this is actually a place
where our expert knowledge makes a big difference because there
are pieces of content that might seem very similar from a
semantic level if you don't really know the meaning of the
words, but you know that the words are kind of similar.
But if you're a human being and especially behavioral scientist
and you read them, you're like, those are very different.
And so we can basically use our logic models to force a
separation between items that are different in terms of their
meaning, where the model might have otherwise put them close
together because of their words being similar.
So there's that piece of it. And then the other piece of it
is helping to set the reward function.
So I again, I use that really simple example of a mammogram
where the highest reward function is achieving the
mammogram, actually getting it done.
But with some of these more complex behaviors we have to
think about, you know, how do we prioritize them?
What is really the objective from a behavioral perspective
and how do we build that into the algorithm?
So that's what it's maximizing for.
It would be really easy to abuse this sort of approach, and I
would suspect that Facebook probably uses some of this
stuff, for example, because of the way that their news feed is
created. And sort of putting the next
thing in front of you that keeps you on the screen like that is
an objective that you could train an agent for.
In our case, we specifically don't want to do that.
I would rather you not even read the messages if I can somehow
otherwise get you to do the behavior.
So we just want to be really careful that whatever we're
rewarding the agent for is the thing that's really most helpful
to our objective. And sometimes that's also
thinking from a behavioral scientist perspective.
If you have somebody who has a lot of healthcare needs, it's
not always the most sort of medically impactful thing you
want to go after first. Because people who have a lot on
their plates in terms of changing their behaviors and
addressing their health might be better served by trying
something easy first, you know, building self efficacy and
notching that small initial win. They may be better served by,
you know, doing something that is easier for them for some
reason. Maybe they live a lot closer to
the facility where certain healthcare appointment happens
than the one where the like, the big appointment happens.
We need to think about those things too.
And that's where AI can be really helpful at parsing out
like complex data about a person and, you know, sort of
identifying what is most likely to work for this person, which
could be a bridge to the LLBM, the large behavior model.
But I don't want to just talk for like 4 minutes you guys say.
Well, this actually brings up a maybe semi interesting question
for me is what if you put a negative incentive on reading of
the e-mail or you know, time spent on the e-mail as opposed
to just a small incentive, really disincentivize that
outcome and then really, you know, outweigh the attendance at
the mammogram for example? Yeah, it.
Happen. So I'm going to give you a
really, I'm sorry, this is a disappointing response.
I don't think that was actually possible because one of the
things that we're actually learning and and this just, it
doesn't matter to us that much because we've never put a huge
reward on interaction with the product.
But e-mail metrics in particular are getting really hard to
measure because some of the changes that like Apple and
Google have made to their e-mail clients.
So all of those engagement metrics are artificially
inflated by some of the design changes they've made recently.
And so we're actually even talking about them less and
less. But what we have been doing that
I think is in the area of your question, people will reply to
us if we send text messages, if they received the intervention
through a text message, a surprising amount of people were
fly back and this is not with a system command, it's with
substantive information. And so we have actually we have
in market now, but it's going to be it's an MVP that's going to
become I think really powerful. We are basically analyzing with
some natural language understanding with an intent
model, what people are replying to us.
And we are using that to sometimes negatively reward the
agent because sometimes people are responding very negatively.
You know, they're they are upset that they're being contacted or
this is at the campaign I'm thinking of as a vaccination
intervention. And vaccines are always a little
bit of a hot button for some people.
And so there's some people who are replying like with
conspiracy theories and political commentary and we're
basically negatively rewarding the agent in some of those cases
to a small amount. And we're also taking actions in
those cases as well. So we we can send replies that
are appropriate to what the person said if like they ask a
question. In the case of somebody who's
really angry, what we do is we suppress the messaging because
they don't wanna hear from us. You're not gonna get through to
that person. No, no.
And it's like, why are we spending the 10th of a set to
piss this person off even more? Let's not do that.
Why won't you get the vaccine? I really want to implant this.
Yeah. So I'm really excited about that
because I think that our bidirectional capabilities are
going to become a really critical part of what we do.
And that becomes a way for us to also gather information about a
person's experience and more quickly identify what barriers
they're actually experiencing and better train those agents.
Like what is the right type of messaging for this person for
this behavior? Can we talk a little bit about
barriers? So you mentioned you do a lit
review and I know you have some a lot of theory that's really
oozing through your models. We talked a lot about Com B.
How does that work with the sort of top down theory driven
systematization versus the bottom up like talking to health
systems and clinicians and trying to understand what did
patients actually say? How did those two come together?
Yeah, that's a challenge. It's not a challenge.
It's, it's a fun part of what we have to pay attention to.
A lot of it is just early documentation.
You know, we, we put all the lit review things and I, I mentioned
we have a document called Strategy Matrix.
We use Excel to build it. It's very glamorous fancy.
But the initial version of that is fairly large because of
course all of the lit review papers are using different
terminology or maybe you're slightly, you know, defining a
barrier differently. So we do some organization and
right sizing of what we find in the literature.
We're recording what we found from the client research and
we're trying to basically bucket it together with what we see in
the literature. And what we typically wind up
with is the clients experiencing things that were already known
to research for the most part, but there's usually a handful of
things that are really specific to them.
What we tend to do is we include them in our design process.
And then when we develop our content, we always, even if
somebody comes to us and they want to use an existing
intervention, we go through a tailoring process and then what
we call content activation to make it live in their
environment. And that's an opportunity.
You know, we have system variables, we have the
opportunity to tweak and change some of the messaging and we can
infuse some of those client specific things in at that
point. And what we typically try to do
is just make sure that we do it in a way that is consistent with
the logic model we've built. So what that might look like is
if we already know that convenience is a barrier for
somebody, we've already identified, you know, the BCT in
the messaging, but we're working with a retail pharmacy that
really it's important to them that the patients know that
they're open more than 9:00 to 5:00.
We can build that into the existing message that already
uses that sort of approach, but now it becomes very customer
specific. And if it's something that's
really unique to that client, we can just build that out as their
own, you know, bundle of content that is activated for them, but
doesn't become part of our core library.
And our CMS, I won't get into it, but our CMS allows us to
sort of this is the master content library, the overall
intervention, but then here's the iterations of it for various
clients. When you're going into a new
intervention, are you starting at 0?
Say we know nothing about this patient.
We are going to send every patient, you know, some randomly
selected message from the library, and then we'll learn to
get to, you know, their optimal message.
Or can you already start from somewhere and say, well, I know
that this patient lives, you know?
20 miles from the clinic. So that's likely to be a barrier
to actually attending the appointment.
Which of those? Both, both and sometimes the
other places as well. So we have some clients that we
work with and it's a net new use case, new intervention and for
whatever reason they are unable or unwilling to share very much
data with us. And in that case, it is
essentially, let's send something randomly and start to
learn from scratch about these individuals.
And I know that Chris, my counterpart on the AI side is
probably would probably jump in and correct me.
It'll be like, it's not really from scratch yet.
But for my, let's just call it from scratch sometimes if it's a
new intervention and it has similarities to other
interventions, we're able to leverage the training of the
Asians for those other interventions.
So a good example here again is vaccination.
We have nudged all different types of vaccines.
We've done COVID, flu, RSV, pneumo, shingles, and there are.
We've done the research to know that the barriers for receiving
those vaccines are not completely overlapping, but
they're a lot overlapping. Covic is the one that's kind of
different, but the others are a lot overlapping.
And so we have been able to basically use agents trained for
other vaccinations to get so that kind of jump start.
And then you specifically mentioned home address.
And I know you didn't do this on purpose, but that's a piece of
information that's particularly useful to us because on my team,
Sarah D dot, my VP of Behavioral Design, has her PhD in health
geography. So she intimately understands
space and place from a behavioral perspective.
And then on the AI side, we have a team member, Rajiv Vatsavai,
who is also faculty at NC State, and his expertise is geospatial
transformation. So between the two of them, we
actually can get a ton of mileage out of somebody's
address and specifically identifying things like
proximity to where the healthcare behavior is
delivered, but also things like, you know, are they near green
space? Are there food deserts depending
on what's relevant for that intervention.
So we can sometimes do transformations of variables
that we receive, or that we have access to derive much more
meaningful information from them.
Wow, awesome. Those are definitely specialties
that I did not. I'm very glad that they exist,
but I had no idea. I know it's kind of wild too,
that we didn't deliberately set out to assemble this super team
of geography people. But.
We did it. Really cool.
Yeah, yeah. That's cool.
And I guess I wanted to in some ways turn the tables a little
bit of what we've explored here in terms of we've talked about
kind of from a behavioral science point of view, what we
can do with AI and the benefit that we can have in our
interventions and so on. But maybe as a hypothetical,
because I think a lot of behavioral scientists is kind of
exploring like, what is my value in this new evolving landscape?
And you know, if you then look at the kind of Euro illyria and
the thing you described, you know, what would be the cost of
removing behavioral science from that equation?
Like how do you see you come in as a behavioral scientist and
adding value and what would be there if you were not there?
Obviously I have a bias. I don't think I would work
without us. No, I really do.
Actually, I, I think it would be pretty bad and I'll give you a
couple of reasons why. So the first is we have actually
done tests where we do our behavioral science messaging
without the AI. So, and there's been, you know,
various reasons that we've done that, whether it's deliberate
or, you know, we were tweaking something.
And but the behavioral science messaging by itself has a huge
impact compared to control. So, you know, I, I think that
having that behavioral lens on the content that we send out to
people, and I actually believe that the logic model behind it
is also really important. I've seen other studies, other
interventions where people create intervention outreach
that's very strong, you know it, and it's what I might call a
kitchen sink behavioral science approach.
Like how much can we put into this messaging to, you know,
just convey all the different tools and reasons why somebody
might be able to do this thing for us?
Any given message really only has one BCT in it, one behavior
change technique. But I think the fact that we get
very specific and spend a lot of time, you know, just really
making sure that comes through extreme.
We go through a quality check process where we have blind
raters review all the items and determine, does this adequately
operationalize the BCT and does it do it in a way that's
distinguished from other BC TS? I think all of that means that
the content by itself is just a huge value add.
But then the other pieces, as I mentioned, there is no universal
learner. That's a thing that Chris says
all the time. You really have to train your AI
against subject matter expertise.
And in our case, we are doing that against, you know, behavior
science and healthcare expertise.
We use an example in our sales presentations that I think is,
you know, it's a little bit intended to be provocative, but
we actually have seen healthcare organizations who hire former
executives from companies like Amazon or Disney, you know,
these companies that are amazing at engaging people and they
bring them in. And I, I so far have not seen a
major success story in improving patient engagement in a
healthcare setting with that expertise.
And I, I think it speaks to the fact that you really do have to
infuse your approach with a deep understanding of well, how human
behavior works, but also how does human healthcare behavior
work in a particular system that provides barriers, limitations,
enables some behaviors. So I, I, I don't think that the
product would work without infusing that type of expertise.
We, we might be able to get it to do something else.
It we might be able to get it to do something that's still like
better than random, but I just don't think that.
I think the delta between what we can do today and what we
would do without VISI is huge and negative.
And if you think about it, that's where most companies are,
right? Lacking the behavioral science.
Well, or or the AI, yeah, I mean, it's funny and and again,
like we're we're a little bit jerks about this.
But you know, Chris and I have been to conferences together.
We've locked the exhibit floor and we'll see these companies
that are like AI decide and between the two of us, we're
like, come on. Yeah, that's not real AI or
that's not real D side. I know.
Exactly which ones you're talking about.
Yes, yes. And that was, again, it gets
back to why we wrote that scoping review and sort of the,
you know, the dual meaning of BS there, because we wanted to
really know for sure what is out there in terms of AI.
Obviously everybody's gonna market in the way that they
think is gonna be most appealing to their customers and things
like that. But some of them are really
doing AI. We wanted to know who they.
Were all right. It is time for our quick fire
round. Are you ready?
I'll tell you what it is. Don't worry.
So this is a game that we've developed.
It's called to AI or Not to AI. And so it's your job to
determine whether the activity that we present to you is well
suited to AI or not. All right, let's begin.
So first one, negotiate a job offer to AI or not to AI?
I think you can use AI in this. I wouldn't directly interface
the hiring folks with the AI, but I, I could use it to like
revise, you know? Hey, this is what I'm thinking
of replying to this e-mail. Can you revise it?
Can you make it more succinct, more professional or getting
stats? Can you tell me what a good
salary might be for this role based in the city?
So I say AI has a role in this. So really as a as a partner, not
a replacement. Correct.
How about file taxes? That makes me nervous.
I think that the risk of being wrong is too high.
I say no AI. OK recommend a mystery book.
Absolutely yes. Write a mystery book.
No. It we've done a lot of work
looking at the quality of content and I think the short
form content can be pretty good, but anytime we ask it to do
something long form or repeated, it's boring.
The Sign. Behavioral interventions for a
mental health app. No, I will not design A
behavioral intervention for a mental health app without a
clinical partner. About Taylor Cat memes to one
specific preferences 100% how personalized will it get?
Well, so I have a cat named Jackie Daytona after What We Do
in the Shadows. So Jackie Daytona, human
bartender. That's so funny, by the way.
I love that. Thank you.
But I've tried really hard to get like Dolly mid journey to
give me Jackie Daytona, human bartender, but he's really a
brown tabby cat type things and they're very they're not where I
want them to be. We're going to keep working on
it all. Right.
Maybe some promise in the future.
Yeah, I'm now just thinking about what we're doing in the
shadows and I completely lost track of of thinking about this
quick fire around, but OK, next one device and investment plan.
Yeah, I think it could. But I this is again where I
would want to review it and then go execute it.
I wouldn't want AI to devise the investment plan and then execute
it. And last one, identify patients
at risk for diabetes. Well, yes, we we actually do a
little bit of that. So yeah, I think that's a great
use case. Awesome.
All right, it's time to ask you the question that we end with
all of our guests this season. Amy, what is your most
controversial opinion about AI? I think AI is not that good yet.
And I also think that we are unnecessarily afraid of it in
the wrong places. So I see a lot of people who are
like, Oh my gosh, AI is going to replace us.
And I just don't think it's good enough to do that yet.
And like I say, I'm on LinkedIn a lot, mostly because I'm told
that's a thing I should be doing.
Obviously, notice a lot of people write their posts with
ChatGPT. And so first of all, I can tell,
you can tell if somebody actually wrote their post this
way, because like I said, ChatGPT might look great in
isolation, but it has its own tics and mannerisms that show up
in it's writing across different pieces.
And like once you know it's voice, it's kind of easy to see
it. And so I think that people
overlook the fact that, you know, I can tell the ChatGPT
wrote that it doesn't mean it's bad, but it's no replacement for
your own voice. And then even things like
LinkedIn fired a lot of their content writers and now they do
that thing where they ask people to contribute to articles.
I'm assuming there's some kind of AI behind that in terms of
generating like which prompts which people see and how they
frame those to you. And I'll tell you, I don't
actually don't read those social articles the way that I
sometimes used to read the ones that LinkedIn staff writers
wrote because there's just like a a difference in kind of
quality and organization that to me as a reader leaves me
unsatisfied. So I just don't think that AI
has nearly reached its capabilities.
And I think that the human in the loop is likely to be
important forever, forever. Maybe not in every case, maybe
not in every case. I mean, we certainly have seen
things where AI has been able to, you know, manage supply
chains and computational sorts of things.
But I just don't think AI is going to replace humans in
nearly the way that some people think it will.
Yeah, I guess time to the first thing is that I'm talking about
memes as well. We're talking about cat memes
before, but a non cat meme that I've seen quite a bit with
imagery is basically first there's this person who says
like I haven't seen a lot AI generated images lately.
And then you see them like with like a Oh my God face.
Like I haven't seen an AI generated images lately.
Yeah, Yeah, it is. It is getting better.
A. Little bit to your point, like
maybe there's some subset where it's like really well done where
you don't notice it, but a lot of it you notice.
And especially, I know also you get recruitment stuff and I I
get people applying for things. And I understand why people use
AI for writing applications and various things.
But it's also like I can see AI was used here.
Like, yeah. So we just hired several roles
on my team and one of them was a writing role, a copywriter role,
and that was the one where we saw a lot of ChatGPT generated
application materials. We didn't see it for the
behavioral science roles for or we didn't notice it for the
behavioral science roles. I'll say that we might have seen
it, but one of the ways that it was most evident was people
would put their work histories. Obviously that's what you do on
a resume and they would have things in there.
That was like 2012 to 2013 worked on precision nudging
messaging. And it's like, I don't think you
did. And I also don't think a human
being would make that mistake, you know, Weird.
Yeah. And And the other thing, again,
because we're reviewing lots of resumes, you start to see
patterns across them. Yeah.
I even think this was not AI, but when I was a grad student
and I was teaching, you know, we'd always get people who
plagiarized on their papers. And almost always the way that I
caught them was they weren't the only one plagiarizing from the
same source. Like I vividly remember one
night reading a paper view, like did I just read this paper or
looked and there were actually 3 copies of essentially the same
paper in my pile. So I think that's a lot of where
AI gets caught is when multiple people use it for similar
purposes and it isn't different enough.
Yeah, it feels different to them, but then to the one who's
receiving all of the AI generated text, Yeah.
Yeah, I did use AII, gave it like my LinkedIn bio, which I've
never liked and had it edited for me.
And I, I what's on there now was edited for me by ChatGPT, but I
wrote the source. Yeah.
Well, I think like that's in the end, I think it will often goes
back to like the importance of the understanding of the
context. And I think that's really is
true for AI as as it's been for us as payroll scientists before.
And yeah, I guess to wrap up, it's such a gift to have you on
the podcast and maybe it like as a little bit of a thank you as
always. I, I was going to message you
this, but I had kind of randomly this experience going to an
office for a client. I was going to start working
with a kind of digital health company and the product manager,
he welcomed me. And then in the, in his office,
he's like, I've, I've read this payable science book.
Do you know about it? And he showed me your book and I
was like, oh, thank God he's showing me that book because
there's so many books that you could have shown me and.
I'll be like rolling my. You would have just been like,
oh. Yeah, but I was like, OK, we're
going to be in good place. Let's get.
Started. Yeah, I guess thank you for all
the work you've been doing and recently now also sharing the
work with AI and so on as well. And thanks for coming on today.
It's really fun to shut. Yeah, and I, I have to thank
both of you too. I can't tell you how many people
I refer to have it weekly. I just think it's one of the
absolute best resources out there and especially for people
who want to learn more, it's so accessible.
And you know, one of the few things that I got where I am
always clicking links. So thank you for the service
that you both do and I'm honored to be able to be a small part of
it. And that's a wrap.
You've been listening to the Behavioral Design Podcast,
brought to you by Habit Weekly and Nuanced Behavior.
Sam and Elaine tell me this season is packed with incredible
insights about behavioral design and AI, so be sure to subscribe
and share the podcast with your friends.
Though you might want to keep it away from your enemies.
In case you haven't noticed, I'm an AI voice.
Yep, pretty crazy. Quite the improvement since last
season's AI outro, don't you think?
If you'd like to collaborate with us at Nuance Behavior,
where we use behavioral design to craft digital products with
Nuance, e-mail us at hello@nuancebehavior.com or book
a call directly on our website, nuancebehavior.com.
A special thanks to the amazing Dave Pizarro for our show music,
and to Mei Chen Yap and April English for their help in
producing and publishing this episode.
Thanks again for tuning in. We'll be back soon with another
exciting conversation where behavioral design and AI
intersect. Happens to Mugatroid.
Oh. It's been hard.
The Dunning Kruger effect is real, y'all.
I'm living it.