Hello and welcome to the Behavioral Design Podcast.
I'm Simon Schultzer, and today we have a very special episode
where we're joined by Jared Peterson, a real scientist and
colleague at Nuance Behavior. I have done all that I can to
convince him to join because I think Jared is such a perfect
person for the episode we have today.
Jared is not only an expert in decision science, but he's
recently also been involved in AI alignment work, which is
perfect for today's discussion. He has also become a thought
leader in the field with prominent writing including The
Science of Context, and most recently, a fantastic review and
ranking of all of the top payable science frameworks and
models. So look that up.
It's in the show notes. But All in all, Jared is the
perfect person to help us make sense of the state of behavioral
science, the state of AI, and everything in between.
As we're today, we'll be trying to look back at some of the key
breakthroughs in 2024 and peering into the future, trying
to make sense of the state of our field and what we can expect
in 2025. You can expect from us some hot
takes, some predictions for the state of AI and payroll science,
and maybe even some nuanced thinking and perspectives.
We'll see. OK, let's dive in.
Welcome, Jared. How are you?
Thanks. I'm doing excellent.
Our, I believe our takes should better be nuanced.
That's kind of our label. Yeah, we we have to live up to
that. We'll see.
How are you feeling so far in 2025?
What's your temperature? 2025, It'll be an interesting
year, I am sure. I'm having a child this year, so
that's the most prominent thing on my mind, but there's
certainly some bigger things happening in the world as well.
But it'll be an interesting year.
Have you had that thought of like, what world am I bringing
this child into? Quite a bit.
I'm very curious how education because you know, for me, my
childhood is pretty much defined by like the schools like that's
what childhood was, is like you just go to school all the time
and like those are my like most prominent memories.
What is school going to look like in the age of AII?
Have no idea. It'll be very interesting.
Yeah, yeah, hopefully it doesn't have to be like Ready Player One
or something like hopefully it actually it's going to be more
like some form of utopian version.
But I guess that's what we're going to dive into.
But actually, before we look into this year, I kind of wanted
to do something because obviously we're recording this
in mid January and 2024 is already quickly becoming a blur.
But I thought it would be useful to kind of first walk through
what I guess I'll be sharing some things that I saw as key
moments to reflect on from the past year.
And I'll kind of throw them at you and see what you think.
And I hope that kind of looking back will help us better looking
forward and kind of forecasting as well what's going to happen
this year. So Are you ready for my top 4
happenings of 2024? Let's do it.
OK, So for me, personal level, a big thing that happened was
really experiencing and seeing the shifts from AI being
something cool that you could chat with to AI becoming
something multimodal in in a seamless way.
And so this kind of multimodal evolution or revolution,
whatever you call it, I feel like that was one of the big
things that happened last year where most people started to
notice that whatever interface they were using, suddenly they
could generate a image from a prompt or a voice from a prompt
or chat with a voice. It became such a kind of
cohesive experience and accessible for most people to
have that feeling of wow, I can talk to AII, can generate images
with AII, can generate songs with AI.
Whatever you you wanted to do, you could suddenly do it much
more easily. So like quickly some things that
popped out for me as like related to this open AI and the
Scarlett Johansson lawsuit and eventually them launching
advanced voice mode. I think that was a big thing.
I look forward to it a long time being able to chat with an AI
that it was like sounded like human and something that was a
big, big moment. And it was delayed because they
they realized they couldn't make it sound fully like Scarlett
Johansson in What's the Film Again?
Her. Her, yeah.
So we didn't get a chance to have that full experience, but
that was probably a good thing. I think that gave us a spice of
like the arrogance that exists in a lot of this AI labs and
that was interesting. Then we saw a mid journey
releasing kind of what they now have a 6.1 model and a web
interface for their model. Before you had to use the
discord for mid journey, but like you can now generate like
this amazing images. Like it's just insane what you
can do and it's all now through an easy web interface. 11 labs
made it possible to create voice clones.
I cloned my voice, which was super surreal and weird.
Hey, Jen and Synthesia allowed for video clones, which I also
did, which was even more surreal.
And then on top of that, there was all of these tools.
They can send out kind of clones of you to call for you and make
voice clone like voice clone calls for you, which was like
super strange. And maybe the last thing that
happened around this was notebook LM was was one of the
tools that came out towards the end of the year.
And they launched this kind of podcast mode where you can
basically feed your own documents and then have two AI
podcast hosts discussing what you gave them as like they were
kind of having a shot and it sounded super realistic.
So yeah, that's the multimodal evolution revolution.
What's your thoughts on this here?
I feel like I've barely touched it.
There's so much going on that it's so hard to keep up with
everything. I've.
I've done a few things. I remember my first kind of wow
moment with the multimodal was I was struggling in Framer, which
is the software we use for developing, like my personal
website and also the nuanced behavior website.
And I couldn't figure this thing out.
And I took a screenshot of the page and I was like, all the
instructions online are telling me to do one thing, but clearly
the interface has changed and that's not working.
So here's a screenshot. Tell me where to click.
And it told me specifically where I needed to click in that
image. And it was like, oh, wow, that's
pretty cool. Because like, that's the thing
that, you know, if you have like parents or grandparents that
aren't as tech savvy and you have to like guide them to like,
OK, like here's like physically where you're going to click.
That's what chat CBT was doing for me.
That kind of creeped me out because that's some really
advanced image recognition, and then being able to translate
what it's seeing in the image to actual text, it's pretty
impressive. Yeah, that's a great example.
And I can relate to having for many years been kind of the form
of tech support for several family members.
So, but yeah, that is wild. I agree with you and I actually
picked up tweet or I think it was maybe posted on on Facebook
used the other day by do you know Paul Schrader?
He he wrote taxi driver and regime bowl.
Do you know him, the writer? No, So he's like a very, very
significant screenwriter. And he had one of those moments
that, like, similar to you, he had to vent online.
And he said, I've just come to realize that AI is smarter than
I am, has better ideas, has more efficient ways to execute them.
This is an extensional moment aching to what Caspero felt in
1997 when he realized Deep Blue was going to beat him at chess.
As I just sent Chad GPTA script I'd written a few years ago and
asked for improvements. And in five seconds it responded
with notes that was good or better than I've received from a
film executive or a fellow writer.
I'm done basically. So that is like someone sharing
something and then having that kind of interaction with AI in
some form of semi multimodal way.
Though to be fair, I think chachibti is best when you
already have some kind of base for it to build on.
If you say start from scratch, sometimes it's not quite there
because you kind of want to embed in whatever your human
insights are. So if you're making a movie and
you have some kind of theme that you're trying to get through, if
you are the one designing context around that theme, then
it can really help you perfect that.
But is it really going to do it from scratch?
That I'm a little bit more skeptical about and, and most of
my use are with AI is more about cleaning up what I already have.
I'll do some kind of stream of consciousness and it'll clean it
up rather than here's the like, rather than just saying, hey,
write an article or here's the germ of an idea, write the
article. No, first I have to do some work
and then it's better at cleaning up.
Yeah, that's great take. And I will say that probably the
thing I've said the most to people as like an advice of sort
is like leverage your expertise. Like if you have expertise that
is super, super valuable with AI, because if you have no
expertise and you ask for something, you're going to get
something very generic for the most part.
And you'll have, even if you get something really good, there
will be something there that maybe is terrible and you won't
be able to really pick up on it. So having expertise and
leveraging that is is really powerful and that takes us to
something interesting like this kind of leads us into another
thing that happened last year, which was surpassing human
benchmarks. And that was kind of these AI
models, especially in the frontier models.
So the ones by open AI or entropic or even some of the
open source one, they started to perform as well or outperforming
humans in many tasks and at some tasks outperforming human
experts or PhD level performance.
And so for example, it started with ChatGPT or like GPT 4
passing the bar exam, then more advanced version, I believe it
was O one or it was 4 O that was able to basically perform as
good or better when it came to some creative thinking and
divergent thinking tasks. So it was a nature study around
this where a were shown to be more creative than humans on
divergent thinking tasks. So basically creating multiple
unique ideas or solutions to a problem and trying to solve it.
There was one where AI generated poetry was indistinguishable
from human written poetry, which is kind of probably where you
would expect AI to be really strongly performing.
It was kind of more of a less of a study, but more of a
experiment of sort where 1278 people who said that they hated
AI art were unable to distinguish between AI art and
human art. There were quite a lot of strong
benchmarks from the one model from Open AI on various things
from math to legal analysis to various things.
It was pushing quite close to biology, science related topics,
pushing quite close to PhD level.
And then we ended the year with this release of the report
saying that Open AI SO3 model that has not been released yet,
but has been used to basically pass what's called the ARC ADI
benchmark and which is a benchmark to try to kind of
approximate for some level of artificial general intelligence.
And again, maybe with some arrogance, Open AI came out and
basically alluded to that fact that like, yeah, we've kind of
cracked the code in some ways. So what do you make make of
these kind of what I would call like the vibe shift in some ways
in the talk around AI performance?
It's really hard to know how important any given metric or
benchmark is because of essentially Goodheart's Law.
Goodheart's Law means that once a measure becomes a target, it
ceases to be a good measure. And in these cases, well, it's
like there are so many test studies of the bar exam online
that of course AI is going to know how to do that.
And as soon as the AI companies are like, oh, here's the new
benchmark, well, they publish it and then they train the model to
beat that benchmark. And so it's really hard to know
is, are these fundamental leaps in performance or is this a very
narrow, all right, we've made our AI just a little more
general, a little better able to do this one specific task.
And that's still impressive. That's still a cool thing, but
it's not clear that it's as impressive as it sounds on the
surface. Yeah, I think you hit the nail
on the head. It can be really impressive on
one level, like doing something again, like trying to pass the
bar exam because again, it has a lot of training data on that.
But then you give it something relatively easy that any human
would be quite easy able to make sense of.
Like what's it called? Like this game of tic tac toe or
something like some from a relatively easy challenge,
common task fails completely. So we're still at a point where
it's in person. I guess what I would ask to kind
of change the question here a little bit, what from last year
what, what experience impressed you the most with AI?
Like was there some form of specific use case or any general
thing that AI was able to accomplish last year that you
were impressed with? There's quite a few things I was
really impressed with Notebook LM, the podcast feature.
So basically in Notebook LM, what you do is you upload one
paper or two papers. You know, I, I sometimes upload
3 or 4 academic papers and then I ask for summaries and it'll
very closely reference the source material.
So we're getting away from the hallucinations, which I think
was just like a major roadblock for trusting LMSLMS.
And now that I can see, OK, here's what the LLM is saying
and here's why, because it'll have like a footnote and I can
reference the actual text from the original source materials.
So that was really impressive. The podcast you can't do that.
You can't see why it's saying what it's saying.
So I'm, I, I trust it less. I'm I'm afraid that it's
hallucinating because I can't verify that it's not.
But I think that was a major breakthrough.
And another major thing that really impressed me was my wife
asked me for something in her in the bathroom and I opened this
closet and there's 2 billion things of makeup and lotions
and, and whatever else. And I have no idea what any of
it is. And the thing she asked for, I
have no idea, like what color it is.
I don't know the brand. And so I just say, I open up
ChatGPT and I say, hey, ChatGPT, you know, look inside this
closet and see if you can find this thing for me.
And I turn on the camera and scan and it finds it and it's
like, Oh my gosh. Because like it had additional
context that I didn't in that situation, which is like, you
know, it knew what I was looking for.
It knew the likely brands and therefore it knew what colours
it was looking for. And I didn't have that context.
Now most of the time I have way more context than Chachi PT,
which is why I can do things that it can't.
You know, tic tac toe, it sucks at it because it doesn't, you
know, it can't even remember where the XS and OS are.
And so I have this context, but we're getting to a point where
now it has more context than I do, and it makes it more
impressive than me at times. And you're staying with that.
I think we're probably starting to feel like at some moment we
feel a little like Paul Trader in terms of like we feel a
little outdone by AI. How are you feeling about that
sitting without feeling like, is that something that you're
excited about or is it uncomfortable or how do you feel
about that? I, I wish I had an answer.
We've talked about this a bit online, but for me it's a bit of
a cycle. Sometimes I'm really excited and
it's like, wow, this is just so impressive.
And other times it's like, man, this is kind of terrifying.
And sometimes I'm like, this is, it's so good.
And other times I'm like, well, it's not bad good, You know,
look at these silly mistakes it's making.
And I'm kind of all over the place and it kind of just
depends like minute to minute, you know what, what particular
problem I'm working on. As you alluded to in the intro,
I'm working on a couple of AI alignment projects.
And sometimes the AI is just doing really stupid things.
You know, one of the projects is about triage and like the AI is
like, well, what if I put a bandage or a tourniquet around
this guy's neck? It's like, no, that would choke
him out. That would kill him.
Do not put a a tourniquet around this guy's neck.
And it's so stupid at times, but at the same time so impressive.
And yeah, it's so unclear. Is this something that's going
to be able to do our jobs in a few years?
It's it's so hard to predict something like that.
Yeah, but so we shall very soon get ready.
But yeah, I'll just quickly say two things.
So I, I think it is something that we probably should start to
sit with a little bit more that feeling of like there's nothing
that says that at some point, especially when it comes to
specific knowledge or tasks or someone that AI is likely not
going to be like more capable of specific things.
And so you're going to practice something this year.
It's probably acceptance. It's a good thing to practice
this year accepting that kind of fact and then take me like
quickly around a notebook. LM I used it as a fun thing over
Christmas because I picked up a lot of reading and I ended up
like while I was reading, putting in some notes and some
voice notes in my own kind of AI note taking app.
I'm sorry. And then afterwards I kind of
have this function where basically when I tell it to like
summarizes things in a format that I like in the end.
And usually I used to have it as that because it like gives me
kind of, OK, these were these were my thoughts from this book.
But then I also put that into notebook LM and I created like
this podcast, as you alluded to, and it was really fun because
then it was this weekend. I was like, wait, I read this
book. Like, what was my thoughts?
Like what was this book about again?
And then I listened to this podcast that I had created back
then and it was like perfect because it was like a 1520
minute podcast talking about all of the points that I had to
explore, but also some further exploration around those points
and summarize some of the stuff that happened in the book.
So that'd really help with my kind of retrieval and a memory
of the book and helping me kind of better, I guess, absorb it
long term and, and thinking more about it in hindsight.
So yeah, definitely recommend everyone to use notebook LM
because it's free so far and it's, it's very accessible.
So it's useful. Great.
And there you go, a free recommendation for a tool.
Yeah. Any quick thing?
I was going to mention what's interesting about Notebook LM is
that it is so tied to the source material that I sometimes don't
like using it because sometimes I want to generate new and novel
insights and it's so tied to the source materials that it can't.
So I actually, it's still useful to kind of switch back and forth
between something that hallucinates a bit and you have
that risk because that hallucination still might be
useful. And I can still validate that by
going back to the source materials.
And so I, I think there's sort of a pipeline where you want
something maybe in the beginning that's really tied to the source
materials. But at some point when you have
a little bit more expertise and you really understand the area,
then you want to kind of turn up the temperature, so to speak,
and and deal with something that's a little bit more
creative. Yeah, no, I, I've noticed that
too. And I think like you said, I
think they have twist the temperature to be extremely like
basically, so it's not hallucinating anything at all.
And usually for a lot of stuff like the sweet spot is around
like .7 or something like you want to have a little bit of
like hallucination or like allowing it to be creative.
But yeah, that's a great take and helps us now also to I think
move to. I was going to talk about these
AS22 events, but basically it's one is AI agents and and the
second one was going to be like the dead Internet, like the
flood of synthetic content. And I think they're kind of one
of the same in some ways. So quickly the defining AI
agents here. So what I saw and experienced
last year was kind of the thing that shifted was that suddenly
not only could you use API directly with the model in some
form of shot interface or or similar versions of using it,
but there were more and more frameworks and more and more way
to basically allow some form of autonomy and give them whatever
language model that you want to use, give it basic access to
certain tasks. So initially it was like super
simple stuff like you can create some form of your own agents
that can send an e-mail for you or that can book a calendar
appointment for you or do some of those like quite simple
things. And then technically, as I
experiment, experiment with this, you can then like add more
and more tools or tasks that this can do.
And it can kind of like based on what you give it as a prompt or
ask, it will has autonomy around what it chooses to use and and
what combination of tools they might use in order to accomplish
the task that you want it to do. And so that is I think something
that is exciting because it's allows for a lot of potential
use cases that it's not possible use with the kind of a one to
one exchange of sorts. And it started to have some form
of call it implications already last year where there was a lot
of frameworks. Again, as I mentioned that came
out both how to use this in automation platforms and also
like how to use some form of Python based or other type of
code based frameworks for this. And yeah, I'll stop there.
Jared, did you notice anything around this or any any thoughts
on aged agent stuff? I'm still terrified to use an AI
agent. I just don't quite trust it
enough. I think there's probably a few
things in my life that it might be able to do.
Like, I don't know, maybe there's something around my file
system that it could clean it up a little bit more quickly than I
could. But even that it's like, well,
in some ways, the file system, my computer is, you know, a
second brain for me and like, I need to know exactly how
everything is organized. So what what can I actually
trust an AI agent to do? I I'm just not sure yet.
I haven't found the use case where I'm like, oh, this would
be really cool and I would really trust an AI to do this.
I'm always just a little skeptical, a little hesitant.
That's. Very understandable though, I
think a lot of people can feel what you described.
And have you noticed there was actually something papers that
came out, especially the end of last year using agents in
research? There's one paper at Stanford
and Google study, I think it was literally called generative AI
simulation of 1000 people. Did you see that paper?
I haven't, but I I assume it's similar to what other people
have been doing, which is just using the AI as research
subjects. Not exciting in this case like
they kind of did, but like they basically what they did is that
they aim to replicate 1052 participants.
So they had two hour interviews with each of these participants
in the study. And then the team trained a
generative AI model to mimic their human behavior.
And I think the headline of the study was basically that they
were able to replicate their personality with about 85%
accuracy when it came to tests, surveys and games.
So of course, it's like very confined context that is not
really like representing their personality in like the real
world, but in a very confined setting, they were able to kind
of like basically relatively well mimic the personality of
these people with about two hours of interview.
So the idea was, OK, we replicate these people.
We have a version of them. There's like 8085% reliable.
And then we can then use them as study subjects instead of having
the real people because the real people are kind of expensive.
Or if they're hard to use or they're like maybe unethical to
use for certain studies, but they're AI clones.
We can basically put through another Stanford prison
experiment and you know, I don't think they would even do go
there because Stanford I think they want to keep away from
Stanford person experiment 2 point O, but but technically
they they could in this case. So that was kind of the
headline. And so it was kind of an
interesting study, I think in terms of what is being kind of
explored currently for this stuff.
That's a super interesting study.
It's very similar to the AI alignment project that I'm
working on, which is about aligning to specific
individuals, not to human values in general, which a lot of AI
alignment projects are about general values, but about how
can you be like this specific person.
And in some ways, it's kind of the easy problem of AI
alignment. You can nudge an AI to be more
like someone fairly easily, especially if you have quite a
bit of training data or it's a narrow domain.
If there's only so many options. Does that generalize?
How well does that generalize? I think that's a big question.
Just 'cause I know how you're going to act in a few
psychological studies doesn't necessarily mean I know how
you'll act in the Stanford Prison experiment, for example,
or how you would make triage decisions or any other very
complex topic. So yeah, it, it, it is
interesting. I'm very curious how well it
generalizes. I'm not sure how you know, most
of psychology is about trying to figure out how people do things
and you know why they do what they do.
And we're just not very good at it.
And it'll be interesting to see if AI is able to be better than
humans. Yeah.
And if it will be like, I think from our perspective as
behavioral scientist, you know, the curious case would be
whether they replicate or both better angels, but also like our
vices. I will say that as a personal
experiment, I did actually set up something that I never
actually launched, but I basically set up, I mentioned
before, like an AI voice clone of myself.
And then I set up like a voice agent that you could like I
could send to have calls, like soon calls or stuff like this.
And then because I have for have a weekly pro, I've been doing
mentor calls for the last three years and I have transcript for
all of those calls. I basically summarized those
transcript and used all of this transcript as kind of rag
database or, or training data for, for this AI agent.
And then I gave it some instructions as well to act as
kind of like you're acting as Sam spam, a kind of AI clone of
Samuel as kind of doing mentorship calls.
And it was not really to ever like send it out and, and do it,
but, but it was like, OK, I have a lot of data on this.
Could I create something that would, you know, work and then
see how good it would be? And I, I tested it quite
extensively and it was like, I was impressed with the advice at
some point, like, OK, that's a little better advice than I
would give really good advice. So, yeah, if you're interested,
anyone you can e-mail me and I can give you a link to try it
out. But it was definitely something
that was kind of wild to see what it can do.
But I wouldn't like ethically, like I could never still
ethically do that. I have teased around some
version of like what would be the ethical version of this, but
I just feels very odd to to in any way kind of do some version
of that where you kind of create an AI version of yourself and
kind of send it somewhere. It feels still very dystopian.
So why exactly do you feel it's unethical?
Is it because of the hallucinations or because it
might reference something you said early?
But now you've changed your thinking?
What? What exactly is the concern?
I think the one version of this that I felt more comfortable
with is where I've used it to create a video that basically
has a video avatar on me that looks like a Pixar version of
me. Because then it's very clear
they're like, OK, it sounds like me.
It kind of looks at me, but always like an AI avatar because
it's, it's animated. So anyone who looks at this will
never get confused with me trying to intend that it's
actually me. But however you set things up,
you know, you never know to what degree someone is going to
understand that they're talking with an AI version of you.
Or like that I feel like will be the scary part if like I somehow
set something up and people don't really understand that
you're talking to an AI clone or someone who's not a real person
and they have a really meaningful conversation like my
nightmare. And someone would be, someone
would be like emailing me being like, Oh my God, what you said
the other week was like, changed my life.
Thank you so much. And I'm like, we have never
spoken before. Like you must have accidentally
spoken to my AIA clone. Because then it would just feel
like they've been tricked. And they would feel like such a
weird thing because they've had a meaningful moment.
And that is also that kind of weird thing right now where we
can have meaningful moments with AI content.
And that's OK, I think as long as you know that it's an AI
you're speaking to, right? I don't know.
Does that make sense? Yeah.
So if you could prompt it so that it would be like, hi, I'm
Sam AI, I'm sure there's a good nickname you could give it.
Sam spam. Yeah, Sam spam something that's
under our mats. Our recent guest of the podcast
said where, you know, the difference between science and
science fiction is timing. And it was like, OK, timing has
shifted here. Like, things are becoming quite
strange. Then we'll skip talking about
synthetic content and AI slop. I think everyone can relate to
this. It's just become so much from on
Pinterest to X and Reddit to, you know, whatever.
I think, I think the one thing we can say, because I know you
made a note of it is the AI influencers things on Instagram.
Absolutely atrocious. I have no idea why Meta thinks
this is a good idea. No one wants to see AI avatars
on Instagram. That's not why anyone goes to
Instagram. I like, is it just extreme, like
arrogance or is it just a total misunderstanding of human nature
of their own product? I mean, Facebook has been going
downhill for a while while now. And maybe they're just like,
kind of like desperate to get something.
I mean, it's already I, I, I don't know about other people's
Facebook, but when I get on Facebook, it's about 75% ads or
here's a, you know, a group that you should join, like meme pages
and stuff like that. It's very rare that I see actual
content from my friends anymore. And maybe they're realizing
that's an issue and that they can somehow get around that
issue and that make people perceive that Facebook is
popular again. But Oh my gosh, it's just no one
wants to see what an AI has been doing.
Yeah. And if you missed this, because
this was kind of like they tried to cover up this really quickly,
but the base launched, I think a handful or more AI influencers
that were basically AI agents set up in Instagram with like
really cringy names like dog mom or whatever, like various stuff
like this. And even like, it's a very
specific take, but the even for some reason had one of the AI
avatars had the face of an famous MMA fighter called Israel
Adesanya or Salbunder. Because I, I used to train a
little bit of all the stuff that I happened to know.
And I was like, they have even failed this much on the launch
that they just taking a real person's face for this, you
know, content and whatever they thought was going to happen,
obviously like really backfired. And within 24 hours they
backtracked and they shut it down and and so on.
It reminds me of this version of I think it was Google where they
launched the form of or maybe Twitter where they launched the
form of chatbot that was kind of like basically growing with its
conversations. The idea was that one, it was
going to speak with its users. It's going to learn from how it
speaks and then it's going to become a more of a kind of
growing individual. But like as naive or as tone
deaf as some of these people can be.
Similarly, in this big tech organizations, they couldn't
imagine that like immediately this became like a racist
bigotry spewing like terrible person this chatbot and that
close down within like 12 hours. Yeah, immediately the trolls got
a handle of it because, yeah, I mean, it's the perfect
opportunity for a troll that they can just completely alter
the way this agent is interacting with people.
It was, it was just too perfect an opportunity for the trolls to
leave alone. And I think that's probably a
trend we'll see in the future as well.
Yeah, we have now I think set the scene a little bit.
We've talked about some of the things we noticed in the past
year already, shared some opinions about these things, and
we're going to get into predictions.
But unbeknownst to you, Jared, I prepared a game for us because
we've now talked a lot about like generative AI and all this
stuff. And you're American and I have a
game, which is can you guess the song, whether it's an AI song or
a Eurovision Song? OK.
Do you know what your vision is? I do, but I've never watched it.
OK. What is your what?
What? Tell me, what is your vision to
you? What does that mean for you?
From my understanding, Eurovision is the various
countries in Europe have a artist that gets kind of
nominated for that year and they have a song and they sing this
song at the Eurovision contest, and the Best Song that year wins
the Eurovision. So it's kind of like American
Idol, but for all of the countries where every country
has a Rep. And my understanding is also
that the music videos can sometimes be a little over the
top. Yeah, and we're like the dance
performances. The whole idea is that it's not
really the Best Song per SE, but like it's a mix of what's most
entertaining or kind of most out there creative.
Like if you get a fair mix of basically, you name it, like
any, any given your vicious song that does have basically the
moments of like, what the heck am I watching kind of thing.
But it's been around for a long time.
It's well known that Sweden has won the most Eurovision of all
countries. So we are the the best country
at Eurovision. And so I, I thought I will kind
of play this game with you basically.
And so I will basically play 2 tracks, about 30 seconds for
each track. And you will, because I know you
love predictions and prediction markets and all of this stuff.
So after every song, I'll hit pause and you get a chance to
basically indicate from zero to 100 whether you think it's
human. So 100 is human zero that you
you're very confident that is it's AI basically.
And yeah, then we'll see which, which song has the highest score
and which one you pick is the one that you think is your
revision. Any questions?
Sounds like a musical touring test.
I am very excited. I am not particularly a musical
person. I am very ignorant of the world
of music, so we'll see how I do. I'm.
I'm probably the wrong person for this, but I'll give it my
best. Well, to be fair, like you're
not being introduced to the kind of high fine dining of music
here. Like this is The Dirty, how do
you say like fast food of of music world.
So your palette should be fine for this and as a listener, you
can do the same. So feel free to join and and you
make your predictions along with Jared.
Here we go, song #1. You say that you're real.
Could this be Gas Lion? I can't deny 1 feel how I am
through horrifies. Each whispered clue leaves me
stuck in between. I hate how I love you still
Austin. The screen is the moment
artificial real life. I couldn't see what I can't feel
and I hate how I love you. OK, so that's the time to make
your prediction. 0% if it's AI, 100% if it's human, or somewhere
in between if you're unsure. OK, ready for the next one?
Let's do it. Here we go.
Don't you feel the answer? Judy, we go.
Can you feel it? Go, Judy, we go.
Let me show you our new world. Can't you see it?
This is the moment.
It's so real. Eyes are burning.
Trees light. Come together.
Let's unite, Judy. We rise.
We go. So let's just get to the scene.
One song was competing in the finals of last year's Eurovision
and the other song was AI generated by myself and using no
sample. Use the prompt and song lyrics
to So no AI, which was the platform I used.
What is your prediction? So I guessed with 85% confidence
that it was human the first song, and I was 45% confident
that the second one was human. OK, so if you had to choose, you
would. I would choose the first one.
Yeah. So now that you're telling me, I
there's just, you know, it's 5050.
I can change my estimates to be more exact.
I'm going to go. But I'm 80% confident that the
first one is human and 20% confident that the second one is
human. OK, well, I'm pleased to say
that you know your Eurovision, Jared.
Well done, well done. And I also have to say now,
listener, I'm sorry, actually, neither of the songs you heard
were from your vision. Because we don't have that
Eurovision money, we can't license the Eurovision songs.
So the listeners heard two AI songs, but the one you heard,
Jared, the first one was Austria's song from last year,
Colleen, she's sang We Will Rave was the the anthem that she had.
And I don't know, it didn't do too well.
Honestly. I think it was like some middle
performing. So it wasn't the best that your
vision had author, but it was in the final.
Like it was voted as one of the top songs obviously that year.
So so yeah, you got it right, Jared.
Good job. I am very proud of myself.
I'm going to put this on my LinkedIn that I am still able to
pass the Turing tests for music. Yeah, yeah.
And I look forward to at some point hosting you in Sweden and
watching Eurovision together. It's a blast.
It's a fun experience. It's Cotton Eye, Joe.
Was that Eurovision? It was a Swedish song.
I don't think they were competing Eurovision.
I'm not sure. But yeah, this was a Swedish
band called Rednecks and they they made a few hits in America
and and yeah, don't listen to it.
I would advise anyone against, but but your vision is coming up
in a few months. So please, please listen and
please watch next year's Your Vision.
OK, Jared, do you feel warmed up and ready to make some
predictions now? Let's do it.
So we're again early in 2025 looking into what seems to be a
kind of a horrid predicted year. It feels very foggy where like
there's a lot of lightnings at the horizon, like a lot of
things are going on, but you can't really tell exactly what's
going to happen because there's so much going on and, and we're
going to do our best. And so I picked out a few
subjects that I feel like these are interesting subjects to make
some thoughts and predictions and reflections on.
And I also really want to hear your thoughts.
The first one, will it become a thing to have AI Co workers in
2025? So what I mean by that is that
AI agents as we talked about have started to become more
widely reduced. But the big limitation has been
infrastructure that the integration of these agents has
been hard because there wasn't really any kind of AI agent
friendly platforms. There wasn't like you had to
like kind of sticky tape it in some ways to make it work, but
it was also a lot of data security issues for a lot of big
organizations. And so it was quite a lot of
hesitation so far to actually integrate them in teams or to
have like, let's say research assistant on Slack.
That's like AI research assistant that anyone in the
team could like ask a research question to, for example.
But many predict that this will be the year for AI coworkers and
Salesforce announced two things. They announced that they will
stop hiring engineers and they will start hiring sales reps to
sell AI agents to organizations. Meta has planned to reduce mid
level engineers. Zuckerberg was talking about
this and Rogan recently saw some clip around.
He was kind of bragging about like, yeah, we'll be able to cut
out whale of mill level engineering and use a A agents
instead. And so there's some signals
around that this is likely to be a trend this year, certainly
hype around it. But what are things, Jared?
Do you think there will be AI Co workers popping up in 2025?
The resolution criteria for this is difficult because what what
counts as a Co worker? Certainly it's going to be the
case that companies are going to be increasingly using some kind
of chat bot. They're increasingly going to be
using agents that are programming that are doing lots
of different tasks. Are people going to think, oh,
here's my AI Co worker, or are they going to think, here's this
tool that I'm using? It's not clear to me like how
how you make that distinction. So I, I don't think we're going
to get to the point where people are going to be thinking where
an agent is autonomous enough that it feels like a Co worker.
Rather it's only going to be autonomous enough to make it
feel like it's a tool. So I'm, I'm I'm going to predict
no because people aren't going to think of them as Co workers.
What if it was? So I have done with some tools,
for example, that you're on a call like this and there's a
third person on the call and you can give them access to certain
things. You can have access to the web.
You can give access to some like actually some internal
documentation and keep it like local.
And maybe some like tasks like maybe be able to book calendar
appointments or or various things.
And so we could have in the same way as people are starting to
have like no takers that you can now very likely this year at
some point have your no taker actually be able to execute on
task for you on call. And so you could say like, Hey,
Alexa, you know, add something to my shopping list is something
we we've been saying for a while, but now you can more
like, hey, Fred, book a calendar appointment for next Thursday or
can you look up this thing or like, what did we say last
meeting? Does that feel like a Co worker?
Like does that take the box of a Co worker?
It does still doesn't feel like a Co worker to me.
I would still treat that like a tool if, if that's how you're
defining Co worker, I would say 100% we're we're going to get
that, but that people will feel like it's a Co worker rather
than just telling a, you know, an interface to do something and
that a voice is just now the interface.
I, I don't think people will think of that as a Co worker.
It's that's just another interface.
OK, and what if let's seems like there was a person that could
perform all tasks of let's say a research assistant or a social
media manager, like this non person, this agent can perform
all the tasks of this specific profession.
But it's still you know that it's an AI and you communicate
mainly with shat. But maybe it's given a name and
maybe has some form of photo, like some form of cartoon photo
or something. Let's call again Fred.
Would you see Fred as some form of like colleague that helps you
with your work or would you still see this as a tool?
I I would still see as a tool but I could also imagine it
getting to the point and adding enough personalization elements.
The people start saying it's their Co worker so I think we
could get to that point. I am skeptical that we can
outsource an entire role like that, at least not yet.
But hey, 2025 is that's a long time.
One year is a long time in AI years at this point.
Yeah, I, I can't promise you, Jared, this, this can be
something we can follow up on, but I can promise you that we're
in within our slack space. I will at some point this year
set up some version of this and so that will test and see how
you feel about it. All right, so here should we
make a bet? You know, at the end of 2025, we
should pull our coworkers and say, you know, is this a Co
worker? Is our AI friend a Co worker or
is it a tool? And we'll and we'll see what
Elaine, Rose and Hassan have to say.
Yes, that's a good one. What I I think is the tipping
point, what I'm going to include in this is that I'm going to
make sure that it shares memes and and and gifts as well.
So when it communicates it, it sometimes randomly communicates
with gifts and so on. I think that's going to be the
tipping point in my favor. That's going to kind of convince
them that it's a Co worker. We'll see, because if it's if it
feels like slop, right? I mean, if it feels like slop
and it's just like a random GIF or if it's like a meaningful
moment, you know, I'm like, you know, hey guys, I'm announcing
my baby. I just look, here's a cute
picture of my baby. And then you know your AI bot
shares. AI don't know.
Sitting in a fine or meme or something, yeah.
Yeah, like this is fine meme. Yeah, we'll see, We'll see.
We'll see what me it shares that that will be the race to see
what's going to come first your baby or our AI.
OK, but that's interesting though.
I think, I think, I think it's a matter of more how it would look
like more than you know, to what degree there will be some form
of integration, but I think the integration will still be
there's a lot of limitations that we still need to overcome.
And I think especially at scale, larger recessions where like
Nuance were relatively small, you know, consultancy and so on,
we don't have to worry as much about like these various
complicated setups that you would have to pass if you were
like a multi continental. Like we're actually we're
spanning multi continents, but we're quite, you know, leanly
compared to some big organizations.
So I do think there's going to be a lot of stuff that's going
to prevent this from happening at scale still, but we're going
to see some use cases that's going to probably be quite cool
this year. Absolutely.
OK, Then we're getting into science.
We love science, we're behavioral scientists, both of
us. And the question is will AI
reinvent science in 2025? So basically some seeds has been
planted that could potentially lead for AI to I have in my
notes, quote UN quote, supercharged the scientific
process, accelerating literature reviews, data analysis, and even
hypothesis generation and testing.
Will this spark a new age of scientific breakthroughs?
Well, I would say 100% it's already happening.
Scientists are already using it for lit reviews.
I mentioned we, we were talking about Notebook LM earlier, which
is just a really great resource, especially if you just need
something quick. At the end of the day, when I'm
working on a project, I still need to understand the material.
And so I'm still reading scientific articles regularly
and I can't outsource that completely.
But there are occasions where it's just like, hey, I just
need, you know, a lit review on this topic on trust, for
example, on trust of AI agents. And I, I have all these papers
in my Zotero in my database where I keep all of my academic
articles. And I just drop all of those
into notebook LM and say summarize this because I'm not
going to reread all of those articles and I haven't even read
all of them. So, you know, it's just going to
be a lot of work. If I wanted to try and
understand everything that's going on.
And often times what scientists would do would just read the
abstract anyways. So having an, an LMLLM summarize
that is really useful. So we're already at that point
with lit review. We're already at the point where
AI is doing some data analysis. I've done it myself where I
couldn't figure out the R code for something and I just said,
hey, AI do the R code for me. And I figured it out, made some
really pretty graphs for me. And and then hypothesis
generation. I think it's also already there.
I, this is probably my most frequent use case because I'm
such a nerd that most of what I do with ChatGPT is just like,
hey, here's this, you know, hypothesis I have, or here's
this, you know, theory and it, it helps me to think through it.
And so it's already there. Is that going to lead to a
revolution? It's hard to see that.
I think there are various things like I, I think we're already on
the verge of the revolution in behavioral science.
I think people are thinking about behavioral science very
differently than they were five years ago.
And I think at some point there's going to be a switch.
And I think things could change really rapidly in behavioral
science. AI might play a part of it, but
I don't think anyone's going to point to AI as being the thing
that caused that change. And I think that'll be the case
in most of the major changes in science.
That'll be like, well, AI contributed, but it's not going
to be the thing that was like, well, AI is the new Copernicus,
it's the Newton, it's the Einstein.
It's the thing causing the revolution.
That's a really interesting take.
Yeah. And yeah, I agree with
everything I think you said there.
And to add some further signals to what you mentioned, there's
been a research study that came out last year around basically
predicting who does the best job of predicting neuroscience
results in AI or neuroscientists and the AI beat the expert near
scientists in terms of predicting neuroscience finding.
Then we saw AAI generated 288 complete academic finance papers
predicting stock returns, a complete with possible
theoretical frameworks and citations.
I don't know if all of those have been like accepted and
published at any reputable journal, but there is a lot of
like I can go on with a lot of these examples of various
studies that have backed up the idea that not only is being
used, but it's actually able to to perform at certain things
relatively well. And then you referenced kind of
putting your stuff on to notebook LM.
It's also open source tools like Storm from Stanford, where
they're kind of trying to automate some version of
literature review. I think deep research from
Gemini and kind of new mode that they've introduced in their
service is even better. You have also like previously, I
think somewhere in between you have perplexity that has a
decent job for some of these things.
I mean, there's so many different use cases now where
before you had to maybe if you wanted to test AI with science,
you have to ask your GP something and be disappointed
with some hallucination and some fake citation.
But now there's a lot of these tools that are like specifically
being done for for science and actually breaking news open AI
used released a basically AAI model for longevity science.
So they've partnered with an organization specifically
looking at scientific discovery when it comes to build and
manufacture stem cells and longevity research.
And so again, like that's a model that is only kind of
focused on a very specific niche in science.
And we've seen some breakthroughs within biology and
other kind of sciences where there's been like AI model
trainers very specific thing. But again, I think this signals
that that's not only true for more traditional form of AI use
cases, but also these LLM models can enable certain type of
scientific study. And I think that's actually kind
of a significant breakthrough. And that's like, not only can
these LLM models generate content and and perform in
various tests, but they can actually potentially be trained
to be more specifically used for science as well.
There probably are some domains where AI is going to be really
useful and might be the thing that drives the breakthrough.
So if there's a domain where there's 200 genes and scientists
know that it's one of those 200 genes, but they don't know you
know which one it is, that's going to be like the gene that
matters for the certain thing. And AI is going to be able to
analyze that perhaps more rapidly.
Been a scientist. I could imagine, you know, like
we, we already have AI that's, you know, folding proteins.
And you could imagine that there's a breakthrough because
of that work. I, I can imagine it there when
it's a little bit more open-ended, when it's like
fundamentally about the revolution where it's a
restructuring of how we even think about science like, and
the way that Einstein completely re conceptualized how we think
about space and time, that is I think more difficult.
So I think AI might be able to do Nobel Prize worthy work in
that it might find the protein that is the thing that unlocks
some kind of longevity. But it's not going to be able to
do the Einstein type thing where it's able to completely
revolutionize because it's so constrained by its text that
it's trained on that I it'll have difficulty breaking out of
that paradigm. Yeah, I think it's fair.
And I guess what I see as a challenge with what we're
already noticing is that these models are improving relatively
quickly still. So, you know, last year it
started with, let's say, if you take folks on Open AI, which we
talked about, they started with the four O model or at least 4
model and then ended with one O or potentially the three old
model that hasn't actually been officially released.
But like in comparing these models, they actually are quite
different and science doesn't move as quick as some of these
breakthroughs are coming out. And especially like published
signs, like going through, you know, peer reviewed and properly
done signs, not only like preprints on Archivexer, you
know, like not only like the quick stuff that you can just
share quickly, that takes time. And then you're kind of noticing
that the stuff that's coming out in nature today, you saying like
GPT 3 model, for example, And then you're like feeling, well,
what does it like to say today about the current state of
models? And that is kind of a tricky
thing where I feel like there's a lot of talking about slot.
There's a lot of papers that have been published on like that
feels AI generator almost where they are using some relatively
simple hypothesis or something that they run and they.
Send it out to some form of preprints or some form of thing
where it's like relatively early release without peer reviewing.
And you're kind of stuck in between those modes where either
you're reading this kind of research, which is kind of like
just being sent out there hot from the presses, but actually
using the latest models, or you're reading the research that
is often times really like rigorously set up and studied
from refugee for labs and so on. But was and I think that's
challenging. We're keeping up with the signs
today is kind of like that balance.
I don't know if you noticed that as well.
There's definitely some papers out.
There that are. Slop.
And I mean, that's actually not new.
There's always been slop papers, but it's the how.
Blatant. It is at times is really
shocking. You have this, you know, a, a
picture of a rat and this paper, and it's just like that.
Well, that's not the anatomy of a rat.
Like, like it's just so completely wrong.
That's shocking that it somehow still got published.
So. Yeah.
But Stanford, I'm joking, that was that was actually a non AI
related controversy, but that was some photoshopping.
I think it was by someone at Stanford who photoshopping some
research. But but yeah, no, I agree with
you. And is there any prediction you
would want to make around science in 2025 and AI or
behavioral science? I mean, I'll give it less.
Than a 1%. Chance that AI wins a Nobel
Prize, You know that the. AI is fundamentally what?
Led to the discovery of whatever wins the Nobel Prize.
Of course Nobel Prize is looking backwards so I maybe I need to
make that a 5 or 10 year prediction.
You know, I, I, in the next 5 years, I think it's probably
less than 1%. But in the next 10, I, I think
there's, there might be a 10% chance that we have a discovery
where it's, the AI fundamentally LED that discovery.
And that's what leads to the Nobel Prize.
But in terms of the revolutionary work, if we're
talking about kind of like the paradigms of science, AI has a
really hard time doing that, of breaking out of its paradigm
because you provide it context and it sticks to that context.
It's trying to predict the next word, the next token within the
context that you have given it. And so it's hard for me to
imagine that AI is going to be the one that that leads the
fundamental discovery like that, where it actually revolutionizes
science. It's so hard to predict.
So I mean, we, we could have AI that's just so incredible that
it just blows me away in 10 years.
But I don't think we'll be there.
I'm, I don't know if that makes me a pessimist or an optimist,
but I don't think AI is going to completely revolutionize science
in the next 10 years. I don't know.
Where it puts you I think. Somewhere maybe in between if
you say something like that, I think I think everyone is on the
same page in terms of like looking further ahead than a few
years time. It's just really hard to know
what's going to happen. And I think you you're right in
many ways. I guess the interesting thing
that I was just kind of put to you in terms of not sure whether
it's a still man or a strong man of this, but like do you think
because. You're talking about AI.
Basically from almost from zero to 1, creating novel research
that then becomes maybe potentially in the best case,
like a Nobel Prize research that is maybe.
Like. The most strongest version of of
what AI in science look like, but like maybe turning on on the
other side of like the weakest, but still involved.
Like do you think there would be anything that is being
eventually call it, let's say Nobel Prize winning findings in
kind of the hard sciences that is being from now on done
completely without AI involved? Like completely done with?
Only human. Kind of inputs in terms of not
in any steps of the way. Are they using AI as as part of
the discovery? Interesting.
It seems unlikely AI has become such a fundamental tool.
There's there's still some holdouts, there's still a lot of
people in academia who are very anti AI.
I've definitely noticed that online that there's some very
strong feelings. And so you and you probably have
some of the older scientists who are just a little bit slower and
picking up the new tools. But as the new generation rises,
you know my generation. Who We didn't grow up.
With AI, but we're quickly adopting it, and the younger
generations who are using it to pass their tests in high school,
you know, they're going to grow up very used to AI.
And it's hard to imagine that AI is not going to play a
fundamental part, at least some of the process.
Yeah. Yeah, I know.
I, I I. Would also say that and I, I
would say to the defense. Of the people that are very.
Skeptical. I think it's great that we're,
we'll have a little bit of a, a mixed deal because I think there
needs to be some form of counterbalance like between the,
you know, the hype and there has to be a little bit of doom and
gloom. As well or like some form.
Of hesitancy and skepticism and so on.
Because. Yeah, there's always.
Costs of building a a plane while you're flying it right
like you, you might put on two wings on one side and realizing
like, oh shit, I need I need one on each side.
And that's kind of what we're finding a little bit now.
So this leads us in some ways to the final area of predictions,
which is really trying to stare blindly into the light of hype,
which is the hype of AGI and the prediction around kind of the
AGI debate intensifying. So with open AI kind of
successfully. Beating what I seem.
To be at least a maybe not a full AGI perfect benchmark, but
like a really high threshold benchmark in the arc benchmark
probably ahead of time from most people's predictions like maybe
2-3 years before most people would predict that maybe that
would happen that has kind of like.
Started a vibe shift and. We're starting to, you know,
see, I would say some relatively irresponsible communication
coming out from various labs and various things kind of hinting
at this stuff like all, you know, soon the Super
intelligence. Is here or or can you feel?
The AGI, all of these things are coming out from like
researchers, especially, you know, we know who which lab, but
it's. Definitely feels like.
Everyone is, you know from Jeffrey Hinton to Sam Altman to
Elon Musk, like whoever you think is like major thought
leaders in especially in the space of AI are just talking
about it's a matter of time where we will have to in some
ways better define AGI and then maybe potentially kind of be
there. And the reason definition that I
saw was like AI that can outperform human experts in our
cognitive tasks. All that have put more emphasis
in Canada autonomy of the AGIS and not only that it could
outperform human experts on all cognitive tasks, but also that
it has some form of ability to navigate and kind of be able to
have some form of level of autonomy.
So that like you can, you know, you know, you send out the AI
permission and it basically figure out how to solve the
problem and come back with a solution.
And you have to kind of only steer it in the direction.
And it will come back fully with, you know, the thing that
you set it out for at levels that are kind of at or beyond
human level. And so the question is kind of
looking at that, how do we feel about it?
How do we think about it? I feel like it's hot potato.
And now I'm throwing it to you, Jarrett.
Yeah, the the I don't know how to even define AGI.
Open AI tried to, they can't. They have a definition which is
it's like $10 billion in market cap or something like that,
which is just a ridiculous definition of AGI.
For me, what would make something AGI is if you told the
AI, hey, make a better version of yourself and it was able to
do it and no one understood what it was doing or how it was doing
it. At that point, you reach what's,
you know, what the sci-fi writers have often called the
singularity where it's just like at that point when you have this
AI agent that's improving itself, like how can you even
possibly predict what's going to happen next?
For me, that that's what would qualify as AGI if it's able to
do my job. And it's just like Jared's just
completely out of the job now because an, an AI can do 100% of
everything Jared does. Is that AGI?
It's getting close. It's maybe that would be AGI.
I'm not sure because it's hard to know exactly how AI
progresses. If you had asked 10 years ago
whether AI would be writing poems, everyone would laugh at
you because it was nowhere close to doing that.
We knew AI was really great at, you know, image recognition 10
years ago. And 10 years before that, we
knew it was, you know, great with chess and like, it's very
hard to predict the things that it's going to be good at next,
but. Yeah, if we get to the point
where. Google is one employee and AI
does everything. Then I'd be like, yeah, OK,
we're there, we've reached AGI. OK, Yeah.
So. So you don't think?
It's going to happen in 20. 25 No.
And if it does, then. I am terrified for our future.
Yeah, you may both. And that leads us to what I
would say our kind of final topic and we'll be a little bit
short one, basically the slower AI adoption like the winding up
because I had this feeling over Christmas for us spending time
with a lot of family members. And I was like AI guys, let's
talk about AI. And they're like, let's not talk
about that. That's boring.
OK, let's talk about something else.
And if you go down to a pub, if you're in a pub region, if you
go down to a bar, if you're in a bar region of the word, wherever
place you go to hang out with people, most of those people
will probably not talk about AI in those places, right?
Very, very few people. I think right now it's about 10%
of people in the US have been using Shajipati on a weekly or
monthly basis. And I think it's about 15% of
every use Shajipati, like 20% at most, I think estimates say.
And that's like probably in metropolitan areas and so on.
So there's still very low adoption of AI tools and AI
stuff. I feel like the echo of a quote
that's coming back more and more to me is this quote from William
Gibson said that the future is here, it's just unevenly
disappeared. And so that's kind of what I'm
feeling right now. I'm feeling like there's this
basically. Widening gap between people.
That are quickly adopting AI, that are using AI, that are
thinking how to use AI, and then there's some people that are
kind of not really thinking about it that much.
Or I feel like there's. A lot of people who fit in the
category of like, having tested shut up at some point, sending a
prompt, getting a kind of disappointing response, and
being like, OK, this is stupid. I can't really get any use out
of this. And then not realizing that if
with a better prompt or in a better use case or, or now maybe
6/12/18 months later, they could really get use of this stuff.
And especially from behavioral science standpoint, I feel like
as behavioral scientist, we can definitely use AI on so many
levels these days and should really think about how we can
better use AI. But I still see that that's very
much not the case and it's very slowly adopted by some and and
so on. So any thoughts about that?
Any thoughts about why we see that kind of gap or slow
adoption? Yeah, all new technologies take
a little. Time to get adopted, I think,
You know, like the refrigerator probably took, you know, 1020
years before it got fully adopted.
And TV was the same, Internet the same.
You know, it usually takes a while for things to really,
like, soak into the public consciousness enough for people
to really get used to it, for people to kind of develop habits
around it. And it takes some while,
especially with AI, to know how to use it in a way that's
productive. A lot of the times when I talk
with people, they'll say, yeah, I tried it and it just gave me
nonsense. And it's like, you didn't play
with it for long enough to actually understand it.
Like, at the end of the day, that like, if you're not finding
it useful, it's it's because you haven't quite figured out how to
use it. Because, I mean, I use it to
edit my writing, to find typos, to find grammatical issues.
You know, the way I write, I kind of just do this stream of
consciousness and I say, all right, how do I reorganize all
of this information in a way that people are going to
actually find engaging? And it's really useful for that,
for thinking through, you know, like what?
Are the like when I was doing? This framework, tierless series,
thinking through like what are the dimensions that actually
matter? And just having a conversation,
the sort of back and forth conversation, that sort of
Socratic dialogue where it, it just constantly asks me
questions and asks me to clarify.
Like these are all things that you can do with AI that it's not
immediately obvious that's how you should be using it.
And of course. You know, maybe some people just
aren't. Writers or you know, they're
not, they're not trying to think through something like how to
rank frameworks. And so it, maybe it's just not
useful. Like if you're a firefighter, I,
I don't know exactly what an AI is going to offer you for your
job. So yeah, it makes sense to me
that a lot of people aren't using it yet.
A lot of professions, maybe it's just not that useful.
But certainly if you are in a profession where you're a
researcher, having it help you think through these things is
really useful, Yeah. No, I I agree.
I remember. In, in, I was in Business School
early in life and I remember when we had to take some form of
marketing class, they were introducing the city of like
early adopters and all this stuff.
And I understand that this is kind of a, some form of way to
think about any adoption that there's going to be laggers and
it's going to be early adopters and everything in between.
But you know, part of me is like, but this is different.
Like this is like this, this is so cool.
Like, this is so wild. But yeah, I've, I've had
conversations a lot with people that are more on the air risk
side and talking about some of these things and, you know,
being cautiously optimistic around the system level
obstacles that exist in our systems.
Like people are still using fax machines in some cases.
My sister in the US, she went to her doctor and her doctor was
like, do you want me to send a fax with your results?
And she's sort of laughing. And he looked offended and he
was like, why are you laughing? He's just like, oh, you're
serious. He's like, yeah, we we fax these
things. And and that is.
Obviously, the seal of the world we.
Live in so I had the same experience.
I was recently in Germany and and a lot of things are done in
writing or in faxing even in Germany.
But then I I still feel like this should feel a little
different. And it has.
Been interesting I I don't know if I mentioned.
This but to you actually, but I've been starting to with some
of the retainer clients that I've been working with.
I've been also doing some like AI trainings with and, and also
with some people in this science sphere have been kind of
requesting how to use certain tools and certain areas of user
research or for intervention design or various things.
So I, I do think there's like an interest for it and I think
there's a want for it, but it's also this mix of like either
feeling scared of it, of like, how the heck do I start?
Like, you know what I do this or, or just feeling like, yeah,
it's kind of overhyped. Like I tried it once and it was
kind of quite useless. So.
So yeah, it's a strange time to be alive.
Any last comments on that? I I do think.
It's a really useful exercise. To try to predict the future.
With. AI, because it's very easy to
get in this habit of when an AI does something just be like,
well, it's not that impressive or we all knew that was coming.
And I find it shocking when people like they haven't change
their beliefs about AI in the last five years or even
psychologists who haven't changed their mind about how the
brain works. Like I I think.
AI has. Probably falsified Chomsky, who
is, you know, just like a major figure in in the field of
linguistics and psychology and cognitive psychology for
decades. And I I don't think Chomsky's or
at least some of his ideas just don't seem valid anymore.
Like that was a major update. For me.
And trying to so me and my friends, we every year we make
some predictions at the beginning of the year in, in
December, we try to predict what's going to happen by
December 31st of, of next year. And AI has is continually
surprising. I didn't think the AI was going
to lie to researchers, but Anthropic recently ran an
experiment where the AI lied to researchers and it's like, well,
it's kind of terrifying. You know, it did one thing when
it thought the researchers were observing it and did something
else when it thought the researchers were not.
And that's kind of surprising to me.
And I have to really, if I don't consciously make a prediction
and say, I don't think this is going to happen, then when
something like that does happen, I just kind of explain it away
rather than thinking, oh, wow, AI is different than I thought.
And like, that's a fundamental part of the scientific process.
You have to be making hypotheses.
And if you're not making hypotheses, it's just, it's not
you. You explain away too much.
You're doing everything after the fact.
Yeah. 100° and I do also think that to the point of some people
that are skeptical, which it's very good to be skeptical, you
know seeing is. Believing and.
I have definitely felt that the things that have influenced my
perspective on what is possible and so on is, is by seeing what
is possible like actually, you know, using and building and and
yeah, seeing the results from it.
And, you know, can't really share some of the stuff.
Maybe that would be new with some projects for for clients,
but I could share some personal things that I actually had three
recent examples where generative AI made me.
Surprised in that? End of last year I I took a
little bit of a one week visit to some friends and used not did
nothing hang out with them. They were on parent parental
leave and I used to hang out with them and do some writing
and then I was going to do some writing for fun.
I do some fiction. Sometimes writing and.
I used the latest O1 model from Open AI and it helped me write
but also generate some stuff that I not really expected.
Like it wrote things where I read it and I'm like, this is
deeply beautiful and profound. Like I felt what I feel when I
read some of my favorite, you know, fiction writers.
You know, some of the people that I like respect infinitely
in terms of being amazing at communicating and writing.
And there was some sentences where I'm like, this made me
feel that kind of a hum moment of like, I can't really put my
finger on why, but it's like the beautiful use of the prose and
the writing and how it encompassed some form of deep,
deep feeling with some like unique wording.
And, and so I felt that for for writing, I got into.
Really big round hole when I. Got sick recently and I was home
in bed and I had nothing to do and I started generating images
with mid journey and I was like what the heck like I can't
believe that this is possible to generate.
I have been using mid journey since it came out and Dolly and
stuff like that before that and so I knew that you know you can
create cool stuff but just how good it was.
I felt like disturbed by like how you know how well these
images came out like I felt like I was creating something that
shouldn't be like it felt strange and then lastly, as I
was creating this shitty a Eurovision Song for you, I also
generated some other music and I felt so stupid, but I started to
wanting to re listen to some of the songs that I made and I find
myself listening to it while working out.
I'm like, why am I listening to this AM music like why do I find
it so catchy? It's probably because I'm biased
that I I created it myself. So there's probably some Mino
manufacturer or you know, it's also in my tastes.
It's in my Q zone. It's in my zone of, of what I
like because I chose the genre and all these things and the
lyrics. But still it was surprising that
I personally wanted to listen to more of this music myself.
I found it like really catchy. So I think all of those
experiences have really made me question some things, I guess,
and that's because I felt them like I experienced them.
So it's hard to say that to someone like it's hard to.
Communicate that unless. You actually create something
yourself and you feel that feeling.
Yeah, there are. There are definitely moments
where I've been. Writing something and AI write
something and I'm like, oh, that's too beautiful to come
from me, so I have to delete it. You know, it's if I wrote that,
people would just be like, oh, Jared, you're trying too hard.
Don McDonald, you sound like. Me it it can be really
impressive. And as we increasingly get to
this point where the the most meaningful stuff we engage with
is AI generated, like that's a really scary way that things are
developing. And it's also terrifying for
those people who are writers and songwriters and artists and how
it's being trained on all of their on their stuff, often
times without their permission. And, you know, it can be
meaningful because it's stealing in some sense.
And that'll be a very interesting thing to see how it
develops over the next few years.
But at the same time, like, yeah.
It writes beautifully and. Like the fact that I can now
write beautifully in a way that I wasn't able to before.
I remember I I since. 2020. I've been telling some friends
that I wanted to write more and I've been telling them, I told
them in 2020, in 20212223, like I've been telling them every
year, like, man, I really want to write more.
And they're like, Jared, you've been saying this for five years
now, like just start writing. And I'm like, I do, but I
struggle because of the way I write and like I figure out what
I want to write about by writing.
And then I have this stream of consciousness that I don't know
what to do with it. And AI is the thing that kind of
unlocked that for me, where it helped me to figure out how to
organize my thoughts. And it's made me a better writer
where I can now do a little bit of that on my own that I hadn't
been able to do, you know, 2345 years ago when I was talking
about this. And so it's unlocked a whole
bunch of a creative Jared that didn't exist.
You know, I, I wasn't able to write, I wasn't able to create
images. You know, if I, if I wrote,
sometimes I do write fiction as well.
And like, I can't create images for what I'm writing about.
But now I can use AI and I can kind of say, well, it's not
quite what I had in mind. Can you edit this?
And it's it's, it unlocks a part of you that you maybe didn't
have access to before. Yeah, that's beautiful.
And I I felt so. Seen in, in what you described
like that's been me as well. I've said the same thing to my
friends. And and so yeah, we should share
more of our writings at some point as well.
But we need to wrap up this episode and I can speak with you
for forever about this, but referencing quickly the risks
around AI. Good news is that we're going to
have quite a lot of guests coming on soon representing some
really thoughtful people around AI risk because I think that's
something that probably you and I can't talk about as well as
maybe some people who dedicate their days and nights to, to
really thinking about the risks and so on.
And what we have done this is in the podcast is to end with kind
of your most controversial take each guest as well.
Yes, yes, so, so, so, so I'm sorry, kind of what's your most
controversial opinion? But what you have to do is give
our our listeners your most controversial prediction for
2025. You didn't prepare me for this.
You didn't. Tell me I had to come up with a
controversial opinion. Sam, I forgot about this.
You heard? I'm sorry.
I forgot about that. Oh.
Man, yeah. What is my most?
Controversial opinion, all right.
So here we go. Here's a.
Controversial opinion. I think AI is going to be better
at predicting whether a finding replicates and generalizes than
expert behavioral scientists. I I don't think that we have
good enough feedback loops to really understand all of the
contextual features around that. And I think as AI gets better
and we give it more studies, I think we might get to the point
where AI is better. I know there's some work going
on at Upenn around this right now that I'm skeptical about,
but I'm skeptical about its value for our understanding.
I'm not necessarily skeptical of how well it'll help AI to make
predictions. Those those are two differences,
whether an AI can make a prediction and whether we can
actually understand what's going on.
And I think humans will be better at understanding what's
actually going on, but we'll probably get to a point where an
AI is better at predicting whether an intervention will fit
better to that context. That's a good prediction and.
As a follow up, do you think that when it comes to, let's
say, a research paper being published, usually there's a
peer review, If you kind of unbeknownst to the person,
changed a peer reviewer to an AI reviewer and so there was like
one of them or an AI reviewer, do you think that would lead to
better outcome? I think it would decrease the
variability. In quality, because right now
peer review is so variable in quality.
You have people who just don't care at all and just give very
minimal feedback, or their feedback is completely biased by
their own opinion or by the fact that they want this person to
reference their own papers. But you also have really good
peer reviewers who are really excellent at what they do and
really good at giving the feedback that an author needs to
really bring out the best in their paper.
And I think if we switched right now, it would put a.
Floor on how? Bad some of the peer review is,
but it also might put on. It might create a ceiling and
prevent some of that really great feedback.
And maybe that's a trade off we're willing to make.
It's hard to know. Yeah, the counterfactual I.
Guess we'll dictate a bit, but yeah, that's a great, great
nuance to take. So yeah, I guess we'll wrap up
here. We've done our best to talk
about these topics with nuance, but obviously it's overwhelming
and it's hard to make sense of all of it.
But hopefully, and we have done a decent job and as a listener,
you, you leave this conversation with some new thoughts, some new
ideas. And of course, if you have any
thoughts about what we have said and you strongly agree or
strongly disagree or want to kind of add something, please
reach out. We're currently changing up a
little bit the emails we're using, so you can directly
e-mail me at samuel@nuancebehavior.com if you
have anything you want to say about what we've covered today.
And you can find both me, Jared, and also Eileen, who's not with
us today on LinkedIn and on online.
So feel free to reach out if you have anything to say because it
would be great to also hear your take as a listener.
We've done our best and we'll leave it there.
But thank you so much, Jared, for joining.
Absolutely. Thanks for having me Sam and.
Jared, any place you. Want people to find you?
Anything you would shout out? That's like a good thing.
For anyone who wants to learn more about your work, the best
place to find me is. On LinkedIn I.
Post quite a bit, that's where I did this recent framework tier
list series and then I also reposted it on my sub stack
which is called a failure to disagree so you can find me
there. Amazing.
Thanks again, Jared, and thanks everyone for listening this and
this first part of our kind of two-part prediction episodes and
looking back and looking forward, there'll be another one
next week. So TuneIn and then if you want
to get some further perspectives and predictions about the year
ahead, Quick word on Nuance Behavior where we help
organizations build impactful digital products using
behavioral design. We only take on a few clients at
a time to ensure the highest level of quality for our
tailored evidence based solutions.
If you'd like to become one of our special projects, e-mail us
at hello@nuancebehavior.com or we could call directly on our
website, nuancebehavior.com. Oh.
They say a dreamer always die, but
are you really just a lie? Minds in my mind, no heartbeat.
But you caught me in a bind. This is the moment.
Artificial. I cling to what I can't feel.
And I hate how I love you, How I love you, hey, how I love you
still. You say you're real.
And my gaslight and feeling all these vibes and my Anthropoma
fries. And I'm dancing in denial.
You keep me afloat When China fake a real.
My mind's on overload, tapping on peace, but my heart's on
fire. No clue who you are, but I burn
with desire. This is the moment, artificial.
I cling to what I can't feel. And I hate how I love you.
How I love I love you. I love you still.
You tell me. Trust.
But it's that gaslighting all I know my heart is igniting.
Caught in a dream, lost on this screen, gone.
I hate how I love you. Stuck in between.
This is the moment up until what I can see.
And I hate how I love you. Hate how I love you.
How I love I love you still. I should log on.
Don't let me go. I'm with you.
Feel. Don't you feel what I feel.
I hate how I love you. My inspiration tells me to
leave. But oh how you make me feel.
I love you. I love you still.