Welcome to the Cerebral Valley Podcast.
I'm Eric Newcomer. We're calling this fear
mongering and forecasting. We're looking to the future.
We're making predictions, or really I think we're going to be
relying on AI and chat bots to make our predictions and then we
will shit all over them. I'm here with Max Child.
Hey, Eric. And James Wilserman.
Hey, hey guys. Ready to make some hot take
predictions here? We're gonna use both Open AI,
Chaji, BT and Anthropic, so we'll also get sort of the meta
sense of who's which bot is making better predictions.
But yeah, excited what? What is AI if not constantly
looking to the future and charting out what could be given
the rate of growth we've seen so far?
So it's very fitting that that in our last episode before this
Ruebel Valley AI Summit on November 20th in San Francisco,
we are going to fiendishly fantasize about what could be to
come. This episode is presented by
Breck's The Financial Stack that founders and VCs can truly bank
on. Imagine what your founders could
do with their runway if they had a banking solution that had no
minimums, no transaction fees, and 20 times the standard FDIC
protection. Plus, they could earn an
industry leading yield while maintaining access to funds
whenever needed. Brex simplifies financial
services for startups so they can focus on building.
Connect your portfolio to the financial stack that one in
three US venture backed startups already use.
Check out brex.com/banking Dash solutions.
Without further ado, James. So for this episode I asked both
ChatGPT and Claude the same question to come up with a list
of predictions about AI for the year 2025.
I also asked the predictions to be hyper specific and
falsifiable so that by the end of 2025 we could easily
determine whether the predictions were correct or
incorrect. The predictions I said could be
spicy, medium, mild. However, however hot Thai spicy.
I didn't get any pepper emojis out on the on the output.
Maybe I should ask for that. But the, the, the predictions I
was, I was pretty happy with how they came out and I asked, you
know, relevant to the AI industry and then I also asked
to get a probability estimate for each prediction.
So then you guys are going to be able to provide me kind of the
over undertake on each prediction.
So we have our own prediction market basically.
Exactly. We basically created a bunch of
possible prediction markets around AI and you guys have to
decide whether you're short or long each prediction.
Nice, how's that? I love it.
Is this is this GPT 01 or is this?
Yeah, this is the ChatGPT 01 preview and Claude Sonnet 3.5.
OK, so the the most thinking oriented AI models.
We're paying the big bucks here. We're premium subscribers for.
We're we're bankrupt now they're running, but no, these these are
the best AI can do in a consumer consumer product.
It it feels if I was pretty impressed, like I was kind of
skeptical actually, that this would work that well.
You know, I think AI has a tendency to provide sort of
generic takes in a lot of ways, but if you prompt it correctly,
I was able to get these really specific, really specific
predictions that. They could keep this is for 12
months from now. Like what's the date?
You have the year 25. Yeah, by the end of 2025,
December 31st. OK, cool.
All right, let's do it. All right, we'll start with O1
preview from Open AI in Chacha BT.
So here's the first prediction So.
We have not heard these before. To be clear, Max and I are
coming in blind. Unless.
You guys, you guys are Co founders so maybe you're in
cahoots. Nope, I got nothing running
blind. All right, so first prediction,
Open AI will release ChatGPT or GBT 5 exceeding 10 trillion
parameters. What do you guys think?
How many parameters is 4? Again, one trillion.
The bots are already smarter than me.
What is that? You're like what's a parameter
isn't. Isn't it like 400 billion?
Can you can we get the number of the current it's like 4 or 500
billion, right? 1.5 trillion.
OK, let's add the. Elevator .76 trillion Is the
rumours OK? 1.
One to two, OK. So this would be, you know.
What's it predicting to rate how many?
A 10X like like A7X basically or whatever.
And we think the current is 1.7 something.
Yeah. Also the the probability for
this prediction is 50% so. Oh, over smash the over on that.
Wait, wait, wait, wait. Sorry, it's is it making a claim
about the name of it? Because I feel like I, I think I
was just listening to Dario on Lex Friedman and like, I feel
like half of his, like Hemming and hawing about what they're
gonna do is like not committing to the name of various models
because you know, they're doing all these like half updates and
stuff. So how much is this a prediction
about what the model is going to be called?
I think we can decide there, but I would say both have to come
true. The GPT 5 naming and the 10
trillion. I think, you know, if I'm
reading the tea leaves of what open what 01 was thinking and
reasoning about here, maybe they're saying, hey, 10 trillion
is a big step up to the point where it would have to almost be
called, you know, GBT. Five, I mean, I don't know, this
is kind of pedantic, but I would smash the over on a model that
it has 10 trillion parameters, but I don't know if they're
gonna call GBT 5 exactly. I feel like Altman has been you.
Committed to over already, I think the.
Name. OK, fine, I'll commit.
Why are you guys so skeptical that it's gonna be called GBT 5?
I'm just chaotic on the names. The names are bad at the moment.
Like a circle that we get you introduced as like open AI chat
GBTO one like they they need to like I I do I mean anthropic has
more colorful names, but they're getting chaotic sure, but so I'm
just take the under on the basis of the name in terms of the
parameters yeah, I'm probably bullish.
I mean, thankfully it's a compound prediction, but yeah,
the under. OK, All right, Well, I will
stick with my over, but yeah, I mean, I think they're basically
running into like the iPhone 4S5 S 6S problem where Apple used to
give the like in between phones, like, you know, an S instead of
a new number. And then I think the marketing
team realized like people want to see the number go up.
Like we don't see, we don't we don't buy new iPhones unless the
number goes from 14 to 15. So I think, I think This is why
GBT is trying to move off these numbers because I think they're
like, nobody wants the new model unless we give it a higher
number, right? And they're like, they don't
even care if we introduce like 4 O and voice mode and you know,
O1 and all this stuff. So I just, I feel like Altman is
severely foreshadowing that they're gonna drop these
numbers. I do think you're predicting.
Yourself. No, I'm taking the over. 50% is
a good price. I just think I understand where
they're coming from with the names.
James, are you making predictions?
You'll give us a quick your quick take.
Yeah, I'll just react to your guys takes.
I think that yeah, you guys are way too focused on the naming
conventions. I think the reason they are
calling things like, you know, 40 and 01 preview is to try to
like, you know, downplay the model a little bit before the
next evolution of the model. Like I think they will stick
with the GPT 456 naming convention, but they will need
it to be kind of step change, you know, improvement on the
model. So I do think A10 trillion
parameter model will get that name.
But there's an open question. I mean, this is gonna be
probably coming up throughout, just like how monumental is A10
trillion parameter model change, right?
Like if it's true if they have consumed basically all the data
there is to consume and set that data has its limits.
That's fair. Then if there's not a step
change in the model capabilities, maybe they don't
give the new name. OK, all right, let's move on.
Good, good first discussion there.
Are you over or under though? I never got that part.
I'm over, OK? I mean if you want me to make
make my own predictions. You've got to come in.
I just. Say I just put.
I'll say where you are. I'll take the over.
All right, so next prediction from Chachi BT we have by the
end of 2025, at least three countries will have implemented
regulations at the national level specifically governing the
use of AI to diagnose healthcare disease.
And what's the price on that? 70% probability.
Short. Way short.
I just think it's really, it takes a long time to write
legislation. I mean, you're saying like
national level entities are going to write complex
legislation about healthcare and AI in the next 14 months.
Yeah, yeah, it's. Interesting, though I'll grab.
Go ahead. I'll let Eric answer, but you're
just I'm. Taking the over, I mean, I don't
want to just do the opposite of Max every time here, but I do
That's good. There are a lot of countries.
There are a lot of countries we've actually seen aggressive
AI regulation already. There's clearly a lot of
appetite. And I don't think this
prediction requires some sort of comprehensive, they just need
some regulation touching sort of health, health diagnostics,
right? Yeah, I, I guess.
I guess the question is like, is some random EU regulator issuing
a decree count as 35? Countries.
Yeah, exactly. I think it should like I think
it. Should.
Well, yeah, of course you think it should.
Like, yeah, but I'm just saying like, you know, I'm this.
Episode isn't gonna be about the nature of predictions more than
this underlying. Game.
You know, for every good game, we've realized the rules are
really what's most important to discuss, not the actual OK the
gameplay. All right, well, I'm still I'm
still grabbing the under here. But yeah, that just seems like a
lot of work for three different nation states to execute and
then in the 12 month time frame would be mine.
Are we getting more point like or so like does he get more
return on his investment for predicting something that was at
70? Yeah, that's a good idea.
I think that we could use something like that.
If you take the over on a low probability prediction and then
at the end of next year you're proven correct, I think we'll
give you give you some more points.
I think we need. ROI like let's let's let's save
the exact one. Right, exactly.
Could we get into another rule here?
Yeah. James, what are you picking?
OK, I am taking the over there and I think, you know, we're
just going to see this come up a lot in the next year is like
people using AI for their jobs, maybe in areas where they're not
supposed to. And I just think that'll be a
key kind of political discussion.
Point is, hey, why is why is my doctor, you know, using ChatGPT,
right? And is that a should that be
legal? So I wouldn't be surprised by
three countries. Taking I want them to be to be
clear or like as an assistant. I think it would be good.
I'm not saying that's a bad thing, but maybe it should be a
little bit regulated, right? There should be something.
I might just, I might just be biased based on living in the
United States where we have just epic legislative gridlock.
But yeah, I mean they. Passed AI regulations in
California. Well, that's true.
I mean, you're. Not the big one.
But you're saying the state, the state government, Yeah.
Yeah, which is like one of. The.
Well, they didn't pass it because the.
They passed other ones, they passed it all there other.
Things OK, this prediction specifically says national,
national. Yes, wouldn't count.
Yeah, all right. But Europe?
'S like anyway, sorry. Moving on prediction, by
December 2025, Tesla will release fully self driving cars
that are legally approved for unsupervised operation on a on
public roads in at least one US state.
What's the price? We are saying 40% probabilities.
Interesting. I'll go over actually I one of
my predictions coming in was around Tesla AV capabilities.
I just think that Waymo's proved it's possible.
Elon's got more chips than anyone.
They've got more video data than anyone.
They're obsessed with this problem.
He thinks it's the entire future of the company.
And then coming down to the ticky tacky state approval bit,
you know, he's got his boy in the White House and he's got
friends in every Republican government in America.
So like fucking Wyoming could approve this shit and he win the
bat. So I think that I'll take the
over at 40%. So it it's to be driving like
without a driver behind the wheel.
Yeah, that's I think unsupervised, meaning you don't.
Have to I still, I still think there's a very likely scenario
where it's like in Texas or Wyoming or whatever you, you
know, you can sort of allow it to drive, but people are
effectively behind the wheel. Like I'm not very bullish on
Tesla's like technology, I don't think.
Are you talking about like remote assistance, like remote
drivers taking over or something or like I think?
That would be that would count or not I feel like.
That would count like that's what Waymo does.
Right, Waymo has remote takeover if there's.
No takeover. Remote takeover, yeah.
But it's a little. It's like a remote.
How is what they have right now not already passing that?
Like people are allowed to use autopilot.
You're talking about Tesla. Tesla you.
From a legal perspective, you have to have a driver in the car
behind the wheel, ready to take over at a moment's notice.
Basically it also has to be supervised, like you know that
you have to be looking at the road or disengage.
Like they have a camera on your eyes that show that checks if
you're watching the road or not and it gets mad at you if you're
like on your phone. I've tried this.
Oh, wow, yeah, yeah, it it literally throws a warning up on
the screen saying I like, I can tell you're looking at your
phone. And I'm like, well, what's the
point of self driving if I can't look at my phone?
Like that's the entire use case. I think I'm I'm taking the over
just because of all this sort of seeming cases where this would
count as approved. I don't think they're close to
Waymo. I'm not bullish on Tesla's
approach, but I think there are lots of ways to sort of fudge
this with humans still sort of behind the wheel.
All right, for bonus points, which is the 1st state.
That will allow this Texas. Texas would be the obvious
choice, but it's also really big and there's a lot of complicated
roads and cities and stuff there, so I don't.
Know, but does it have to be the whole state for the rules or?
Yeah, Yeah. It's a state government, right?
Yeah. Wait the whole state.
It says one state. Will approve it.
That's interesting. You mean like Waymo is not in
the whole state, it's in San Francisco, right?
Or I think. I think it would just have to be
not the whole state, but you know, somewhere within somewhere
in the state. Yeah, like the whole state.
I would change my whole. Yeah, yeah.
I think it has certain. Things I guess, like I wasn't
even thinking you'd be able to own a Tesla but only use it for
full self driving if you're in one city.
Yeah, that's interesting. All right, Texas.
All right, We'll see how that plays out.
James. Yeah, my take taking the over as
well. I think you guys are, you guys
are spot on. And yeah, I think I will
actually go a little contrarian and take California as the first
thing. Wow, OK, wow all.
Right guys, let's move on to the next prediction we have and
another prediction from ChatGPT. By the end of 2025, AI generated
content will constitute at least 50% of online news articles
published by a major news outlet, probability 30%.
What's major? Again, I think this is a key
question here. Like how major?
Mitigated after the fact. What would be capital M major?
I mean like I'm. Watching the I'm taking the
over, I think Conde Nast is clearly major.
I think Forbes is probably major.
And I think if you're just talking articles, you know,
we've seen a lot of media just be willing to produce a lot of
articles to, you know, get get gobbled up by SEO.
And so it's not necessarily the majority of their homepage, but
the majority of the articles. One outlet.
I say yes. I think, yeah, I think it's 30.
I'd grab the over. I just think 3 to one odds on
that or two to one odds sounds pretty good.
I think to your point, like yeah, the SB Nation network I
could see or Vox or whatever, like just somebody is gonna give
in to the SEO, the SEO temptation to have a crap load
of articles generated by AI that people are going to link to.
So will. They talk about that publicly.
Do you think like will they? No, I'm sure there will be some
journalist that gets leaked to in a bar where where Eric lives
in Brooklyn and then then somebody will write an.
Angry story about I don't think Axios would necessarily do it,
but there's definitely a breed of sort of business first media
publication that would brag about this to seem cutting edge.
I bet someone will write a cranky article before somebody
brags brags about it. But before.
Yeah, yeah. We'll see.
AI itself break the news that. Not until it hooks up to the
Bitcoin network and starts getting paid for it.
I'll take the under on 30% I'm sure have to litigate what
content a major outlet, but I'm skeptical that it'll hit 50% of
articles like I mean, at least in the next year.
I think that I think that it'll maybe start, you know, kind of
long form like some some indie Blogger or something.
We'll be generating hundreds of thousands of articles or
something with AI next year, but I'm skeptical that a major
outlet will be doing that at that scale next year, but we'll
see. I just like.
Media outlets. I just go with the, yeah, the
Logan Roy attitude here, which is that money wins.
And there's a lot of private equity firms that own these now
and. If they're not doing this,
they'll they'll yeah, management will get fired.
This is an amazing way to cost cut and it's also an amazing way
to get links from Google search. So I just think it's going to be
very tempting for a lot of these companies.
All right, next prediction. By December 31st, 2025, the
first AI generated feature length film will be released in
theaters, with the script and visuals both entirely crafted by
AI. 25% probability under. I'm crushing the under on that.
I just think that you have to do a deal with theaters like get
distribution and Hollywood's going to lose their effing minds
if somebody tries to do this in the next year.
So I just think the the Gears of entertainment don't turn that
quickly. I'm not saying somebody won't
generate a feature film by the end of 25 that plausibly could
be in a theater, although I I would actually debate that as
well. But I think that the odds that
it's like actually released in theaters.
I'm I'm just super under on that.
They barely, they barely let Netflix into the Oscars.
And your mistake is misunderstanding the
entertainment industry. Even though you're you grew up,
you grew up in LA area, but. The you can pay to play in
theaters any day. Like there are all these sort of
documents, there are all these films that sort of pay to get
theatre distribution so that they account for stuff and you
can absolutely do that. I'm taking the over.
There are tons, tons of companies with an interest to
have this achieved and somebody will just buy the theatre plays
to make it so you. Have the like, buy the US
Constitution angle here, which is somebody's just gonna fucking
do it because it's gonna sound cool basically, and pay to play
the theaters. OK?
I mean, there's already, there's already a.
Film that Sarah, my wife saw that you know, there are some
boundaries like it was shot by humans, but then it reorders
based on AI, you know, so that was already in theaters.
So I think. This is a good, this is a good
robust debate. I like that.
You're like, you guys believe in the forces of change and I
believe in the entrenched hating institutions.
I haven't taken my. I haven't taken my.
Well, I was referencing the the legislation discussion from
earlier, but yeah, this is a meta point about how quickly the
Gears of power turn. And you guys are like really
quick. And I'm like, I don't know.
So, all right, James, what do you think?
I am also taking the under on this one.
Yes, and not for any remote reason around theater
distribution, but just purely from model capability.
Like the specification of this prediction is that the script
and visuals have to entirely be created by AII think that we are
not there on visuals, we're barely there on scripting.
I don't think we're there on scripting either, by the way.
I haven't. Seen the script as you started
the episode with is that there's a lot of control in models over
prompting. And so it's like if they prompt,
they could prompt a whole script into it and then output a script
and say it was AI. And that I just think there's
such a strong incentive for the result to be it was AI that like
humans can put a lot of thought into it and get what they want.
I don't think it'll be hard to do a script.
I mean, it won't be a good script, but I do think it'll be
hard to create a full feature length.
Does it say it says feature length?
I don't know that Probably 120 minutes or something of visuals
like a that's extremely expensive to create, and also B,
it's like very hard to get any sort of consistency of the
characters or anything. I'm just skeptical of the
visual. Aspect.
I think our guys at Runway will personally make sure this
happens. Yeah, probably the guys.
It's probably the way Eric wins is at the runway.
Our friends at Runway just bribe.
Have a theater chain into doing right?
Yeah. They listen to this episode,
they listen to this episode and they're like.
They're like Eric's got some skin in the game and we're gonna
help him out here. We do this, Eric is gonna brag
about us on the next. Year's yeah.
I Episodes I. Don't know they haven't even
released Sora you know yet or. It's coming soon.
Somebody was just one of the competitors.
I think Runway maybe was saying we think Sora is about to come,
come all right. All right, good discussion.
Let's switch over to Claude and see how anthropics predictions
do and how you guys respond to the probabilities.
And then we can have a little discussion at the end of who's
better at predicting, but meta discussion but starting with.
We're already having a discussion about our opinions on
their predictions, and then we're going to have a higher
level of discussion about our opinions of the relative
predictive power of the two models.
And then we're going to have AI watch this represent and react
to our predictions. It's all good.
Well, I guess on a certain
level, you know, think about AI is all about intelligence.
So it's like, how do we introspect our introspection
even more anyway? Sure.
Yeah, somebody. Someday we'll be teaching a
class in university. About this episode.
All. Right, too self indulgent.
All right, all right. Starting with Claude's first
prediction, by December 31st, 2025, at least three major
smartphone manufacturers among Apple, Samsung, Google and
Xiaomi will release phones with dedicated AI Co processors
capable of running large language models with at least 7
billion parameters entirely on device probability 75%.
And the parameters ones are where I'm like.
I don't even know how many parameters they can run on
device right now. Can't they run 7 billion locally
already? Is that wrong?
I think that the Apple intelligence chip can run 7
billion, right? I thought.
It's at least three to four, you know. 3 billion. 3 billion.
Yeah. OK, three to four.
All right, I got it. I nailed.
It Apple's current. Just to reiterate there, Apple's
current on device LLM uses 100% local LLM with three billion
parameters and then can also connect to external servers for
larger parameter models. And what's the price on this?
Over 75% or probability 75%? OK, well I'm gonna grab the
under on that cuz I think dedicated Co processor is really
what's holding me back here cuz that requires like a separate
chip essentially to go into the hardware.
I'm a avid follower of Apple rumors and I have not gotten any
sense that they're interested in adding chips like separate chips
to their system to do this kind of thing.
And so you'd basically have to get Xiaomi, Google and Samsung
to all do this separately chip in the next year.
And there's not really clear sentiment that that's necessary
to run these models. So yeah, I'll take, I'll take
the under at 75 for sure. You sold me under.
Hardware Hardware timelines are really long, and it that that's
only a slightly larger model than they can already run
locally, so it's just not clear. To me, the parameters make
sense, I And it's also not all these come on device for Google.
Like, is that really in their mission?
Like there's so much like, we'll send everything to the cloud.
Like, are they really? Yeah, Apple's the one selling
like everything will happen on device.
I think I'd like grab the over at like 10% on this, but that's
75 I think under big time. Yeah.
I mean, I guess I'm hearing what you're saying on the dedicated
aspect, but like I feel like you could quibble with that
semantically of like as long as it's a GPU at all that is
dedicated to running AI tasks. Like that's what the.
GPU. They already have a GPU, right?
I mean, yeah. OK, so they have essentially a
GPU and then there's a set of cores, right?
I guess it's. Just what is the neural engine I
guess is what? I'm yeah, it's like are we
counting the current neural engine on there for like 8 years
already as as a yes to this because.
Then it's like. Sure.
Like I guess I mean like, but I don't think that's the spirit of
the prediction, right, OK. Like the the thing they've had
on the phones for the last eight years.
Is I don't. Know.
So far, Cha Chi BT is beating Anthropic because I feel like
there's. A quality of prediction.
I don't love this. This.
Whatever, Yeah, Anyway. James, Just pick, pick and we'll
litigate everything afterwards. Yeah, I'll take the over and
I'll try to litigate my take that the neural ending.
All right, Great, great. All right.
Prediction #2 Anthropic will release a model that achieves a
score above 90% on the Unified Benchmark for AI Reasoning, or
Bar Evaluation Suite, which will become the first model to
outperform the human baseline. Human expert baseline of 87%,
probability 65%. Where are they at right now?
Unified. What was the the metric?
It was unified. Benchmark for unified, benchmark
for AI reasoning. We're all going to frantically
Google for that, yeah? It's funny that we have
anthropic predicting its own intelligence.
Capabilities I know. I mean, a key part of this is
the claim that it's the first also, right?
It says first, right? Yeah, it'll be the first model.
For this, for the record, is this benchmark real?
I'm I can't find this benchmark. Is this is this benchmark?
Being my God, that would be hallucinated.
I think it's my God. I can't find this.
Unified benchmark for AI reasoning.
It's like I was like God. I was like, I've never heard of
this benchmark like. I love it.
It's crazy. It's hallucinate a benchmark.
I mean, can you find it? Like it says something about our
deference that at first I'd. Wear like I did not defer.
I didn't. No, no, no, I assume.
That it's. Do do not loot me into this.
Definitely. I feel like I I immediately
googled it and I was like, what I've never heard of this.
I've heard of like MMLU. I've heard of like, you know, a
couple other evals, but I've never heard of the bar like or
whatever. Maybe this is like internal
philanthropic. Maybe this is like like, we'll
all be proven idiots when this becomes a benchmark, right?
And then? They leaked this.
Yeah, OK. I guess I will.
I'll take the under given I'm not sure this benchmark.
Is real I guess if it doesn't exist we can create it and you
know y'all rule it and you can make it true.
We'll have to do some more Googling.
Later OK, but I don't think this benchmark exists for my own
recent. Just very quick for the listener
to get to the substance. I know we're focused on
technicalities. Do you think someone will
surpass sort of human intelligence overall by the end
of next year? And do you think it will be
anthropic first? Just quick 10.
In some kind of evaluation, you're saying basically like,
yeah, yeah, for sure. I, I sort of think we're like
already there. But I guess it depends what
invented benchmark. Exactly.
That's the take I was going to get, which is we have sort of
quietly passed humans on a lot of things already.
And we're just this, you know, since people, you know, smartest
in every domain still remains smarter than it.
Like we've sort of under undersold how exciting it is for
it to be smarter than the average human on on a lot of
stuff. Totally.
Next prediction from Claude. At least 5 Fortune 500 companies
will replace more than 25% of their middle management
positions with AI systems for task delegation and performance
monitoring, publicly acknowledging this transition in
their annual reports. Probability 25%. 25% chance that
25% of middle management positions get axed.
What if they become ICS? If they become individual
contributors where they don't get fired, is this met?
I think that this would be met because I think that we're
saying past delegation and performance monitoring, meaning
it's most most of the management responsibilities.
I'll, I'll, I'll take that over. And my reasoning is essentially
just that there's a lot of companies, there's in fact 500
companies in the Fortune 500 and.
A lot. There's some.
Number. Yeah, yeah.
There's a pretty high number, somewhere around 500 of them.
And and plenty of those companies are not growing or
shrinking and need to cut talent in some way or another to manage
the business. And this will be the greatest
freaking like cover stories. Yeah, exactly.
Smoke and mirrors. Like, you know, look over here,
like the left hand's doing some AI doing layoffs because they're
right. Yes, right, right.
This is going to be an amazing way to drop in your earnings
report that like we had to layoff a bunch of people, you
know, but it's cool because it's, it's about the future dog.
We're adopting AI. So I think that, yeah, I just
totally buy that, you know, 5 out of 500, we'll need to do
that in the next year. And it's just an amazing cover
story for doing layoffs. What's the percentage?
Eric James. They need to replace more than
25% of their management positions and the probability is
25% that this occurs. Wow, so it's a much better
return for me to say. Yes, yeah, you get a, you get a
4X return on the Yes. Yeah, over.
I mean, yeah, I, I think the, the covers you guys, I'm, I'm
not, I'm not as excited about AI in middle management
necessarily, but I do think there are too many middle
manager like we have this whole separate trend of like Facebook
and companies saying let's just like be flatter organizations
overall. And I feel like we're in that
trend. And I do agree that the AI trend
is sort of going to get mixed in with that, but I'm not convinced
that AI is going to come for the middle managers first.
I think it's going to supplement sort of the coders and
frontline. You know, I to me, the
deployment model is more like people are actually doing the
work, figuring out how AI makes them better and sort of adding
it to their just making themselves look better to their
bosses. And I think that's how my model
for how AI is getting deployed in workplaces, which is just
sort of like people hacking shit together and delivering results.
But I'm still taking the over. Yeah.
I, I don't know, you kind of convinced me to take the under
there. I think that I think that AI
will have a much bigger impact on sort of IC roles.
And you know, we've seen this already and some of the customer
service oriented companies sort of announcing and bragging that
they can hire less representatives.
So I think we're more likely to see that first before companies
start taking this AI approach on management, but could be wrong.
25% is a big number at a Fortune 5.
Exactly like are are they gonna? I I, I think I'm gonna I've my
whole argument is the under. I feel like I should just like.
No, you're not allowed to switch.
You you really you? You made me not switch on like
2. On the fence, I could see the
advantage you. Know you're on the fence, you
know, No, I'm committed or I'm over.
Yeah. Committed.
It's funny that I made the whole case for the I.
Know, I know. Thank you.
Thank you for. Your all right, all right.
Once you say the words, you're. Locked.
Yeah, once you say the words, you're locked, Bear.
It locked. All right, let's go to the next
prediction from Claude. This prediction is that the deep
mind or let's say Google broadly, will demonstrate an AI
system capable of discovering at least one novel pharmaceutical
compound that also passes phase one clinical trials, reducing
that typical discovery to trial timeline by around 50%.
We will just specifically say that it has to have discovered a
compound and passed phase one trials within the next year,
probability 20%. I'm gonna take the Max world
view, which is like a lot of these are predictions about
society. And I definitely don't think
drug approvals are going to sort of just get much faster because
AI companies say they should. So I'm going to take the under.
Who's Running the FDAI guess is another.
Yeah, exactly. Oh yeah, that's.
Trump I'm only conflicted because the Trump administration
is about to take over. But they don't like Google,
Google or they're not going to like Fast Track Google.
That's the last company. That's the company they want to
hit the most. Yeah, they don't like the FD.
Or sorry, Yeah, they don't like Google.
They also like pretend they didn't do Operation Warp Speed
even though that was like the coolest thing ever.
I don't really know how long, like Phase 1 specifically.
I know I don't know a lot. About.
I mean, I know the entire half. The time, right?
Yeah, so. I know the entire.
I don't know. I think that's just like kind of
an additive less necessary component of the prediction
reducing the typical discovery. OK.
So well it just says bypassing phase one clinical trial within
a year, it is reducing the. Typical would be very fast.
Would be a 50% improvement in speed I guess.
I know the entire time is like passage.
It's like 10. 10 to 15 years. And so it's on the order of
years for sure for phase one. Gosh, yeah, I think I just got
to stick with my, I don't believe that the the
institutions change as fast as the technology here and take the
under even though 20% with the new administration is is
appealing, that's just feels like a high bar to pass phase
one trials. It does seem like phase one
involves actual experiments on yeah, which AI cannot speed up.
I just read Dario's essay and like one of the main points that
he makes in his bowl case on AI is just a lot of the limits will
be human systems and things where you have to run real world
experiments and even if it has good ideas, the experiments
could take a while anyway. All right, would you say?
I'm taking the under too. I think that it just seems too
fast to occur. Would you guys take the over on
Google entering phase one trials or something with something?
Yeah, I think there'll be a discovery or some.
Yeah, yeah. Registering for them or
something? Over 20%, I would take the
around 20%. Obviously Demis at DeepMind is
quite passionate about biology, is a major application for AI.
So yeah, I mean, I think that would be a cool thing for them
to demo, say, hey, look, we made a drug.
Yeah. All right.
Let's do one more from Claude. All right, prediction.
The first international treaty specifically governing AI
development and deployment would be ratified by 15 countries or
more, including three of the following countries, US, China,
EU, UK. What's the percentage on this?
Is 5% like? Yeah.
What is this like a 1%? They're legally binding
commitments on things like model evaluation and a standard.
What is the percentage they gave?
Yeah, 50%. Oh my God, this is that's the
Claude's. That's the I feel like that we
needed much better. That's like that's the easiest.
Under Under. Claude's like, under Claude's
like dreaming of this fantasy safety world or it's something
where where every country has come together.
China, or like I didn't get the memo.
You don't need the US or China because you only need three of
these entities. So you could have EUUK and Japan
all entering some sort of treaty.
OK, interesting. EUI would believe I mean UK
though is sort of I mean I definitely am picking the.
Under I just feel like the yeah, the UK is a mess economically
right now. Like our, you know, is the
Labour Party really going to be like, yeah, guys, we're not
fixing the economy, but we signed this dumb AI treaty that
none of you care about. Like, I, I don't know, I, I
think they got bigger fish to fry there.
Yeah, under, under A50 for sure. Yeah, I agree with you guys
taking the under on that. Yeah, I think this is Claude's
own fever dream of hoping for. A better place, a better world,
a better world. AI where Claude itself is highly
regulated, Yes. Yeah, OK, great.
Great work everyone. That was fun.
How do you guys think ChatGPT did compared to Claude?
What do you like about the different predictions?
Well, it seemed like Claude made-up some stuff.
I did think it had a little bit of a safety ISM trend line.
So I don't know. They're all blur to me but it
felt like the first half. Were.
More interesting in the second. It seemed like GBT's predictions
were more entertaining as well as more somewhat realistic.
I guess. So I think I would give GBT the
clear victory in this round of entertaining and reasonably
accurate predictions. Maybe you prompted?
For O1 for reasoning models. Yeah, you prompted exactly the
same stuff. Or.
Exact same prompt. Yeah, yeah.
Right. Well, that's a that makes me
more excited for the full version of O1 coming out.
Yeah, that would that's probably one of our one of our
predictions. I think that's coming out this
year, right, is the rumor. I mean, the zoom out thing that
I took from this is just how much human systems interplay
with the things we want AI to deliver.
And even if we are bullish on rapid development of AI, there
are a lot of things where we're like, well, the human systems
will still effectively limit what it can do.
That's self driving drug development, sort of everything
under the sun. And so I think that's
interesting to come out of this. How much?
Yeah, this this sort of societal piece is still sort of a strong
clever on what AI can accomplish.
I feel like the way we get to that treaty is that something
horrible goes on, right? Yeah, exactly right.
Like, yeah. Something cool.
Yeah. We underestimated Black Swan.
We have to narrowly escape from some horrible thing to get a
binding treaty between a bunch of countries, I think.
Oh. Yeah, then we'll look really
dumb. Yeah.
It's like they, you know, AI does figure out how to like.
Although we just had a paper. Clips or whatever to send a
nuclear. Bomb we just had a pandemic and
nothing came out of that like hey.
Move on. It's not like.
Some tragedies that bother us more than others.
Yeah, but the US, China and the EU didn't get together and be
like, hey, maybe we should like control pandemics a little more
carefully. Like, you know, let's let's
regulate that a little bit. It's like, Nah, we we got
through it. Don't worry about it.
Let's just get on the record on yeah, I mean the core just
because it is a prediction episode and like we, we got into
sort of the technicals like do you think the models will be
significantly smarter next year, like by the end of next year in
that they will we'll see the same basic rate of progress,
meaning these are like PhD level.
Thinkers I think by the next year, I think next year the
overall take from this discussion like you said is the
difference between things that can be facilitated by software,
things that can be require hardware and then things that
require institutions and government and so on right and I
think next year I think it's going to be a crazy year for
leaps forward and software. I think that GBT 5 is going to
come I think 01 or O2 or whatever the GBT 5 level version
of the thinking model is going to come.
I think these computer use API's, as I said in a previous
episode, or just an incredibly powerful interface to the
Internet and the software world for these models.
And so I think it's going to be like a crazy, crazy software a
year. I'm I think it'll be the
beginning of the impact on hardware and institutions.
But it feels like the old Intel like tick tock strategy where
you have one year where you have huge progress and then one year
of refinement. And that's like the tick and the
talk or whatever. I think 24 was kind of a talk
year where we didn't have really dramatic progress in the models,
but we started to see the existing models eat away at all
these, you know, tools and software that we have.
And now next year, I think we have a big, big tick here.
And that'll be the foundation for like, you know, societal
transformation for years and decades to come.
So I'm I'm very long model progress in 25.
I think that next year we'll we'll be talking about this
trend of AI getting slower and slower.
So we started out. Just going slower and adventure
define. Slower.
All right. No, no, no, I don't know.
I'm not talking about reasoning capability.
I'm talking about inference time essentially.
And basically what you, what we've seen already is like with
O1, right? And now it takes, you know, 5 to
30 seconds to return an answer, but it can be better.
What we're hearing is that, you know, internally, at least in
open AI, they feel like that's a whole new scaling paradigm to
get better and better reasoning is, is inference time compute.
And what I think, you know, we'll see is like, like Max
said, a lot of these computer use models are opening up
opportunities for kind of agentic systems.
And we'll start to see, you know, people like ourselves
using these models that take hours to return US information,
right? So Eric could go talk to an
agent and just be like, hey, could you go research this thing
for an upcoming article I'm writing?
Or, you know, Max and I could have one of these agents go off
and kind of research some game genre that we're looking to
build, right? But it will just take longer and
longer to return. And, and, you know, I think
that'll be a really interesting trend.
And I think that, you know, maybe that's the prediction for
our, our own internal usage of how things are, how we're going
to be changing how we work with these age, these AI systems.
But I also think more specific prediction for like industry or
or enterprise would be that we're going to see a lot of use
cases in, in sort of the finance space for these agents, right?
Like there will be new forms of hedge funds or investment groups
that are like using these agents to actually trade.
And, you know, maybe we'll see that primarily in crypto, but I
could also see that occurring in sort of traditional assets.
But I do think we're going to see a lot of like adoption of
these agentic models to like trade their own portfolios on
behalf of their, you know, human kind of owners or overseers, I
guess. I mean, this isn't a prediction.
And I think it sort of I mean, I think one thing we are seeing is
that intelligence can be out there and most a lot of people
don't know how to use it, right. So I think we're going to see
obviously this uptick in applications taking existing,
you know, the intelligence that has been proven with open AI,
Anthropic and others and making it much more straightforward for
people to use it in particular use cases.
And I think there's going to be a ton of money there.
So that's applications, which we've talked a lot about.
And then I think the other piece that you're sort of touching on
is that they're going to be these sort of specialists that
appreciate how intelligent these models are.
And they're going to figure out either how to use it and make
money on it directly or become service providers for people who
haven't really figured it out and do things better and say,
all right, just pay me. I'll solve this problem.
Secretly I'm using AI or not so secretly I'm using AI so I can
do it cheaper than you. And I think that's how we're
going to see these two models. There's sort of two ways people
are going to implement AI. And I think so we'll see more AI
applications and AI service companies.
And, you know, the AI applications will almost
certainly lag the actual power, the thinking power of AI.
So it'll be an interesting sort of push and pull there.
For sure. Cool.
Yeah, Super fun. Yeah.
Well, I feel, I don't know, James, you'll try and come up
with the score of those predictions.
I don't know. I think, I think to me they were
more like what the draft I am invested, ego invested in the
outcome. I feel like these predictions
were more thought experiment than necessarily to me, the sort
of perfect proxy for what we think, how accurately we see
what's coming in AI, but certainly enjoyed it.
That was a fun, fun concept. Fun episode.
Yeah. Yeah, Greg.
Well, like really really demonstrated that AI was better
at creating the grist for the content.
Than we are. Yeah, exactly.
I mean, we were supposed to come up with predictions as like,
yeah. I got a couple.
Ideas. I mean it shows that sometimes
like just having throwing shit at the wall to react to you is
better than like overthinking my perfect prediction.
And getting and getting your hot picks live, right?
Getting. Yeah, exactly.
Having to think through it. Seeing the thinking, that's
something that, you know, AI is just starting to learn how to
do. You know, humans still went on
the showing our thought process. Anyway, we will be back after
Cerebral Valley and we will be publishing all the talks.
We're going to be throwing you our favorite clips and then we
will be back with Max and James after the conference, reacting
to everything that happened on stage.
So come back and see us, newcomer.
All right. Thanks so much.
Thank you.