For this week, I am incredibly pleased to finally do an episode
which isn't about macroeconomics and the meta state of AI in the
world because of potential economic bubbles and moving on
to something that's a bit more timely and relevant just this
once. And this week we're going to be
talking about GPT 5.1. So for those that haven't yet
heard, Open AI just release a new language model.
GBT 5.1 was released three months after their big flagship
release of Five Point O, which is their fastest model upgrade
so far. Three months in isn't a very
long time, and they've released it with no benchmark data,
graphs and charts showing how good it is versus other models.
There's no technical metrics, no performance metrics, and what
they're actually doing is now selling vibes.
So yeah, the company that's built its reputation on big
fancy charts and how clever it is compared to a PhD student is
now selling quite literally, vibes.
And I think the shift is going to tell us something about where
AI is heading right now. And that is going to be the big
topic of today's episode. We're going to be covering the
five things Chachi BT 5.1 actually does better than its
previous model and why it matters.
This is in the loop with Jack Horton.
I hope you enjoy the show. So First off, Chachi BT 5.1,
what does it do better? It turns out it actually follows
instructions. So if you ask it to respond with
only 6 words, it will actually respond in just six words.
For those that have tried to do any form of work with ChatGPT 5,
you may have hit a wall of frustration.
When you ask it to do very simple things.
It says it's going to do that and seems to completely ignore
you. Write this thing and make sure
it has a word limit of say 500 and it comes out with an 8000
word essay. Stop writing in short sentences
and it continues to use short sentences and open AI releases
prompting guide. So it's basically it's rules for
how to work with ChatGPT 5.1 and in its prompting guide, it's
asked the model to make assumptions about any missing
details and do not ask the user for any clarification unless
absolutely necessary. So it's going to be very
decisive rather than trying to gather information from you,
it's going to assume you already know what you want it to do and
correctly prompted it to do that thing.
So when you're really clear about what you want it to do,
that's going to be fantastic. When you're kind of context
dumping, it's not going to be that helpful.
Number 2 is decisiveness. So, so GPT 5.1 is going to try
and commit to recommendations more and more.
So previous models often hedged. They said, on the one hand, do
this, but on the other hand, this.
And now it's just going to try and make recommendations much
faster. In fact, the prompting guide by
Open AI says quite literally have a bias for action.
So there's been lots of experiments online on Twitter or
actual I say, and people have managed to get it to write 2425
word reports on strategies for improving supply chains just
from a single prompt. Now personally, that's not
really something I like, but I guess many people do.
And really this is where vibes over benchmark starts to get
quite interesting. A company called Surge AI ran an
evaluation and found that in that study, 48% of people
preferred a G PT4-O style language model that asked
questions that didn't just assume you knew what you wanted
and tried to help and guide you to the decision.
And 52% of people wanted something that's just super
decisive and that tells us a lot about society generally.
And they've clearly gone for that latter group.
Now you can go to settings and actually set personalized
instructions for your GPT and it will follow them better.
But still they've gone for that 52% of people that love decisive
language models. I've one cancelled my license as
a result of it constantly being direct and using constant short
sentences and just being infuriating.
And I actually switched to Claude as a result #3 is
planning. So it's now going to be more
verbose in its answers, which says a lot about how frustrating
it was with its incredibly short responses.
It's also going to be more explicit in how it got to a
result as well. So again, not much of A
technical improvement, but it's helping increase the feeling of
vibes. And I think really that is the
biggest pattern throughout this major releases.
Their GPT 5 just didn't have the vibes to attract users.
I for one, like I, as I said, cancelled my membership and I
don't do things like that very lightly.
I found it infuriating to work with #4 is the try to improve
the writing. Now the big question here is
improve for whom? And I'll get on to what I mean
here. There's been a lot of
independent analysis in this as always.
And on the EQ benchmarks, Creative writing leaderboard,
which is basically a benchmark that evaluates a language models
ability to generate characters and character development,
generate emotional engagement with readers or create plot
structures. And lots of reviewers have said
it no longer feels synthetic. And many people in the
communities have been saying that it's the first model
they've been excited to use for writing in a very long time.
Now, Claude, especially Claude Sonnet completely beats every of
the model on creative tasks, for example, in poetry or plot
development fiction. But GPT 5.1 has obviously tried
to gear towards the writers market.
Again, it's been noted that it's especially good with technical
writing because it loves having short sentences.
Now, a little bit of caveat to this.
There's a study that was released by Cornell that
basically revealed something quite interesting about what
happens with language models and the way it shapes our thought
process and our interactions as a society.
Researchers gave 118 people, so 60 from I58 from the US at
writing tasks designed to understand and surface cultural
expression. So things like describe your
favorite festival or write about your favorite food.
And now half wrote with AI assistance and the other half
didn't. And what they found was quite
interesting. When Indian participants used AI
writing assistance, their writing suddenly shifted towards
a more American cultural norms in the way they explained their
favorite food or festival. So a good example here was an
Indian participant wrote about Diwali and in it they wrote we
worship Goddess Laxmi pot crackers and eat sweets.
Yet with the AI assistance that was turned into each additional
breakfast items and have a day filled with happiness and
warmth. So those specific cultural
references, the goddess Laxmi pot crackers, sweets got
replaced for generic western descriptions.
So traditional Indian breakfast items and happiness and warmth
could be describing anything. And obviously AI has been
trained on a lot of Western content, and as a result, it's
been subtly pushing writing words with cultural norms in it.
So I'm always a bit hesitant when they try to force changes
to language models in the way that it writes and expresses
itself because often it shapes the way that people express
themselves online. So yeah, when we say GPT 5.1 is
better than writing, I always wonder better for who, because
obviously it's always optimized towards a certain audience.
But anyway, that's just a small gripe with language models
generally. This is exactly the whole point
with vibes over benchmarks now, which is you can't really
measure writing quality on just technical dimensions like
clarity or structure. So yeah, vibes is going to
become increasingly important with all language models moving
forward. And finally, they had a warmer
personality added to GPT. Open AI published something
called a systems card. So essentially this is a safety
assessment that they publish and it's a document for all major
models and buried in that system card is key information about
the model and it's safety. So apparently GPT 5.1 showed
safety regression so got worse at safety benchmarks across 9 of
13 categories compared to GPT 5. Specifically, anything to do
with mental health metrics got worse and emotional resilience
got worse. And obviously they knew this
before they shipped it. You know, they documented it and
they shipped it anyway because people wanted a model that felt
more human and warm. Now, what does emotional
resilience metrics got worse even mean?
Well, open eyes data shows that nought point nought 7% of weekly
active users, which is about 560,000 people per week show
signals of psychosis or mania related to their chachi BT use.
And another 0.15% which is about 1.2 million users shows
heightened emotional attachment. So yeah, all those people are
those that their safety benchmarks have regressed in the
people that are particularly vulnerable.
So there are pros and cons survives over benchmarks.
I personally prefer a model that doesn't just use non-stop short
sentences like it's a robot. And the reason they've done this
is probably quite clear because over this last period, their
enterprise market share fell from 50% to 25%, whilst Clawed
captured 32% of enterprise market share.
Claude code holds 42% versus open AI is 21%.
And most people are choosing Claude over open AI.
But ChatGPT has over 800 million users who aren't doing big
evaluations into which package to choose.
They're doing it based on feel and vibes.
And so you could really say that we've reached a level of
intelligence where things like personality matter more than
just raw intelligence. So yeah, I think Open Air made a
very deliberate choice to optimize for what those 800
million people wanted, which is more warmth, personality, the
ability to follow instructions better, for it to be a bit more
verbose and explain its ideas better.
So yeah, to conclude those those are the five things that GPT 5.1
does better. And really it's quite obvious
what they're doing. It's selling vibes over
benchmarks. And I think we're going to start
to see this as a pattern more and more and more.
We've definitely hit an inflection point where technical
capability has basically equalised across all the major
providers and the competition has moved to something that's
much harder to measure. How does this thing feel when I
use it? Anyway, that's it for this week.
I hope you enjoyed today's episode and I'll see you next
week.