A new word just entered the workplace lexicon, Work slop.
If you haven't heard about it yet, you will do soon.
Stanford Research has just coined the term and it's
spreading like wildfire online. So this Stanford report that
came out this September spoke to one of the biggest complaints
when it comes to AI. The most of the work it produces
is a bit crap, with the study finding that 41% of workers who
were surveyed reported receiving work slop from their colleagues
in just the past month. In today's episode, we're
discussing the rise of work slot, the real financial cost of
it, whether this is all just noise, and how we can all fix it
going forward. This is in the loop with Jack
Horton. I hope you enjoy the show.
So I think let's start with what workstop actually is.
So workstop is the term that's used for basically AI generated
content and work content specifically that kind of looks
polished but is actually just a bit rubbish and crap.
And in practice, this looks like, you know, slide decks that
or reports maybe that look really good when you I guess
give it that 5 second glance or you skim through it really
quickly. But as soon as you really dig
into it or want to use it for something practical at work, it
has wrong numbers or repeating texts or every word of a
headline or a sentence is capitalized or the argument in
it is just again, a bit weak and, and repetitive and again,
crap. Now I have got a take on the
real cause of this, which we'll come on to.
But in a survey of over 1100 US workers, 40% of people said they
received AI Workslop in just the past month.
And according to this study, it takes about two hours of extra
work to fix all of this workslop.
And as a result, the researchers estimated that it costs about
$186 per employee per month, which equates to about $9
million a year for a 10,000 person company to fix of it.
Now, of course, there's an element of probable exaggeration
when they are reporting or self reporting the impact of
Workslop, but I think it does speak to a real thing at work
right now. And at this point, I think you
probably know the people at work who do this, you know the team
members who send sloppy AI work. And I think as AI becomes more
mature and widespread, those team members are going to pay a
price. Because, you know, about half of
the surveyed workers reported they regarded slop senders as
less creative, as less capable, or even reliable.
As one redditor said, everyone hands me that AI generated slop
to edit and I have to cut 75% of it.
Usually it's the same info repeated in slightly different
ways. Another quote the study
referenced makes this point even more clear.
If you use AI to pretend to do 20 tasks, you're a Trojan
horsing bunch of work to other people.
Basically by doing rubbish work you cause other team members to
have to do more work to fix it. Now workshop is nothing new,
it's just our new I guess blind spot enabled by technology.
In the past it would have seemed like a rush piece of work that's
just clearly copied and pasted from a Google article somewhere.
And in the past it was probably even harder to spot and even
harder to create as well. But now it's just so simple to
do on mass because AI is so good at generating words.
And this kind of mirrors a lot of the things that as a company
we've had to speak to each other about.
South Barry, who's the CEO and Co founder of Mindset, compares
it to say giving people a Formula One car.
And just because you've been given a Formula One car doesn't
mean you can drive a Formula One car.
Yes it can go fast, but you will most likely crash if you're not
really thinking through how to drive it.
Because data from very well regarded and big studies have
shown that AI can accelerate work when it's used right.
You know a recent MIT and Nielsen study found that ChatGPT
users writing a business document finished about 59%
faster and produce better outputs than those that didn't
have AI. So it's not that AI will
definitely cause slop, it just needs help so it doesn't produce
slop. And if it is slop, then I would
look at the person, not the tool.
For me, it's when people take their conversation, say when
writing or coding and deciding, oh, that's just good enough,
That'll do. You have to create tests, you
have to interrogate, you have to read and importantly understand
the topic yourself enough to call rubbish.
I say rubbish instead of swearing.
But yeah, you have to use initiative to get it from that
65% or 80% done, to bring it together to be something that's
perfect and ready to send. And every medium has its own
nuances here. I'm not saying that this is a
blanket rule. Whether you're writing, coding,
creating art, you can't really expect things to be done without
applying some level of brain power and analysis to the thing
that you're creating. Now, I'm not saying that people
don't care. I know many don't, but many also
do. And this is where I think
something called the efficiency trap starts to creep in with AI
work slop. Now I think this is a really
interesting take on how companies were very clever
people who probably have the best intentions start creating
rubbish. So someone called Doctor
Cornelia Walther at Wharton has documented what she calls the
efficiency trap, and it basically explains why work slot
can just proliferate in an organization despite everybody's
best intentions not to do so. It folds in four predictable
stages. So stage 1, according to her,
was that workers experiment with AI quite cautiously.
They use it selectively, They maintain lots of control.
Productivity genuinely improves. Tasks that might have taken days
might actually now take hours, and everybody gets quite
excited. Stage 2 is management really
noticing and getting involved, operating under resource
optimization objectives. They're going to increase their
expectations of a team's output. And the logic makes sense.
If technology drives us to deliver more and less time, why
not request more deliverables? And as this happens, AI becomes
very normalised in a person's work process and what was a tool
is now a habit. This is where you get to stage 3
and teams to meet these escalating demands delegate
increasingly complex tasks to AI and that I guess selective
assistance evolves into reliance task requiring independent
analysis and critical thinking become AI driven by default.
You know, dependency accelerates and a lack of processes around
all of this work then leads to more and more work slop
happening. And this is where she describes
stage 4. Each productivity improvement,
she says, becomes that new baseline deadlines reduce, so
you have less time to do more because the number of projects
that get started increases, the complexity of those projects
also increases. And efficiency gains are almost
permanently incorporated into performance standards.
And workers reach what Walther calls technological addiction,
which is feeling psychologically incapable of meeting demands
without using AI. And this workslop increases.
And by the way, this is one of the reasons that I don't think
AI is a bubble, because I have witnessed this.
I feel this. I've been a part of enabling
this. Now, your first instinct might
be, well, people just need better training because that's
what a lot of the time, I guess people default to when talking
about AI work now. Now I think there's also a human
element to this, which I want to explain in a little bit more
detail. Your first instinct might easily
become, you know, just train people and give them more
information and training. And I think that's the easy one
to default to. And you'd definitely be half
right. But here's where there is
probably a little bit of nuance. A big study from BCG, Harvard,
MIT and Wharton examined 758 consultants, roughly 7% of BCG's
individual contributors that were working with CHAT 2BT4.
Now for tasks that were within AI's capabilities, training
really helped. Below average performers
improved by 43% and above average performers gained a 70%
improvement too. So the quality of their outputs
jumped, you know, over 40% compared to those control
groups. That's a big jump.
But where it gets quite interesting, I think, is for
those tasks that are a little bit outside the A IS capability
set, or where that capability is stretched.
Training didn't help. In fact, training actually made
those people who had received training worse.
The people who had received that training on how to create the
prompts were more likely to accept A IS outputs, thinking
that if they prompted it correctly, then the response
must also be correct. So training created
overconfidence without any critical evaluation skills,
domain expertise or good processes to prevent this
happening. Now Professor Ethan Mollick,
who's a fantastic thinker in this space, actually called this
the jagged technological frontier, which is where a IS
capabilities aren't uniform. This means that some tasks are
just incredibly easy for AI, whilst tasks that seem almost
identical become really difficult for AI to achieve.
Now that boundary can be super unpredictable and training alone
doesn't help people navigate it. On Reddit's R slash GPTA,
computer science professor captured this perfectly.
Only 20% of students could effectively use ChatGPT, but for
that 20% it was incredibly impressive.
The differentiator wasn't prompt engineering expertise, it was
just a bit of critical thinking, domain expertise, and knowing
when to ignore the AI. And that's why I think the
reality truly is that this isn't a technology problem.
It's a human systems problem. So how do we fix this mess?
Well, I think, to put it plainly, domain expertise
matters more than ever. If you're not sure what good
looks like, AI can generate plausible first drafts that may
go way over your head. But if you do know your field,
you're the one that's going to be able to spot that mess.
The hallucination, the nonsense that comes out of it.
You see a lot online writers or coders complaining that AI often
makes very subtle errors that for a novice that really doesn't
understand the craft would struggle to notice.
This also ties to a really interesting analogy made by MIT
and Harvard. In their study, they referred to
a Centaur versus a Cyborg. So they said that some workers
act like Centaur. So dividing tasks between AI and
themselves. You know, ChatGPT, give me an
outline. I'll write the content, you
create the review, and others try to become true cyborgs,
fully blending AI into their workflow, sometimes over relying
on that AI system to produce the work.
The most effective teams tend to use a hybrid centauric approach.
Leverage AI for speed, whether it's creating certain templates
and processes or drafting certain text, but with an expert
human steering it to make sure the quality is still really
high. I think of course, another
really important one is training.
Many workers obviously dive into lots of new tools with very
little training or coaching. And if you don't understand the
underlying ways of or underlying processes, should I say you're
going to really struggle to get good outputs.
You know, BCG recently found that only about 1/3 of employees
actually feel properly trained on AI.
Then #3 is what I think is arguably the most important
factor, which is process and quality control breakdowns.
Now, I think few workplaces will have a very good review cycle
around AI drafts. So in the past, say an e-mail or
a document or some slides would be created by a person and then
reviewed by a person. But now AI is able to create so
much of it that probably often skips a review cycle, especially
when people are operating under a deadline.
Again, thinking back to the efficiency trap, now there's
multiple drafts and multiple review cycles that can happen
incredibly quickly. So we have two drafts that are
built by product management. You create a document which is
about essentially the outline of the brief, the things that this
thing must do and why. We then have a specification
document created, which is essentially a planning document,
which takes the vision of the big feature raw product that
you're going to build into an actionable plan for developers.
At both of those stages, you have a review cycle.
Devs will then use clawed code to create an entire
implementation plan, which is then reviewed and then we go
ahead and do the work. Now that can happen 6070% faster
than normal processes because it doesn't require 8 meetings to
get to that end plan. It can happen in literally 30
minute segments and an individual ideating with AI to
get to the right plan. And the same goes for writing.
You know, Gen. AI can ideate and apply for the
writing and help with the research.
Then you can handpick the research and then you can draft
and bring it together yourself. Each should have its own review
cycle. And again, all of these
processes should be baked in with guardrails, contexts on the
business, the process, the frameworks, the even template
outputs. Anyway, go down to the 4th and
final factor that would help fix this massively, and that's
expectations. I think many managers have
jumped on AI expecting kind of magic equals better and more
equals better. And I think it just creates a
trap where workers obviously feel pressure to just crank out
more content for the sake of it. Now we have to talk about levers
to pull in a project. You can pull the speed lever,
you can pull the scope lever, but if you're always trying to
pull that speed lever, then often the quality lever is also
being pulled. And I think solving this is just
as much about resetting expectations.
For example, training programs should focus on ideating with AI
to create your plan and then let humans do some heavy thinking
and actually produce good work. Anyway to to conclude the
episode that this story should be quite clear by now.
Worksop isn't an AI problem, it's a symptom of under prepared
people and processes. You know, when we rush it or
under train ourselves or hide its use or don't put process
around this Formula One car, we get sloppy results.
And conversely, when an organization invests in smart AI
habits, they're going to get real fantastic benefits.
And I think for listeners, the take away is quite actionable.
If you want good AI output at work or for personal use, treat
it like any tool. Learn to use it properly, double
check its work, and loop humans into the process and assume that
it's not going to get the right answer straight away.
Anyway, thanks for listening. That's it for this episode and
I'll see you next week.