1 00:00:00,400 --> 00:00:04,080 A new word just entered the workplace lexicon, Work slop. 2 00:00:04,240 --> 00:00:06,320 If you haven't heard about it yet, you will do soon. 3 00:00:06,600 --> 00:00:09,400 Stanford Research has just coined the term and it's 4 00:00:09,400 --> 00:00:12,200 spreading like wildfire online. So this Stanford report that 5 00:00:12,200 --> 00:00:15,880 came out this September spoke to one of the biggest complaints 6 00:00:15,880 --> 00:00:19,720 when it comes to AI. The most of the work it produces 7 00:00:19,720 --> 00:00:24,880 is a bit crap, with the study finding that 41% of workers who 8 00:00:24,880 --> 00:00:28,880 were surveyed reported receiving work slop from their colleagues 9 00:00:28,880 --> 00:00:30,840 in just the past month. In today's episode, we're 10 00:00:30,840 --> 00:00:34,600 discussing the rise of work slot, the real financial cost of 11 00:00:34,600 --> 00:00:38,360 it, whether this is all just noise, and how we can all fix it 12 00:00:38,360 --> 00:00:40,840 going forward. This is in the loop with Jack 13 00:00:40,840 --> 00:00:42,440 Horton. I hope you enjoy the show. 14 00:00:55,360 --> 00:00:57,920 So I think let's start with what workstop actually is. 15 00:00:58,160 --> 00:01:02,320 So workstop is the term that's used for basically AI generated 16 00:01:02,320 --> 00:01:05,960 content and work content specifically that kind of looks 17 00:01:05,960 --> 00:01:08,600 polished but is actually just a bit rubbish and crap. 18 00:01:09,320 --> 00:01:11,960 And in practice, this looks like, you know, slide decks that 19 00:01:12,520 --> 00:01:15,280 or reports maybe that look really good when you I guess 20 00:01:15,280 --> 00:01:18,120 give it that 5 second glance or you skim through it really 21 00:01:18,120 --> 00:01:20,800 quickly. But as soon as you really dig 22 00:01:20,800 --> 00:01:24,000 into it or want to use it for something practical at work, it 23 00:01:24,000 --> 00:01:28,600 has wrong numbers or repeating texts or every word of a 24 00:01:29,040 --> 00:01:33,160 headline or a sentence is capitalized or the argument in 25 00:01:33,160 --> 00:01:38,360 it is just again, a bit weak and, and repetitive and again, 26 00:01:38,360 --> 00:01:40,840 crap. Now I have got a take on the 27 00:01:40,840 --> 00:01:42,440 real cause of this, which we'll come on to. 28 00:01:42,440 --> 00:01:48,640 But in a survey of over 1100 US workers, 40% of people said they 29 00:01:48,640 --> 00:01:51,120 received AI Workslop in just the past month. 30 00:01:51,120 --> 00:01:54,640 And according to this study, it takes about two hours of extra 31 00:01:54,640 --> 00:01:57,280 work to fix all of this workslop. 32 00:01:58,360 --> 00:02:01,720 And as a result, the researchers estimated that it costs about 33 00:02:01,720 --> 00:02:07,000 $186 per employee per month, which equates to about $9 34 00:02:07,000 --> 00:02:09,840 million a year for a 10,000 person company to fix of it. 35 00:02:10,039 --> 00:02:13,280 Now, of course, there's an element of probable exaggeration 36 00:02:13,280 --> 00:02:17,040 when they are reporting or self reporting the impact of 37 00:02:17,040 --> 00:02:20,760 Workslop, but I think it does speak to a real thing at work 38 00:02:20,760 --> 00:02:22,520 right now. And at this point, I think you 39 00:02:22,520 --> 00:02:25,040 probably know the people at work who do this, you know the team 40 00:02:25,040 --> 00:02:30,400 members who send sloppy AI work. And I think as AI becomes more 41 00:02:30,400 --> 00:02:33,320 mature and widespread, those team members are going to pay a 42 00:02:33,320 --> 00:02:35,800 price. Because, you know, about half of 43 00:02:35,800 --> 00:02:38,920 the surveyed workers reported they regarded slop senders as 44 00:02:38,920 --> 00:02:42,280 less creative, as less capable, or even reliable. 45 00:02:42,480 --> 00:02:46,040 As one redditor said, everyone hands me that AI generated slop 46 00:02:46,040 --> 00:02:48,040 to edit and I have to cut 75% of it. 47 00:02:48,560 --> 00:02:50,960 Usually it's the same info repeated in slightly different 48 00:02:50,960 --> 00:02:52,640 ways. Another quote the study 49 00:02:52,640 --> 00:02:55,200 referenced makes this point even more clear. 50 00:02:55,400 --> 00:02:58,480 If you use AI to pretend to do 20 tasks, you're a Trojan 51 00:02:58,480 --> 00:03:01,280 horsing bunch of work to other people. 52 00:03:01,400 --> 00:03:05,320 Basically by doing rubbish work you cause other team members to 53 00:03:05,320 --> 00:03:09,160 have to do more work to fix it. Now workshop is nothing new, 54 00:03:09,720 --> 00:03:13,040 it's just our new I guess blind spot enabled by technology. 55 00:03:13,640 --> 00:03:17,320 In the past it would have seemed like a rush piece of work that's 56 00:03:17,320 --> 00:03:21,160 just clearly copied and pasted from a Google article somewhere. 57 00:03:21,320 --> 00:03:24,280 And in the past it was probably even harder to spot and even 58 00:03:24,280 --> 00:03:28,720 harder to create as well. But now it's just so simple to 59 00:03:28,720 --> 00:03:32,120 do on mass because AI is so good at generating words. 60 00:03:32,240 --> 00:03:35,800 And this kind of mirrors a lot of the things that as a company 61 00:03:35,800 --> 00:03:37,560 we've had to speak to each other about. 62 00:03:38,240 --> 00:03:42,400 South Barry, who's the CEO and Co founder of Mindset, compares 63 00:03:42,400 --> 00:03:45,520 it to say giving people a Formula One car. 64 00:03:46,040 --> 00:03:48,160 And just because you've been given a Formula One car doesn't 65 00:03:48,160 --> 00:03:49,920 mean you can drive a Formula One car. 66 00:03:50,200 --> 00:03:54,400 Yes it can go fast, but you will most likely crash if you're not 67 00:03:54,400 --> 00:03:56,080 really thinking through how to drive it. 68 00:03:56,120 --> 00:03:59,880 Because data from very well regarded and big studies have 69 00:03:59,880 --> 00:04:02,840 shown that AI can accelerate work when it's used right. 70 00:04:03,320 --> 00:04:06,800 You know a recent MIT and Nielsen study found that ChatGPT 71 00:04:06,800 --> 00:04:09,760 users writing a business document finished about 59% 72 00:04:09,760 --> 00:04:13,280 faster and produce better outputs than those that didn't 73 00:04:13,280 --> 00:04:15,280 have AI. So it's not that AI will 74 00:04:15,440 --> 00:04:18,920 definitely cause slop, it just needs help so it doesn't produce 75 00:04:18,920 --> 00:04:21,480 slop. And if it is slop, then I would 76 00:04:21,480 --> 00:04:23,480 look at the person, not the tool. 77 00:04:24,240 --> 00:04:27,160 For me, it's when people take their conversation, say when 78 00:04:27,160 --> 00:04:30,880 writing or coding and deciding, oh, that's just good enough, 79 00:04:30,880 --> 00:04:33,440 That'll do. You have to create tests, you 80 00:04:33,440 --> 00:04:36,560 have to interrogate, you have to read and importantly understand 81 00:04:36,560 --> 00:04:41,040 the topic yourself enough to call rubbish. 82 00:04:41,240 --> 00:04:42,640 I say rubbish instead of swearing. 83 00:04:42,800 --> 00:04:45,640 But yeah, you have to use initiative to get it from that 84 00:04:45,640 --> 00:04:50,280 65% or 80% done, to bring it together to be something that's 85 00:04:50,280 --> 00:04:52,920 perfect and ready to send. And every medium has its own 86 00:04:52,920 --> 00:04:54,840 nuances here. I'm not saying that this is a 87 00:04:54,840 --> 00:04:59,200 blanket rule. Whether you're writing, coding, 88 00:04:59,200 --> 00:05:03,800 creating art, you can't really expect things to be done without 89 00:05:03,800 --> 00:05:09,440 applying some level of brain power and analysis to the thing 90 00:05:09,440 --> 00:05:11,520 that you're creating. Now, I'm not saying that people 91 00:05:11,520 --> 00:05:14,160 don't care. I know many don't, but many also 92 00:05:14,160 --> 00:05:16,280 do. And this is where I think 93 00:05:16,560 --> 00:05:19,960 something called the efficiency trap starts to creep in with AI 94 00:05:19,960 --> 00:05:32,850 work slop. Now I think this is a really 95 00:05:32,850 --> 00:05:37,170 interesting take on how companies were very clever 96 00:05:37,170 --> 00:05:40,690 people who probably have the best intentions start creating 97 00:05:40,690 --> 00:05:43,370 rubbish. So someone called Doctor 98 00:05:43,570 --> 00:05:48,410 Cornelia Walther at Wharton has documented what she calls the 99 00:05:48,410 --> 00:05:52,520 efficiency trap, and it basically explains why work slot 100 00:05:52,520 --> 00:05:55,720 can just proliferate in an organization despite everybody's 101 00:05:55,720 --> 00:06:00,480 best intentions not to do so. It folds in four predictable 102 00:06:00,480 --> 00:06:03,320 stages. So stage 1, according to her, 103 00:06:03,320 --> 00:06:06,320 was that workers experiment with AI quite cautiously. 104 00:06:06,520 --> 00:06:09,720 They use it selectively, They maintain lots of control. 105 00:06:09,920 --> 00:06:13,840 Productivity genuinely improves. Tasks that might have taken days 106 00:06:13,840 --> 00:06:16,640 might actually now take hours, and everybody gets quite 107 00:06:16,640 --> 00:06:19,120 excited. Stage 2 is management really 108 00:06:19,240 --> 00:06:22,120 noticing and getting involved, operating under resource 109 00:06:22,120 --> 00:06:25,280 optimization objectives. They're going to increase their 110 00:06:25,280 --> 00:06:29,400 expectations of a team's output. And the logic makes sense. 111 00:06:29,400 --> 00:06:33,400 If technology drives us to deliver more and less time, why 112 00:06:33,400 --> 00:06:36,960 not request more deliverables? And as this happens, AI becomes 113 00:06:36,960 --> 00:06:40,880 very normalised in a person's work process and what was a tool 114 00:06:40,880 --> 00:06:43,240 is now a habit. This is where you get to stage 3 115 00:06:44,000 --> 00:06:47,680 and teams to meet these escalating demands delegate 116 00:06:47,840 --> 00:06:53,440 increasingly complex tasks to AI and that I guess selective 117 00:06:53,440 --> 00:06:58,160 assistance evolves into reliance task requiring independent 118 00:06:58,160 --> 00:07:02,080 analysis and critical thinking become AI driven by default. 119 00:07:02,480 --> 00:07:05,360 You know, dependency accelerates and a lack of processes around 120 00:07:05,360 --> 00:07:08,600 all of this work then leads to more and more work slop 121 00:07:08,600 --> 00:07:10,560 happening. And this is where she describes 122 00:07:10,560 --> 00:07:13,960 stage 4. Each productivity improvement, 123 00:07:13,960 --> 00:07:17,960 she says, becomes that new baseline deadlines reduce, so 124 00:07:17,960 --> 00:07:20,880 you have less time to do more because the number of projects 125 00:07:20,880 --> 00:07:24,280 that get started increases, the complexity of those projects 126 00:07:24,520 --> 00:07:27,840 also increases. And efficiency gains are almost 127 00:07:27,840 --> 00:07:30,360 permanently incorporated into performance standards. 128 00:07:31,080 --> 00:07:35,800 And workers reach what Walther calls technological addiction, 129 00:07:36,040 --> 00:07:39,760 which is feeling psychologically incapable of meeting demands 130 00:07:39,760 --> 00:07:43,960 without using AI. And this workslop increases. 131 00:07:44,040 --> 00:07:47,760 And by the way, this is one of the reasons that I don't think 132 00:07:47,840 --> 00:07:50,560 AI is a bubble, because I have witnessed this. 133 00:07:50,560 --> 00:07:53,960 I feel this. I've been a part of enabling 134 00:07:53,960 --> 00:07:55,720 this. Now, your first instinct might 135 00:07:55,720 --> 00:07:58,760 be, well, people just need better training because that's 136 00:07:58,760 --> 00:08:01,760 what a lot of the time, I guess people default to when talking 137 00:08:01,760 --> 00:08:04,400 about AI work now. Now I think there's also a human 138 00:08:04,400 --> 00:08:06,800 element to this, which I want to explain in a little bit more 139 00:08:06,800 --> 00:08:09,160 detail. Your first instinct might easily 140 00:08:09,160 --> 00:08:11,920 become, you know, just train people and give them more 141 00:08:11,920 --> 00:08:15,200 information and training. And I think that's the easy one 142 00:08:15,200 --> 00:08:16,920 to default to. And you'd definitely be half 143 00:08:16,920 --> 00:08:19,000 right. But here's where there is 144 00:08:19,000 --> 00:08:22,640 probably a little bit of nuance. A big study from BCG, Harvard, 145 00:08:22,640 --> 00:08:28,880 MIT and Wharton examined 758 consultants, roughly 7% of BCG's 146 00:08:28,880 --> 00:08:32,280 individual contributors that were working with CHAT 2BT4. 147 00:08:33,000 --> 00:08:36,320 Now for tasks that were within AI's capabilities, training 148 00:08:36,320 --> 00:08:38,240 really helped. Below average performers 149 00:08:38,240 --> 00:08:42,640 improved by 43% and above average performers gained a 70% 150 00:08:42,720 --> 00:08:45,280 improvement too. So the quality of their outputs 151 00:08:45,280 --> 00:08:47,840 jumped, you know, over 40% compared to those control 152 00:08:47,840 --> 00:08:49,720 groups. That's a big jump. 153 00:08:50,800 --> 00:08:53,920 But where it gets quite interesting, I think, is for 154 00:08:53,920 --> 00:08:57,280 those tasks that are a little bit outside the A IS capability 155 00:08:57,280 --> 00:08:59,840 set, or where that capability is stretched. 156 00:09:00,160 --> 00:09:03,400 Training didn't help. In fact, training actually made 157 00:09:03,400 --> 00:09:05,960 those people who had received training worse. 158 00:09:07,000 --> 00:09:09,760 The people who had received that training on how to create the 159 00:09:09,760 --> 00:09:13,680 prompts were more likely to accept A IS outputs, thinking 160 00:09:13,680 --> 00:09:16,240 that if they prompted it correctly, then the response 161 00:09:16,240 --> 00:09:18,600 must also be correct. So training created 162 00:09:18,600 --> 00:09:22,440 overconfidence without any critical evaluation skills, 163 00:09:22,440 --> 00:09:25,160 domain expertise or good processes to prevent this 164 00:09:25,160 --> 00:09:27,200 happening. Now Professor Ethan Mollick, 165 00:09:27,200 --> 00:09:31,400 who's a fantastic thinker in this space, actually called this 166 00:09:31,400 --> 00:09:34,600 the jagged technological frontier, which is where a IS 167 00:09:34,600 --> 00:09:38,880 capabilities aren't uniform. This means that some tasks are 168 00:09:38,880 --> 00:09:42,840 just incredibly easy for AI, whilst tasks that seem almost 169 00:09:42,840 --> 00:09:46,040 identical become really difficult for AI to achieve. 170 00:09:46,280 --> 00:09:49,240 Now that boundary can be super unpredictable and training alone 171 00:09:49,240 --> 00:09:52,480 doesn't help people navigate it. On Reddit's R slash GPTA, 172 00:09:52,480 --> 00:09:55,160 computer science professor captured this perfectly. 173 00:09:55,640 --> 00:09:59,680 Only 20% of students could effectively use ChatGPT, but for 174 00:09:59,680 --> 00:10:01,720 that 20% it was incredibly impressive. 175 00:10:02,280 --> 00:10:05,920 The differentiator wasn't prompt engineering expertise, it was 176 00:10:05,920 --> 00:10:09,000 just a bit of critical thinking, domain expertise, and knowing 177 00:10:09,000 --> 00:10:12,040 when to ignore the AI. And that's why I think the 178 00:10:12,040 --> 00:10:15,840 reality truly is that this isn't a technology problem. 179 00:10:15,880 --> 00:10:30,320 It's a human systems problem. So how do we fix this mess? 180 00:10:30,440 --> 00:10:32,920 Well, I think, to put it plainly, domain expertise 181 00:10:33,360 --> 00:10:35,880 matters more than ever. If you're not sure what good 182 00:10:35,880 --> 00:10:40,000 looks like, AI can generate plausible first drafts that may 183 00:10:40,080 --> 00:10:43,560 go way over your head. But if you do know your field, 184 00:10:43,560 --> 00:10:46,040 you're the one that's going to be able to spot that mess. 185 00:10:46,040 --> 00:10:48,360 The hallucination, the nonsense that comes out of it. 186 00:10:48,520 --> 00:10:53,080 You see a lot online writers or coders complaining that AI often 187 00:10:53,080 --> 00:10:56,600 makes very subtle errors that for a novice that really doesn't 188 00:10:56,600 --> 00:10:59,640 understand the craft would struggle to notice. 189 00:10:59,840 --> 00:11:04,040 This also ties to a really interesting analogy made by MIT 190 00:11:04,040 --> 00:11:05,880 and Harvard. In their study, they referred to 191 00:11:05,880 --> 00:11:09,480 a Centaur versus a Cyborg. So they said that some workers 192 00:11:09,680 --> 00:11:12,640 act like Centaur. So dividing tasks between AI and 193 00:11:12,640 --> 00:11:14,760 themselves. You know, ChatGPT, give me an 194 00:11:14,760 --> 00:11:17,520 outline. I'll write the content, you 195 00:11:17,520 --> 00:11:22,000 create the review, and others try to become true cyborgs, 196 00:11:22,400 --> 00:11:26,480 fully blending AI into their workflow, sometimes over relying 197 00:11:26,480 --> 00:11:29,040 on that AI system to produce the work. 198 00:11:29,600 --> 00:11:33,720 The most effective teams tend to use a hybrid centauric approach. 199 00:11:34,200 --> 00:11:37,800 Leverage AI for speed, whether it's creating certain templates 200 00:11:37,800 --> 00:11:41,320 and processes or drafting certain text, but with an expert 201 00:11:41,320 --> 00:11:45,040 human steering it to make sure the quality is still really 202 00:11:45,040 --> 00:11:46,520 high. I think of course, another 203 00:11:46,520 --> 00:11:48,120 really important one is training. 204 00:11:48,320 --> 00:11:50,800 Many workers obviously dive into lots of new tools with very 205 00:11:50,800 --> 00:11:53,520 little training or coaching. And if you don't understand the 206 00:11:53,520 --> 00:11:57,880 underlying ways of or underlying processes, should I say you're 207 00:11:57,880 --> 00:11:59,560 going to really struggle to get good outputs. 208 00:11:59,760 --> 00:12:02,800 You know, BCG recently found that only about 1/3 of employees 209 00:12:02,800 --> 00:12:04,800 actually feel properly trained on AI. 210 00:12:04,960 --> 00:12:08,320 Then #3 is what I think is arguably the most important 211 00:12:08,800 --> 00:12:12,160 factor, which is process and quality control breakdowns. 212 00:12:12,320 --> 00:12:15,440 Now, I think few workplaces will have a very good review cycle 213 00:12:15,440 --> 00:12:19,040 around AI drafts. So in the past, say an e-mail or 214 00:12:19,040 --> 00:12:23,080 a document or some slides would be created by a person and then 215 00:12:23,080 --> 00:12:26,720 reviewed by a person. But now AI is able to create so 216 00:12:26,720 --> 00:12:29,960 much of it that probably often skips a review cycle, especially 217 00:12:29,960 --> 00:12:31,640 when people are operating under a deadline. 218 00:12:31,640 --> 00:12:33,800 Again, thinking back to the efficiency trap, now there's 219 00:12:33,800 --> 00:12:37,080 multiple drafts and multiple review cycles that can happen 220 00:12:37,080 --> 00:12:40,480 incredibly quickly. So we have two drafts that are 221 00:12:40,480 --> 00:12:43,320 built by product management. You create a document which is 222 00:12:43,320 --> 00:12:46,440 about essentially the outline of the brief, the things that this 223 00:12:46,440 --> 00:12:49,240 thing must do and why. We then have a specification 224 00:12:49,240 --> 00:12:51,600 document created, which is essentially a planning document, 225 00:12:51,800 --> 00:12:55,680 which takes the vision of the big feature raw product that 226 00:12:55,680 --> 00:12:58,520 you're going to build into an actionable plan for developers. 227 00:12:59,280 --> 00:13:01,640 At both of those stages, you have a review cycle. 228 00:13:01,800 --> 00:13:04,320 Devs will then use clawed code to create an entire 229 00:13:04,320 --> 00:13:08,040 implementation plan, which is then reviewed and then we go 230 00:13:08,040 --> 00:13:12,080 ahead and do the work. Now that can happen 6070% faster 231 00:13:12,240 --> 00:13:15,320 than normal processes because it doesn't require 8 meetings to 232 00:13:15,320 --> 00:13:19,000 get to that end plan. It can happen in literally 30 233 00:13:19,000 --> 00:13:22,240 minute segments and an individual ideating with AI to 234 00:13:22,240 --> 00:13:25,280 get to the right plan. And the same goes for writing. 235 00:13:25,280 --> 00:13:27,320 You know, Gen. AI can ideate and apply for the 236 00:13:27,320 --> 00:13:29,000 writing and help with the research. 237 00:13:29,240 --> 00:13:32,160 Then you can handpick the research and then you can draft 238 00:13:32,160 --> 00:13:34,640 and bring it together yourself. Each should have its own review 239 00:13:34,640 --> 00:13:36,240 cycle. And again, all of these 240 00:13:36,240 --> 00:13:39,720 processes should be baked in with guardrails, contexts on the 241 00:13:39,720 --> 00:13:43,640 business, the process, the frameworks, the even template 242 00:13:43,640 --> 00:13:46,040 outputs. Anyway, go down to the 4th and 243 00:13:46,040 --> 00:13:49,000 final factor that would help fix this massively, and that's 244 00:13:49,000 --> 00:13:51,880 expectations. I think many managers have 245 00:13:51,880 --> 00:13:55,120 jumped on AI expecting kind of magic equals better and more 246 00:13:55,120 --> 00:13:57,640 equals better. And I think it just creates a 247 00:13:57,640 --> 00:14:00,680 trap where workers obviously feel pressure to just crank out 248 00:14:00,680 --> 00:14:03,960 more content for the sake of it. Now we have to talk about levers 249 00:14:03,960 --> 00:14:06,640 to pull in a project. You can pull the speed lever, 250 00:14:07,600 --> 00:14:10,360 you can pull the scope lever, but if you're always trying to 251 00:14:10,360 --> 00:14:14,680 pull that speed lever, then often the quality lever is also 252 00:14:14,680 --> 00:14:17,120 being pulled. And I think solving this is just 253 00:14:17,120 --> 00:14:19,480 as much about resetting expectations. 254 00:14:19,600 --> 00:14:23,240 For example, training programs should focus on ideating with AI 255 00:14:23,240 --> 00:14:27,120 to create your plan and then let humans do some heavy thinking 256 00:14:27,400 --> 00:14:41,560 and actually produce good work. Anyway to to conclude the 257 00:14:41,560 --> 00:14:45,120 episode that this story should be quite clear by now. 258 00:14:45,320 --> 00:14:49,400 Worksop isn't an AI problem, it's a symptom of under prepared 259 00:14:49,400 --> 00:14:52,800 people and processes. You know, when we rush it or 260 00:14:52,800 --> 00:14:56,200 under train ourselves or hide its use or don't put process 261 00:14:56,200 --> 00:14:59,920 around this Formula One car, we get sloppy results. 262 00:15:00,640 --> 00:15:04,000 And conversely, when an organization invests in smart AI 263 00:15:04,000 --> 00:15:06,960 habits, they're going to get real fantastic benefits. 264 00:15:07,720 --> 00:15:10,840 And I think for listeners, the take away is quite actionable. 265 00:15:11,080 --> 00:15:15,120 If you want good AI output at work or for personal use, treat 266 00:15:15,120 --> 00:15:17,920 it like any tool. Learn to use it properly, double 267 00:15:17,920 --> 00:15:21,680 check its work, and loop humans into the process and assume that 268 00:15:22,040 --> 00:15:24,320 it's not going to get the right answer straight away. 269 00:15:24,760 --> 00:15:27,080 Anyway, thanks for listening. That's it for this episode and 270 00:15:27,080 --> 00:15:28,520 I'll see you next week.