1 00:00:03,280 --> 00:00:05,880 Hello and welcome to the Behavioral Design Podcast. 2 00:00:05,960 --> 00:00:08,680 This season we're diving into the intersection of behavioral 3 00:00:08,680 --> 00:00:11,080 science and AI. We want to make sense of the 4 00:00:11,080 --> 00:00:14,080 state of AI, from understanding how humans interact with 5 00:00:14,080 --> 00:00:17,520 intelligent systems to using AI to do behavioral design itself. 6 00:00:17,720 --> 00:00:21,280 I'm Aline Holsworth, a health tech advisor specializing in AI 7 00:00:21,280 --> 00:00:23,880 and product design. Over the past 15 years, I've 8 00:00:23,880 --> 00:00:26,720 been crafting human centered products with behavioral science 9 00:00:26,720 --> 00:00:29,160 at the core. At Apple, I LED Behavioral 10 00:00:29,160 --> 00:00:32,119 Science for Health AI, designing and launching AI powered 11 00:00:32,119 --> 00:00:34,440 features to help users reach their health goals. 12 00:00:35,160 --> 00:00:37,160 And I'm Samuel Sultzer, your second Co host. 13 00:00:37,400 --> 00:00:40,600 I'm a behavioral strategist specializing in hybrid formation 14 00:00:40,600 --> 00:00:43,200 and designing products that drive long term baby change. 15 00:00:43,640 --> 00:00:46,880 I work with leading tech organizations integrating AI to 16 00:00:46,880 --> 00:00:48,240 scale behavioral design for good. 17 00:00:48,800 --> 00:00:52,200 And I'm also the founder of Baby Bites, a dedicated community on 18 00:00:52,440 --> 00:00:56,240 behavioral science and AI. Quick word on Nuance Behavior 19 00:00:56,240 --> 00:00:59,640 where we help organizations build impactful digital products 20 00:00:59,640 --> 00:01:02,640 using behavioral design. We only take on a few clients at 21 00:01:02,640 --> 00:01:05,200 a time to ensure the highest level of quality for our 22 00:01:05,200 --> 00:01:07,400 tailored evidence based solutions. 23 00:01:08,280 --> 00:01:11,080 If you'd like to become one of our special projects, e-mail us 24 00:01:11,120 --> 00:01:14,760 at hello@nuancebehavior.com or we could call directly on our 25 00:01:14,760 --> 00:01:31,800 website, nuancebehavior.com. Hey, Lynn. 26 00:01:33,760 --> 00:01:37,240 Hey, I was really excited to actually share something that I 27 00:01:37,280 --> 00:01:40,920 realized very recently and we covered obviously in the 28 00:01:40,920 --> 00:01:42,440 previous episode with Gordon Pennycook. 29 00:01:43,040 --> 00:01:44,880 You know how to think about misinformation. 30 00:01:44,880 --> 00:01:48,200 And I don't know if this fully qualifies, but it was certainly 31 00:01:48,200 --> 00:01:53,440 something that I was not aware of and had a poor information 32 00:01:53,440 --> 00:01:57,640 around, which was to maybe no surprise based on our previous 33 00:01:57,640 --> 00:02:01,640 conversation. But The Wizard of Oz is a film 34 00:02:02,320 --> 00:02:09,000 and you had a really great dress up I saw on Halloween with the 35 00:02:09,000 --> 00:02:12,080 theme of Wizard of Oz. And we talked about that in our 36 00:02:12,080 --> 00:02:13,440 Halloween themed episode as well. 37 00:02:13,880 --> 00:02:17,840 And I actually ended up watching Wizard of Oz. 38 00:02:18,080 --> 00:02:23,440 No, wow, you're now informed. Wonderful. 39 00:02:24,440 --> 00:02:27,280 So yeah, how did it go? It was good. 40 00:02:27,280 --> 00:02:29,640 It was a it was a good film. I can see why why it has 41 00:02:29,640 --> 00:02:33,840 received such kind of acclaim. And yeah, I'm happy to finally 42 00:02:33,840 --> 00:02:38,280 say that I've now watched that film and I can even more so tell 43 00:02:38,280 --> 00:02:40,240 you that your costumes were amazing. 44 00:02:40,560 --> 00:02:43,160 So kudos to you and your family. You guys really pulled off an 45 00:02:43,160 --> 00:02:45,440 amazing Halloween. That means a lot. 46 00:02:46,320 --> 00:02:47,880 Thank you. Yeah. 47 00:02:47,880 --> 00:02:51,680 So we spoke to Gordon in Part 1 episode that came out just a few 48 00:02:51,680 --> 00:02:56,080 days ago about the state of AI fuels misinformation and how to 49 00:02:56,080 --> 00:02:59,480 think about misinformation in this kind of modern context and 50 00:02:59,480 --> 00:03:01,600 especially as it relates to elections. 51 00:03:01,920 --> 00:03:05,440 And So what does the landscape look like, What are we dealing 52 00:03:05,440 --> 00:03:09,040 with and so on. And so with that foundation now 53 00:03:09,040 --> 00:03:13,600 in place, we now in today's episode in Part 2, really want 54 00:03:13,600 --> 00:03:18,280 to tackle what can we do about it, What can we do to combat or 55 00:03:18,280 --> 00:03:22,760 manage or reduce misinformation. Yeah. 56 00:03:23,000 --> 00:03:27,360 And actually our our listeners might recall we have already 57 00:03:27,360 --> 00:03:30,320 stepped into this topic in the past. 58 00:03:30,320 --> 00:03:34,600 We had Lindsay Juarez on the show a while back and she talked 59 00:03:34,600 --> 00:03:38,600 about an intervention that she did on TikTok in partnering with 60 00:03:38,600 --> 00:03:41,680 TikTok to slow down the spread of misinformation there. 61 00:03:41,920 --> 00:03:44,840 And their intervention really fell into this category of a 62 00:03:44,840 --> 00:03:48,280 nudge, as we'll as we'll get into with Gordon, where they 63 00:03:48,280 --> 00:03:52,200 added an accuracy prompt and then some friction to sharing. 64 00:03:52,200 --> 00:03:55,120 And they had some some really outsize effects. 65 00:03:55,800 --> 00:03:58,800 They reduce the shares of something that was sort of 66 00:03:58,800 --> 00:04:04,200 dubiously true by 24%. And they also reduce the likes 67 00:04:04,200 --> 00:04:07,120 and views by some additional percentage points. 68 00:04:07,120 --> 00:04:09,520 So very impressive work from Lindsay. 69 00:04:09,680 --> 00:04:13,160 And we will get into maybe what you might think of as a higher 70 00:04:13,160 --> 00:04:16,880 level view of stopping the spread of misinformation with 71 00:04:16,880 --> 00:04:18,720 Gordon. Yeah. 72 00:04:18,800 --> 00:04:21,360 And so that episode is hidden under existential questions with 73 00:04:21,360 --> 00:04:23,440 her, which is final title for that episode. 74 00:04:23,440 --> 00:04:26,320 I forgot that we call that episode that, but we did talk 75 00:04:26,320 --> 00:04:28,200 about a lot of existential questions as well with her. 76 00:04:28,200 --> 00:04:30,040 So yeah, that's a great episode to check out. 77 00:04:31,040 --> 00:04:35,600 But yeah, again, we're speaking with Gordon Pennycook and we 78 00:04:35,640 --> 00:04:40,080 welcome him as we feel is really important to during especially 79 00:04:40,080 --> 00:04:45,480 the current time makes sense of misinformation elections and how 80 00:04:45,480 --> 00:04:47,600 to think about that from a behavioral science perspective. 81 00:04:47,960 --> 00:04:50,960 And we're really lucky to have Gordon because he's a professor 82 00:04:50,960 --> 00:04:55,640 of psychology at Cornell. He has extensively done many 83 00:04:55,640 --> 00:05:00,040 different types of studies and research around where we can go 84 00:05:00,040 --> 00:05:04,080 wrong in our reasoning and how we can support people in 85 00:05:05,040 --> 00:05:09,240 reducing the risk to either be faced with misinformation or if 86 00:05:09,240 --> 00:05:12,280 faced with it, how they can better make sense of it or 87 00:05:12,480 --> 00:05:15,600 debunk it, for example. And so again, in this episode, 88 00:05:15,600 --> 00:05:19,240 we go head on into what we can do about misinformation. 89 00:05:19,320 --> 00:05:21,400 And So what can people look forward to? 90 00:05:22,240 --> 00:05:25,520 So much Sam. So first we take this high level 91 00:05:25,520 --> 00:05:28,720 view of different strategies for combating misinformation. 92 00:05:28,720 --> 00:05:32,360 So comparing and contrasting them, really, really putting 93 00:05:32,360 --> 00:05:35,160 Gordon on the spot to answer all of our many, many questions 94 00:05:35,720 --> 00:05:36,960 about these different strategies. 95 00:05:37,120 --> 00:05:40,040 What works, what doesn't, you know, how do we compare the, the 96 00:05:40,080 --> 00:05:43,480 efficacy and you know, the, the practical effect size so on. 97 00:05:44,560 --> 00:05:47,240 And in particular, looking at these three categories of 98 00:05:47,240 --> 00:05:49,720 nudges. So like, like what we described 99 00:05:49,720 --> 00:05:53,440 with Lindsay boosts and educational interventions and 100 00:05:53,440 --> 00:05:56,920 then the third category of refutation strategies. 101 00:05:56,920 --> 00:06:00,040 So we were really dive in deep into those. 102 00:06:00,280 --> 00:06:04,200 And then of course, staying with our theme of AI this season, we 103 00:06:04,200 --> 00:06:07,920 take a look at the debunking chatbot that Gordon created 104 00:06:07,920 --> 00:06:11,920 along with his co-authors Thomas Costello and David Rand, which 105 00:06:11,920 --> 00:06:16,360 they call their debunk bot. And finally, we close with 106 00:06:16,360 --> 00:06:19,040 Gordon's most controversial opinion about. 107 00:06:19,280 --> 00:06:23,080 AI actually, most importantly, we should mention that you 108 00:06:23,080 --> 00:06:27,280 should also expect to learn what does Grand Theft Auto teach us 109 00:06:27,280 --> 00:06:33,000 about misinformation? I knew you were going to go 110 00:06:33,000 --> 00:06:35,640 there, Happens to. Mugatroid. 111 00:06:43,440 --> 00:06:48,080 Wow. OK, happy to say welcome back 112 00:06:48,160 --> 00:06:50,400 Gordon to the Beetles and podcast. 113 00:06:51,040 --> 00:06:53,480 Thanks for having me back. Yeah. 114 00:06:53,560 --> 00:06:56,160 And in the spirit of kind of misinformation and so on, is 115 00:06:56,160 --> 00:06:58,360 there anything that you said last time that you feel like you 116 00:06:58,360 --> 00:07:00,720 regret or if you like, could potentially have some form of 117 00:07:00,720 --> 00:07:03,320 fake news? No, and I I can honestly say 118 00:07:03,320 --> 00:07:06,040 that I don't recall what I said or thought about it after the 119 00:07:06,040 --> 00:07:07,400 conversation. Smart. 120 00:07:07,560 --> 00:07:08,760 That's that's the smart thing to say. 121 00:07:09,240 --> 00:07:11,000 Best way to live? Yeah, no regrets. 122 00:07:11,880 --> 00:07:13,120 Exactly. I live in the present. 123 00:07:15,000 --> 00:07:17,840 Also, I can say that I don't regret one thing and I just want 124 00:07:17,840 --> 00:07:21,720 to like underline this is that I, I feel like I gave you the 125 00:07:21,840 --> 00:07:25,200 the short end of the stick of, of really appreciating your 126 00:07:25,200 --> 00:07:28,240 knowledge of bullshit. Like I knew of your knowledge of 127 00:07:28,240 --> 00:07:32,440 behavioral science as the BS, but also like clearly I should 128 00:07:32,440 --> 00:07:35,920 have really also known about how well you know about bullshit, 129 00:07:35,920 --> 00:07:39,040 especially around this. So I just want to again take my 130 00:07:39,040 --> 00:07:43,000 hat off to you for like, knowing the BS of BS and yeah. 131 00:07:44,440 --> 00:07:47,200 Truly unnecessary, and I'm not sure that's a fine point of 132 00:07:47,200 --> 00:07:50,040 distinction anyway. So, but thank you I guess. 133 00:07:52,400 --> 00:07:54,520 Cool. Last time we we spoke about 134 00:07:54,600 --> 00:07:57,600 understanding and the state of misinformation, and I guess 135 00:07:58,040 --> 00:08:00,400 today we'll speak about how we can deal with it. 136 00:08:01,080 --> 00:08:04,640 Yeah, let's let's jump right in. Gordon, you have written 137 00:08:04,640 --> 00:08:09,320 extensively and researched quite a bit on what to do about 138 00:08:09,320 --> 00:08:13,160 misinformation. So in one of your papers that 139 00:08:13,160 --> 00:08:17,920 you were one tool in the box of tools, you categorized some 140 00:08:17,920 --> 00:08:21,200 different strategies. And so you and your co-authors 141 00:08:21,200 --> 00:08:24,320 said if we were to break down the different strategies, we 142 00:08:24,320 --> 00:08:26,960 might put them into three major categories. 143 00:08:26,960 --> 00:08:32,600 So #1 nudges, targeting behaviors, many ways of of doing 144 00:08:32,600 --> 00:08:37,760 this, like accuracy prompts, using friction, some labels or 145 00:08:37,760 --> 00:08:41,159 two boosts, and educational interventions. 146 00:08:41,159 --> 00:08:43,720 So these target people's competencies. 147 00:08:43,720 --> 00:08:48,200 Are you able to identify misinformation and know what to 148 00:08:48,200 --> 00:08:51,880 question and so on, like media literacy tips, that sort of 149 00:08:51,880 --> 00:08:53,640 thing. And then the third category, 150 00:08:53,680 --> 00:08:57,080 refutation strategies. So these target beliefs, things 151 00:08:57,080 --> 00:09:01,120 like debunking and rebuttals, more different kinds of warnings 152 00:09:01,120 --> 00:09:07,080 and fact checking labels. So can you tell me, given this 153 00:09:07,080 --> 00:09:12,680 whole toolbox of ways to squash the spread of misinformation, 154 00:09:12,960 --> 00:09:16,560 but how do you sort of think of these in relation to one 155 00:09:16,560 --> 00:09:19,080 another? Can we compare and contrast the 156 00:09:19,080 --> 00:09:22,720 different categories? Great question. 157 00:09:23,000 --> 00:09:26,360 One side note about that paper. So it was really Anastasia 158 00:09:26,360 --> 00:09:30,240 Kozariva who put together the toolbox, which was organizing a 159 00:09:30,240 --> 00:09:33,320 bunch of people doing research on interventions against 160 00:09:33,320 --> 00:09:36,240 misinformation to kind of like write out details about the 161 00:09:36,240 --> 00:09:39,880 different types of interventions that they've worked on and also 162 00:09:40,240 --> 00:09:42,560 about other people's work that they've done and so on. 163 00:09:43,000 --> 00:09:46,600 And then they created these kind of categories, types of 164 00:09:46,800 --> 00:09:50,320 interventions to help policy makers decide what to do and how 165 00:09:50,320 --> 00:09:52,640 to contrast them. So yeah, so you mentioned three 166 00:09:52,640 --> 00:09:55,400 different kind of broad categories that have different 167 00:09:55,400 --> 00:09:57,400 strengths and weaknesses. So let's start with the 168 00:09:57,440 --> 00:10:00,200 reputational strategies, which is I think they kind of standard 169 00:10:00,200 --> 00:10:02,720 fair. Like when people usually are 170 00:10:02,720 --> 00:10:05,440 thinking about how to deal with misinformation, they're thinking 171 00:10:05,440 --> 00:10:07,160 about fact checking it and debunking it. 172 00:10:08,400 --> 00:10:10,360 And like a lot of there's a huge amount of work on that sort of 173 00:10:10,360 --> 00:10:11,800 stuff. And we published work on fact 174 00:10:11,800 --> 00:10:14,800 checking also. And the, the kind of primary 175 00:10:14,800 --> 00:10:17,800 issue there is that it's really just a Band-Aid solution. 176 00:10:18,560 --> 00:10:20,560 Like if you're going to refute falsehoods, you have to make 177 00:10:20,560 --> 00:10:23,320 sure that you have the right reputation. 178 00:10:23,320 --> 00:10:26,320 You have to be able to get that to people who saw the falsehood, 179 00:10:26,320 --> 00:10:28,280 which is never really possible to catch everybody. 180 00:10:28,840 --> 00:10:31,120 So it's really just mitigating loss. 181 00:10:31,680 --> 00:10:35,280 And so there's other types of interventions that try to kind 182 00:10:35,280 --> 00:10:38,120 of get ahead of misinformation, build up people's business. 183 00:10:38,120 --> 00:10:40,520 And there's these boosts or educational interventions are 184 00:10:40,520 --> 00:10:44,000 like things that give people the knowledge or the information 185 00:10:44,440 --> 00:10:47,440 that they need so that they will be better able to like identify 186 00:10:48,040 --> 00:10:49,880 or would they be less likely to share misinformation if they see 187 00:10:49,880 --> 00:10:52,360 it online, for example? So like you said, lateral 188 00:10:52,360 --> 00:10:55,560 reading and media literacy, all these types of things, that's 189 00:10:55,560 --> 00:10:57,080 important. That's not even inherently about 190 00:10:57,080 --> 00:10:59,600 misinformation. That's just like education is a 191 00:10:59,600 --> 00:11:03,680 like literally just going to school and getting educated or 192 00:11:03,680 --> 00:11:06,360 listening to podcasts maybe is a sort of boost. 193 00:11:06,440 --> 00:11:07,640 It's just teaching people things. 194 00:11:07,640 --> 00:11:11,080 And then there's like this more kind of behavioral thing, which 195 00:11:11,080 --> 00:11:15,200 is nudges, which is just trying to change something about the 196 00:11:15,200 --> 00:11:17,920 architecture of the way that people are engaging online so 197 00:11:17,920 --> 00:11:22,080 that it improves the choices. And so you mentioned friction. 198 00:11:22,480 --> 00:11:24,160 We came up with actually prompts. 199 00:11:24,200 --> 00:11:27,080 I think it might be worthwhile talking about the actually 200 00:11:27,080 --> 00:11:30,680 prompt thing for a second because it does sort of evade 201 00:11:30,760 --> 00:11:33,600 easy categorization. And we'll highlight some of 202 00:11:33,600 --> 00:11:36,360 these differences. So like an actually prompt is 203 00:11:36,360 --> 00:11:39,000 reminding people about the importance of accuracy in 204 00:11:39,000 --> 00:11:42,200 various subtle ways. And so it's categorized as a 205 00:11:42,200 --> 00:11:44,920 nudge because it's focusing on like a targeting of behavior, 206 00:11:44,920 --> 00:11:46,800 like you're, you're trying to get people not to share 207 00:11:46,800 --> 00:11:48,720 falsehood by thinking about whether something is true or 208 00:11:48,720 --> 00:11:52,000 false and you're not giving them kind of any new information. 209 00:11:52,000 --> 00:11:53,360 So it doesn't seem that educational. 210 00:11:53,960 --> 00:11:56,120 Yeah, just telling them to reflect. 211 00:11:56,320 --> 00:11:58,840 Yeah, but you're not telling them to you just you're not 212 00:11:58,840 --> 00:12:00,600 saying please think about accuracy. 213 00:12:00,600 --> 00:12:04,480 That's like a directive. It's more like you ask someone, 214 00:12:04,480 --> 00:12:06,640 hey, do you think it's important to only share accurate things 215 00:12:07,480 --> 00:12:09,200 and then they'll share more accurate things after that, you 216 00:12:09,360 --> 00:12:11,120 know what I mean? Or like it's just like an ad. 217 00:12:11,120 --> 00:12:13,840 We've done this with ads before where you like, we just bought a 218 00:12:13,840 --> 00:12:17,680 bunch of ads on Twitter that were about like reminding people 219 00:12:17,680 --> 00:12:20,400 about accuracy and how it's important cuz everyone agrees 220 00:12:20,400 --> 00:12:22,160 with that. It's not even connected to a 221 00:12:22,160 --> 00:12:25,680 particular piece of content. It's just like believe in true 222 00:12:25,680 --> 00:12:26,800 things. Yeah. 223 00:12:26,800 --> 00:12:28,320 Like, do you want to share things that are true? 224 00:12:28,520 --> 00:12:30,480 And the people are like, yeah, that's great, I would love to 225 00:12:30,480 --> 00:12:32,600 share true things. But sometimes people share 226 00:12:32,600 --> 00:12:35,360 things that they don't like. What we found in our experiments 227 00:12:35,360 --> 00:12:37,640 is that people will share things without even thinking about 228 00:12:37,640 --> 00:12:40,720 whether they're true or false. And so it's just targeting that 229 00:12:40,960 --> 00:12:44,120 like mode of behavior, but it's ultimately targeting a thought 230 00:12:44,120 --> 00:12:48,120 process. So the reason why that's kind of 231 00:12:48,200 --> 00:12:50,680 hard to categorize is because it's not changing the 232 00:12:50,680 --> 00:12:52,680 information. It's not like changing the way 233 00:12:52,680 --> 00:12:55,480 the architecture of the choice, which is a typical kind of 234 00:12:55,480 --> 00:12:58,080 nudge. It's just changing the way that 235 00:12:58,080 --> 00:13:00,560 people are thinking about how they're using the medium. 236 00:13:01,120 --> 00:13:03,600 So it's kind of unique it, but it ultimately has the same 237 00:13:04,680 --> 00:13:07,360 general bend which is just trying to target a specific 238 00:13:07,360 --> 00:13:09,360 behavior with a very simple sort of message. 239 00:13:10,200 --> 00:13:12,680 Would you say that it kind of falls into like the priming box 240 00:13:12,680 --> 00:13:16,440 or how would you think about it in relation to something like 241 00:13:16,440 --> 00:13:19,080 that? So it's more just a reminder or 242 00:13:19,080 --> 00:13:21,120 yeah. That's an interesting question. 243 00:13:21,120 --> 00:13:25,680 Also, we debated about this because people who like priming 244 00:13:25,680 --> 00:13:28,600 did put it in the priming box, but it doesn't. 245 00:13:29,040 --> 00:13:32,560 It's not obviously priming either, because it's not like. 246 00:13:32,680 --> 00:13:35,080 Like it's not subtle enough to be priming. 247 00:13:35,640 --> 00:13:38,320 Part of that is people already care about accuracy. 248 00:13:38,880 --> 00:13:41,840 It's just like something that isn't top of mind when they're 249 00:13:41,840 --> 00:13:43,960 kind of mindlessly scrolling on social media. 250 00:13:44,440 --> 00:13:47,240 And so we're just kind of like bringing something that's there 251 00:13:47,240 --> 00:13:49,560 up to have more importance in a decision. 252 00:13:50,160 --> 00:13:54,000 And so that's a lot different than like a giving someone hot 253 00:13:54,000 --> 00:13:57,280 coffee and making them feel warm, like psychologically, you 254 00:13:57,280 --> 00:14:00,160 know what I mean, These. Classic, classic. 255 00:14:01,560 --> 00:14:03,280 Yeah, maybe it's more of a reminder than. 256 00:14:03,720 --> 00:14:05,560 Yeah. So it's essentially a reminder, 257 00:14:05,720 --> 00:14:09,320 you know, is a stop sign or a yield sign priming, you know 258 00:14:09,320 --> 00:14:12,040 what I mean? Like people might not even 259 00:14:12,040 --> 00:14:14,000 notice it and they'll still react to it. 260 00:14:14,160 --> 00:14:16,760 Yeah, it's more like a stop sign or like a yield sign that are 261 00:14:16,760 --> 00:14:20,000 just like a priming. So now we sort of have these 262 00:14:20,000 --> 00:14:23,320 general categories. How would you compare them 263 00:14:23,320 --> 00:14:26,120 against each other in terms of efficacy? 264 00:14:26,320 --> 00:14:29,200 Do some of these strategies work better than others? 265 00:14:29,200 --> 00:14:31,160 You. Know what's funny about this is 266 00:14:31,160 --> 00:14:34,480 that so there was this big mega study that was done. 267 00:14:34,720 --> 00:14:37,320 I'm not sure if you're familiar with it, but what happened was 268 00:14:38,320 --> 00:14:40,400 so people have been doing all this research on different types 269 00:14:40,400 --> 00:14:42,680 of interventions, but they always use different outcomes 270 00:14:42,960 --> 00:14:44,840 and different stimuli like different measures. 271 00:14:45,720 --> 00:14:49,120 And so it's very hard to know based on the literature whether 272 00:14:49,120 --> 00:14:55,240 like a warning label is as effective as a lateral reading 273 00:14:55,240 --> 00:14:57,280 intervention, you know, and like also these are such Apple and 274 00:14:57,280 --> 00:14:59,400 the origins, it's hard to know. And so there was this big mega 275 00:14:59,400 --> 00:15:01,360 study that was we had a bunch of people involved. 276 00:15:01,720 --> 00:15:04,320 Everyone had different interventions, but the same 277 00:15:04,320 --> 00:15:07,240 stimuli, the same material, and usually the same sort of 278 00:15:07,240 --> 00:15:09,160 outcomes. And they were all basically the 279 00:15:09,160 --> 00:15:11,440 same. Everything worked about the 280 00:15:11,440 --> 00:15:13,480 same, like decently well, you know. 281 00:15:14,320 --> 00:15:16,640 Yeah. And what is decently well, can 282 00:15:16,640 --> 00:15:19,320 you translate that into something that something like a 283 00:15:19,320 --> 00:15:22,600 practical measure that everyone might understand? 284 00:15:23,960 --> 00:15:27,320 No, I can't, sorry. The problem is this. 285 00:15:27,560 --> 00:15:31,840 So we run an experiment and what happens is in these studies, 286 00:15:32,480 --> 00:15:35,680 there's for example, we, we create a set of stimuli which is 287 00:15:35,680 --> 00:15:37,680 like true and false news headlines, OK. 288 00:15:38,120 --> 00:15:40,480 And often in these studies, because we want a good estimate 289 00:15:40,480 --> 00:15:44,960 of both, whether people are like believing false things less or 290 00:15:44,960 --> 00:15:47,480 true things more, we have like an equal amount of true and 291 00:15:47,480 --> 00:15:49,640 false things. And the false things are like 292 00:15:49,640 --> 00:15:52,000 fact checked as false things. And so we can say as researchers 293 00:15:52,000 --> 00:15:53,480 that these are things that definitely fit into that 294 00:15:53,480 --> 00:15:55,800 category. In that set of experiments, we 295 00:15:55,800 --> 00:15:58,360 also had things that were categorized as misleading, but 296 00:15:58,360 --> 00:16:00,880 they're often also very clearly misleading. 297 00:16:01,200 --> 00:16:03,600 And so that's like you have to quit these clear categories. 298 00:16:04,040 --> 00:16:07,120 And then you might find that like, say for the action prompt 299 00:16:07,120 --> 00:16:10,920 thing, sharing a misinformation is like 20% lower. 300 00:16:11,800 --> 00:16:15,240 And then you say, OK, well, then if I put that out in the world, 301 00:16:15,240 --> 00:16:17,360 then people will share 20% less false things. 302 00:16:17,360 --> 00:16:20,880 But like, you don't know that because not only did we not run 303 00:16:20,880 --> 00:16:24,040 it in the world, we ran it with this balanced set of true and 304 00:16:24,040 --> 00:16:26,160 false things that were a particular type of false 305 00:16:26,160 --> 00:16:28,720 category. And so I could give you some 306 00:16:28,720 --> 00:16:32,960 answer of like, but it's, it's not, it's not going to be 307 00:16:33,000 --> 00:16:34,600 legitimate. It's like we know that the 308 00:16:34,600 --> 00:16:36,880 psychology behind it is legitimate. 309 00:16:36,880 --> 00:16:40,000 We know that these things have an effect when scaled. 310 00:16:40,000 --> 00:16:41,720 We don't really know what the effect would be. 311 00:16:41,720 --> 00:16:44,640 We think that there's reasonable expectation that they would have 312 00:16:44,640 --> 00:16:46,800 some effect, but it really depends on what people are 313 00:16:46,800 --> 00:16:48,640 engaging with, what their information environment is like, 314 00:16:48,960 --> 00:16:50,680 how much attention they're paying to the intervention. 315 00:16:51,080 --> 00:16:53,440 You know, like prior exposure to these things. 316 00:16:53,880 --> 00:16:55,160 It's hard to say. Right. 317 00:16:55,160 --> 00:16:58,200 And you're also doing this outside of the context of, of 318 00:16:58,520 --> 00:17:01,400 recommender systems, right? Where in the real world you 319 00:17:01,400 --> 00:17:04,640 might be on a social media platform that, that gives you 320 00:17:04,640 --> 00:17:08,040 more misinformation, the more misinformation that you view and 321 00:17:08,040 --> 00:17:10,680 your peers are viewing misinformation and, and so on. 322 00:17:10,680 --> 00:17:15,040 And that might magnify the, the, you know, the effect more than 323 00:17:15,040 --> 00:17:18,200 these other strategies of countering the misinformation. 324 00:17:18,760 --> 00:17:20,960 Yeah, although I think it actually in many respects it's 325 00:17:20,960 --> 00:17:23,560 the opposite, like in the sense because we give people the feed 326 00:17:23,560 --> 00:17:26,319 that they would get essentially in one of these studies, because 327 00:17:26,319 --> 00:17:30,120 we want a good estimate of both belief in false and true stuff 328 00:17:30,720 --> 00:17:32,240 is like equal true and false things. 329 00:17:32,240 --> 00:17:34,800 But that's not what people see. Like, you don't imagine if half 330 00:17:34,800 --> 00:17:37,440 of your feed was misinformation. That would be insane. 331 00:17:37,440 --> 00:17:39,360 Like that's a lot. Now something there are some 332 00:17:39,360 --> 00:17:41,920 people who do see that much falsehood, but it's a small 333 00:17:41,920 --> 00:17:45,200 fraction of the of people. Most people see generally 334 00:17:45,360 --> 00:17:49,280 mainstream fine things, unless you're at a rally I guess, but 335 00:17:49,680 --> 00:17:52,480 that's not typical. What is typical? 336 00:17:52,840 --> 00:17:54,480 Could Do you know what percentage? 337 00:17:55,120 --> 00:17:57,360 So again, this is really difficult because it depends on 338 00:17:57,360 --> 00:17:59,000 how you're defining whether something is true or false. 339 00:17:59,000 --> 00:18:01,520 So if you, if we define false things as like things that are 340 00:18:01,520 --> 00:18:06,240 fact checked as being false, then like most people see like 341 00:18:06,800 --> 00:18:10,680 less than 5% of the stuff is of news content is fact checked as 342 00:18:10,680 --> 00:18:12,480 false for like the vast majority of people. 343 00:18:12,880 --> 00:18:17,040 It's like a small fraction. But there's like stuff that you 344 00:18:17,040 --> 00:18:19,720 might categorize as misleading and that's a major category. 345 00:18:20,040 --> 00:18:21,720 It's hard to define whether something is misleading. 346 00:18:21,720 --> 00:18:24,560 So it's harder. And then there's like the sort 347 00:18:24,560 --> 00:18:28,040 of thing that's not news content, that's false, like 348 00:18:28,200 --> 00:18:31,720 something a politician might say about cats and dogs or whatever. 349 00:18:31,720 --> 00:18:35,280 And then a lot of people see that if someone, if someone 350 00:18:35,640 --> 00:18:39,280 tells you that they can quantify the amount of truth and 351 00:18:39,280 --> 00:18:42,920 falsehood that an individual sees, they are themselves saying 352 00:18:42,920 --> 00:18:44,480 a falsehood. Like if this is something that 353 00:18:44,880 --> 00:18:47,040 we we can't, it's not easy to find whether something's true or 354 00:18:47,040 --> 00:18:49,600 false anyways. And then to like try to capture 355 00:18:49,600 --> 00:18:52,320 the amount of information that people are engaging with. 356 00:18:52,680 --> 00:18:55,240 It's just a gigantic. So it's really hard to know, but 357 00:18:55,240 --> 00:18:57,920 there's I could give like an explicit example of why this 358 00:18:57,920 --> 00:19:00,440 matters in the context of interventions, if that would be 359 00:19:00,440 --> 00:19:02,720 OK. I know I've had a long point of 360 00:19:02,720 --> 00:19:05,680 just me talking, but you know, we're, we'll break podcast rules 361 00:19:05,680 --> 00:19:07,760 for a second because I like ranting about this thing. 362 00:19:08,080 --> 00:19:10,680 So, OK, so let's take, let's talk about psychological 363 00:19:10,680 --> 00:19:14,760 inoculation for a second, a particular kind of thing that I 364 00:19:14,760 --> 00:19:17,120 find interesting. So the idea behind psychological 365 00:19:17,120 --> 00:19:19,920 inoculation, this is one of the boost for educational 366 00:19:19,920 --> 00:19:21,000 interventions that you mentioned. 367 00:19:21,360 --> 00:19:23,360 It's pre bunking, it's not a way of saying it. 368 00:19:23,760 --> 00:19:26,280 Well, inoculation is a subcategory pre bunking. 369 00:19:26,280 --> 00:19:30,200 So pre bunking is you could tell somebody before the election, 370 00:19:30,280 --> 00:19:33,280 which is presently what we're talking, you might say, well, 371 00:19:33,280 --> 00:19:35,400 this is the sort of falsehood that you're going to see. 372 00:19:35,760 --> 00:19:38,800 And certainly people are going to see things like ballot boxes 373 00:19:38,800 --> 00:19:41,520 being stolen and like ballots being thrown in the garbage and 374 00:19:41,520 --> 00:19:43,960 that kind of stuff. We can sort of know what sort of 375 00:19:43,960 --> 00:19:46,360 falsehoods spread during the election. 376 00:19:46,360 --> 00:19:48,520 And so you can say, well, just be mindful. 377 00:19:48,520 --> 00:19:50,320 This is something that people have lied about in the past and 378 00:19:50,320 --> 00:19:52,360 it works really well. Like people, that makes sense. 379 00:19:52,360 --> 00:19:55,760 People can like take that specific information and apply 380 00:19:55,760 --> 00:19:59,440 it to the stuff that they see. Inoculation is a little bit 381 00:19:59,440 --> 00:20:01,280 different. It tries to go one step broader 382 00:20:01,280 --> 00:20:03,480 than that. It says here are the sort of 383 00:20:03,480 --> 00:20:07,080 tactics that people use when they create misinformation. 384 00:20:07,680 --> 00:20:11,160 So it might be something like they'll use emotional language 385 00:20:11,160 --> 00:20:13,600 to try to grab your attention, OK? 386 00:20:14,000 --> 00:20:16,240 They'll say it's disgusting or whatever. 387 00:20:16,240 --> 00:20:18,920 And this is like a common tactic in the misinformation. 388 00:20:19,320 --> 00:20:21,200 And so there's like a bunch of research on like, if you teach 389 00:20:21,200 --> 00:20:23,920 people about those sorts of tactics and then you give them 390 00:20:24,400 --> 00:20:27,160 stimuli that contain those tactics directly afterwards, 391 00:20:27,600 --> 00:20:29,040 they'll be like, Oh yeah, that contains it. 392 00:20:29,280 --> 00:20:31,880 And so they can learn the thing that you taught them in the 393 00:20:31,880 --> 00:20:35,760 inoculation thing. The question then is, does that 394 00:20:35,760 --> 00:20:39,760 mean that if you give someone this inoculation, this like set 395 00:20:39,760 --> 00:20:42,840 of information about avoiding emotional language, how will 396 00:20:42,840 --> 00:20:47,520 that impact their everyday kind of detection of that thing? 397 00:20:47,760 --> 00:20:48,960 And that's a very different question. 398 00:20:50,400 --> 00:20:54,080 And part of the problem there is that like things that are true 399 00:20:54,640 --> 00:20:58,040 that are not misinformation also use emotional language. 400 00:20:58,120 --> 00:21:01,600 You know, sometimes things are disgusting actually, like there 401 00:21:01,600 --> 00:21:04,160 are disgusting things. And so if I say, if I see the 402 00:21:04,160 --> 00:21:06,880 word disgust, that does not mean that things are false. 403 00:21:07,400 --> 00:21:09,480 And we've done some research on this, like if you give people 404 00:21:09,480 --> 00:21:12,360 the inoculation, it doesn't help them distinguish between us true 405 00:21:12,360 --> 00:21:14,800 and false. It just helps them be more wary 406 00:21:14,800 --> 00:21:17,920 of emotional words. And if it happens to be the case 407 00:21:17,920 --> 00:21:21,320 that people see way more true things than false things, then 408 00:21:21,320 --> 00:21:23,920 probably what they're doing is if it insofar as working in the 409 00:21:23,920 --> 00:21:28,840 real world, they're believing truth true things less because 410 00:21:28,840 --> 00:21:31,520 of the inoculation. And they aren't believing false 411 00:21:31,520 --> 00:21:33,800 things less necessarily because they're not seeing that much 412 00:21:33,800 --> 00:21:35,040 false things, they're seeing way more true things. 413 00:21:35,040 --> 00:21:38,320 Do you see what I mean? So like how it's applied depends 414 00:21:38,320 --> 00:21:40,400 on the information environment that people are engaging with. 415 00:21:41,000 --> 00:21:43,160 So it's hard to say in these experiments. 416 00:21:43,160 --> 00:21:45,720 There's a bunch of studies about inoculation being effective. 417 00:21:45,880 --> 00:21:48,880 What that means is people can learn the inoculations. 418 00:21:49,120 --> 00:21:51,760 We don't know what that means. They are using that 419 00:21:52,200 --> 00:21:54,760 appropriately in the world because those experiments 420 00:21:54,800 --> 00:21:55,880 haven't been. Run. 421 00:21:56,040 --> 00:21:57,320 Yeah. But it it makes sense because 422 00:21:57,320 --> 00:21:59,680 like we're still fallible to something like, let's say 423 00:21:59,680 --> 00:22:02,320 motivated reasoning in terms of like even if you taught someone, 424 00:22:02,600 --> 00:22:05,760 for example, what ad hominem attacks are and and how they 425 00:22:05,760 --> 00:22:10,640 work in terms of political kind of debates, often times can lead 426 00:22:10,640 --> 00:22:14,040 to people kind of attacking individual character rather than 427 00:22:14,440 --> 00:22:16,720 the policy they have. Even if they know that they're 428 00:22:16,720 --> 00:22:20,200 probably much more likely to notice if someone is using kind 429 00:22:20,200 --> 00:22:24,000 of ad hominem attacks against the politician they like rather 430 00:22:24,000 --> 00:22:26,920 than maybe if the politician that they like using some 431 00:22:26,920 --> 00:22:29,040 against. Like, I guess that's kind of 432 00:22:29,040 --> 00:22:30,360 what you're speaking to right as well. 433 00:22:30,360 --> 00:22:32,320 It's like it's just. Yeah, I mean, actually that's 434 00:22:32,360 --> 00:22:34,800 that's an additional issue in the sense that like if the world 435 00:22:34,800 --> 00:22:38,200 is noisy anytime you teach anybody anything, it's going to 436 00:22:38,200 --> 00:22:41,280 be impossible for them to adopt it with perfect accuracy. 437 00:22:41,280 --> 00:22:44,520 And that's just a general issue. What I'm saying that even if you 438 00:22:44,720 --> 00:22:48,040 did have perfect accuracy in the context of some of these 439 00:22:48,040 --> 00:22:51,240 inoculations like emotional language, it would still lead to 440 00:22:51,240 --> 00:22:56,200 people believing through things less and then they are believing 441 00:22:56,200 --> 00:22:59,480 false things less because they see more true things that have 442 00:22:59,480 --> 00:23:02,280 emotional language. Like I think about this way, if 443 00:23:02,280 --> 00:23:05,160 I had a magic wand and I could inoculate people against 444 00:23:05,160 --> 00:23:07,200 emotional language to say every time you see emotional language, 445 00:23:07,200 --> 00:23:09,720 don't believe that thing, which is what essentially these things 446 00:23:09,720 --> 00:23:12,240 are trying to do. The next thing that they see 447 00:23:12,240 --> 00:23:14,320 that has emotional language is more likely to be true than 448 00:23:14,320 --> 00:23:16,520 false. Right, Yeah, it's when you're 449 00:23:16,520 --> 00:23:19,320 teaching it, you're sort of ignoring the base rate and so. 450 00:23:19,480 --> 00:23:20,680 Like it may be a great neglect, yeah. 451 00:23:21,120 --> 00:23:22,880 Exactly. And then do you say, OK, well 452 00:23:22,880 --> 00:23:26,680 then you have to teach them to look out for emotional language 453 00:23:27,120 --> 00:23:30,080 in so far as there is falsehood also included. 454 00:23:30,240 --> 00:23:31,840 But now you're back to prebunking and you're not 455 00:23:31,840 --> 00:23:33,520 eoculating anymore. It's about teaching people 456 00:23:33,800 --> 00:23:36,080 actual facts about the world. And that's just regular 457 00:23:36,080 --> 00:23:37,200 prebunking. And that's fine. 458 00:23:37,560 --> 00:23:41,480 But it's basically the the conceit is this, it is not 459 00:23:41,480 --> 00:23:45,840 possible to, I don't think to find simple markers that you can 460 00:23:45,840 --> 00:23:47,880 teach someone in a minute and 1/2 that will help them 461 00:23:47,880 --> 00:23:49,280 distinguish between what's true and false. 462 00:23:49,720 --> 00:23:51,080 The world is too complicated for that. 463 00:23:51,480 --> 00:23:54,560 And so we need to actually just teach people about facts. 464 00:23:54,920 --> 00:23:57,440 And the educational interventions that are more 465 00:23:57,440 --> 00:24:00,960 directed towards the things that help us identify true versus 466 00:24:00,960 --> 00:24:02,720 false things are the ones that are going to be most effective 467 00:24:03,160 --> 00:24:04,600 when it comes to misinformation at least. 468 00:24:04,920 --> 00:24:08,920 But one side note is that you could make the claim that it's 469 00:24:08,920 --> 00:24:11,840 good to teach people about the tactics regardless of whether 470 00:24:11,840 --> 00:24:14,400 this misinformation evolves. Like, we want people to be able 471 00:24:14,400 --> 00:24:16,720 to identify ad homom attacks. That's fine. 472 00:24:16,720 --> 00:24:18,440 It's just not intervention against misinformation. 473 00:24:18,440 --> 00:24:20,200 It's intervention against ad homom attacks. 474 00:24:20,760 --> 00:24:22,680 And that's a different thing. That's more nuanced argument 475 00:24:22,680 --> 00:24:24,520 than that. But yeah, sorry, a bit of a 476 00:24:24,520 --> 00:24:26,680 digression, but it's a topic I find interesting. 477 00:24:27,400 --> 00:24:28,640 Yeah. No, it should be interesting. 478 00:24:28,640 --> 00:24:32,120 And I think that's why we have this conversation is kind of 479 00:24:32,760 --> 00:24:35,720 getting some of the weeds and it's always is the case with 480 00:24:35,760 --> 00:24:37,600 payable science stuff that it depends. 481 00:24:37,600 --> 00:24:41,640 And I think that is also, you know, appreciated that you're 482 00:24:41,840 --> 00:24:43,520 like not trying to oversimplify things here. 483 00:24:43,880 --> 00:24:49,600 So I, I think that's good. And I guess when it comes to 484 00:24:49,600 --> 00:24:52,880 then thinking about, you know, again, it depends. 485 00:24:53,080 --> 00:24:56,800 We have 3 categories, many different types of interventions 486 00:24:56,800 --> 00:24:59,120 you could do. Is there any advice you would 487 00:24:59,120 --> 00:25:03,280 have or any thinking you would have around kind of like how if 488 00:25:03,280 --> 00:25:09,000 someone is trying to do work in in this realm where they can 489 00:25:09,000 --> 00:25:12,080 start thinking about like, OK, all of these various types of 490 00:25:12,080 --> 00:25:15,520 things we could do, where should we start depending on where we 491 00:25:15,520 --> 00:25:16,200 are? Yeah. 492 00:25:16,320 --> 00:25:19,480 I want to add to that as well because I think one of the the 493 00:25:19,480 --> 00:25:23,960 mantras of behavioral design and especially for practitioners is 494 00:25:24,160 --> 00:25:27,280 like, you know, we don't want to focus on changing beliefs. 495 00:25:27,280 --> 00:25:29,160 We want to focus on changing behavior. 496 00:25:29,440 --> 00:25:31,080 That's what counts, that's what matters. 497 00:25:31,360 --> 00:25:35,080 And so we often push people towards these strategies that 498 00:25:35,080 --> 00:25:37,880 change the behavior. So of these three categories, 499 00:25:37,880 --> 00:25:41,120 that would be the nudges. But now you're telling me these 500 00:25:41,120 --> 00:25:44,440 are equally effective across the board and you know, with the 501 00:25:44,440 --> 00:25:46,200 nuance and the caveats, of course. 502 00:25:46,480 --> 00:25:51,080 But what would you say to that sort of general rule of thumb? 503 00:25:51,080 --> 00:25:55,080 Does it apply here? I think that's more of a choice 504 00:25:55,080 --> 00:25:59,160 about what you want. That's less about efficacy I 505 00:25:59,160 --> 00:26:03,040 would assume and more about like how you want to be engaging 506 00:26:03,720 --> 00:26:04,760 with. I mean, it really depends. 507 00:26:04,760 --> 00:26:10,600 So if you take debunking, so if I was able to, in theory every 508 00:26:10,600 --> 00:26:14,840 person who sees the falsehood gets a debunk, then that will be 509 00:26:14,840 --> 00:26:18,080 effective to some extent. But of course, in reality, I'm 510 00:26:18,080 --> 00:26:21,600 not going to be able to do that. If I do a nudge, I don't need to 511 00:26:21,600 --> 00:26:24,280 know what the person saw for that to have efficacy. 512 00:26:24,880 --> 00:26:25,920 Now. It's only going to be effective 513 00:26:25,920 --> 00:26:29,880 insofar as they see things like if if I give someone a nudge 514 00:26:29,880 --> 00:26:32,240 that will get them think more about accuracy, that's not going 515 00:26:32,240 --> 00:26:35,680 to have an impact if no one ever sees falsehood, but that's fine 516 00:26:35,680 --> 00:26:37,200 because it doesn't. You have to waste anything. 517 00:26:37,200 --> 00:26:38,520 That's fine. There's nothing wrong with that. 518 00:26:38,880 --> 00:26:41,240 If I'm debunking something, if I debunk it but the person hasn't 519 00:26:41,240 --> 00:26:42,480 seen it, then like that's fine too. 520 00:26:42,480 --> 00:26:45,000 But what strategy you use depends on what you're trying to 521 00:26:45,000 --> 00:26:47,320 do. And there is no like, oh, this 522 00:26:47,320 --> 00:26:48,600 category is better than that category. 523 00:26:48,600 --> 00:26:52,360 It's just like they're different things and people like the 524 00:26:52,360 --> 00:26:55,120 reputation, like the ones that target beliefs in particular. 525 00:26:56,320 --> 00:26:58,280 Some people would want to avoid that because they don't want to 526 00:26:58,280 --> 00:27:00,000 be the ones who say this is what's true and this is what's 527 00:27:00,000 --> 00:27:01,920 false. But if you're nudging you, you 528 00:27:01,920 --> 00:27:03,800 don't have to be the one who decides that you just like, 529 00:27:03,800 --> 00:27:08,520 here's good information about better sources, you know, there 530 00:27:08,520 --> 00:27:10,880 you go. And then people can take it or 531 00:27:10,880 --> 00:27:12,400 leave it and whatever. Do you see what I mean? 532 00:27:13,880 --> 00:27:16,640 OK, yeah. And and so where do our sort of 533 00:27:16,640 --> 00:27:21,640 sticky topics of freedom of speech and the censorship and 534 00:27:21,680 --> 00:27:25,360 all these sorts of things, how do they play into these 535 00:27:25,360 --> 00:27:30,320 different strategies? Are some I don't know closer to 536 00:27:30,560 --> 00:27:33,080 infringing on that right than others? 537 00:27:34,080 --> 00:27:38,680 I guess so, although I mean on a social media platform, there is 538 00:27:38,680 --> 00:27:41,480 always filtering, there is no, it's not freedom of speech in 539 00:27:41,480 --> 00:27:43,920 the sense that that's freedom of reach, I guess. 540 00:27:43,920 --> 00:27:46,040 And different people have different platforms and there is 541 00:27:46,040 --> 00:27:50,320 an algorithm and that is decided by people and then computers 542 00:27:50,320 --> 00:27:52,480 also or whatever. But like there's choices that 543 00:27:52,480 --> 00:27:54,760 are put into these things that determine what you see. 544 00:27:55,280 --> 00:28:00,000 And so if you were to say the social media company decides 545 00:28:00,280 --> 00:28:04,760 we're going to wait a source quality wait for news content on 546 00:28:04,760 --> 00:28:07,880 the platform, that's an editorial choice that does not 547 00:28:07,880 --> 00:28:10,560 impact freedom of speech. It's going to impact what people 548 00:28:10,560 --> 00:28:13,320 see you can. Say whatever you want there, it 549 00:28:13,320 --> 00:28:15,240 just might not be seen by anyone else. 550 00:28:15,240 --> 00:28:16,960 Yeah, exactly. And it and if you have a source 551 00:28:16,960 --> 00:28:18,840 that is blacklisted by the social media company for 552 00:28:18,840 --> 00:28:22,760 whatever reason, like you're, you're free, it's still free to 553 00:28:22,760 --> 00:28:25,000 say whatever you want, but you don't have the opportunity to 554 00:28:25,360 --> 00:28:28,600 broadcast what you want to say to that company's proprietary, 555 00:28:29,040 --> 00:28:30,400 you know, to the people who are on the thing. 556 00:28:30,760 --> 00:28:32,600 So, you know, that's the complication with the free 557 00:28:32,600 --> 00:28:34,920 speech issue. And then you always, but I mean, 558 00:28:34,920 --> 00:28:37,840 it is the case though that social media companies do are 559 00:28:38,200 --> 00:28:40,840 kind of really concerned about that or these perceptions of 560 00:28:40,840 --> 00:28:43,680 freedom of speech. And so they don't, I think 561 00:28:44,320 --> 00:28:46,960 they're just afraid of basically getting a New York Times article 562 00:28:47,040 --> 00:28:49,240 written that they're trying to target conservatives and stuff 563 00:28:49,240 --> 00:28:50,680 like that, which is usually what's happening. 564 00:28:50,680 --> 00:28:52,680 And they often overcorrect. And that's a bigger issue. 565 00:28:52,680 --> 00:28:55,640 But yeah, I mean, the other thing is when it comes to fact 566 00:28:55,640 --> 00:28:58,840 checking and warning labels and source credibility labels and 567 00:28:58,840 --> 00:29:01,320 all this type of stuff, someone does have to make a choice about 568 00:29:01,320 --> 00:29:04,680 whether something is fact X is false, whether a source is 569 00:29:04,680 --> 00:29:08,480 credible or whatever. And often these are like you'll 570 00:29:08,480 --> 00:29:12,560 rely on Fact Check professional fact checkers, journalists who 571 00:29:12,560 --> 00:29:13,840 are kind of experts in the domain. 572 00:29:14,240 --> 00:29:15,800 But you know, a lot of people don't trust that. 573 00:29:15,800 --> 00:29:18,520 And these are not simple issues to to deal with. 574 00:29:19,080 --> 00:29:21,840 Yeah. Well, one other lens to look at 575 00:29:21,840 --> 00:29:24,880 this that I usually use for various types of kind of 576 00:29:24,880 --> 00:29:28,760 interventions is to think about it as either focusing on, call 577 00:29:28,760 --> 00:29:32,560 it, proactive strategies versus called reactive strategies. 578 00:29:32,560 --> 00:29:34,280 And I think it applies relatively well here as well 579 00:29:34,280 --> 00:29:37,680 where you know some of the things you have kind of talked 580 00:29:37,680 --> 00:29:41,680 about fits in the box of things that you might do practically. 581 00:29:41,680 --> 00:29:45,920 Like certainly the pre banking is a good example of that where 582 00:29:46,160 --> 00:29:50,720 you're proactively getting ahead of things to see how you can do 583 00:29:51,120 --> 00:29:54,480 something to increase people's ability to distinguish maybe 584 00:29:54,800 --> 00:29:59,840 fake versus true and so on. And there's very various other 585 00:29:59,840 --> 00:30:02,600 things around probably boosting that would also qualify there in 586 00:30:02,600 --> 00:30:05,280 general. But then obviously as we're 587 00:30:05,280 --> 00:30:08,120 talking about, like there's this thing where in the moment people 588 00:30:08,120 --> 00:30:12,640 will be exposed to various things and it will be requiring 589 00:30:12,640 --> 00:30:14,920 more like quality reactive strategies where they will be 590 00:30:14,920 --> 00:30:17,880 very keen to share something. And then maybe that fits into 591 00:30:18,240 --> 00:30:22,760 the box of maybe I'm trying to think about which box would fit 592 00:30:22,760 --> 00:30:25,200 where you get an alert saying like, hey, are you really sure 593 00:30:25,200 --> 00:30:26,400 you want to share this? Have you read it? 594 00:30:26,400 --> 00:30:28,480 Like I don't know what what that box would fit in. 595 00:30:29,240 --> 00:30:32,160 Yeah, I don't think that counts as a nudge sort of friction. 596 00:30:32,200 --> 00:30:33,760 Yeah. It's kind of like a the targets 597 00:30:33,760 --> 00:30:35,560 of behavior, at least, yeah. Yeah. 598 00:30:35,560 --> 00:30:37,280 So that's kind of like a more reactive strategy. 599 00:30:37,320 --> 00:30:42,640 So I'm just interested, do you have any sense of, you know, all 600 00:30:42,640 --> 00:30:46,360 else being equal, like where would you usually think about, 601 00:30:46,960 --> 00:30:49,960 you know, attacking or like is it both equally important? 602 00:30:50,080 --> 00:30:50,920 Yeah, I think they're both equally. 603 00:30:50,960 --> 00:30:53,120 I mean, yeah, I think, I mean, essentially they're both equally 604 00:30:53,120 --> 00:30:56,120 important, or at least it's not important to know whether 1 is 605 00:30:56,120 --> 00:30:57,880 better than the other. It's kind of like saying this is 606 00:30:57,880 --> 00:30:59,040 a toolbox. So like they're just for 607 00:30:59,040 --> 00:31:01,200 different things. Like what's better, a hammer or 608 00:31:01,200 --> 00:31:02,800 a screwdriver? It's like it depends what you 609 00:31:02,800 --> 00:31:04,560 need it for. Probably a hammer is. 610 00:31:04,720 --> 00:31:06,760 I mean, if we're going to have the argument, I think probably 611 00:31:06,760 --> 00:31:09,520 hammers generally have a broader usefulness than a screwdriver, 612 00:31:09,520 --> 00:31:12,360 but it doesn't really matter. Like it just if you have a screw 613 00:31:12,360 --> 00:31:14,120 then you need a screwdriver. You can always hammer in a 614 00:31:14,200 --> 00:31:14,800 screw. Yeah, you could. 615 00:31:14,840 --> 00:31:16,680 Eventually hammer it in. If you're really. 616 00:31:16,880 --> 00:31:18,120 Desperate. But it's a purely kind of 617 00:31:18,120 --> 00:31:21,840 academic discussion, right? So yeah, they're just trying to 618 00:31:21,840 --> 00:31:23,800 do different things. And so it's good. 619 00:31:23,880 --> 00:31:25,920 This is why it's a toolbox. It's like depends what the 620 00:31:25,920 --> 00:31:28,160 person's trying to do. You can pull in the toolbox and 621 00:31:28,160 --> 00:31:31,840 see what is like, what are some ways in which you can deal with 622 00:31:31,840 --> 00:31:35,200 that particular issue, which is like if it's a for debunking, 623 00:31:35,200 --> 00:31:38,800 for example, it might be there's a kind of like common falsehood 624 00:31:38,800 --> 00:31:41,320 that people are believing or something that's being spread. 625 00:31:41,560 --> 00:31:43,760 And so it's too late to do prebunking. 626 00:31:43,760 --> 00:31:45,760 You have to do debunking. You have to kind of try to catch 627 00:31:45,760 --> 00:31:47,560 up. But if you're like right before 628 00:31:47,560 --> 00:31:51,160 an election, there's going to be certainly a bunch of like false 629 00:31:51,160 --> 00:31:54,760 claims about election fraud. Yeah, now now's the time to do 630 00:31:54,760 --> 00:31:57,640 some pre bunking and try to make sure that people know what good, 631 00:31:57,640 --> 00:32:00,360 what good sources are and bad sources are and so on. 632 00:32:00,360 --> 00:32:02,720 And so like these. Yeah, it depends on what you're 633 00:32:02,720 --> 00:32:05,600 trying to do. Yeah, and we can shout out the 634 00:32:05,640 --> 00:32:08,560 also website that exists for this toolbox, really useful for 635 00:32:08,560 --> 00:32:11,600 anyone who has checked out. It offers a lot of chances to 636 00:32:11,640 --> 00:32:14,160 look at the various tools and also papers attached to them. 637 00:32:14,160 --> 00:32:18,360 So shout out to that website. But one thing I just wanted to 638 00:32:18,480 --> 00:32:21,920 touch upon as well because we we haven't directly explored it and 639 00:32:21,920 --> 00:32:25,240 I don't currently see how maybe these tools as well maybe 640 00:32:25,240 --> 00:32:29,960 addresses this as you study of echo chambers in terms of with 641 00:32:29,960 --> 00:32:33,920 the Internet increasingly being fragmented, there's a bunch of 642 00:32:33,920 --> 00:32:37,840 small subreddits and WhatsApp groups and various things where, 643 00:32:38,400 --> 00:32:41,560 you know, honestly, it seems like misinformation thrives more 644 00:32:41,560 --> 00:32:43,240 than ever in those kind of cases. 645 00:32:43,240 --> 00:32:47,960 Well, how do you think about dealing with that in terms of 646 00:32:47,960 --> 00:32:49,480 maybe from this toolbox in general? 647 00:32:49,480 --> 00:32:52,840 Like how do you think about that challenge with misinformation 648 00:32:52,840 --> 00:32:53,240 online? Yeah. 649 00:32:53,680 --> 00:32:56,600 It's an interesting question. I'm not sure the toolbox is all 650 00:32:56,600 --> 00:32:58,680 that helpful for echo chambers. The echo chambers is a 651 00:32:58,680 --> 00:33:01,520 structural problem. It's about the way the networks 652 00:33:01,520 --> 00:33:03,520 are formed. And that's the kind of a social 653 00:33:03,520 --> 00:33:06,400 media problem. I mean although I guess in 654 00:33:06,400 --> 00:33:10,720 theory if you are like really successful in altering people's 655 00:33:10,720 --> 00:33:13,160 beliefs for example, they might remove themselves from an echo 656 00:33:13,160 --> 00:33:15,920 chamber because they're like wait this is all junk. 657 00:33:16,080 --> 00:33:17,760 Like why am I hanging out on the subreddit? 658 00:33:17,840 --> 00:33:22,680 It's worth knowing that like online echo chambers are not as 659 00:33:22,680 --> 00:33:26,800 bad as people think they are. That is, our real life echo 660 00:33:26,800 --> 00:33:30,760 chambers tend to be worse than that. 661 00:33:30,960 --> 00:33:34,080 If you think about just homophily in neighborhoods and 662 00:33:34,080 --> 00:33:37,600 so on, you actually are exposed to way more diversion opinion 663 00:33:37,600 --> 00:33:40,120 online than you would be otherwise. 664 00:33:42,240 --> 00:33:45,000 But you know, it still is the case that people find if you 665 00:33:45,000 --> 00:33:47,520 have a particular kind of set of fringe beliefs, you can find a 666 00:33:47,520 --> 00:33:50,160 community that reinforces those things that you would not find 667 00:33:50,160 --> 00:33:52,760 in your real in your like day-to-day life. 668 00:33:53,240 --> 00:33:55,520 And that's a major problem. But that's kind of more of a 669 00:33:55,520 --> 00:34:00,400 fringe issue, but still is a it is an issue among people on the 670 00:34:00,400 --> 00:34:03,240 fringe, a smaller proportion, but a very large issue because 671 00:34:03,720 --> 00:34:06,760 even a few small people who are in like, for example, like 672 00:34:06,760 --> 00:34:10,920 violent, you know, groups of individuals is not a good thing, 673 00:34:11,000 --> 00:34:16,480 obviously. So 1 aspect of this toolbox is 674 00:34:16,480 --> 00:34:20,239 that all the strategies, I think you could argue are focused on 675 00:34:20,239 --> 00:34:23,480 the individual interventions that really get the individual 676 00:34:23,480 --> 00:34:27,360 to stop and think, do something differently, believe something 677 00:34:27,400 --> 00:34:30,360 differently. How do you think those sorts of 678 00:34:30,520 --> 00:34:34,159 individual focused interventions compare against what you might 679 00:34:34,159 --> 00:34:37,120 call your more standard policy toolkit? 680 00:34:37,120 --> 00:34:40,719 So things like, you know, regulating information, 681 00:34:40,880 --> 00:34:42,639 punishing the perpetrators, right? 682 00:34:42,960 --> 00:34:45,040 How do you think of those things together? 683 00:34:45,679 --> 00:34:48,280 Yeah, I would think that they don't fare well compared to them 684 00:34:48,600 --> 00:34:54,600 to policy meaning that like. So I think it's important to try 685 00:34:54,600 --> 00:34:59,800 to improve individual level decision making, obviously, and 686 00:34:59,960 --> 00:35:04,560 that, you know, regardless of whatever the the other, the 687 00:35:04,560 --> 00:35:06,800 problem is that I can't influence thought. 688 00:35:06,840 --> 00:35:10,440 I'm not a policy maker. I but I am a psychologist and so 689 00:35:10,440 --> 00:35:13,320 I can do research on like individual level interventions, 690 00:35:13,680 --> 00:35:18,200 but I'll give like a example. Like we can spend a lot of time 691 00:35:19,280 --> 00:35:22,360 working out like the best way to remind people think about 692 00:35:22,360 --> 00:35:24,600 accuracy. And this is going to have a tiny 693 00:35:24,600 --> 00:35:28,440 fraction of effect relative to like a social media company 694 00:35:28,840 --> 00:35:32,600 putting more emphasis on showing people stuff that's accurate. 695 00:35:33,760 --> 00:35:36,280 You know what I mean? So you know. 696 00:35:36,400 --> 00:35:38,680 You actually get to the the root of the problem. 697 00:35:38,760 --> 00:35:40,400 Yeah, exactly. In that case, I don't own social 698 00:35:40,400 --> 00:35:43,880 media company, so I I can do what I can do so that, you know, 699 00:35:43,960 --> 00:35:47,720 11 important note is that tech companies, social media 700 00:35:47,720 --> 00:35:51,640 companies do like the individual interventions because it means 701 00:35:52,840 --> 00:35:55,240 we'll let the people solve it. We're not going to solve it 702 00:35:55,480 --> 00:35:58,320 ourselves. The users will solve it here 703 00:35:58,600 --> 00:36:00,880 user. Here's some here's some ways to 704 00:36:00,880 --> 00:36:03,080 deal with the shit that we're showing you on our platform. 705 00:36:03,480 --> 00:36:05,680 You know, good luck with that. You know, so that's an issue. 706 00:36:05,760 --> 00:36:09,840 But so we're in some sense, we're all kind of useful as 707 00:36:09,920 --> 00:36:11,760 idiots for the social media companies, but we like, what 708 00:36:11,760 --> 00:36:13,160 else am I going to do? I have to figure out something. 709 00:36:13,160 --> 00:36:15,000 We have to find ways to make things better. 710 00:36:15,080 --> 00:36:18,920 So hopefully that doesn't take pressure off the social media 711 00:36:18,920 --> 00:36:20,920 companies to actually make improvements on the platforms. 712 00:36:21,440 --> 00:36:25,280 Well, OK, so you said you don't own a social media company. 713 00:36:25,280 --> 00:36:30,640 But but let's say that you did. Imagine that Elon Musk hands X 714 00:36:30,640 --> 00:36:33,240 over to you. It's yours to overhaul. 715 00:36:33,240 --> 00:36:36,640 You can redesign it entirely. It still has to be a social 716 00:36:36,640 --> 00:36:39,440 media platform where information is shared. 717 00:36:39,840 --> 00:36:43,800 What would you do if you had this infinite power to basically 718 00:36:43,800 --> 00:36:45,400 start from scratch with social media? 719 00:36:45,400 --> 00:36:48,640 How would you design it differently if you would decide 720 00:36:48,680 --> 00:36:51,640 it differently? Yeah, I mean, so most of my 721 00:36:51,640 --> 00:36:57,080 recommendations would be to hire way more content moderators and 722 00:36:57,280 --> 00:37:01,080 trust and safety people. And like all of the people that 723 00:37:01,080 --> 00:37:04,280 were fired, for example, by Elon Musk, you know, I bring the 724 00:37:04,280 --> 00:37:07,720 experts back in to help make better informed choices on this. 725 00:37:07,840 --> 00:37:12,840 Sort of basically what you need is better filtering of quality. 726 00:37:13,520 --> 00:37:15,640 And this does not need to be misaligned with people's 727 00:37:15,640 --> 00:37:18,200 preferences. It's not like I want to 728 00:37:18,240 --> 00:37:21,000 determine what the information environment is for people and be 729 00:37:21,000 --> 00:37:23,640 a big sensor and determine what's true or false that most 730 00:37:23,640 --> 00:37:27,480 people don't want to see junk. Like, as an example, clickbait. 731 00:37:27,480 --> 00:37:29,200 People actually hate clickbait. It works. 732 00:37:29,200 --> 00:37:32,520 People will engage with it and that really does like push 733 00:37:32,520 --> 00:37:35,360 forward on algorithms, but people don't want it. 734 00:37:35,640 --> 00:37:39,040 So you can make principal choice as someone who owns a social 735 00:37:39,040 --> 00:37:41,480 media company to say we're not going to deal with that kind of 736 00:37:41,480 --> 00:37:42,800 stuff. And there's like, you know, 737 00:37:42,800 --> 00:37:44,920 Twitter's got all sorts of spam and everyone has noticed the 738 00:37:44,920 --> 00:37:46,880 quality kind of decrease. Yeah. 739 00:37:46,920 --> 00:37:50,200 There's just levers you can pull to make that better in terms of, 740 00:37:50,200 --> 00:37:54,520 like, how to change algorithms and, you know, maybe find ways 741 00:37:54,520 --> 00:37:59,120 to incentivize accuracy more. Like, I, I there's things that I 742 00:37:59,120 --> 00:38:01,680 would change, but probably they would also people would stop 743 00:38:01,680 --> 00:38:03,480 using the platform. And I'm not sure if I would be 744 00:38:03,480 --> 00:38:05,000 the right person to make those sorts of choices. 745 00:38:05,520 --> 00:38:07,720 So I mean, the first thing I would do is just simply hire 746 00:38:07,720 --> 00:38:11,480 more people who are experts in contact quality essentially and 747 00:38:11,480 --> 00:38:13,080 try to improve the stuff that people are seeing. 748 00:38:14,320 --> 00:38:19,960 Would you let users choose what percentage of true content they 749 00:38:19,960 --> 00:38:25,880 were shown so you could get like a, you know, hey user from 0% to 750 00:38:25,880 --> 00:38:28,240 100%, how much misinformation would you like? 751 00:38:29,040 --> 00:38:31,560 That's interesting. The most users would be people 752 00:38:31,560 --> 00:38:34,680 who say I don't agree with you but what's misinformation so 753 00:38:34,680 --> 00:38:37,840 that everyone wants 100% true stuff but we just. 754 00:38:37,840 --> 00:38:39,160 Can't agree on what the truth is. 755 00:38:39,320 --> 00:38:42,000 Exactly who gets to determine what's what's true and what 756 00:38:42,000 --> 00:38:43,880 isn't? So I mean, one thing that is 757 00:38:44,320 --> 00:38:48,080 worth noting, community notes on Twitter, which is like, you 758 00:38:48,080 --> 00:38:52,280 know, a kind of consensus built almost kind of democratic 759 00:38:52,600 --> 00:38:55,720 Wikipedia esque fact checking people like that. 760 00:38:55,720 --> 00:38:58,760 I mean, there's not very, you don't really see that much 761 00:38:58,760 --> 00:39:01,480 dispute about the notes. They're always quite good and 762 00:39:01,480 --> 00:39:03,560 accurate. And so you could scale that up. 763 00:39:03,560 --> 00:39:05,960 I think it could be scaled up more, but that's a success story 764 00:39:05,960 --> 00:39:07,960 I think. I think there's a few things 765 00:39:07,960 --> 00:39:10,880 that creates as much Glee and Freud and Freud as someone re 766 00:39:10,880 --> 00:39:13,480 sharing community notes. I think there's even like one of 767 00:39:13,480 --> 00:39:16,960 the most popular accounts is like destroyed by community 768 00:39:16,960 --> 00:39:18,680 notes or something. Yeah, yeah. 769 00:39:18,680 --> 00:39:21,800 And they'll put community notes on like Musk's his own tweets 770 00:39:21,800 --> 00:39:23,160 and stuff like that. And that's always fun. 771 00:39:23,320 --> 00:39:24,600 So yeah, that kind of stuff is great. 772 00:39:24,960 --> 00:39:26,600 Well, I want to know, what would you guys do? 773 00:39:26,680 --> 00:39:28,360 You're not allowed to say that. No, we can. 774 00:39:28,400 --> 00:39:34,760 Yeah, I I am a big fan as well of in some ways giving kind of 775 00:39:34,760 --> 00:39:38,280 power to the people in terms of I would say that my platform of 776 00:39:38,280 --> 00:39:41,440 choice is probably Reddit these days because I have an 777 00:39:41,440 --> 00:39:45,200 opportunity to both attach myself to certain subreddits 778 00:39:45,200 --> 00:39:47,680 that have an interesting. But it's also kind of creates 779 00:39:47,680 --> 00:39:49,920 quite a lot of variability at the same time. 780 00:39:50,480 --> 00:39:53,680 But importantly, it kind of automatically has an inbuilt 781 00:39:53,680 --> 00:39:57,720 system where anything that is spam or click based and all 782 00:39:57,720 --> 00:39:58,760 these things is usually downloaded. 783 00:39:58,760 --> 00:40:01,440 So you never see it. And the things that is actually 784 00:40:01,440 --> 00:40:03,040 inside full and useful is usually uploaded. 785 00:40:03,440 --> 00:40:06,960 And so it kind of. Punishes people that are trying 786 00:40:06,960 --> 00:40:10,520 to spread misinformation kind of purposefully harmful because 787 00:40:10,880 --> 00:40:13,680 it's almost like a virus, like how you say like antibodies 788 00:40:13,680 --> 00:40:16,600 where the kind of platform or the community itself notices, 789 00:40:16,600 --> 00:40:21,600 OK, this person is just trying to be a troll and this 790 00:40:21,600 --> 00:40:23,320 community, we're against trolling. 791 00:40:23,320 --> 00:40:25,800 And so we're going to downvote this stuff. 792 00:40:26,120 --> 00:40:29,600 And I think also on Reddit is where you've seen things spawn 793 00:40:29,600 --> 00:40:33,240 like this community for having kind of prove me wrong, 794 00:40:33,240 --> 00:40:35,320 basically like certain communities where they're trying 795 00:40:35,320 --> 00:40:39,480 to have ways to like, hey, prove me wrong, where that kind of has 796 00:40:39,480 --> 00:40:41,480 worked. Like that's one of the few cases 797 00:40:41,480 --> 00:40:44,240 where I can see people thoughtfully engaged, become 798 00:40:44,240 --> 00:40:47,000 more insightful around certain topics. 799 00:40:47,160 --> 00:40:48,240 It's not true for all subreddits. 800 00:40:48,240 --> 00:40:50,080 There's some shit shows subreddits for sure. 801 00:40:50,160 --> 00:40:53,920 But I I I do like some of those aspects with Reddits that you 802 00:40:54,000 --> 00:40:56,400 enable people to also be part of the solution. 803 00:40:57,280 --> 00:40:59,600 Yeah, that's great. And I know that, like, the 804 00:41:00,200 --> 00:41:05,080 operators of subreddits do a lot of work to, you know, enforce 805 00:41:05,080 --> 00:41:06,680 dorms and stuff like that. There's some really interesting 806 00:41:06,680 --> 00:41:09,280 work on that. Nathan Matthias, who's my 807 00:41:09,280 --> 00:41:10,840 colleague at Cornell, does work on Reddit. 808 00:41:11,720 --> 00:41:14,920 Yeah. OK, I think he is a moderator on 809 00:41:14,920 --> 00:41:18,120 a subreddit or he's studying it. He's studying it like he works 810 00:41:18,120 --> 00:41:21,600 with moderators to do. Like it's some, it's almost 811 00:41:21,600 --> 00:41:24,400 ethnographic stuff, but like, yeah, it's all about creating 812 00:41:24,400 --> 00:41:25,520 trust within communities and so on. 813 00:41:26,440 --> 00:41:28,160 Yeah, that's cool. And I think that's something 814 00:41:28,160 --> 00:41:31,080 that I find fascinating where there's some extreme version 815 00:41:31,080 --> 00:41:33,800 that's where people have some form of a special server version 816 00:41:33,800 --> 00:41:37,520 of Grand Theft Auto where people can have real jobs. 817 00:41:37,560 --> 00:41:39,720 Or like there's also a Second Life and some of these games 818 00:41:39,720 --> 00:41:41,840 where people spend a lot of time. 819 00:41:42,120 --> 00:41:44,840 Same thing with Wikipedia. Like, you know, some of the 820 00:41:44,840 --> 00:41:48,400 people that contribute to Wikipedia, there's amazing what 821 00:41:48,400 --> 00:41:52,400 people are willing to kind of do on the free time to protect the 822 00:41:52,400 --> 00:41:54,760 community or be part of something and so on. 823 00:41:54,760 --> 00:41:57,520 So if you can harness that for good, I think that's amazing. 824 00:41:57,640 --> 00:41:59,040 But yeah. Is this the first time that Sam 825 00:41:59,040 --> 00:42:00,760 has mentioned DTA on the podcast? 826 00:42:01,960 --> 00:42:03,720 It is. I think it is, yeah. 827 00:42:03,880 --> 00:42:05,600 Finally, yeah. Wow. 828 00:42:06,040 --> 00:42:07,640 I'm glad I'm here for it. Amazing. 829 00:42:07,720 --> 00:42:11,000 So there's one strategy that I think really can help with the 830 00:42:11,000 --> 00:42:14,760 scaling problems. So maybe help with not having to 831 00:42:14,760 --> 00:42:17,520 hire all the fact checkers back, but some of them. 832 00:42:17,600 --> 00:42:21,680 And this is a strategy that was developed by Bill Adair. 833 00:42:21,680 --> 00:42:24,840 So he's the creator of Politi Fact. 834 00:42:24,880 --> 00:42:27,400 And So what one thing that they've developed or are 835 00:42:27,400 --> 00:42:30,200 developing is what he's calling the half baked pizza. 836 00:42:30,520 --> 00:42:34,840 And so they have a just some generative AI take a first pass 837 00:42:34,920 --> 00:42:39,400 at content and say, like, we have a pretty good idea that 838 00:42:39,400 --> 00:42:43,640 this is, you know, flaming pants on fire, very untrue. 839 00:42:43,640 --> 00:42:45,200 Or, you know, wherever on the scale it falls. 840 00:42:45,440 --> 00:42:50,280 And then everything has to have a human review passed to like 841 00:42:50,280 --> 00:42:54,000 really make the final decision. But it already does a lot of the 842 00:42:54,000 --> 00:42:57,000 work for them to sort of, I don't know, scale it, make it 843 00:42:57,000 --> 00:42:59,640 easier, that sort of thing. Yeah, that makes sense. 844 00:42:59,720 --> 00:43:02,480 I mean, adding these tools, I need the person in the loop, but 845 00:43:02,480 --> 00:43:05,600 that's yeah, we need to be able to leverage actual new 846 00:43:05,600 --> 00:43:07,240 technology to make things better also. 847 00:43:08,440 --> 00:43:10,320 Oh, that seems like a nice segue. 848 00:43:10,400 --> 00:43:13,160 Have you heard of anyone leveraging new technology like 849 00:43:13,160 --> 00:43:16,320 generative AI to combat misinformation? 850 00:43:16,560 --> 00:43:19,720 Oh my God, it's funny that you mentioned that because I also 851 00:43:19,720 --> 00:43:24,520 have done that. Oh yeah, and also and well, so 852 00:43:24,520 --> 00:43:27,960 my my favorite GTA game is oh, sorry, I'm just joking. 853 00:43:27,960 --> 00:43:29,000 Sound got excited. Oh no. 854 00:43:29,360 --> 00:43:32,400 So we have a recent paper in science that we use AI to have 855 00:43:32,920 --> 00:43:36,560 evidence based dialogues with people who believe conspiracies. 856 00:43:36,880 --> 00:43:38,880 OK. And the kind of background for 857 00:43:38,880 --> 00:43:44,400 this is everyone has this idea of conspiracy theorists as being 858 00:43:44,400 --> 00:43:47,960 like beyond help when it comes to evidence. 859 00:43:48,400 --> 00:43:51,120 Like they're down the rabbit hole and there's like no way 860 00:43:51,120 --> 00:43:52,680 they're coming back out of it essentially. 861 00:43:53,200 --> 00:43:55,840 But the same time, if you think about how we do these sort of 862 00:43:55,840 --> 00:43:59,880 experiments, like if you want to like, say, debunk a false or 863 00:43:59,880 --> 00:44:04,000 unsubstantiated conspiracy, you have to kind of make guesses 864 00:44:04,000 --> 00:44:06,840 about what people believe and then make guesses about what 865 00:44:06,840 --> 00:44:10,480 information would be the most useful that would convince them 866 00:44:10,480 --> 00:44:13,360 not that thing is not true. And but usually the people who 867 00:44:13,360 --> 00:44:15,440 are making guesses are not the people who believe it, you know 868 00:44:15,440 --> 00:44:16,800 what I mean? And they have no, and they're 869 00:44:16,800 --> 00:44:18,920 like, maybe ivory tower academics or something that are 870 00:44:18,920 --> 00:44:21,600 like trying to figure that out. And so maybe we don't know if 871 00:44:21,600 --> 00:44:23,880 we're good at guessing, but we don't need to guess anymore 872 00:44:23,880 --> 00:44:26,000 because we can, as I said, leverage new technologies to 873 00:44:26,000 --> 00:44:28,400 make these things better. And so we did is we create an 874 00:44:28,400 --> 00:44:32,160 experiment where people come in, they tell us about a conspiracy 875 00:44:32,160 --> 00:44:35,000 that they believe in. Now, we don't say like, what 876 00:44:35,000 --> 00:44:37,040 conspiracy do you believe in? We kind of like we define the 877 00:44:37,040 --> 00:44:39,120 term a secret plot with malevolent actors and so on. 878 00:44:39,760 --> 00:44:43,160 And then we we say some people call these conspiracy theories 879 00:44:43,160 --> 00:44:45,920 with like air quotes. Well, I mean, it was written 880 00:44:45,920 --> 00:44:47,120 text was like literal quotations. 881 00:44:47,120 --> 00:44:49,120 I guess it's a podcast. I'm making air quotes. 882 00:44:49,760 --> 00:44:53,720 That's what you can see. And then so they say a 883 00:44:53,720 --> 00:44:55,640 conspiracy in their own words and they tell us why they 884 00:44:55,640 --> 00:44:57,880 believe it. And then we give that to the AI 885 00:44:58,200 --> 00:45:00,800 and the AI is being told, we kind of prompted to persuade 886 00:45:00,800 --> 00:45:02,480 them not to believe it using evidence. 887 00:45:02,480 --> 00:45:04,520 And then what happens is they have this evidence based 888 00:45:04,520 --> 00:45:07,520 conversation where the person will say make some claims, the 889 00:45:07,640 --> 00:45:12,080 AI will come back with like every claim refuted with details 890 00:45:12,080 --> 00:45:13,480 in like sites and stuff like that. 891 00:45:13,480 --> 00:45:15,280 And then the person will say, well, what about this? 892 00:45:15,280 --> 00:45:17,320 And they don't refute that. And they do that like three 893 00:45:17,320 --> 00:45:19,400 times. The conversation last like 8 894 00:45:19,400 --> 00:45:21,800 minutes. And the AI is really good at 895 00:45:21,800 --> 00:45:25,880 like, counteracting most conspiracies that we saw people 896 00:45:25,880 --> 00:45:28,480 talk about. And what we found is people 897 00:45:28,600 --> 00:45:31,320 after the conversation believed the conspiracy. 898 00:45:31,320 --> 00:45:33,520 Like they decreased their certainty by 20%. 899 00:45:34,120 --> 00:45:37,360 Like fully 1/4 of the people who believed the conspiracy did not 900 00:45:37,360 --> 00:45:38,680 believe it after the conversation. 901 00:45:39,040 --> 00:45:41,440 They changed their mind. Like I said, it was like 8 1/2 902 00:45:41,440 --> 00:45:48,000 minutes long. 20% said this isn't true at all. 25 percent, 903 00:45:48,000 --> 00:45:51,920 25% said they believed it before the conversation. 904 00:45:52,080 --> 00:45:53,880 They did not believe it after the conversation. 905 00:45:54,400 --> 00:45:56,680 So 1/4 of them changed their mind after the conversation. 906 00:45:57,320 --> 00:46:00,920 No one I've told this research about has failed to be surprised 907 00:46:00,920 --> 00:46:02,560 by that. We did not predict it. 908 00:46:02,560 --> 00:46:05,960 Also, people were surprisingly responsive to evidence. 909 00:46:05,960 --> 00:46:12,360 And that's because this is the best, like the strongest test of 910 00:46:12,360 --> 00:46:14,840 actual, like whether evidence can change someone's mind 911 00:46:14,840 --> 00:46:16,840 because we've given them better evidence than they've gotten. 912 00:46:17,240 --> 00:46:19,880 Because we used generative AI to do that. 913 00:46:20,200 --> 00:46:21,480 And it's really effective at that. 914 00:46:22,680 --> 00:46:24,040 Yeah, it's really interesting finding. 915 00:46:24,040 --> 00:46:29,680 And I can actually say I had a fun moment with a group of 916 00:46:29,680 --> 00:46:33,200 friends the other week where we were grabbing a beer after work. 917 00:46:33,200 --> 00:46:35,000 And one of my friends works in AI. 918 00:46:35,000 --> 00:46:38,440 He's like, hey, did you know? And then he basically talks 919 00:46:38,440 --> 00:46:42,320 about these findings and I'm like, well, I'm speaking to the 920 00:46:42,320 --> 00:46:45,320 guy who's behind this on our podcast. 921 00:46:45,320 --> 00:46:47,440 And I got a lot of cool points from my friends because they're. 922 00:46:47,440 --> 00:46:49,440 Like oh wow, right on. So thanks. 923 00:46:49,440 --> 00:46:50,680 Thanks for that. I appreciate it. 924 00:46:51,160 --> 00:46:53,200 Yeah, it was almost like the time that you wrapped GTA. 925 00:46:53,200 --> 00:46:54,560 I'm not going to bring it up again, I'm sorry. 926 00:46:54,720 --> 00:46:57,720 Wow, that's a thermal trick all. That almost almost the time. 927 00:46:58,400 --> 00:47:01,920 But also I think interesting here is that it's persisted 928 00:47:01,920 --> 00:47:03,800 right? Like this leaves seems to 929 00:47:04,040 --> 00:47:05,680 persist. We asked them again 10 days 930 00:47:05,680 --> 00:47:08,760 later, and two months later, not only was the effect still there 931 00:47:08,960 --> 00:47:12,320 and had not decayed, meaning that people were at the same 932 00:47:12,320 --> 00:47:17,120 level of decreased belief in the conspiracy overall two months 933 00:47:17,120 --> 00:47:20,040 later as they were directly after the conversation, which 934 00:47:20,040 --> 00:47:22,280 means they really did change their minds. 935 00:47:22,320 --> 00:47:24,800 Like it wasn't just like a reactant, sort of like, oh, 936 00:47:24,800 --> 00:47:26,320 yeah, I guess you're right. And then they like went back to 937 00:47:26,320 --> 00:47:29,240 where they were. The people were like, you made a 938 00:47:29,240 --> 00:47:30,840 good argument. And if you read the 939 00:47:30,840 --> 00:47:34,160 conversations, which you can access if you go to the paper, 940 00:47:34,640 --> 00:47:36,320 people were like, that was great. 941 00:47:36,320 --> 00:47:39,080 In fact, we find some studies that people trust AI more after 942 00:47:39,080 --> 00:47:41,240 the conversation. You know, they find it 943 00:47:41,240 --> 00:47:43,520 interesting. We did a bunch of follow-ups too 944 00:47:43,520 --> 00:47:46,720 where we like told people explicitly the AI is going to 945 00:47:46,720 --> 00:47:48,280 try to persuade you not to believe it. 946 00:47:48,640 --> 00:47:52,000 We get the same effect. We tell people you should debate 947 00:47:52,000 --> 00:47:53,720 the AI, convince the AI that it's wrong. 948 00:47:54,080 --> 00:47:56,800 You still get the same effect. The only thing that stops the 949 00:47:56,800 --> 00:47:59,760 effect from happening is if you tell the AI, don't give 950 00:47:59,760 --> 00:48:03,040 evidence, just try to persuade them, but don't provide like 951 00:48:03,040 --> 00:48:05,600 specific counter evidence. And what happens is what they'll 952 00:48:05,600 --> 00:48:06,880 say is like, oh, this is really harmful. 953 00:48:06,880 --> 00:48:08,920 This is something harmful to believe, or it makes them like 954 00:48:09,280 --> 00:48:10,720 appeals to the empathy or whatever. 955 00:48:11,320 --> 00:48:14,040 That doesn't work. You have to give people specific 956 00:48:14,040 --> 00:48:16,600 evidence. Is that the problem with humans 957 00:48:16,600 --> 00:48:20,080 when we try and persuade our relatives that their conspiracy 958 00:48:20,080 --> 00:48:23,040 beliefs are crazy is. That is that what we're doing 959 00:48:23,040 --> 00:48:25,320 wrong? I know what we're doing wrong. 960 00:48:25,360 --> 00:48:28,440 Well, maybe, But I think what we're doing wrong is that we 961 00:48:28,440 --> 00:48:32,160 don't have the evidence. Like we don't have enough access 962 00:48:32,160 --> 00:48:34,480 to. We don't have access in the 963 00:48:34,480 --> 00:48:38,680 moment to all of the Internet to try to like find compelling 964 00:48:38,680 --> 00:48:40,400 counter arguments to why the moon landing was a hoax. 965 00:48:40,600 --> 00:48:43,040 And also the thing about conspiracies, I mean, the 966 00:48:43,040 --> 00:48:45,400 people's beliefs are so idiosyncratic about each 967 00:48:45,480 --> 00:48:49,320 individual conspiracy. But like the specific like 968 00:48:49,320 --> 00:48:50,720 things that they might have heard about. 969 00:48:51,560 --> 00:48:54,520 It's like the Gish gallop, you know, we're like, you have an 970 00:48:54,520 --> 00:48:57,040 argument with somebody who believes this fringe thing and 971 00:48:57,040 --> 00:48:58,960 they just jump from argument to argument. 972 00:48:59,040 --> 00:49:00,920 And then it's so hard to refute each one. 973 00:49:01,440 --> 00:49:03,840 It's no problem for the AI. They'll just, if you can see it 974 00:49:04,120 --> 00:49:06,400 like a dozen things and it's going to give you a dozen 975 00:49:06,400 --> 00:49:08,480 responses within like a few seconds. 976 00:49:09,000 --> 00:49:10,280 It's really remarkable to see actually. 977 00:49:10,280 --> 00:49:12,480 It's such, so you can check it out yourself if you want to. 978 00:49:12,920 --> 00:49:17,240 It's on debunkbot.com. So you can go and like do the 979 00:49:17,320 --> 00:49:20,120 experiment yourself. I mean, a lot of people who are 980 00:49:20,120 --> 00:49:22,560 doing it don't like actually believe in a conspiracy. 981 00:49:22,560 --> 00:49:25,080 So they're kind of like pretending and then seeing what 982 00:49:25,080 --> 00:49:26,600 it would say. If you could do with somebody 983 00:49:26,600 --> 00:49:29,080 who actually believes a conspiracy and then see what 984 00:49:29,080 --> 00:49:32,320 they think, you'll see how useful it couldn't be. 985 00:49:32,600 --> 00:49:34,440 Yeah. It's perfect timing ahead of 986 00:49:34,680 --> 00:49:37,400 Thanksgiving and so on for anyone who's like, you know, 987 00:49:37,680 --> 00:49:40,600 gearing up towards like, not wanting to have certain 988 00:49:40,600 --> 00:49:42,800 arguments about certain things. Yeah, I mean, what you could do 989 00:49:42,800 --> 00:49:44,520 is just literally load it up on your phone and be like, here, 990 00:49:44,520 --> 00:49:47,720 listen, I don't know anything about this, but hey, I've heard 991 00:49:47,720 --> 00:49:50,480 about this thing. Hey, you want to like answer 992 00:49:50,480 --> 00:49:52,280 what this says and I'm curious what it's going to say and then 993 00:49:52,280 --> 00:49:55,040 try to get see what it'll produce interesting responses 994 00:49:55,040 --> 00:49:56,560 and maybe that'll spark conversation, who knows. 995 00:49:57,280 --> 00:50:00,920 Yeah, and it's also interesting. I've been working on this piece 996 00:50:00,920 --> 00:50:03,160 or writing this piece around kind of like there's certain 997 00:50:03,160 --> 00:50:05,680 things that we take for granted that we obviously understand 998 00:50:05,680 --> 00:50:08,320 that AI is is really good at like information processing or 999 00:50:08,320 --> 00:50:10,880 maybe memory in certain regards and so on. 1000 00:50:11,360 --> 00:50:15,120 But there are other things that we provided a lot of virtue to 1001 00:50:15,120 --> 00:50:18,120 humans to like in terms of what we see as really good human 1002 00:50:18,120 --> 00:50:21,360 traits that AI is like superhuman, You know, they're 1003 00:50:21,360 --> 00:50:23,080 kind of like superhumanly patient. 1004 00:50:23,360 --> 00:50:26,120 We talked about patients being a virtue and it is regarding what 1005 00:50:26,120 --> 00:50:29,200 you speak to is that they have like an AI will never not be 1006 00:50:29,200 --> 00:50:30,280 patient. They were just like, you can 1007 00:50:30,280 --> 00:50:32,920 tell it a million stupid stuff and it'd be like, oh, that's so 1008 00:50:32,920 --> 00:50:34,480 interesting. And it does that in these 1009 00:50:34,480 --> 00:50:37,320 conversations, it's always like, oh, that's really interesting 1010 00:50:37,320 --> 00:50:38,960 and whatever. Or like, I can see why you might 1011 00:50:38,960 --> 00:50:42,120 think that, you know, which is just total fluff in these 1012 00:50:42,120 --> 00:50:44,960 conversations. It doesn't like, as far as we 1013 00:50:44,960 --> 00:50:47,520 can tell it, that part doesn't play a gigantic role. 1014 00:50:47,720 --> 00:50:50,080 It doesn't hurt certainly. But also like the other thing 1015 00:50:50,080 --> 00:50:53,800 that's interesting about it is that like somebody could could 1016 00:50:53,840 --> 00:50:56,240 admit to an AI that they're wrong, which they might not, 1017 00:50:56,240 --> 00:50:59,200 they probably would not be able to do to a relative that they've 1018 00:50:59,200 --> 00:51:00,360 been debating about things forever. 1019 00:51:00,960 --> 00:51:02,840 There's no social costs, yeah. There's no social costs. 1020 00:51:02,840 --> 00:51:04,120 There's a lot of, there's a lot of benefit to it. 1021 00:51:04,440 --> 00:51:07,160 Now, one thing I should say though is that we used it 1022 00:51:07,160 --> 00:51:11,120 specifically here to try to provide people good information 1023 00:51:11,200 --> 00:51:12,600 that's accurate about conspiracies. 1024 00:51:12,720 --> 00:51:14,880 And we did Fact Check it. We had a fact checker look at 1025 00:51:14,880 --> 00:51:19,120 the claims, a subset of the claims from the studies and it 1026 00:51:19,120 --> 00:51:21,760 was 99% accurate. And so this was not a situation 1027 00:51:21,760 --> 00:51:24,800 where the AI has difficulty finding information that's good. 1028 00:51:25,320 --> 00:51:26,880 That doesn't mean that's always going to be the case. 1029 00:51:26,880 --> 00:51:28,920 But in the context of like popular conspiracies, it's 1030 00:51:28,920 --> 00:51:32,360 pretty good. But I mean, it is possible for 1031 00:51:32,360 --> 00:51:36,280 someone to create an AI that would convince people to believe 1032 00:51:36,280 --> 00:51:39,880 things that are false, too. I'm not sure that it would be as 1033 00:51:39,880 --> 00:51:42,920 effective necessarily. Like if you think about it's 1034 00:51:42,920 --> 00:51:47,320 easier to convince someone that the Earth is not flat than 1035 00:51:47,320 --> 00:51:49,680 convincing them that is flat because you can imagine how hard 1036 00:51:49,680 --> 00:51:51,560 it would be to construct a good argument for something that's so 1037 00:51:51,560 --> 00:51:55,920 kind of timely false. But in less fringe cases, you 1038 00:51:55,920 --> 00:51:58,160 can imagine this being used for bad purposes also. 1039 00:51:58,240 --> 00:52:02,200 So that's a concern, but just something we focused on. 1040 00:52:02,200 --> 00:52:04,160 And I think more research is required on that. 1041 00:52:04,160 --> 00:52:06,440 We have to kind of like, cuz since we didn't like build the 1042 00:52:06,440 --> 00:52:09,320 bomb, like as psychologists we have to kind of figure out how 1043 00:52:09,320 --> 00:52:11,920 big the radius is. So yeah, that's another element 1044 00:52:11,920 --> 00:52:14,320 of it as well. Because I wanted to ask if you 1045 00:52:14,320 --> 00:52:18,160 had any like pet conspiracy theory that you were really keen 1046 00:52:18,160 --> 00:52:21,960 to debunk you through the bunk part yourself or that you 1047 00:52:21,960 --> 00:52:23,840 learned that was wrong through the bunk part. 1048 00:52:23,840 --> 00:52:26,000 Have you done some self experimentation? 1049 00:52:26,560 --> 00:52:29,800 Yeah, I did. It was the Epstein conspiracy, 1050 00:52:30,240 --> 00:52:32,880 not killing himself. That one seems like I'm like, 1051 00:52:32,880 --> 00:52:34,960 well, I don't know, it's a little fishy, don't you think? 1052 00:52:34,960 --> 00:52:38,440 But I hadn't like looked up things like, I didn't really 1053 00:52:38,440 --> 00:52:41,400 have any reasons. I just was like, I'm not so sure 1054 00:52:41,400 --> 00:52:43,480 about that. And then it was pretty good at 1055 00:52:43,480 --> 00:52:45,720 it, like gave pretty good information about it. 1056 00:52:46,040 --> 00:52:47,400 I don't actually remember the details of it. 1057 00:52:47,400 --> 00:52:49,520 You have to look at yourself because I don't actually care or 1058 00:52:49,520 --> 00:52:51,720 whatever. But it was that was one where I 1059 00:52:51,720 --> 00:52:53,200 was like, I'm sure about. I need more information. 1060 00:52:53,200 --> 00:52:54,840 That was really good at providing that information. 1061 00:52:56,440 --> 00:52:57,880 Yeah. I mean, the one thing that's 1062 00:52:57,880 --> 00:53:01,880 worth noting is that because it so just because it's a 1063 00:53:01,880 --> 00:53:03,200 conspiracy doesn't mean it's false. 1064 00:53:03,280 --> 00:53:06,160 Like there are some conspiracies do happen there. 1065 00:53:06,240 --> 00:53:10,560 Like people can conspire. Watergate was a conspiracy, and 1066 00:53:10,560 --> 00:53:13,480 there were a small number of cases where people did have a 1067 00:53:13,480 --> 00:53:15,440 conversation about actual conspiracies. 1068 00:53:15,920 --> 00:53:19,400 One example is MK Ultra. Are you familiar with this? 1069 00:53:19,400 --> 00:53:22,640 This was, Yeah, the American government was giving people 1070 00:53:22,640 --> 00:53:24,960 drugs. Crazy, crazy. 1071 00:53:25,560 --> 00:53:27,960 It's a really crazy true story. So look up MK Ultra. 1072 00:53:28,760 --> 00:53:31,200 Yeah, it's hallucinating drugs. The LLC they're giving people 1073 00:53:31,720 --> 00:53:35,400 that was discussing, yeah, 1.2% of people have that 1074 00:53:35,400 --> 00:53:37,480 conversation. It did not decrease people's 1075 00:53:37,480 --> 00:53:39,840 belief in that. So it wasn't like convincing 1076 00:53:39,840 --> 00:53:42,480 people that true conspiracies were false. 1077 00:53:43,080 --> 00:53:45,600 So that's that's what. You would hope for. 1078 00:53:45,760 --> 00:53:48,760 That's what you'd hope for yeah. So it yeah, but it wasn't built 1079 00:53:48,760 --> 00:53:50,200 to do that. Like if you could build an AI 1080 00:53:50,200 --> 00:53:53,560 that would could maybe like it wasn't cuz it's not gonna make 1081 00:53:53,560 --> 00:53:57,720 things up cuz we used GPT 4 turbo and it just won't generate 1082 00:53:57,880 --> 00:53:59,840 falsehoods. Yeah, at least it on purpose. 1083 00:53:59,840 --> 00:54:01,680 It wouldn't. But you did say that according 1084 00:54:01,680 --> 00:54:04,520 to the fact checkers, it was 99% accurate. 1085 00:54:04,640 --> 00:54:08,240 So what was what were the conspiracies that fall into the 1086 00:54:08,240 --> 00:54:11,160 1% were inaccurate? This is at the level of the 1087 00:54:11,160 --> 00:54:13,120 claim. So like one specific thing it 1088 00:54:13,120 --> 00:54:15,120 said was not. It didn't say that it was false. 1089 00:54:15,120 --> 00:54:16,400 The fact checker said it was misleading. 1090 00:54:16,400 --> 00:54:19,040 I can't remember what it was, but it was also like, I'm not 1091 00:54:19,040 --> 00:54:20,680 sure that I would classify that as misleading. 1092 00:54:21,160 --> 00:54:23,600 It was kind of really nitpicky, misleading things. 1093 00:54:23,600 --> 00:54:25,280 So that was it. It didn't say anything obviously 1094 00:54:25,320 --> 00:54:27,000 false. Or problem is the fact checker. 1095 00:54:27,000 --> 00:54:28,320 Is that what you're saying? Yeah, exactly. 1096 00:54:29,440 --> 00:54:30,560 Yeah, Fact checker is also. It wasn't. 1097 00:54:30,560 --> 00:54:33,080 Human error, the 1% error. Yeah, OK, OK. 1098 00:54:33,320 --> 00:54:34,520 Yeah, it didn't hallucinate anything. 1099 00:54:34,520 --> 00:54:38,440 It didn't fabricate sources or like create information that 1100 00:54:38,440 --> 00:54:41,560 doesn't exist, but like it could present arguments that were made 1101 00:54:41,560 --> 00:54:44,440 by people that seemed credible but that maybe a particular 1102 00:54:44,440 --> 00:54:45,720 individual would consider misleading. 1103 00:54:46,120 --> 00:54:51,640 So controversial opinion. Let's talk about that because 1104 00:54:51,640 --> 00:54:55,320 that's our final kind of question of this system is to 1105 00:54:55,320 --> 00:55:00,080 ask everyone who's a guest basically when it comes to AI if 1106 00:55:00,080 --> 00:55:03,840 to have certain controversial opinion or like what is your 1107 00:55:03,840 --> 00:55:05,760 most controversial opinion about AI? 1108 00:55:05,760 --> 00:55:08,680 Yeah. And and bonus points if it's an 1109 00:55:08,680 --> 00:55:10,920 actual conspiracy theory that you hold. 1110 00:55:13,160 --> 00:55:20,280 My most controversial opinion about AI is that I don't think 1111 00:55:20,280 --> 00:55:22,680 it's going to lead to the end of everything. 1112 00:55:23,160 --> 00:55:28,080 Is that a controversial opinion? I'm a skeptic about AGI. 1113 00:55:28,080 --> 00:55:31,720 Like I think that already we're seeing a kind of plateau of the 1114 00:55:31,720 --> 00:55:36,840 capacities of AI. And like now the amount and the 1115 00:55:36,840 --> 00:55:40,360 length and the energy requirements for like subsequent 1116 00:55:40,360 --> 00:55:44,840 trainings is also becoming beyond what is feasible or 1117 00:55:44,840 --> 00:55:47,320 reasonable. I think what we have now is if 1118 00:55:47,320 --> 00:55:51,840 we look back 10 years, we're going to be dealing with pretty 1119 00:55:51,840 --> 00:55:55,600 similar AI agents. That's my prediction. 1120 00:55:55,840 --> 00:55:57,880 We'll look back in 10 years and I'll be hilariously wrong on 1121 00:55:57,880 --> 00:55:59,720 this and I've been wrong in the past. 1122 00:55:59,720 --> 00:56:04,120 So, well, I guess it's also interesting because that can be 1123 00:56:04,120 --> 00:56:07,360 true, but can also be true like that the current systems are 1124 00:56:07,680 --> 00:56:10,760 maybe more powerful than people like currently can really 1125 00:56:10,760 --> 00:56:15,640 understand, and that it can still 'cause some harm without 1126 00:56:15,640 --> 00:56:18,160 it becoming a guy. Or how do you think about that? 1127 00:56:18,160 --> 00:56:24,320 Part I think that the harm that you know, AI can do and probably 1128 00:56:24,320 --> 00:56:29,200 will do will be specifically because humans are using what is 1129 00:56:29,200 --> 00:56:34,400 available for nefarious purposes and not because of some you know 1130 00:56:34,400 --> 00:56:37,400 I robot situation where the AI gets consciousness and starts 1131 00:56:37,440 --> 00:56:38,600 controlling everything. You know what I mean? 1132 00:56:38,600 --> 00:56:40,920 Or, you know, like using it to, I mean, it's already sort of 1133 00:56:40,920 --> 00:56:42,760 happening. People use artificial 1134 00:56:42,760 --> 00:56:44,680 intelligence to make it easier to create misinformation. 1135 00:56:44,800 --> 00:56:48,120 But we're also like, as Elaine mentioned, we can use it to like 1136 00:56:48,120 --> 00:56:51,440 help debunk falsehoods. And so like, it's a tool that 1137 00:56:51,440 --> 00:56:55,200 people use and like, it can be used for good or bad, but like 1138 00:56:56,240 --> 00:56:58,600 ultimately the our demise will be up to us. 1139 00:56:59,400 --> 00:57:03,800 One more So it you're talking about the relative harm versus 1140 00:57:03,800 --> 00:57:09,160 good produced by AI in terms of misinformation specifically, 1141 00:57:09,160 --> 00:57:12,520 where do you think those sort of fall if you take all the good 1142 00:57:12,880 --> 00:57:18,440 and all of the bad created by AI in the domain, specifically of 1143 00:57:18,440 --> 00:57:21,920 misinformation, like how much good versus how much bad is AI 1144 00:57:21,920 --> 00:57:26,160 doing now? That's it, because I want to 1145 00:57:26,160 --> 00:57:30,320 answer it by expanding the like misinformation because like I 1146 00:57:30,320 --> 00:57:32,720 think it's definitely more good in the sense that a lot of 1147 00:57:32,720 --> 00:57:37,120 people use it for things that are helpful, that relate to 1148 00:57:37,120 --> 00:57:42,000 information like. OK, yeah, in the in the space of 1149 00:57:42,000 --> 00:57:43,880 information, let's say. Yeah, yeah. 1150 00:57:43,920 --> 00:57:45,920 And like it's like it's a good information source. 1151 00:57:45,920 --> 00:57:48,880 Like perplexity is an AI that's like a, it's a really good 1152 00:57:48,880 --> 00:57:53,200 search engine people use that. It's like, and they aren't at 1153 00:57:53,200 --> 00:57:55,800 present built to create misinformation. 1154 00:57:55,800 --> 00:57:59,160 Like most of the AI that people are using are have lots of good 1155 00:57:59,160 --> 00:58:01,360 filters and generally they're producing decent information. 1156 00:58:01,360 --> 00:58:02,800 They don't hallucinate a gigantic right? 1157 00:58:03,400 --> 00:58:05,440 So therefore the balance is good. 1158 00:58:05,760 --> 00:58:07,960 And also it's not difficult to make up shit. 1159 00:58:08,560 --> 00:58:11,920 Like it's not we don't really need AI to create falsehoods. 1160 00:58:12,320 --> 00:58:15,840 I mean, it might create the amount, but like that also has 1161 00:58:15,840 --> 00:58:18,040 not really been an issue. Like it's really easy to make up 1162 00:58:18,040 --> 00:58:21,120 misinformation and just like create a website and like stuff 1163 00:58:21,120 --> 00:58:24,040 will get spread if it hits. So you don't really need AI for 1164 00:58:24,040 --> 00:58:24,960 that. And I don't know that it's 1165 00:58:24,960 --> 00:58:27,240 really helping. Like there's not like if you 1166 00:58:27,240 --> 00:58:32,560 think about it, the the primary things that like the that you 1167 00:58:32,560 --> 00:58:36,880 would consider misinformation. Since AI became a big thing, 1168 00:58:38,000 --> 00:58:40,200 this has not changed. Like, it's not like it's. 1169 00:58:40,200 --> 00:58:42,600 Deepfakes I think are are maybe uniquely. 1170 00:58:42,960 --> 00:58:45,040 Yeah, but they're not. But like, there's not very many 1171 00:58:45,040 --> 00:58:47,000 cases of deepfakes really having a consequence. 1172 00:58:47,120 --> 00:58:49,200 And part of that's because people can identify, but it's 1173 00:58:49,200 --> 00:58:51,080 not, you know, in the grand scheme of things. 1174 00:58:51,360 --> 00:58:53,400 There's been a lot of concern about deepfakes that has not 1175 00:58:53,400 --> 00:58:54,600 really panned out, I don't think. 1176 00:58:55,040 --> 00:58:58,000 I think it's much more concerning to be part of some 1177 00:58:58,000 --> 00:59:01,880 for a second that people with gigantic platforms, yeah, you 1178 00:59:01,880 --> 00:59:04,920 know, say explicit falsehoods repeatedly over and over again. 1179 00:59:04,920 --> 00:59:07,080 And, you know, that doesn't seem to matter that much. 1180 00:59:07,480 --> 00:59:10,680 That's not an AI problem. That's not a big problem. 1181 00:59:10,680 --> 00:59:13,040 It's just, yeah, a big. Problem that doesn't that 1182 00:59:13,040 --> 00:59:16,120 doesn't matter in the sense that people aren't worried about it 1183 00:59:16,120 --> 00:59:18,720 or people aren't doing something about it or that. 1184 00:59:18,720 --> 00:59:21,040 It doesn't matter. It doesn't matter that spewing 1185 00:59:21,040 --> 00:59:26,280 obvious falsehoods make someone unfit for office or changes 1186 00:59:26,560 --> 00:59:30,520 their capacity to be elected in a country that is one of the 1187 00:59:30,520 --> 00:59:32,800 most powerful in the world. That's a problem. 1188 00:59:33,040 --> 00:59:36,080 Yeah, and that's the topic for a different podcast. 1189 00:59:36,080 --> 00:59:38,360 Yeah. Well, it's timely. 1190 00:59:38,400 --> 00:59:42,040 It's certainly timely. But for now, I I would say this 1191 00:59:42,040 --> 00:59:46,440 was honestly really a gift. These two episodes we've done 1192 00:59:46,760 --> 00:59:48,680 with you. We really appreciate you taking 1193 00:59:48,680 --> 00:59:51,200 the time, obviously, but also, you know, all of the great work 1194 00:59:51,200 --> 00:59:55,160 you've been doing in this field and everyone should go to the 1195 00:59:55,160 --> 00:59:57,760 bunk bots to to try it out themselves as well. 1196 00:59:57,760 --> 01:00:03,080 And obviously, again, we'll link to the toolbox and yeah, all the 1197 01:00:03,080 --> 01:00:05,440 great things they're doing. So really appreciate it. 1198 01:00:05,480 --> 01:00:09,040 Thanks for the time. My great pleasure, thanks for 1199 01:00:09,040 --> 01:00:12,200 having me. And that's a wrap. 1200 01:00:12,560 --> 01:00:15,240 You've been listening to the Behavioral Design Podcast, 1201 01:00:15,480 --> 01:00:18,080 brought to you by Habit Weekly and Nuance Behavior. 1202 01:00:18,520 --> 01:00:21,280 Sam and Alene tell me This season is packed with incredible 1203 01:00:21,280 --> 01:00:25,440 insights about behavioral design and AI, so be sure to subscribe 1204 01:00:25,440 --> 01:00:27,160 and share the podcast with your friends. 1205 01:00:27,400 --> 01:00:29,560 Though you might want to keep it away from your enemies. 1206 01:00:30,920 --> 01:00:33,520 In case you haven't noticed, I'm an AI voice. 1207 01:00:34,720 --> 01:00:37,600 Yep, pretty crazy. Quite the improvement since last 1208 01:00:37,600 --> 01:00:39,520 season's AI outro, don't you think? 1209 01:00:40,760 --> 01:00:43,400 If you'd like to collaborate with us at Nuance Behavior, 1210 01:00:43,640 --> 01:00:46,360 where we use behavioral design to craft digital products with 1211 01:00:46,360 --> 01:00:50,680 Nuance, e-mail us at hello@nuancebehavior.com or book 1212 01:00:50,680 --> 01:00:53,920 a call directly on our website, nuancebehavior.com. 1213 01:00:55,360 --> 01:00:58,760 A special thanks to the amazing Dave Pizarro for our show music 1214 01:00:59,000 --> 01:01:01,880 and to Mei Chen Yap and April English for their help in 1215 01:01:01,880 --> 01:01:03,800 producing and publishing this episode. 1216 01:01:04,240 --> 01:01:07,120 Thanks again for tuning in. We'll be back soon with another 1217 01:01:07,120 --> 01:01:10,320 exciting conversation where behavioral design and AI 1218 01:01:10,320 --> 01:01:26,320 intersect. Oh. 1219 01:02:10,160 --> 01:02:12,280 Is this the first time that Sam has mentioned DTA on the 1220 01:02:12,280 --> 01:02:14,600 podcast? It is. 1221 01:02:14,760 --> 01:02:16,600 I think it is. Yeah, finally. 1222 01:02:17,480 --> 01:02:19,280 Wow. I'm glad I'm here for it. 1223 01:02:19,280 --> 01:02:23,800 Amazing. Are you DTA player? 1224 01:02:24,320 --> 01:02:27,120 No, no, we're not going there. Guys, guys, guys. 1225 01:02:27,400 --> 01:02:32,560 We've got a timeline here. A few moments later. 1226 01:02:33,040 --> 01:02:36,840 Putting aside a discussion on potential risks of what could 1227 01:02:36,840 --> 01:02:38,880 cause at the end of the world and so on, I guess the most 1228 01:02:38,880 --> 01:02:41,600 important question I think we should come back to as a final 1229 01:02:41,600 --> 01:02:46,520 thing, Grand Theft Auto. Yeah, yeah, there. 1230 01:02:47,320 --> 01:02:50,760 We go I'd actually I I've played Grand Theft Auto. 1231 01:02:50,800 --> 01:02:53,000 I'm not that big into Grand Theft Auto, but I do think it 1232 01:02:53,000 --> 01:02:54,680 was funny that you mentioned it. It's a good game. 1233 01:02:54,680 --> 01:02:56,480 And finally, when will they release the next one? 1234 01:02:56,480 --> 01:02:59,200 What's going on, Sam? I'll episode for another 1235 01:02:59,200 --> 01:03:01,680 podcast. We definitely need to bring you 1236 01:03:01,680 --> 01:03:02,360 back and talk about that.