1 00:00:03,280 --> 00:00:05,880 Hello and welcome to the Behavioral Design Podcast. 2 00:00:05,960 --> 00:00:08,680 This season we're diving into the intersection of behavioral 3 00:00:08,680 --> 00:00:11,080 science and AI. We want to make sense of the 4 00:00:11,080 --> 00:00:14,080 state of AI, from understanding how humans interact with 5 00:00:14,080 --> 00:00:17,520 intelligent systems to using AI to do behavioral design itself. 6 00:00:17,720 --> 00:00:21,280 I'm Aline Holsworth, a health tech advisor specializing in AI 7 00:00:21,280 --> 00:00:23,880 and product design. Over the past 15 years, I've 8 00:00:23,880 --> 00:00:26,720 been crafting human centered products with behavioral science 9 00:00:26,720 --> 00:00:29,160 at the core. At Apple, I LED Behavioral 10 00:00:29,160 --> 00:00:32,119 Science for Health AI, designing and launching AI powered 11 00:00:32,119 --> 00:00:34,440 features to help users reach their health goals. 12 00:00:35,160 --> 00:00:37,160 And I'm Samuel Sultzer, your second Co host. 13 00:00:37,400 --> 00:00:40,600 I'm a behavioral strategist specializing in hybrid formation 14 00:00:40,600 --> 00:00:43,200 and designing products that drive long term baby change. 15 00:00:43,640 --> 00:00:46,880 I work with leading tech organizations integrating AI to 16 00:00:46,880 --> 00:00:48,240 scale behavioral design for good. 17 00:00:48,800 --> 00:00:52,200 And I'm also the founder of Baby Bites, a dedicated community on 18 00:00:52,440 --> 00:00:56,240 behavioral science and AI. Quick word on Nuance Behavior 19 00:00:56,240 --> 00:00:59,640 where we help organizations build impactful digital products 20 00:00:59,640 --> 00:01:02,640 using behavioral design. We only take on a few clients at 21 00:01:02,640 --> 00:01:05,200 a time to ensure the highest level of quality for our 22 00:01:05,200 --> 00:01:07,400 tailored evidence based solutions. 23 00:01:08,280 --> 00:01:11,080 If you'd like to become one of our special projects, e-mail us 24 00:01:11,120 --> 00:01:14,760 at hello@nuancebehavior.com or we could call directly on our 25 00:01:14,760 --> 00:01:31,520 website, nuancebehavior.com. Sam, Hey, Elaine, Do you 26 00:01:31,520 --> 00:01:38,080 remember way, way back in the end of last year, in late 2024, 27 00:01:38,080 --> 00:01:41,320 which in AI time is basically a decade ago? 28 00:01:41,360 --> 00:01:43,920 Yeah. Do you remember we organized a 29 00:01:43,920 --> 00:01:47,640 series of of LinkedIn polls? Do you know what I'm? 30 00:01:47,880 --> 00:01:50,480 Talking about, oh, I don't remember a lot of things from 31 00:01:50,480 --> 00:01:52,280 last year, but I do remember those polls. 32 00:01:52,360 --> 00:01:55,480 Yes, I do remember them. And we were really interested in 33 00:01:55,480 --> 00:01:59,960 using AI for different parts of the behavioural design process. 34 00:01:59,960 --> 00:02:04,240 So if you think of the the process just very generally as 35 00:02:04,240 --> 00:02:08,600 like starting with discovery, so trying to aggregate information 36 00:02:08,600 --> 00:02:12,920 and understand the landscape of a key behaviour, identify that 37 00:02:12,920 --> 00:02:15,360 key behaviour, you know, basically like doing a lit 38 00:02:15,360 --> 00:02:19,160 review and understanding what's out there and then moving on to 39 00:02:19,320 --> 00:02:23,200 diagnosis. So this often is, you know, some 40 00:02:23,200 --> 00:02:28,400 sort of behaviour mapping or a UX audit looking at the the user 41 00:02:28,400 --> 00:02:32,160 journey and so on. And then maybe building on those 42 00:02:32,160 --> 00:02:35,680 two phases to do some design, some behavioral design with the, 43 00:02:36,040 --> 00:02:40,200 you could be creating a product or you know, refining an 44 00:02:40,200 --> 00:02:42,080 existing product, something like that. 45 00:02:42,600 --> 00:02:45,400 And then finally ending in a testing phase. 46 00:02:45,680 --> 00:02:47,680 And of course, we're sort of glossing over the fact that you 47 00:02:47,680 --> 00:02:50,320 can do testing in any of these phases, but let's just say we 48 00:02:50,320 --> 00:02:53,160 end with testing that sort of final product. 49 00:02:53,800 --> 00:02:58,360 So we were curious, given the advent of AI and how how we're 50 00:02:58,360 --> 00:03:01,200 seeing all kinds of different applications out there in the 51 00:03:01,200 --> 00:03:07,880 world, what do our friends and followers think it would be 52 00:03:08,040 --> 00:03:13,200 really well suited to replacement or the help from AI 53 00:03:13,200 --> 00:03:15,600 tools. And so, you know, all the 54 00:03:15,600 --> 00:03:20,200 caveats apply with this is a, you know, a very unscientific, 55 00:03:20,240 --> 00:03:23,960 crowdsourced, convenient sample of like our, you know, not very 56 00:03:23,960 --> 00:03:26,640 many LinkedIn friends. Don't say that. 57 00:03:27,000 --> 00:03:28,600 I think we have pretty good reach on LinkedIn. 58 00:03:29,280 --> 00:03:34,520 OK, sure, but for science's sake, small samples this is 59 00:03:34,520 --> 00:03:38,400 these are not the many thousands that you would get on M jerk or 60 00:03:38,400 --> 00:03:41,720 prolific or whatever. So yes, we did these polls. 61 00:03:42,200 --> 00:03:48,720 It was more of AI think, impetus for conversation than anything 62 00:03:48,720 --> 00:03:49,960 else. We really wanted to see like 63 00:03:49,960 --> 00:03:55,080 what are our friends intuitions in these areas and what did we 64 00:03:55,080 --> 00:03:56,160 find? Do you remember? 65 00:03:56,680 --> 00:03:59,760 No, but I'm not actually super intrigued to hear because it 66 00:03:59,760 --> 00:04:02,600 feels like in hindsight, that was like a really interesting 67 00:04:02,600 --> 00:04:05,560 time to ask this because at least for me, I would say 68 00:04:06,560 --> 00:04:11,720 working closely with AII, think around last fall around October, 69 00:04:11,720 --> 00:04:16,760 November, that's where a lot of AI models became kind of taking 70 00:04:16,760 --> 00:04:19,839 the next level leap into something that we're dealing 71 00:04:19,839 --> 00:04:23,600 with today. And I, I feel like so much has 72 00:04:23,600 --> 00:04:25,600 happened since then. So I, I don't know, I'm super 73 00:04:25,600 --> 00:04:28,160 keen to hear actually what, what did people say? 74 00:04:29,440 --> 00:04:33,120 Well, and and honestly, many of the deep researchers that we 75 00:04:33,120 --> 00:04:35,640 have today, we're not even out when we ran this poll. 76 00:04:35,640 --> 00:04:37,840 So the land, the landscape looked very different then. 77 00:04:38,280 --> 00:04:42,160 However, I don't think that the, you know, this is just my, my 78 00:04:42,160 --> 00:04:45,120 hypothesis, but I don't think the opinions would be different 79 00:04:45,120 --> 00:04:46,960 if we asked people the same question now. 80 00:04:47,200 --> 00:04:51,240 So the vast majority of our respondents said that the 81 00:04:51,280 --> 00:04:54,560 discovery phase, you know, like doing a inventory of the 82 00:04:54,560 --> 00:04:57,760 landscape and seeing what existing literature says, 83 00:04:58,040 --> 00:05:02,240 basically doing a late review that this is the best phase 84 00:05:02,560 --> 00:05:04,680 suited for outsourcing to AI tools. 85 00:05:04,680 --> 00:05:08,240 Does that sort of feel right to you? 86 00:05:09,120 --> 00:05:14,400 Yeah, it is interesting. I, I think in some ways I agree 87 00:05:14,400 --> 00:05:17,920 with you that like probably people would say something 88 00:05:17,920 --> 00:05:21,240 similar today, but it's also interesting, like does that 89 00:05:21,240 --> 00:05:25,680 reflect how AI, what AI is capable of or is it also 90 00:05:25,680 --> 00:05:28,480 reflecting kind of like what people are afraid of it being 91 00:05:28,480 --> 00:05:32,320 capable of? Yeah, yeah, we're we're only 92 00:05:32,320 --> 00:05:36,200 talking about people's subjective opinions with all of 93 00:05:36,200 --> 00:05:38,800 their biases baked in, not the reality of the situation. 94 00:05:38,880 --> 00:05:42,560 Well, I feel scarier that AI could be able to do the second 95 00:05:42,560 --> 00:05:46,120 and third steps, especially rather than the first step in 96 00:05:46,120 --> 00:05:48,520 the journey. Like it feels if AI could do a 97 00:05:48,520 --> 00:05:53,360 really good job around kind of better running interventions, 98 00:05:53,360 --> 00:05:57,360 designing them or understanding and experimenting like those is 99 00:05:57,360 --> 00:06:00,480 probably a little bit feels a little bit more closer to home 100 00:06:00,480 --> 00:06:02,800 as well. It might be reflected, you know, 101 00:06:02,800 --> 00:06:06,760 that AI might not be always as good and evolved in those areas 102 00:06:06,880 --> 00:06:09,680 depending on the context. But I also do think it's it's a 103 00:06:09,680 --> 00:06:14,080 little more scary to to imagine for a lot of behavioral 104 00:06:14,080 --> 00:06:16,960 scientists. I think you're right that it is 105 00:06:17,040 --> 00:06:19,640 maybe threatening to one's identity as a behavioral 106 00:06:19,640 --> 00:06:23,680 scientist to think the later phases could be automated by AI. 107 00:06:23,680 --> 00:06:28,920 However, if you think about what are regularly touted strengths 108 00:06:29,000 --> 00:06:34,080 of at least generative AI, that's really consistent with 109 00:06:34,080 --> 00:06:38,280 the late review phase. So like scanning massive amounts 110 00:06:38,320 --> 00:06:41,320 of, you know, whatever it is data, in this case, it's 111 00:06:41,320 --> 00:06:43,800 research papers. So you just taking like 112 00:06:43,800 --> 00:06:47,680 thousands and thousands of papers and assessing them very 113 00:06:47,680 --> 00:06:52,080 quickly, summarizing research findings like how how great is 114 00:06:52,080 --> 00:06:56,160 ChatGPT at summarizing anything? Like excellent often does a 115 00:06:56,160 --> 00:06:58,920 really, really great job. And then extracting key 116 00:06:58,920 --> 00:07:02,320 insights, right? Organizing based on themes, kind 117 00:07:02,320 --> 00:07:06,240 of grouping similar concepts together, doing this sort of 118 00:07:06,400 --> 00:07:10,600 thematic clustering and what you could call topic modeling, 119 00:07:10,600 --> 00:07:12,800 right? Kind of this like organizing and 120 00:07:12,800 --> 00:07:15,760 structuring and finding similarities between things. 121 00:07:15,760 --> 00:07:19,120 That's really what generative AI is really, really good at. 122 00:07:19,440 --> 00:07:23,120 And so you might think, OK, like, yeah, that that that 123 00:07:23,120 --> 00:07:26,920 should lead to a really, really excellent literature review. 124 00:07:28,120 --> 00:07:31,280 Yeah, no, that is fair. I guess what is interesting as 125 00:07:31,280 --> 00:07:34,760 well though is that we are saying that, you know, I think 126 00:07:34,760 --> 00:07:38,120 it was just this week that there was this kind of like first 127 00:07:38,120 --> 00:07:40,760 paper in. I try to remember if it was in 128 00:07:40,760 --> 00:07:44,000 biology or if it was another science where basically it was 129 00:07:44,760 --> 00:07:48,600 AI who researched the literature review of it. 130 00:07:48,600 --> 00:07:51,480 Kind of like understanding the context, came up with a novel 131 00:07:51,760 --> 00:07:56,160 idea to test, like the hypothesis tested it wrote the 132 00:07:56,160 --> 00:07:59,160 paper and then said oh really for review. 133 00:08:00,720 --> 00:08:04,000 So you know. I don't like that at all. 134 00:08:04,600 --> 00:08:06,720 No, it feels wrong, right? It feels wrong. 135 00:08:06,720 --> 00:08:09,360 And I think I, I guess we talked about it before on the podcast 136 00:08:09,360 --> 00:08:13,560 that like these days, I think it's as important to understand, 137 00:08:13,960 --> 00:08:17,920 you know, what they I can do as, as when it's most useful and, 138 00:08:17,920 --> 00:08:21,720 and important to to use it versus when to have human in the 139 00:08:21,720 --> 00:08:25,760 loop and and so on. And, and when, you know, it's 140 00:08:25,760 --> 00:08:27,560 stronger or weaker and and so on. 141 00:08:27,920 --> 00:08:30,320 But it's still interesting that that happens. 142 00:08:30,360 --> 00:08:33,120 Yeah. I mean, just think of the poor 143 00:08:33,120 --> 00:08:35,799 editors at these journals who are volunteering their time, and 144 00:08:35,799 --> 00:08:41,039 now they're met with this deluge of new article submissions, 145 00:08:41,320 --> 00:08:46,080 these poor, poor academics. I know, I know. 146 00:08:46,080 --> 00:08:48,560 I've never felt more sympathy for academia. 147 00:08:49,520 --> 00:08:52,880 No, it is AI has both made a lot of people in academia their life 148 00:08:52,880 --> 00:08:56,240 easier in many ways in terms of what is helping people do, but 149 00:08:56,240 --> 00:09:00,360 also, you know, AI slop is, is a real thing. 150 00:09:00,360 --> 00:09:03,840 And, and I think we see also a lot of research papers are used 151 00:09:04,320 --> 00:09:09,280 kind of put out in numbers that we've never seen before that 152 00:09:09,600 --> 00:09:12,240 yeah, it's probably not so kind of thoughtfully put together 153 00:09:12,360 --> 00:09:16,560 because they're used incentivize to like generate paper ideas or 154 00:09:16,560 --> 00:09:18,960 paper published. And so, yeah. 155 00:09:19,080 --> 00:09:25,400 But it sounded like in your example that there was truly a 156 00:09:25,400 --> 00:09:30,000 new insight that was generated through this synthesis and 157 00:09:30,000 --> 00:09:33,640 hypothesis generate. So like that seems quite good. 158 00:09:34,040 --> 00:09:39,120 I think it that's kind of a step beyond what I was describing as 159 00:09:39,120 --> 00:09:43,120 the literature review capability of your sort of typical 160 00:09:43,120 --> 00:09:46,480 synthesis of what's out there. And it's sort of like predicting 161 00:09:46,800 --> 00:09:49,600 the the next step forward, identifying gaps, and then 162 00:09:49,600 --> 00:09:51,200 trying to answer those questions. 163 00:09:51,680 --> 00:09:53,640 You know, Google recently launched this, you know, Co 164 00:09:53,640 --> 00:09:57,600 scientist as a kind of initiative to support scientists 165 00:09:57,880 --> 00:10:00,640 who are doing good work and make their life easier with AI and 166 00:10:00,640 --> 00:10:03,680 seeing AI as a collaborator. And I probably would say that 167 00:10:03,680 --> 00:10:08,200 for many of those areas is where AI serves a lot of value as a 168 00:10:08,400 --> 00:10:12,840 collaborator maybe rather than the one you want to give full 169 00:10:12,880 --> 00:10:17,240 freedom to. I do think, you know, it's easy 170 00:10:17,240 --> 00:10:21,400 to say that it's impossible for AI to be original. 171 00:10:21,400 --> 00:10:24,200 You know, it's based on historical data exponency. 172 00:10:24,400 --> 00:10:28,480 But I do think that also brings a little bit of a maybe not a 173 00:10:28,480 --> 00:10:30,000 accurate view of what AI is today. 174 00:10:30,000 --> 00:10:35,440 I think it is moved beyond that. And I think it does kind of 175 00:10:35,440 --> 00:10:39,080 challenge some of those notions in that it is able to, obviously 176 00:10:39,080 --> 00:10:45,480 if given the autonomy and access to certain things, it can run 177 00:10:45,680 --> 00:10:47,000 experiments, it can do certain things. 178 00:10:47,280 --> 00:10:49,640 And you know, one of the things that a lot of labs are working 179 00:10:49,640 --> 00:10:55,920 on now is basically creating a environment that is set up 180 00:10:56,600 --> 00:10:59,080 artificially. So basically like taking some 181 00:10:59,080 --> 00:11:01,960 form of real environment in the world and then setting that 182 00:11:01,960 --> 00:11:05,400 environment in a kind of an artificial setting and is mainly 183 00:11:05,480 --> 00:11:07,960 done now for like robotic training. 184 00:11:07,960 --> 00:11:11,400 So imagine you have to train a robot to walk and move and do 185 00:11:11,400 --> 00:11:15,280 some things with kind of the the idea that they have to 186 00:11:15,280 --> 00:11:17,960 understand how the gravity of the world works and how to catch 187 00:11:17,960 --> 00:11:22,520 a ball and all of these things. But now I think NVIDIA has been 188 00:11:22,520 --> 00:11:25,480 quite proud to announce this, where they basically have been 189 00:11:25,480 --> 00:11:28,600 able to set up environments where these robots can be 190 00:11:28,600 --> 00:11:33,080 trained as kind of digital twins of themselves. 191 00:11:33,080 --> 00:11:34,880 Digital twins? Yeah, exactly. 192 00:11:34,920 --> 00:11:37,480 Yeah, you you have a virtual environment that's mapped to the 193 00:11:37,480 --> 00:11:39,280 real world. But yeah, you can't actually 194 00:11:39,280 --> 00:11:41,600 send your robots over throughout the entire world. 195 00:11:42,160 --> 00:11:43,160 Right. And I guess that that would be 196 00:11:43,160 --> 00:11:44,360 interesting to like what do you think? 197 00:11:44,360 --> 00:11:46,880 Like wouldn't it be possible if you have better and better 198 00:11:46,880 --> 00:11:49,120 environments like that with better and better replicas of 199 00:11:49,120 --> 00:11:52,760 digital twins, if you give access to those environments to 200 00:11:53,200 --> 00:11:56,360 AI, like it will be more able to run and test experiments and do 201 00:11:56,360 --> 00:12:00,560 various things and potentially come up with some findings. 202 00:12:01,200 --> 00:12:04,920 You could say that, well, it's still resulting as they're like, 203 00:12:04,920 --> 00:12:06,760 well, what do we know about the real world? 204 00:12:06,800 --> 00:12:10,480 Because it's been, you know, set up in a virtual lab setting, but 205 00:12:10,480 --> 00:12:14,880 it's still kind of at some point becomes uncomfortably similar to 206 00:12:14,920 --> 00:12:18,600 a physical lab setting where we still ask those questions. 207 00:12:18,600 --> 00:12:20,480 So I don't know what do you think? 208 00:12:23,080 --> 00:12:29,120 For me though, I wonder how much of it is coming to conclusions 209 00:12:29,120 --> 00:12:32,400 based on the new evidence that's been generated rather than 210 00:12:32,400 --> 00:12:36,360 really creating kind of a general mental model of the 211 00:12:36,360 --> 00:12:40,080 world and having this more generalized understanding. 212 00:12:40,760 --> 00:12:43,960 I'm not going to speak for robots, but I am thinking of 213 00:12:44,240 --> 00:12:48,840 maybe synthetic users and doing more behavioral design kind of 214 00:12:48,840 --> 00:12:51,680 research. And the reason I ask this or the 215 00:12:51,680 --> 00:12:54,440 reason that I suspect that there's a difference here is 216 00:12:54,440 --> 00:12:58,840 that I've at least in the examples that I've seen from, 217 00:12:59,200 --> 00:13:02,960 you know, AI generated lit reviews and so on, is that 218 00:13:02,960 --> 00:13:07,200 there's this real like dearth and critical thinking. 219 00:13:07,480 --> 00:13:09,960 There's the ability to really kind of extract from what's 220 00:13:09,960 --> 00:13:12,960 there. But moving beyond that, I'm 221 00:13:12,960 --> 00:13:18,120 pretty cynical about at least current ability to move beyond 222 00:13:18,120 --> 00:13:24,040 that in terms of this like novel insights and going to the like 223 00:13:24,080 --> 00:13:28,240 actually the next step that doesn't directly come out of 224 00:13:28,240 --> 00:13:32,440 what's already there. And this actually, as I was 225 00:13:32,440 --> 00:13:35,240 looking back at our LinkedIn polls, this came through this 226 00:13:35,240 --> 00:13:38,360 idea of like, you know, what's uniquely human. 227 00:13:38,360 --> 00:13:44,080 There were several categories that zero people chose as being 228 00:13:44,320 --> 00:13:46,280 well suited to be replaced by AI. 229 00:13:46,280 --> 00:13:49,120 And those things were like stakeholder alignment, 230 00:13:49,480 --> 00:13:52,920 connecting business to behaviour, product strategy, 231 00:13:53,080 --> 00:13:57,280 basically all of these like larger systems thinking and 232 00:13:57,280 --> 00:14:03,680 critical thinking skills that we think really are uniquely at 233 00:14:03,680 --> 00:14:06,040 least at least our current conceptualization of what's 234 00:14:06,040 --> 00:14:09,280 uniquely human. That is kind of where we seem to 235 00:14:09,280 --> 00:14:12,320 be drawing the line. Yeah, that is interesting. 236 00:14:12,960 --> 00:14:15,960 I think another word for that in some ways is, is maybe a taste. 237 00:14:15,960 --> 00:14:19,320 I think I hear that a lot in AI circles around kind of like, 238 00:14:19,920 --> 00:14:22,320 well, if AI can generate a lot of things, they can do a lot of 239 00:14:22,320 --> 00:14:26,480 things, that is great. But it still seems to be kind of 240 00:14:27,320 --> 00:14:29,320 often times if you have high expertise in something, you have 241 00:14:29,320 --> 00:14:33,800 also developed a high taste of what certain things, how they 242 00:14:33,800 --> 00:14:36,400 work and what works and what doesn't work in in making those 243 00:14:36,400 --> 00:14:41,920 kind of decisions is quite hard. And yeah, I do think that is 244 00:14:42,720 --> 00:14:45,120 maybe like similar to what you're saying, like it's kind of 245 00:14:45,120 --> 00:14:49,880 a lack of taste. Exactly, yeah. 246 00:14:49,960 --> 00:14:52,760 We know immediately when we see something that is just 247 00:14:52,760 --> 00:14:56,280 absolutely wrong, whereas the AI may not. 248 00:14:57,040 --> 00:15:03,320 OK, so let's talk about how AI lit reviews actually perform. 249 00:15:04,160 --> 00:15:07,440 Yeah. We have how people think they 250 00:15:07,440 --> 00:15:11,840 may do, but what do we know about how they really do? 251 00:15:12,520 --> 00:15:14,000 Yeah. So that's a really interesting 252 00:15:14,000 --> 00:15:15,600 thing. So basically I can give a little 253 00:15:15,600 --> 00:15:22,520 back story on this. So I noticed that there's been a 254 00:15:22,520 --> 00:15:24,720 lot of tools for a long time around, kind of like helping you 255 00:15:24,720 --> 00:15:26,040 research. And I've been using a lot of 256 00:15:26,040 --> 00:15:28,920 them myself in terms of, I think we talked about it many times 257 00:15:28,920 --> 00:15:33,360 around some of the early tools like illicit and, and some 258 00:15:33,360 --> 00:15:36,720 others, the research rabbit and so on that kind of like helped 259 00:15:36,720 --> 00:15:39,440 pull together scholar. Yeah, scholar. 260 00:15:39,760 --> 00:15:42,920 So various tools that were kind of like made a little easier to 261 00:15:42,920 --> 00:15:48,800 search for research papers and compile them in ways that would 262 00:15:48,800 --> 00:15:52,040 take much longer before. But then end of last year, I 263 00:15:52,040 --> 00:15:57,520 think a big thing that came out was that Gemini, the Google shot 264 00:15:57,520 --> 00:16:02,200 based kind of model launched this deep research feature where 265 00:16:02,200 --> 00:16:05,560 basically they allowed for you to ask question and then it 266 00:16:05,560 --> 00:16:09,840 would scour the web and the somebody was like published, 267 00:16:09,840 --> 00:16:12,720 researched. And it will basically provide 268 00:16:12,720 --> 00:16:15,920 you a report of like, here's what the science says. 269 00:16:15,960 --> 00:16:18,800 Here's the basically the answer to your question in the report 270 00:16:18,800 --> 00:16:20,760 form. And that was in some ways, I 271 00:16:20,760 --> 00:16:23,480 think for many, including myself, quite exciting. 272 00:16:23,480 --> 00:16:25,440 The idea of like, OK, I can put in a prompt. 273 00:16:25,680 --> 00:16:28,480 It would also like make sure that I'm clear. 274 00:16:28,480 --> 00:16:30,640 So would like follow up with some questions like, is this 275 00:16:30,640 --> 00:16:33,040 what you mean? Did you mean that in what 276 00:16:33,040 --> 00:16:35,040 context and so on. So the record found the prompt 277 00:16:35,040 --> 00:16:37,400 and then it will go to work and you would see it. 278 00:16:37,600 --> 00:16:41,560 And the thing that gave a lot of people kind of this excitement 279 00:16:41,560 --> 00:16:44,800 feeling is that you see the sources kind of like tick up. 280 00:16:44,800 --> 00:16:47,560 So it started with like five sources and a 10/20. 281 00:16:47,680 --> 00:16:50,640 And in some cases like 5075, it's like it's, you see, it's 282 00:16:50,640 --> 00:16:53,640 kind of going through and scouring the web and you're kind 283 00:16:53,640 --> 00:16:57,240 of like anticipating and waiting for what will the report say, 284 00:16:57,240 --> 00:16:59,480 Like what will it do? So I think that was quite 285 00:16:59,480 --> 00:17:03,360 exciting. But what I feel like is that 286 00:17:03,360 --> 00:17:06,079 we're living in a strange time where what used to take three 287 00:17:06,079 --> 00:17:08,680 years now take three months and what took three months take 3 288 00:17:08,680 --> 00:17:10,480 weeks. And kind of the example of that 289 00:17:10,640 --> 00:17:17,280 is, you know, earlier this year, that was like the only tool for 290 00:17:17,280 --> 00:17:18,920 that. There was no other tool that 291 00:17:18,920 --> 00:17:22,280 would like do that kind of call it agent based, sending out, 292 00:17:22,280 --> 00:17:27,040 searching the web and so on. Suddenly then Open AI launched a 293 00:17:27,040 --> 00:17:29,120 tool and they creatively named it the same thing. 294 00:17:29,200 --> 00:17:32,000 Deep research, yeah. Thanks. 295 00:17:34,120 --> 00:17:36,200 And and then, you know, a lot of others follow. 296 00:17:36,200 --> 00:17:41,280 There were Chinese DeepSeek, there was Kimi from China as 297 00:17:41,280 --> 00:17:43,280 well. And then some of the other like 298 00:17:43,280 --> 00:17:46,800 established tools did some versions of this as well. 299 00:17:46,800 --> 00:17:49,800 And then we had Storm from, you know, an open source tool from 300 00:17:50,240 --> 00:17:52,680 Stanford. And so a lot of these tools kind 301 00:17:52,680 --> 00:17:54,760 of like began to promise similar things. 302 00:17:55,000 --> 00:17:57,560 And I saw a lot of hype online. Like we saw like we've talked 303 00:17:57,560 --> 00:17:59,360 about it. We saw a lot of hype and a lot 304 00:17:59,360 --> 00:18:02,440 of people basically declaring that OK, you don't need a PhD 305 00:18:02,440 --> 00:18:06,720 research anymore, you know, these tools can do the job and 306 00:18:07,400 --> 00:18:08,480 and so on. In. 307 00:18:08,800 --> 00:18:11,360 Seconds in seconds. And so basically, I think what 308 00:18:11,360 --> 00:18:16,720 we were interested in is with nuance, we wanted to better 309 00:18:17,200 --> 00:18:20,360 communicate to our clients and the people we work with. 310 00:18:20,640 --> 00:18:24,480 But also I think we tried to give nuance like that's the 311 00:18:24,480 --> 00:18:25,960 whole goal, what we're trying to do. 312 00:18:26,400 --> 00:18:29,680 And so we also thought about as an opportunity to be like, hey, 313 00:18:30,480 --> 00:18:34,440 let's look at these tools, let's compare them and see how well do 314 00:18:34,440 --> 00:18:38,440 they actually do. Like what can we actually say 315 00:18:38,440 --> 00:18:42,800 right now about how good are these tools at doing literature 316 00:18:42,800 --> 00:18:44,560 reviews? So that's a little bit 317 00:18:44,560 --> 00:18:50,000 background on on this project. Oh, so, so, so exciting. 318 00:18:50,000 --> 00:18:53,760 I mean, I, I, this feels like it's so needed because all I see 319 00:18:53,760 --> 00:18:56,120 is the hype. I don't see a systematic 320 00:18:56,120 --> 00:19:00,240 analysis that anyone has done looking through these different 321 00:19:00,240 --> 00:19:01,720 tools. I just see like, Oh well, this 322 00:19:01,720 --> 00:19:03,120 person likes this one and this one. 323 00:19:03,800 --> 00:19:06,360 It's like I just want to get away from the hype and know 324 00:19:06,360 --> 00:19:10,200 what's real. Yeah, and honestly, I was kind 325 00:19:10,200 --> 00:19:13,040 of keen for that as well, because you're always in this 326 00:19:13,040 --> 00:19:14,720 current set of things. You're always feeling like 327 00:19:15,480 --> 00:19:16,680 there's some tool that you haven't tried. 328 00:19:16,680 --> 00:19:18,600 And there's always like, oh, what about this tool? 329 00:19:18,600 --> 00:19:21,880 What about that tool? And you see some some screenshot 330 00:19:21,880 --> 00:19:26,160 and you see some LinkedIn post and each tool is promised to be 331 00:19:26,160 --> 00:19:30,280 like, you know, the, the best thing ever and so on. 332 00:19:31,200 --> 00:19:34,720 So basically what we ended up doing is that we put that as a 333 00:19:34,720 --> 00:19:37,400 challenge, like could we kind of benchmark and test these tools? 334 00:19:37,760 --> 00:19:40,880 It was very much a group effort. You're involved. 335 00:19:40,880 --> 00:19:43,520 And basically I wanted new ones in some ways were involved. 336 00:19:44,000 --> 00:19:49,160 But special shout out to Josh Panjwani, who I think him and I 337 00:19:49,640 --> 00:19:50,960 were the ones kind of leading this. 338 00:19:51,040 --> 00:19:55,880 And what we started doing is basically looking at what are 339 00:19:56,960 --> 00:19:59,880 all tools that are promising this, basically trying to make 340 00:19:59,880 --> 00:20:02,560 sense of. How many are they? 341 00:20:02,560 --> 00:20:06,360 Like, how many are out there? Are they US based, China based? 342 00:20:06,360 --> 00:20:11,680 You know, what's the landscape? And so we did that and then we 343 00:20:11,680 --> 00:20:15,320 basically narrowed down to Indiana 8 tools that we all had 344 00:20:15,320 --> 00:20:19,600 access to that we could use. Some free, some quite expensive 345 00:20:19,680 --> 00:20:23,480 and some you probably heard a lot, maybe some you haven't 346 00:20:23,480 --> 00:20:28,720 heard about. And we then basically went 347 00:20:28,720 --> 00:20:33,720 through and wanted to test five different criteria for basically 348 00:20:33,720 --> 00:20:37,840 how well can it do at creating an AI generated literature 349 00:20:37,840 --> 00:20:41,160 review. So obvious things I wanted to 350 00:20:41,160 --> 00:20:43,960 see like what's the cost of using it? 351 00:20:44,680 --> 00:20:48,400 How much time does it take to to run this tool? 352 00:20:49,000 --> 00:20:51,600 Then importantly, how good is it? 353 00:20:53,480 --> 00:20:56,720 Is it, is it good both in terms of writing like doesn't provide 354 00:20:56,720 --> 00:21:00,320 a good written report or an output, but also citation 355 00:21:00,320 --> 00:21:01,920 quality? Like does it actually provide 356 00:21:02,040 --> 00:21:07,280 reliable good sided sources that is not hallucinated, but real 357 00:21:07,440 --> 00:21:12,640 and good from reputable journals and from also relevant sources? 358 00:21:13,680 --> 00:21:17,520 And then lastly, given that all of these were generated based on 359 00:21:17,520 --> 00:21:22,520 the prompts, we want to see how good are these models at 360 00:21:22,520 --> 00:21:25,840 following the prompts. So within the prompts, we 361 00:21:25,840 --> 00:21:29,080 basically specified certain things like the specific 362 00:21:29,080 --> 00:21:32,600 research question we wanted to have answered, but also the year 363 00:21:32,600 --> 00:21:35,320 in question that we wanted to get research papers from, the 364 00:21:35,320 --> 00:21:39,120 citation style, APA style citations that we wanted to have 365 00:21:39,600 --> 00:21:42,280 and and so on and so forth. So we wanted to see how good is 366 00:21:42,280 --> 00:21:44,600 it at following our instructions basically. 367 00:21:46,480 --> 00:21:50,200 And so, without giving too much away, how did they do? 368 00:21:50,280 --> 00:21:51,760 What did we find? Yeah. 369 00:21:51,760 --> 00:21:55,080 So we're currently compiling like the full report around this 370 00:21:55,080 --> 00:21:57,800 and it should be out very, very soon, probably very close to 371 00:21:58,320 --> 00:21:59,800 this podcast. But I can say quite a lot of 372 00:21:59,800 --> 00:22:01,720 interesting things already that we have seen. 373 00:22:03,520 --> 00:22:09,400 So where to start? I think 1 interesting thing is 374 00:22:09,400 --> 00:22:12,320 that a big worry for people is hallucinations. 375 00:22:13,640 --> 00:22:16,480 Basically that if you ask AI something it's going to give you 376 00:22:16,480 --> 00:22:17,920 an answer whether it's true or not. 377 00:22:19,600 --> 00:22:24,560 And interesting findings. There was no account of any 378 00:22:24,560 --> 00:22:28,680 hallucinations in any report. None of the tools hallucinated 379 00:22:28,680 --> 00:22:31,880 in any way. What is your definition of a 380 00:22:31,880 --> 00:22:34,320 hallucination? So it's making up basically a 381 00:22:34,320 --> 00:22:36,280 citation that is not real, so it's giving you a. 382 00:22:36,360 --> 00:22:39,080 Citation OK specifically for citations. 383 00:22:39,200 --> 00:22:39,880 Exactly. Yeah. 384 00:22:39,880 --> 00:22:41,320 So it's a good, good thing to clarify. 385 00:22:41,320 --> 00:22:44,400 So it might make the wrong conclusion some point, it might 386 00:22:44,760 --> 00:22:47,440 claim certain things, but it. Which I did see that. 387 00:22:47,480 --> 00:22:48,240 OK. Yeah. 388 00:22:48,680 --> 00:22:51,920 But it's not going to claim it based on a made-up citation. 389 00:22:53,280 --> 00:22:58,320 So again, that is not looking at their accuracy in terms of how 390 00:22:58,320 --> 00:23:00,560 well they're making inclusions and using citations. 391 00:23:00,560 --> 00:23:04,760 But it's like, at least we know citations that are used are real 392 00:23:04,760 --> 00:23:07,560 citations in all of the tools we we tested. 393 00:23:08,720 --> 00:23:13,160 And so that should at least give some level of like, hey, we 394 00:23:13,160 --> 00:23:15,720 don't have to worry about at least these tools just making up 395 00:23:15,720 --> 00:23:19,600 citations if we use them, which is kind of a big finding in some 396 00:23:19,600 --> 00:23:19,960 ways. Right? 397 00:23:20,760 --> 00:23:22,320 Yeah. For all the tools that you found 398 00:23:22,320 --> 00:23:24,640 this, all of them, Yeah, that's great. 399 00:23:24,840 --> 00:23:31,640 Yeah, then when we look at how much time it took let a span of 400 00:23:31,840 --> 00:23:35,080 some tools were able to generate a literature review in less than 401 00:23:35,080 --> 00:23:40,520 a minute on average and some it took around 35 plus minutes. 402 00:23:40,960 --> 00:23:45,040 So that was a span that they took, which is pretty big span, 403 00:23:45,040 --> 00:23:47,880 but also like 35 minutes is still relatively short. 404 00:23:48,200 --> 00:23:50,480 That's. Nothing. 405 00:23:50,920 --> 00:23:55,000 I mean this would take you so long otherwise, at least a day. 406 00:23:55,560 --> 00:24:00,520 Yeah, exactly. And I guess going into more 407 00:24:00,520 --> 00:24:04,920 details, I I'm kind of interested, what do you think 408 00:24:04,920 --> 00:24:09,480 did best shyness tools or American tools? 409 00:24:11,200 --> 00:24:16,800 Oh, come on. Best in what regard? 410 00:24:16,920 --> 00:24:20,240 In general, like in All in all categories like we, we can see a 411 00:24:20,240 --> 00:24:24,720 pretty overarching tendency around quality well. 412 00:24:24,760 --> 00:24:30,040 All right, despite my disappointment with the United 413 00:24:30,040 --> 00:24:35,160 States, I still have a America favoritism I would say. 414 00:24:35,160 --> 00:24:38,880 And yeah, I think probably the US based tools performs better. 415 00:24:41,000 --> 00:24:44,320 And yes, they did so. All right. 416 00:24:45,080 --> 00:24:48,840 In recitation quality, in terms of writing quality, in terms of 417 00:24:50,120 --> 00:24:55,760 prompt accuracy and sensitivity, tools like Open AI, Deep 418 00:24:55,760 --> 00:25:00,880 Research, Geminis and the likes performed better than the likes 419 00:25:00,880 --> 00:25:05,040 of Kimi or DeepSeek on those things. 420 00:25:05,320 --> 00:25:08,600 What I would say that both Kimi and DeepSeek, which was the two 421 00:25:08,600 --> 00:25:12,920 Chinese models that we we looked at, they were extremely fast. 422 00:25:13,720 --> 00:25:20,640 So they were the ones that were basically quickest at generating 423 00:25:20,640 --> 00:25:22,720 things and both of them are free. 424 00:25:22,920 --> 00:25:27,800 And so if you wanted to use, you know, either deep research from 425 00:25:27,800 --> 00:25:32,840 Open AI or from Gemini, it would cost you at least $20.00 per 426 00:25:32,840 --> 00:25:35,120 month or something, which is still run to the sheep, right. 427 00:25:35,400 --> 00:25:40,840 But Kimi and Dipstick are both free, but with a caveat that you 428 00:25:40,840 --> 00:25:43,560 might be giving away data to the Chinese government. 429 00:25:43,880 --> 00:25:45,840 So there's there's always a cost, right? 430 00:25:45,840 --> 00:25:47,320 Nothing is completely free. Yeah. 431 00:25:49,640 --> 00:25:51,120 In all cases. Yeah. 432 00:25:53,200 --> 00:25:55,440 I mean, she's like, I guess to hear from you, what would you 433 00:25:55,440 --> 00:25:57,600 want to know because you don't know the results. 434 00:25:57,680 --> 00:26:00,200 So what is you're interested in terms of finding out? 435 00:26:01,560 --> 00:26:06,000 So I'll, I'll just for context, say that I rated many of these 436 00:26:06,000 --> 00:26:11,080 generated lit reviews and was overall pretty disappointed. 437 00:26:11,080 --> 00:26:17,560 There was one that I read that I would actually use a single 11 438 00:26:17,560 --> 00:26:20,080 output. Everything else I would like, 439 00:26:20,240 --> 00:26:24,040 you know, even if I rated it, I don't know, six or seven out of 440 00:26:24,040 --> 00:26:28,000 10, I wouldn't use it. I would not it, it was of no 441 00:26:28,000 --> 00:26:30,680 value to me. Even if I, you know, could would 442 00:26:30,680 --> 00:26:34,080 say like, oh, it's organized OK, Or you know, the citations are, 443 00:26:34,360 --> 00:26:37,720 you know, relevant and has appropriate structure, etcetera. 444 00:26:38,560 --> 00:26:45,560 So what I want to know what's the one? 445 00:26:46,640 --> 00:26:49,520 What's the best? Tell me the one I should use? 446 00:26:50,080 --> 00:26:51,760 That's my big question. I know. 447 00:26:52,000 --> 00:26:55,440 Well, interestingly to your point, what we saw is basically 448 00:26:55,440 --> 00:26:59,400 that across these three quality categories, so citation quality, 449 00:27:00,040 --> 00:27:05,000 writing quality and prompt sensitivity or accuracy, none of 450 00:27:05,000 --> 00:27:08,880 these performed best across the board. 451 00:27:09,520 --> 00:27:14,960 So there were some that were better in one or two categories, 452 00:27:15,240 --> 00:27:18,160 but then really bad in 1/3, for example. 453 00:27:18,560 --> 00:27:21,960 And so there's not one tool that is better than all of them 454 00:27:22,080 --> 00:27:26,720 across the board. So as an example, we can say 455 00:27:26,720 --> 00:27:30,680 that like when it comes to citation quality and prompt 456 00:27:30,680 --> 00:27:37,520 accuracy, elicits and maybe Site AI were the ones that performed 457 00:27:37,520 --> 00:27:40,360 best there, but they're also some of the ones that performed 458 00:27:40,360 --> 00:27:41,880 the worst when it came to writing. 459 00:27:43,240 --> 00:27:46,520 Yeah. So that is currently where we're 460 00:27:46,520 --> 00:27:49,880 at right now, where we're getting some tools that are 461 00:27:50,400 --> 00:27:53,320 better and better at specifically helping us make 462 00:27:53,320 --> 00:27:57,640 sense of the literature, but they might not be the ones that 463 00:27:57,640 --> 00:28:00,480 we want to rely on for also compiling the findings and 464 00:28:00,480 --> 00:28:01,960 writing and making sense of them. 465 00:28:03,000 --> 00:28:10,120 So I think in some ways this reinforces my kind of belief 466 00:28:10,120 --> 00:28:13,120 that I had before this, which basically these tools can be 467 00:28:13,120 --> 00:28:17,200 extremely valuable today. And you will almost be fooled to 468 00:28:17,200 --> 00:28:21,880 completely disregard the tools that are out there, but you 469 00:28:21,880 --> 00:28:24,280 would also be fooled to completely rely on them for 470 00:28:24,280 --> 00:28:29,000 everything. And keeping human in the loop is 471 00:28:29,000 --> 00:28:31,520 basically still a really important factor. 472 00:28:33,800 --> 00:28:39,800 Yeah, I think my main take away from this is that take any of 473 00:28:39,800 --> 00:28:42,960 the outputs, it's like 110th of a literature review. 474 00:28:42,960 --> 00:28:46,640 It's not, you know, there are major gaps, complete 475 00:28:46,840 --> 00:28:51,480 misinterpretations of the data sometimes, which could be really 476 00:28:51,480 --> 00:28:55,480 misleading. So there was one case in one of 477 00:28:55,480 --> 00:28:59,080 the better literature reviews that I reviewed, unfortunately, 478 00:28:59,080 --> 00:29:06,000 where correlation was completely confused with causation. 479 00:29:06,000 --> 00:29:10,640 And it described the impact of financial literacy on financial 480 00:29:10,640 --> 00:29:13,760 well-being. And came to the conclusion that 481 00:29:13,760 --> 00:29:16,960 based on this association between, you know, people having 482 00:29:16,960 --> 00:29:20,440 higher financial literacy and better outcomes in financial 483 00:29:20,440 --> 00:29:23,720 well-being, that like this is an important area to focus 484 00:29:23,720 --> 00:29:26,280 interventions on. Well, like actually, we know 485 00:29:26,280 --> 00:29:30,680 from so, so much research that financial literacy interventions 486 00:29:30,680 --> 00:29:33,840 actually have so negligible, like a fraction of a person 487 00:29:33,920 --> 00:29:38,880 percentage of an impact on one's actual financial decisions and 488 00:29:38,880 --> 00:29:41,840 their well-being. So, you know, without having the 489 00:29:41,840 --> 00:29:46,080 expertise to interpret the results that you're getting, I 490 00:29:46,080 --> 00:29:48,920 think it's really kind of dangerous in leading people 491 00:29:48,920 --> 00:29:54,760 towards the wrong conclusions. I agree and I would say we put a 492 00:29:54,760 --> 00:29:57,240 lot of thought into that. The prompt we would give the 493 00:29:57,240 --> 00:30:01,280 models, and we purposely gave them probably a way beyond 494 00:30:01,280 --> 00:30:03,760 average quality of prompt in yourself details this 495 00:30:03,760 --> 00:30:07,440 instruction clarity around what we wanted and we would expect 496 00:30:07,440 --> 00:30:11,160 that in the wild most people use these tools with much worse 497 00:30:11,160 --> 00:30:15,200 prompts. Yeah, you're really setting it 498 00:30:15,200 --> 00:30:18,520 up for success. Yeah, and and even then, as you 499 00:30:18,520 --> 00:30:23,160 say, this happens, and I like to liken this to them, look back at 500 00:30:23,160 --> 00:30:25,320 like, what would humans do in this context? 501 00:30:25,560 --> 00:30:28,680 What could we expect of humans? And like kind of what we're 502 00:30:28,680 --> 00:30:31,440 seeing now is what we expect from like maybe intern 503 00:30:31,440 --> 00:30:35,360 researchers where they don't really understand the thing 504 00:30:35,360 --> 00:30:38,360 they're being asked to do. And so they kind of make sense 505 00:30:38,360 --> 00:30:40,520 of it. They go out and look for 506 00:30:40,520 --> 00:30:42,120 findings to put together something. 507 00:30:42,120 --> 00:30:47,360 But they don't have enough kind of taste or expertise or context 508 00:30:47,720 --> 00:30:51,640 to know actually what matters and, and what to neglect and 509 00:30:51,640 --> 00:30:55,400 just put everything together and put it into report and give it 510 00:30:55,400 --> 00:30:56,680 to be like here, here, here's what he asked for. 511 00:30:56,680 --> 00:30:59,640 And you're like, well, it's not really what I asked for. 512 00:30:59,640 --> 00:31:03,160 Like this is kind of a mismatch of a bunch of different things 513 00:31:03,160 --> 00:31:05,440 where kind of what I asked for, but not really. 514 00:31:07,480 --> 00:31:10,040 So that is kind of what I would say as a tendency for a lot of 515 00:31:10,040 --> 00:31:13,560 these, and especially the ones, you know, like Open AI, Gemini, 516 00:31:14,480 --> 00:31:18,080 Both of them are also importantly relying on what they 517 00:31:18,080 --> 00:31:22,240 can find online, not what they can find in research papers, but 518 00:31:22,240 --> 00:31:25,400 what they can find in like research abstracts online. 519 00:31:26,880 --> 00:31:29,520 And so you know that whatever they tell you, they haven't read 520 00:31:29,520 --> 00:31:32,200 the methodology, they haven't read the sign of the study, they 521 00:31:32,480 --> 00:31:35,040 haven't read any of that. They've only read the abstract 522 00:31:35,040 --> 00:31:38,400 of the study. That's a huge problem in itself. 523 00:31:38,480 --> 00:31:42,360 And then if you take that a step further, they've not only not 524 00:31:42,640 --> 00:31:46,360 accessed that information, they've certainly not critically 525 00:31:46,360 --> 00:31:49,120 examined it. Like, if you look at, like, 526 00:31:49,200 --> 00:31:53,040 every single behavioral science lab or any kind of science lab 527 00:31:53,040 --> 00:31:56,960 itself, like a very common exercise is some sort of journal 528 00:31:56,960 --> 00:32:01,040 club where you will read an article and then assess, sit and 529 00:32:01,040 --> 00:32:04,600 say, like, OK, like, did you know, what is the validity with 530 00:32:04,600 --> 00:32:07,320 the internal and external validity of this paper? 531 00:32:07,320 --> 00:32:10,280 And you analyze its methods. You say, what are the 532 00:32:10,280 --> 00:32:12,760 shortcomings? What's the sample size? 533 00:32:12,760 --> 00:32:17,640 You know, on and on and on. And every single paper that I've 534 00:32:17,640 --> 00:32:21,280 ever analyzed has something wrong with it. 535 00:32:21,280 --> 00:32:24,080 There's no perfect research study out there. 536 00:32:24,080 --> 00:32:27,440 And all of these literature review tools are completely 537 00:32:27,440 --> 00:32:31,480 glossing over that entirely. There's no critical analysis at 538 00:32:31,480 --> 00:32:32,960 all. It's just we're taking at face 539 00:32:32,960 --> 00:32:35,880 value whatever the abstract says that you found what you said you 540 00:32:35,880 --> 00:32:38,360 found. And so often that's not the 541 00:32:38,360 --> 00:32:40,560 case. Yeah. 542 00:32:40,560 --> 00:32:43,000 And I think that's something that people can relate to in 543 00:32:43,000 --> 00:32:48,880 that lot of their models are trained to be kind of trust what 544 00:32:48,880 --> 00:32:51,480 we say to them. So like if we say that the Earth 545 00:32:51,480 --> 00:32:54,520 is flat, a lot of models traditionally would say, oh, 546 00:32:54,520 --> 00:32:56,240 yeah, sorry, I was wrong, it is flat. 547 00:32:57,080 --> 00:32:58,920 Now they are better. Like a lot of the models are a 548 00:32:58,920 --> 00:33:01,680 little bit more willing to push back, especially the reasoning 549 00:33:01,680 --> 00:33:04,560 models, they're a little bit more able to say, hey, actually, 550 00:33:04,560 --> 00:33:06,440 I don't know if are you joking with me? 551 00:33:06,440 --> 00:33:09,960 Like they would actually kind of understand that it's probably 552 00:33:09,960 --> 00:33:12,720 not true if you say something completely out there. 553 00:33:13,120 --> 00:33:16,240 But again, a lot of these things are possible. 554 00:33:16,240 --> 00:33:18,240 If you don't have a critical eye, you can use gloss over 555 00:33:18,360 --> 00:33:22,520 whatever potential aspects that could be wrong and so on. 556 00:33:22,840 --> 00:33:26,640 So so I agree with you. Like that's a huge issue still 557 00:33:27,240 --> 00:33:29,800 with any of this is that they used take everything at face 558 00:33:29,800 --> 00:33:34,200 value and basically compile everything as it was the same 559 00:33:34,200 --> 00:33:37,840 quality and and so on. And even if we'd instructed to 560 00:33:37,840 --> 00:33:41,520 say like, Hey, weight the ones with higher sample size more 561 00:33:41,520 --> 00:33:44,200 than the ones who don't, that's still not something that usually 562 00:33:44,200 --> 00:33:46,520 this ones are really good at doing. 563 00:33:46,880 --> 00:33:50,400 And that's also like, that's one proxy, but it's, it's like what 564 00:33:50,400 --> 00:33:52,720 makes for good research papers is really massive to determine, 565 00:33:52,720 --> 00:33:54,480 right? Right. 566 00:33:54,600 --> 00:33:55,720 Yeah, yeah. What's the, what's the 567 00:33:55,720 --> 00:33:58,200 experimental design like? Is it an RCT? 568 00:33:58,200 --> 00:34:00,680 What? What were the actual conditions 569 00:34:00,680 --> 00:34:01,800 that were included? How? 570 00:34:01,800 --> 00:34:04,800 What comparisons were made? Yeah, yeah, there's there. 571 00:34:04,800 --> 00:34:08,159 It's hard to find rules of thumb for the quality of a paper. 572 00:34:08,679 --> 00:34:10,400 Yeah. Can I say something fun actually 573 00:34:10,400 --> 00:34:11,880 around this? Yeah. 574 00:34:12,000 --> 00:34:17,080 So one of the two kind of prompts that we gave 11 prompt 575 00:34:17,080 --> 00:34:19,600 was around, as you said, like financial well-being and it's 576 00:34:19,600 --> 00:34:23,080 basically around UK population and younger adults. 577 00:34:24,040 --> 00:34:26,280 So that was the the thing. And it's really funny how 578 00:34:26,280 --> 00:34:29,400 sometimes it took some of those things, like it took either 579 00:34:29,400 --> 00:34:34,199 younger adults or UK and it just came up with completely like 580 00:34:34,320 --> 00:34:36,560 studies that had to do with something with well-being. 581 00:34:36,560 --> 00:34:41,920 Like, like how likely are, you know, people that are 18 to go 582 00:34:41,920 --> 00:34:44,960 to the gym? And then based on based on this 583 00:34:44,960 --> 00:34:48,600 finding, we can say that like. No, yeah. 584 00:34:48,639 --> 00:34:51,120 Yeah. So it wasn't always great at 585 00:34:51,120 --> 00:34:54,360 some of those things. Yeah, and I noticed a lot of 586 00:34:54,520 --> 00:34:59,800 COVID era research was included across many of the reviews and 587 00:34:59,800 --> 00:35:04,320 and I just like that was such a unique time like looking it just 588 00:35:04,320 --> 00:35:07,320 feels a little non generalizable. 589 00:35:08,280 --> 00:35:09,880 Yeah, yeah. No, I agree. 590 00:35:10,440 --> 00:35:15,480 So maybe obviously we'll release all of this very soon and many 591 00:35:15,480 --> 00:35:17,320 more things to talk about. But I think 1 interesting thing 592 00:35:17,320 --> 00:35:23,320 here that is somewhat sad I guess is that we don't know 593 00:35:23,320 --> 00:35:25,760 which model was the best, but we can definitely say which is the 594 00:35:25,760 --> 00:35:28,040 worst. Oh good. 595 00:35:29,040 --> 00:35:32,760 Tell me. It is the storm model by 596 00:35:34,320 --> 00:35:38,120 basically it's a Stanford lab and bummer, it's a bummer. 597 00:35:39,160 --> 00:35:40,840 I was looking forward to trying that one out. 598 00:35:41,120 --> 00:35:44,240 Yeah, yeah, I know once we tested it a little bit deeper 599 00:35:44,240 --> 00:35:46,720 and really evaluated how it performed across the various 600 00:35:46,720 --> 00:35:51,680 settings and and prompts, basically it wasn't able to 601 00:35:52,280 --> 00:35:54,920 perform very well across any of these things. 602 00:35:54,920 --> 00:35:58,160 And it was, yeah, by far the worst of all of the ones. 603 00:35:58,480 --> 00:36:02,120 So the only one that's kind of like open source and free and 604 00:36:02,120 --> 00:36:04,800 American in this case, like it's both American open source and 605 00:36:04,800 --> 00:36:08,320 and free. That was the worst one and by 606 00:36:08,320 --> 00:36:11,280 margin so that like I would almost now having went through 607 00:36:11,280 --> 00:36:13,440 this, I would recommend people not to use coast arm. 608 00:36:13,440 --> 00:36:16,080 That's probably the only one I would like definitely say don't 609 00:36:16,080 --> 00:36:18,600 use it. It's just going to be mainly 610 00:36:19,360 --> 00:36:23,720 going through blog posts rather than research papers and oh. 611 00:36:24,560 --> 00:36:28,120 Wow, that's shocking. Yeah, that is shocking. 612 00:36:28,120 --> 00:36:30,440 Yeah, bummer. It is a bummer. 613 00:36:30,440 --> 00:36:33,520 And they kind of also asked with many of these. 614 00:36:33,560 --> 00:36:36,960 The tricky thing that I think I'm most worried about still is 615 00:36:36,960 --> 00:36:40,880 that the people who are quote, UN quote buying literature 616 00:36:40,880 --> 00:36:44,720 reviews are often the ones that are not like really good at 617 00:36:44,720 --> 00:36:46,480 doing it themselves. Like they don't really know what 618 00:36:46,480 --> 00:36:50,720 a good one looks like. And so they could be like a lot 619 00:36:50,720 --> 00:36:54,600 of people that would say like, oh, actually, I used to hire a 620 00:36:54,600 --> 00:36:56,600 researcher to put together my literature reviews. 621 00:36:56,800 --> 00:36:58,080 Now I'm going to just do them myself. 622 00:36:58,080 --> 00:37:01,920 I heard this tool Storm was out there and I'm just going to do 623 00:37:01,920 --> 00:37:05,240 it with storm and then storm generous report. 624 00:37:05,240 --> 00:37:07,920 It has a lot of citations, but you have to like click on the 625 00:37:07,920 --> 00:37:12,440 citations to see, oh, it's a blog posts, It's a this, it's a 626 00:37:12,440 --> 00:37:14,840 that. So if you squint it looks really 627 00:37:14,880 --> 00:37:16,600 legit because it has cited like 35. 628 00:37:16,600 --> 00:37:20,760 Sources. Wow, that's a shame. 629 00:37:21,320 --> 00:37:26,600 So I guess this in some ways marks a first initiative that 630 00:37:26,600 --> 00:37:29,600 we're going to publish around something fun that we're doing 631 00:37:29,600 --> 00:37:34,360 with Nuance, which is basically we've been working with AI for a 632 00:37:34,360 --> 00:37:37,280 long time now. You know, even before Nuance, 633 00:37:37,280 --> 00:37:40,320 both you and I were involved in a lot of AI related initiatives 634 00:37:40,320 --> 00:37:43,480 and so on. But we're now going to 635 00:37:43,480 --> 00:37:46,960 specifically because we feel like as an example of this, 636 00:37:47,200 --> 00:37:50,200 there's a need for a little bit of thought leadership and help 637 00:37:50,200 --> 00:37:52,120 people navigating the state of AI. 638 00:37:52,920 --> 00:37:56,720 We're going to actually have AAI Lab dedicated to specifically 639 00:37:56,720 --> 00:38:00,160 exploring these kind of questions and hopefully sharing 640 00:38:00,160 --> 00:38:03,160 some important findings around, you know, what works, what 641 00:38:03,160 --> 00:38:08,280 doesn't work, and hopefully as much of what tools to use and 642 00:38:08,280 --> 00:38:11,240 some of those things, but also like really helping make sense 643 00:38:11,240 --> 00:38:15,720 of when to use some of these things because it's becoming 644 00:38:15,720 --> 00:38:19,720 more of a not so much if AI can be useful, but more how it can 645 00:38:19,720 --> 00:38:21,320 be useful and when you should use it. 646 00:38:21,440 --> 00:38:25,240 And so a lot of our initiatives will be looking at stuff like 647 00:38:25,240 --> 00:38:29,240 this, looking at kind of how we can automate aspects of our 648 00:38:29,240 --> 00:38:32,080 work, but also looking at AI adoption. 649 00:38:32,080 --> 00:38:34,920 How can we do that? Well, because that's not a 650 00:38:34,920 --> 00:38:36,880 problem at all, right? Everyone is doing the adoption. 651 00:38:37,040 --> 00:38:43,520 Perfectly, yeah. Not so that is also something 652 00:38:43,520 --> 00:38:45,840 we're going to be sharing more on and we're doing already some 653 00:38:45,840 --> 00:38:51,080 work on that with clients and yeah, and then many other 654 00:38:51,080 --> 00:38:53,960 interesting things. And actually I wanted to shout 655 00:38:53,960 --> 00:38:55,960 out something that you recently posted. 656 00:38:56,320 --> 00:39:00,360 It's maybe not exactly related to our AI lab, but it is a 657 00:39:00,360 --> 00:39:06,440 really fun AI example and and also relates to something that 658 00:39:06,440 --> 00:39:09,520 is also something new for us, which is our behavioral redesign 659 00:39:09,520 --> 00:39:12,480 series. And so kind of reimagining 660 00:39:12,840 --> 00:39:17,920 products if they had a little more of a payroll spice for good 661 00:39:17,920 --> 00:39:22,160 or bad, I guess. Yeah, some are spicier than 662 00:39:22,160 --> 00:39:24,080 others. Yeah. 663 00:39:24,320 --> 00:39:28,840 And and you shared one that I really loved this week where I 664 00:39:29,120 --> 00:39:32,000 think it was a little bit of a personal story as well behind 665 00:39:32,000 --> 00:39:33,680 that. Could you share the Peloton 666 00:39:34,440 --> 00:39:37,480 example that you redesigned? Oh, yeah. 667 00:39:37,480 --> 00:39:41,240 I mean, so this is something I think about a lot is the sort of 668 00:39:41,640 --> 00:39:47,560 flexibility and forgiveness of the various apps out there that 669 00:39:47,560 --> 00:39:49,960 are trying to help you reach some goal that you have. 670 00:39:50,080 --> 00:39:52,800 And, you know, you're committed to reaching that goal. 671 00:39:52,800 --> 00:39:56,960 But also you're a human and life gets in the way and you have 672 00:39:56,960 --> 00:40:00,200 hard days, you have easier days. And on those hard days. 673 00:40:00,640 --> 00:40:05,000 I have found that unfortunately basically 0 apps have any 674 00:40:05,000 --> 00:40:09,080 tolerance or any acknowledgement that like you had a hard day, 675 00:40:09,080 --> 00:40:10,600 you didn't get any sleep last night. 676 00:40:10,600 --> 00:40:13,040 Let's maybe readjust your plan. Yeah. 677 00:40:13,040 --> 00:40:15,520 And I think like my favorite example of that is, you know, 678 00:40:15,520 --> 00:40:18,280 Fitbit has been around for a long time now and it's been like 679 00:40:18,880 --> 00:40:22,760 kind of getting people to get their daily step counts in and 680 00:40:22,760 --> 00:40:25,080 famously gone. A lot of people crazy where they 681 00:40:25,080 --> 00:40:28,320 like, they realize before going to bed they haven't reached 682 00:40:28,320 --> 00:40:30,160 their step count and so they start running in. 683 00:40:30,280 --> 00:40:33,600 David Sedaris in particular. Exactly. 684 00:40:34,480 --> 00:40:37,440 Yeah, But that comes to mind as an example of like, we've been 685 00:40:37,720 --> 00:40:39,280 faced with this for a long time now. 686 00:40:39,280 --> 00:40:42,680 And if it's like still, that is still kind of a burden given to 687 00:40:42,680 --> 00:40:46,520 a lot of users where it's just like, you know, you have a goal 688 00:40:46,520 --> 00:40:49,760 do it doesn't matter if context shifts, you're expected to do 689 00:40:49,760 --> 00:40:53,920 the same thing every day. Yeah, it's always a nudge to get 690 00:40:53,920 --> 00:40:58,720 you to achieve your goal, not to be kind to yourself and maybe 691 00:40:58,720 --> 00:41:01,160 make you more likely to reach your long term goal. 692 00:41:01,640 --> 00:41:05,600 And so the, the idea with this Peloton design was, I mean, 693 00:41:05,640 --> 00:41:11,400 taking in your sleep data and really like presenting that to 694 00:41:11,400 --> 00:41:14,760 the user, acknowledging that you had a really hard night last 695 00:41:14,760 --> 00:41:16,520 night. I can see that, you know, you 696 00:41:16,520 --> 00:41:19,080 didn't get very much sleep. You were, you know, up three 697 00:41:19,080 --> 00:41:22,240 times at night feeding the baby. And, you know, maybe maybe 698 00:41:22,240 --> 00:41:24,920 Peloton doesn't know that you were feeding the baby, But I, I 699 00:41:24,920 --> 00:41:27,440 can add that personal context here. 700 00:41:27,840 --> 00:41:29,840 And then saying, like, let's adjust your plan. 701 00:41:29,840 --> 00:41:32,440 I see that you were going to do a really intense hit workout 702 00:41:32,440 --> 00:41:35,080 this morning. Maybe we should swap a yoga 703 00:41:35,240 --> 00:41:37,520 instead. And that feels like something 704 00:41:37,520 --> 00:41:41,480 that you could actually accomplish, which maybe helps 705 00:41:41,480 --> 00:41:45,880 you stick to your goal in general of regularly exercising, 706 00:41:45,880 --> 00:41:49,800 but isn't so demotivate like there's just zero chance that 707 00:41:49,800 --> 00:41:51,720 you're going to do a hip workout in that state. 708 00:41:51,920 --> 00:41:55,360 So, you know, some sort of compromise and actively 709 00:41:55,360 --> 00:41:57,800 suggesting that as a kindness to the user. 710 00:41:58,440 --> 00:41:59,960 Yeah. And I love that because I think 711 00:42:00,200 --> 00:42:02,960 we've often talked about the city of well, it's great to have 712 00:42:02,960 --> 00:42:07,200 an if time plan around kind of like, you know, if I get tired 713 00:42:07,200 --> 00:42:09,840 and I will do this thing instead or if I get sick, I will do that 714 00:42:09,840 --> 00:42:12,800 thing instead. But I think it's really hard for 715 00:42:12,800 --> 00:42:14,840 people, Like it's hard for me, it's hard for anyone to really 716 00:42:14,840 --> 00:42:18,840 have those backup if then plans always available or always 717 00:42:18,840 --> 00:42:22,280 thought out in advance. And so why shouldn't our 718 00:42:22,280 --> 00:42:24,920 technology help us? Yeah. 719 00:42:25,040 --> 00:42:28,520 And I feel like what oftentimes I see, what we see probably a 720 00:42:28,520 --> 00:42:32,920 lot of people do is they're going to kind of a fuck it mode 721 00:42:32,920 --> 00:42:37,600 instead where they say this heat workout, I don't have energy for 722 00:42:37,600 --> 00:42:40,640 that fuck it. I'm just going to then do the 723 00:42:40,640 --> 00:42:45,160 complete opposite, you know? And so, yeah, I think giving 724 00:42:45,160 --> 00:42:49,400 people this kind of like, you know, if then plan provided for 725 00:42:49,400 --> 00:42:53,000 them and also like really making it easy to do something instead 726 00:42:53,000 --> 00:42:57,120 of nothing. Yeah, I think it's key that this 727 00:42:57,120 --> 00:43:01,560 prompt was to substitute your HIT workout with a yoga. 728 00:43:01,560 --> 00:43:05,520 It wasn't like, you know, forget about your goal altogether. 729 00:43:06,240 --> 00:43:10,440 What, you know, while there is a time and place for that as well. 730 00:43:10,440 --> 00:43:14,760 And you can sort of see that in Marissa Sharif's research with 731 00:43:14,760 --> 00:43:18,160 emergency reserves. Another way to sort of continue 732 00:43:18,360 --> 00:43:22,160 towards towards your long term goal is to just do something so 733 00:43:22,160 --> 00:43:25,440 you like feel like you've progressed and you're sort of 734 00:43:26,040 --> 00:43:28,040 psychologically consistent with your goal. 735 00:43:28,040 --> 00:43:30,840 Even if you didn't do the hard thing, you at least did 736 00:43:30,840 --> 00:43:35,480 something that's consistent. Yeah, I, I, I think this is such 737 00:43:35,480 --> 00:43:38,960 a obvious thing that needs to be in the world. 738 00:43:40,200 --> 00:43:43,760 I mean, to know, you know, how this has been for you on a 739 00:43:43,760 --> 00:43:47,560 personal level, because I know, you know, you've recently had a 740 00:43:47,560 --> 00:43:50,000 baby and you I know you have a Peloton. 741 00:43:50,480 --> 00:43:53,440 So how have you dealt with this without Peloton supporting you? 742 00:43:53,440 --> 00:43:59,840 Like how has this been? Well, yeah. 743 00:43:59,840 --> 00:44:04,240 Well, so maybe I should say the reason that I created this 744 00:44:04,240 --> 00:44:07,280 design is because I think there's a very real need for it. 745 00:44:07,280 --> 00:44:13,800 I have gone the fuck it route and just not done anything but 746 00:44:13,800 --> 00:44:16,400 this is my cry for help. Like I want to get back into it 747 00:44:16,400 --> 00:44:20,520 but it's really really hard. Yeah, yeah. 748 00:44:20,680 --> 00:44:22,120 I don't have a peloton, by the way. 749 00:44:22,560 --> 00:44:23,360 Yeah. That's fine. 750 00:44:24,040 --> 00:44:28,520 So I guess hopefully we'll see some changes around this and 751 00:44:29,280 --> 00:44:35,640 certainly a worthwhile nudge to the industry to reimagine how 752 00:44:35,640 --> 00:44:37,080 this user experience could look like. 753 00:44:37,480 --> 00:44:41,200 And yeah, I think this is one of many redesigns, I think. 754 00:44:41,200 --> 00:44:42,360 How many have we published so far? 755 00:44:44,000 --> 00:44:49,440 Oh my gosh, at least maybe 10:15 so far, and we've got a whole 756 00:44:49,440 --> 00:44:52,120 lot on the way. Yeah, yeah. 757 00:44:52,120 --> 00:44:55,880 So check out I think in the links and then also we we can 758 00:44:55,880 --> 00:44:59,920 see on the Nuance website we list all of them, but also, you 759 00:44:59,920 --> 00:45:02,080 know, on LinkedIn we share them as well. 760 00:45:02,920 --> 00:45:08,000 So I guess I wanted to share a bit what was going on, some 761 00:45:08,080 --> 00:45:10,000 interesting findings, some interesting stuff. 762 00:45:10,760 --> 00:45:16,200 Yeah, lots of fun, exciting initiatives launching with the 763 00:45:16,320 --> 00:45:19,040 AI Lab and Behavioral Redesign series. 764 00:45:19,040 --> 00:45:23,560 So yeah, just stay tuned. And we also have a bunch of 765 00:45:23,560 --> 00:45:26,920 really exciting guests coming up on the podcast soon. 766 00:45:27,040 --> 00:45:29,320 That is true and I'm really looking forward to it. 767 00:45:29,320 --> 00:45:33,520 And I think, honestly, what I felt is that this podcast has 768 00:45:33,520 --> 00:45:38,760 been really a therapy session every time we've recorded an 769 00:45:38,760 --> 00:45:42,160 episode, especially also because we've had so many, I think, 770 00:45:42,160 --> 00:45:45,120 incredible guests. And, you know, I've tried to 771 00:45:45,120 --> 00:45:48,040 make sense personally, you know, what is the state of AI? 772 00:45:48,040 --> 00:45:49,400 What is the state of behavioral science? 773 00:45:49,800 --> 00:45:52,080 And it's been such a gift to, like, having recently talked to, 774 00:45:52,080 --> 00:45:57,160 like, Sandra Matz, Susan Murphy, like all these people who are 775 00:45:57,160 --> 00:46:01,080 really both pioneers but also leaders in a lot of aspects that 776 00:46:01,080 --> 00:46:06,760 we're kind of thinking about. And yeah, as you say, some more 777 00:46:07,560 --> 00:46:09,400 similar conversations coming up soon. 778 00:46:10,120 --> 00:46:11,960 But thanks again for everyone listening and supporting the 779 00:46:11,960 --> 00:46:15,600 podcast. Really appreciate all of the 780 00:46:15,840 --> 00:46:20,840 yeah love we received as well. So that's a wrap up for today, 781 00:46:21,040 --> 00:46:26,240 but see you very soon for more explorations on all things label 782 00:46:26,240 --> 00:46:28,840 science and AI. And that's a wrap. 783 00:46:29,160 --> 00:46:31,880 You've been listening to the Behavioral design podcast 784 00:46:32,160 --> 00:46:34,680 brought to you by Habit Weekly and Nuanced Behavior. 785 00:46:35,080 --> 00:46:37,360 Sam and Alene tell me. This season is packed with 786 00:46:37,360 --> 00:46:41,240 incredible insights about behavioral design and AI, so be 787 00:46:41,240 --> 00:46:43,800 sure to subscribe and share the podcast with your friends. 788 00:46:44,040 --> 00:46:46,240 Though you might want to keep it away from your enemies. 789 00:46:47,560 --> 00:46:50,200 In case you haven't noticed, I'm an AI voice. 790 00:46:51,360 --> 00:46:54,160 Yep, pretty crazy. Quite the improvement since last 791 00:46:54,160 --> 00:46:56,160 season's AI outro, don't you think? 792 00:46:57,400 --> 00:47:00,080 If you'd like to collaborate with us at Nuance Behavior, 793 00:47:00,280 --> 00:47:03,000 where we use behavioral design to craft digital products with 794 00:47:03,000 --> 00:47:07,320 Nuance, e-mail us at hello@nuancebehavior.com or book 795 00:47:07,320 --> 00:47:10,560 a call directly on our website, nuancebehavior.com. 796 00:47:12,040 --> 00:47:15,440 A special thanks to the amazing Dave Pizarro for our show music 797 00:47:15,680 --> 00:47:18,560 and to Mei Chen Yap and April English for their help in 798 00:47:18,560 --> 00:47:20,480 producing and publishing this episode. 799 00:47:20,920 --> 00:47:23,760 Thanks again for tuning in. We'll be back soon with another 800 00:47:23,760 --> 00:47:27,000 exciting conversation where behavioral design and AI 801 00:47:27,000 --> 00:47:29,800 Intersect happens to. Megatroid. 802 00:48:02,360 --> 00:48:02,440 The.