1 00:00:00,520 --> 00:00:03,600 What's going on, everyone? Today we are talking AI hype 2 00:00:03,600 --> 00:00:06,120 versus real business value with Manish Dasur. 3 00:00:06,120 --> 00:00:09,240 He's working at PwC. He's been in the data and AI 4 00:00:09,240 --> 00:00:15,160 game for over 20 years, helping clients over 100 different 5 00:00:15,160 --> 00:00:20,800 clients navigate the AI disruption field and also 6 00:00:21,360 --> 00:00:26,520 evaluate and deliver business value through the strategic 7 00:00:26,840 --> 00:00:33,200 adoption and application of data AI and now agentic AI efforts. 8 00:00:34,600 --> 00:00:37,760 Huge shout out to PwC for supporting this episode. 9 00:00:37,760 --> 00:00:45,840 Let's get into it. I'll tell you what I was 10 00:00:45,840 --> 00:00:49,280 building real fast because I think this is in line with what 11 00:00:49,280 --> 00:00:52,600 we're going to talk about right now as far as value and looking 12 00:00:52,600 --> 00:00:56,640 at costs. I built a little widget that can 13 00:00:56,640 --> 00:01:02,600 track the spend, my API spend across different providers and I 14 00:01:02,600 --> 00:01:06,120 gamified it a little bit. So every X amount of money you 15 00:01:06,120 --> 00:01:11,920 get to choose if it's $20 increments or $100 or $1000, it 16 00:01:11,920 --> 00:01:16,560 will go cha ching. And so that, you know, you just 17 00:01:16,560 --> 00:01:21,840 spent some cash and you want to make sure that you have eyes on 18 00:01:21,840 --> 00:01:23,600 that. A lot of times we don't even 19 00:01:23,600 --> 00:01:26,320 realize how much we're spending, especially if it's going through 20 00:01:26,320 --> 00:01:30,320 the API. And next thing you know, maybe 21 00:01:30,320 --> 00:01:34,440 it's your boss or your boss's boss that's trying to drill down 22 00:01:34,440 --> 00:01:36,880 and say, hey, what's going on here? 23 00:01:37,040 --> 00:01:40,760 What are you getting from this? Yeah, Yeah, I couldn't. 24 00:01:40,760 --> 00:01:42,440 I couldn't agree more. I mean, look, I'm glad you're 25 00:01:42,440 --> 00:01:43,880 tracking it. I think it's something that's 26 00:01:43,880 --> 00:01:46,080 also surprising a few organizations, right? 27 00:01:46,080 --> 00:01:49,360 Like all of the AI programs they launched, they were all 28 00:01:49,360 --> 00:01:54,000 predicated on a ROIROY equation that they, they got themselves 29 00:01:54,000 --> 00:01:55,560 aligned to and approved internally. 30 00:01:56,080 --> 00:01:59,280 And that ROI equation starts to really get some pressure and 31 00:01:59,280 --> 00:02:02,520 stress if you're not careful about the compute cost and the 32 00:02:02,520 --> 00:02:04,920 tokenization costs and all those things that can add up. 33 00:02:04,920 --> 00:02:07,960 So I, I've got a lot of clients right now thinking about how do 34 00:02:07,960 --> 00:02:10,240 I apply fin OPS to that discipline? 35 00:02:10,240 --> 00:02:12,440 And to your point, they haven't gotten quite to the Vegas to 36 00:02:12,440 --> 00:02:16,080 Ching sound, but certainly are thinking about in the same way. 37 00:02:16,080 --> 00:02:17,680 How do you really mitigate that cost? 38 00:02:18,280 --> 00:02:22,240 Yeah, I've heard the term tokenomics become more and more 39 00:02:22,440 --> 00:02:28,640 popular and it's really just looking at your spend and 40 00:02:28,920 --> 00:02:31,320 recognizing where's the value from it. 41 00:02:31,560 --> 00:02:35,480 I was talking actually at length last week with a bunch of fin 42 00:02:35,480 --> 00:02:39,440 OPS folks on how they're classifying it and how they're 43 00:02:39,440 --> 00:02:44,040 deciding what is valuable spend versus what is a little bit 44 00:02:44,040 --> 00:02:45,760 supervaluous. Yeah. 45 00:02:45,760 --> 00:02:50,360 I think that you have some cool thoughts on where and how to 46 00:02:50,360 --> 00:02:54,600 justify your different initiatives with AI. 47 00:02:54,920 --> 00:03:00,200 And so maybe we can even just like start with what is getting 48 00:03:00,200 --> 00:03:02,760 value from AI look like in your opinion? 49 00:03:03,360 --> 00:03:05,280 Yeah. Well, let's first, let's first 50 00:03:05,280 --> 00:03:07,640 set a target, right? When I think about clients who 51 00:03:07,640 --> 00:03:11,400 are doing this, well, I think they're reporting about like a 52 00:03:11,400 --> 00:03:15,400 30% improvement on value, right? So if they were, for example, if 53 00:03:15,400 --> 00:03:18,320 they were measuring code production or software 54 00:03:18,320 --> 00:03:21,280 development, life cycle production for their teams, what 55 00:03:21,280 --> 00:03:24,600 they're reporting is, hey, previous to giving my team a 56 00:03:24,600 --> 00:03:29,840 series of AI tools to help them through SDLC, prior to that, we 57 00:03:29,840 --> 00:03:33,080 were seeing about 100 Sprint points come out of that Sprint, 58 00:03:33,240 --> 00:03:34,720 right? So I gave my team a task. 59 00:03:34,760 --> 00:03:37,760 They're going to go develop a bunch of code, and in the past 60 00:03:37,760 --> 00:03:41,120 they're kind of averaging 100 Sprint points of output per 61 00:03:41,120 --> 00:03:43,520 Sprint. Now I've enabled them with a 62 00:03:43,520 --> 00:03:47,400 series of tools, arming them to accelerate to the software 63 00:03:47,400 --> 00:03:50,080 development life cycle. These tools are often writing 64 00:03:50,080 --> 00:03:53,600 user stories, They're often writing test scenarios in cases. 65 00:03:53,800 --> 00:03:56,400 Obviously they're doing code assistant code generation and 66 00:03:56,400 --> 00:03:59,200 debugging and things like that. And what they're kind of 67 00:03:59,200 --> 00:04:03,400 reporting back is now that team is able to get through 130 68 00:04:03,720 --> 00:04:06,440 points per Sprint, right? So they're kind of measuring it 69 00:04:06,440 --> 00:04:07,840 that way. They're measuring the output 70 00:04:07,840 --> 00:04:10,200 that way. If in the past it was 100 71 00:04:10,200 --> 00:04:13,600 points, now it's 130 points. So that's one example of what 72 00:04:13,600 --> 00:04:16,279 good looks like, right? It's kind of a 30% efficiency 73 00:04:16,279 --> 00:04:18,360 gain. If I think about some other 74 00:04:18,360 --> 00:04:21,360 clients, they're applying AI to do some, some more aggressive 75 00:04:21,360 --> 00:04:25,160 workflows in finance or HR and supply chain, also reporting 76 00:04:25,160 --> 00:04:28,640 about a 30% automation efficiency in those cases. 77 00:04:28,640 --> 00:04:32,280 So when we think about how do organizations get value from AI, 78 00:04:32,280 --> 00:04:34,520 let's first start with what do we mean, what is value? 79 00:04:34,520 --> 00:04:37,640 And I think 30% roughly is a good target for organizations to 80 00:04:37,640 --> 00:04:40,680 think about when they think about value from AI, from 81 00:04:40,680 --> 00:04:41,920 automation and what it can drive. 82 00:04:42,400 --> 00:04:45,960 Now, how to get to that value is always the challenge, right? 83 00:04:45,960 --> 00:04:50,040 And, and there's a few points I would share, Dmitri 1, focus on 84 00:04:50,040 --> 00:04:53,560 the workflow, not the use case right? 85 00:04:53,560 --> 00:04:57,520 In, in, in client examples that I get to work with clients that 86 00:04:57,520 --> 00:05:00,200 have clients that describe their AI initiative. 87 00:05:00,200 --> 00:05:03,360 As you know, we're targeting this pilot case here and this 88 00:05:03,360 --> 00:05:06,720 proof of concept here and we're working on this use case here. 89 00:05:07,040 --> 00:05:10,080 Often times I feel like they don't see the end to end the 90 00:05:10,200 --> 00:05:12,760 EBITDA value that they would want to see. 91 00:05:13,200 --> 00:05:16,480 Whereas when clients are saying, hey, look, I'm going to focus on 92 00:05:16,480 --> 00:05:20,320 a workflow, I'm going to embed AI into that workflow. 93 00:05:20,800 --> 00:05:26,160 And if the workflow previously took me 20 steps and 100 humans, 94 00:05:26,280 --> 00:05:30,960 now it'll take me 5 steps, 5 AI agents and 15 humans. 95 00:05:31,200 --> 00:05:35,080 And that delta is the value in EBITDA value that they can take 96 00:05:35,080 --> 00:05:37,240 home. So if you think about it like 97 00:05:37,480 --> 00:05:41,120 I've got an accounts payable workflow or I've got an accounts 98 00:05:41,120 --> 00:05:44,920 receivable workflow, and I'm going to embed AI into that 99 00:05:44,920 --> 00:05:47,120 workflow, right? So I know in step one, a human's 100 00:05:47,120 --> 00:05:50,080 going to do that a Step 2, and AI agent's going to do that Step 101 00:05:50,080 --> 00:05:51,760 3, the invoice is going to get created. 102 00:05:51,960 --> 00:05:55,320 So when I start applying AI to the actual workflow and I can 103 00:05:55,320 --> 00:05:59,040 look at the workflow end to end and compare what my old one was 104 00:05:59,040 --> 00:06:01,760 to what my new one was, that's how I'm seeing clients really 105 00:06:01,760 --> 00:06:04,840 get EBITDA value, not disconnected use cases, 106 00:06:05,760 --> 00:06:10,560 connected workflow redesign with AI generating the value. 107 00:06:10,560 --> 00:06:13,560 So that would be kind of .1. So let's not get into isolated 108 00:06:13,560 --> 00:06:15,520 tools. Let's think about a connected 109 00:06:15,520 --> 00:06:18,120 workflow and how AI really changed that workflow all. 110 00:06:19,160 --> 00:06:21,160 Right. I want to double click on the 111 00:06:21,160 --> 00:06:23,720 idea. They're real fast about how you 112 00:06:23,720 --> 00:06:28,000 were saying that 30% increase. And one thing that I've heard 113 00:06:28,000 --> 00:06:33,280 folks talk about is how they will look back a year ago and 114 00:06:33,280 --> 00:06:38,840 recognize certain big pushes that they've had to do and what 115 00:06:38,840 --> 00:06:41,480 productivity looked like around there, how much they code they 116 00:06:41,480 --> 00:06:44,480 were shipping, whatever these metrics were that they were 117 00:06:44,480 --> 00:06:48,800 tracking. And they can then compare that 118 00:06:48,800 --> 00:06:52,600 to now. And it's, it's not necessarily 119 00:06:52,600 --> 00:06:57,960 apples to apples, but it's a much better way of deciding, 120 00:06:58,400 --> 00:07:03,320 hey, how much are we spending on AI versus how much more are we 121 00:07:03,320 --> 00:07:05,160 producing? Yeah. 122 00:07:05,160 --> 00:07:10,520 And then you can see that in a way that isn't just asking your 123 00:07:10,520 --> 00:07:17,040 employees to tell me, how much more value do you think AI is 124 00:07:17,040 --> 00:07:19,720 providing to you? How much more productive are 125 00:07:19,720 --> 00:07:21,720 you? That self reporting metric I 126 00:07:21,720 --> 00:07:25,400 think is a little bit biased. And I'm always weary when I see 127 00:07:25,400 --> 00:07:27,880 those. But I just want, I wanted to 128 00:07:27,880 --> 00:07:29,400 talk about that. And there's so many other things 129 00:07:29,400 --> 00:07:31,960 on this workflow piece that I want to dive into, but I know 130 00:07:31,960 --> 00:07:33,000 you were going to say something else. 131 00:07:33,840 --> 00:07:35,480 Well, I mean, look, I think your point's a good one. 132 00:07:35,480 --> 00:07:39,640 In fact, every transformation initiative that I've seen really 133 00:07:39,640 --> 00:07:43,320 be successful is always challenged to prove value 134 00:07:43,640 --> 00:07:46,680 ongoing basis, right. I want to every transformation, 135 00:07:46,680 --> 00:07:48,680 if within the first quarter you want to be able to start to 136 00:07:48,680 --> 00:07:51,720 demonstrate value, you want to be able to update or increase 137 00:07:51,720 --> 00:07:55,200 that value quarter over quarter. To your point, it's got to be 138 00:07:55,200 --> 00:07:58,520 very, very quantifiable, right? So the other maybe point I'll 139 00:07:58,520 --> 00:08:01,160 add to how do organizations make sure they get value? 140 00:08:01,160 --> 00:08:04,880 What I've seen a lot is organizations will seek to prove 141 00:08:04,880 --> 00:08:08,880 the ROI and they'll set up a champion challenger kind of 142 00:08:08,880 --> 00:08:11,440 test, right? So as they roll out AI 143 00:08:11,440 --> 00:08:15,600 automation, they'll have a control group working in the old 144 00:08:15,600 --> 00:08:19,400 process and they'll have a test group working in the new AI 145 00:08:19,400 --> 00:08:21,880 automated process. And at least for a couple of 146 00:08:21,880 --> 00:08:24,600 cycles, they'll compare the outputs, they'll compare the 147 00:08:24,600 --> 00:08:25,880 results. Depending on the use case, the 148 00:08:25,880 --> 00:08:28,560 comparison might be different, but they'll compare the results 149 00:08:28,560 --> 00:08:32,400 so they can get themselves to a quantifiable value statement and 150 00:08:32,400 --> 00:08:34,960 then find a way to tweak it, improve it over time, right? 151 00:08:34,960 --> 00:08:38,559 So I think that's super important to, to have that 152 00:08:38,559 --> 00:08:41,080 measure of whether you're measuring cost or speed or 153 00:08:41,080 --> 00:08:43,880 revenue uplift, whatever it is that you're measuring, but have 154 00:08:43,880 --> 00:08:47,120 that champion challenger kind of mindset set up that way. 155 00:08:47,680 --> 00:08:49,760 To your point, you're not guessing it, you're not talking 156 00:08:49,760 --> 00:08:51,480 about it. You're actually demonstrating 157 00:08:51,480 --> 00:08:53,520 with some numbers. And that, I think is a is a 158 00:08:53,520 --> 00:08:55,080 winning formula for organizations. 159 00:08:55,400 --> 00:08:59,000 Yeah. You know, I saw that and I think 160 00:08:59,000 --> 00:09:03,120 these days there is no software engineer that I know that 161 00:09:03,120 --> 00:09:06,000 doesn't want to use some kind of a coding agent. 162 00:09:06,280 --> 00:09:13,200 So it's not necessarily what are we going to do AI versus no AI 163 00:09:13,200 --> 00:09:15,960 with especially software engineering and even product 164 00:09:15,960 --> 00:09:19,560 development, that type of thing, those internal type tools when 165 00:09:19,560 --> 00:09:23,480 we're dealing with software. I think the really difficult 166 00:09:23,480 --> 00:09:27,360 part here is you've got some software engineers that are 167 00:09:27,360 --> 00:09:31,920 burning a ton and getting a ton done and then you've got other 168 00:09:31,920 --> 00:09:36,480 software engineers, AKA me, who are burning a ton and not 169 00:09:36,480 --> 00:09:39,680 getting a lot done. And so how do you distinguish 170 00:09:39,680 --> 00:09:45,400 between that efficient burning of tokens versus the non 171 00:09:45,400 --> 00:09:47,200 efficient ones? Yeah. 172 00:09:47,200 --> 00:09:48,800 No, it's a great point. I think we're going to talk 173 00:09:48,800 --> 00:09:51,440 about a little bit of the the fin OPS angle as well. 174 00:09:51,440 --> 00:09:54,160 And look, I don't think a lot of organizations have found that 175 00:09:54,160 --> 00:09:57,480 answer quite yet, right? I do think a lot of 176 00:09:57,480 --> 00:10:01,840 organizations have been pushing AI everywhere, AI for everyone, 177 00:10:02,200 --> 00:10:03,880 right? And, and while that's fantastic 178 00:10:03,880 --> 00:10:06,480 and the right thing to do, the, the cost of that can be quite 179 00:10:06,480 --> 00:10:09,120 high, right? But I do think now organizations 180 00:10:09,120 --> 00:10:10,240 are starting to shift a little bit. 181 00:10:10,240 --> 00:10:13,040 They're thinking about, do I use the right model for the right 182 00:10:13,040 --> 00:10:14,640 use case? I don't have to use the same 183 00:10:14,640 --> 00:10:17,760 model for all use cases. So I can kind of right size the 184 00:10:17,760 --> 00:10:20,120 model to the use case I'm driving that can help me with 185 00:10:20,120 --> 00:10:23,280 some cost. I can do workflow engineering or 186 00:10:23,280 --> 00:10:26,240 prompt engineering in a way that can help me manage my cost. 187 00:10:26,480 --> 00:10:27,800 Right. So those, and then of course, 188 00:10:27,800 --> 00:10:30,480 I've got the fin OPS angle talking about, to your point, 189 00:10:30,480 --> 00:10:32,840 the cha ching thing that you were describing kind of every 190 00:10:32,840 --> 00:10:35,680 time I use, every time I use that compute, use that token, 191 00:10:35,840 --> 00:10:38,960 I've got a measurement that way. And like I said, I, I think all 192 00:10:38,960 --> 00:10:42,720 of that has to be baked into the ROI equation of the program of 193 00:10:42,720 --> 00:10:44,720 the transformation that we're seeking to drive. 194 00:10:45,520 --> 00:10:48,840 But I will tell you overall, I do think organizations are 195 00:10:48,840 --> 00:10:52,640 reporting good benefits. 30% benefits are hard to ignore when 196 00:10:52,640 --> 00:10:54,600 you think about the size of these organizations. 197 00:10:54,600 --> 00:10:59,000 If you can drive a significant reduction in operating cost or a 198 00:10:59,000 --> 00:11:03,320 significant growth in revenue generation, then it does at the 199 00:11:03,320 --> 00:11:06,440 large AI benefits will exceed the value associated to it. 200 00:11:06,920 --> 00:11:11,560 Yeah, so you gave us the point #1 which was these workflows and 201 00:11:11,840 --> 00:11:14,000 trying to figure out where you're plugging AI into. 202 00:11:14,320 --> 00:11:20,960 I wanted to mention that I have a theory that the reason you saw 203 00:11:21,240 --> 00:11:26,880 so many successful use cases with support chat bots and 204 00:11:26,880 --> 00:11:31,800 support agents. Yeah, was because that is a 205 00:11:31,800 --> 00:11:35,320 field where there's very clear SO. 206 00:11:35,320 --> 00:11:39,120 PS And because of that, it's all mapped out. 207 00:11:39,480 --> 00:11:42,400 It's easier to create these workflows like you're talking 208 00:11:42,400 --> 00:11:46,360 about and go end to end. And if you really need to, you 209 00:11:46,360 --> 00:11:48,920 can escalate up to a human when there's something that will 210 00:11:48,920 --> 00:11:51,160 stump the agent or it's not in the books. 211 00:11:51,160 --> 00:11:55,720 But as a friend of mine told me, especially in call centers, you 212 00:11:56,440 --> 00:12:00,320 routinely get folks who are very new at their jobs and all 213 00:12:00,320 --> 00:12:02,840 they're doing is looking through the SO. 214 00:12:02,840 --> 00:12:05,440 PS Yeah. And they're doing a poor job at 215 00:12:05,440 --> 00:12:07,160 that, and an agent can do it much better. 216 00:12:07,160 --> 00:12:11,360 So that idea of like having the workflows defined and then being 217 00:12:11,360 --> 00:12:15,960 able to plug in the agents or the LMS where you can makes a 218 00:12:15,960 --> 00:12:19,600 ton of sense to me. I look and customer care is an 219 00:12:19,600 --> 00:12:21,440 excellent example. So let's double click on that 220 00:12:21,440 --> 00:12:25,240 for a second because I got to tell you a fun story as as as 221 00:12:25,240 --> 00:12:28,680 important it is as it is to architect it into the workflow. 222 00:12:28,680 --> 00:12:31,640 And I agree with you, customer care is a fantastic example for 223 00:12:31,640 --> 00:12:33,480 that. It is still freaking hard. 224 00:12:33,720 --> 00:12:36,480 And let me tell you, let me give you a couple of things that in 225 00:12:36,480 --> 00:12:39,720 my opinion, Stump, I had a, I had an opportunity to work with 226 00:12:39,720 --> 00:12:42,320 a large telecommunications company and they were, they were 227 00:12:42,320 --> 00:12:46,280 seeking to do exactly what you described, 80 percent, 75% of 228 00:12:46,280 --> 00:12:47,960 the calls coming into my call center. 229 00:12:48,120 --> 00:12:51,320 I want an AI agent to be able to handle them and only take it 230 00:12:51,320 --> 00:12:54,560 into a human when I need to for that remaining 2025%. 231 00:12:54,560 --> 00:12:57,080 So that's the, that's the, that's the attempt, that's the 232 00:12:57,080 --> 00:13:00,400 workflow automation. So I, I would say the two 233 00:13:00,400 --> 00:13:03,280 hardest challenges we found in that work was 1. 234 00:13:04,280 --> 00:13:07,080 As you know, when you start with a model, it's relatively 235 00:13:07,080 --> 00:13:09,640 generic. You have to give it an context 236 00:13:09,640 --> 00:13:13,120 and training to make it specific and useful in the workflow that 237 00:13:13,120 --> 00:13:15,560 we're talking about, right? So for example, I've got a 238 00:13:15,560 --> 00:13:17,800 generic customer call centre type of model. 239 00:13:18,040 --> 00:13:21,880 It needs to be trained in my company's policies, inventory 240 00:13:21,880 --> 00:13:24,920 plans, things like that, so that when a customer calls, I've got 241 00:13:24,920 --> 00:13:28,440 relevant information to give it. So when you think about training 242 00:13:28,440 --> 00:13:32,320 a model like that, a customer could be calling about anything. 243 00:13:32,480 --> 00:13:35,640 They could be calling about my invoice, my trip, my phone, 244 00:13:35,640 --> 00:13:38,240 what's wrong with it, etcetera. So the context, the amount of 245 00:13:38,240 --> 00:13:41,320 training you have to give that model is quite unique. 246 00:13:41,320 --> 00:13:45,520 And some of that training comes from previous call logs that the 247 00:13:45,520 --> 00:13:48,000 customer care reps have written down and written down. 248 00:13:48,120 --> 00:13:50,360 So of course we're going to feed that into the model, right? 249 00:13:50,360 --> 00:13:54,640 But the model also needs to know about when Dimitri calls in. 250 00:13:54,640 --> 00:13:57,400 Let's let's use an example. Dimitri calls in and says, hey, 251 00:13:57,400 --> 00:14:00,560 I'm travelling through Asia next month. 252 00:14:00,920 --> 00:14:04,880 Mr. cell phone provider, would you plead at please add roaming 253 00:14:04,880 --> 00:14:07,240 to my plan, right? So I can go around Asia and use 254 00:14:07,240 --> 00:14:09,720 my phone. Now for an AI agent to be able 255 00:14:09,720 --> 00:14:12,640 to handle that call, first it needs to know who Dimitri is. 256 00:14:12,640 --> 00:14:15,520 So it's going to need a way to connect into this. 257 00:14:15,800 --> 00:14:20,360 I'll call it ACDPA customer data platform, customer 360, whatever 258 00:14:20,360 --> 00:14:23,800 terminology you use that so the, so the model knows who Dimitri 259 00:14:23,800 --> 00:14:25,280 is. How many lines does he have? 260 00:14:25,680 --> 00:14:29,120 Has he paid his bill on time? What phone does he have? 261 00:14:29,400 --> 00:14:31,200 Does he, how often does he travel? 262 00:14:31,200 --> 00:14:33,080 What plan does he have today? Right. 263 00:14:33,080 --> 00:14:34,640 So it needs to know who Dimitri is. 264 00:14:35,200 --> 00:14:37,600 Then once it knows who Dimitri is, it's going to ask you some 265 00:14:37,600 --> 00:14:40,120 clarifying questions. What countries are you traveling 266 00:14:40,120 --> 00:14:41,480 to? What dates are you going to be 267 00:14:41,480 --> 00:14:43,960 traveling in? Based on the answers you give 268 00:14:43,960 --> 00:14:46,800 it, it needs to update the algorithm and go into the 269 00:14:47,320 --> 00:14:51,960 inventory of plans I could offer you and select the one that we 270 00:14:51,960 --> 00:14:53,400 think is the right one for you, Right? 271 00:14:53,400 --> 00:14:56,280 Then I've got to make that offer to Dimitri on the phone. 272 00:14:57,160 --> 00:14:59,960 Dimitri is going to say, yeah, OK, I like that plan, give me 273 00:14:59,960 --> 00:15:01,720 that one. He's going to confirm the cost 274 00:15:01,720 --> 00:15:04,560 is, I don't know, an extra $10 a month, and he's going to say 275 00:15:04,560 --> 00:15:07,440 good at it for that month. Then the model needs to have the 276 00:15:07,440 --> 00:15:11,000 authority to go back into your invoice and confirm that you are 277 00:15:11,000 --> 00:15:14,400 now going to pay 10 more, charge your credit card and update your 278 00:15:14,400 --> 00:15:16,680 plan. That's quite a complicated 279 00:15:16,680 --> 00:15:19,160 transaction. It needs to connect to your 280 00:15:19,160 --> 00:15:23,760 billing system is likely written in one like 1 platform. 281 00:15:23,760 --> 00:15:26,800 Your, your, your planning system is likely in another platform. 282 00:15:27,040 --> 00:15:29,120 Your customer system is likely in another platform. 283 00:15:29,120 --> 00:15:32,120 So there's a bunch of API connectivity, a bunch of 284 00:15:32,120 --> 00:15:35,120 integration that you've got to solve for bunch of context and 285 00:15:35,120 --> 00:15:36,360 training. You've got a software. 286 00:15:36,720 --> 00:15:39,120 So it, you know, for those kinds of things, it can be quite 287 00:15:39,120 --> 00:15:41,280 complicated. But that's why I think having 288 00:15:41,280 --> 00:15:44,400 the right governance approach, right structure approach and and 289 00:15:44,400 --> 00:15:47,640 a plan to attack that workflow end to end is super critical. 290 00:15:48,160 --> 00:15:52,520 Don't forget you if I'm calling you on the phone, you have to be 291 00:15:52,520 --> 00:15:55,640 doing this with very low latency. 292 00:15:56,040 --> 00:15:57,400 That's right. That's right. 293 00:15:58,280 --> 00:16:00,920 And you remember like the models of the past, they were taking 294 00:16:00,920 --> 00:16:05,280 voice, going to text, generating a response in text and bringing 295 00:16:05,280 --> 00:16:08,280 it back to voice, which created a lot of latency, right? 296 00:16:08,280 --> 00:16:10,680 But now in the new world, we can go voice to voice. 297 00:16:10,680 --> 00:16:13,400 We don't have to go to voice to text to text to back to voice. 298 00:16:13,720 --> 00:16:15,760 So we can go to voice to voice with the latency is certainly 299 00:16:15,760 --> 00:16:18,400 starting to get better. But I still think training 300 00:16:18,400 --> 00:16:20,960 models, contextualizing models and then applying all the 301 00:16:20,960 --> 00:16:24,680 integration you need with all the enterprise corporate systems 302 00:16:24,840 --> 00:16:28,000 that models need to pull from and interact with it in order to 303 00:16:28,160 --> 00:16:31,800 satisfy the customer query is quite a tall order, right? 304 00:16:31,800 --> 00:16:34,800 And that's why I think organizations need a a practical 305 00:16:34,800 --> 00:16:36,160 approach to solving that problem. 306 00:16:37,040 --> 00:16:39,360 Yeah. So there we go. 307 00:16:39,360 --> 00:16:42,400 That's the workflows on point number one. 308 00:16:42,760 --> 00:16:45,680 Yeah, you had more points and I remember I cut you off. 309 00:16:45,680 --> 00:16:48,760 So it keep it rocking on the different pieces that you want 310 00:16:48,760 --> 00:16:50,920 to hit on. Well, the last thing I look, 311 00:16:50,920 --> 00:16:53,120 we've talked about focus on workflows. 312 00:16:53,160 --> 00:16:56,040 We've talked about, you know, use the champion challenger 313 00:16:56,040 --> 00:16:58,760 model to really to really prove out the value. 314 00:16:59,160 --> 00:17:01,400 The Third Point I would just tell you is I do think it's very 315 00:17:01,400 --> 00:17:05,160 important for leaders to build the culture of innovation right 316 00:17:05,160 --> 00:17:09,520 at the, the only places where I've really seen AI be adopted 317 00:17:09,520 --> 00:17:13,599 is when leaders are rewarding innovation and pushing the 318 00:17:13,599 --> 00:17:16,079 organization to reimagine how they do it. 319 00:17:16,359 --> 00:17:18,680 It's a hard thing to do, right? People have been running these 320 00:17:18,680 --> 00:17:22,280 workflows for two decades in their jobs, in their roles, and 321 00:17:22,280 --> 00:17:24,240 now we're saying, hey, that workflow is going to change. 322 00:17:24,720 --> 00:17:27,520 So if you can in a, in a leadership way, in an 323 00:17:27,520 --> 00:17:31,800 organizational way, find a way to reward a culture of 324 00:17:31,800 --> 00:17:35,280 innovation, reward a culture of let's try something new. 325 00:17:35,280 --> 00:17:38,520 Let's adopt something new. I think that's super critical to 326 00:17:38,520 --> 00:17:40,160 your success as well. And I'll just tell you, man, 327 00:17:40,160 --> 00:17:43,400 like I think our, we did some surveys on this stuff. 328 00:17:43,400 --> 00:17:47,400 I think we saw like 88% of companies say they're doing AI, 329 00:17:47,840 --> 00:17:52,040 but only 33% of companies say they're scaling AI, right? 330 00:17:52,040 --> 00:17:54,560 So I still think there's quite a barrier here. 331 00:17:54,840 --> 00:17:59,680 And as organizations go from I have 10 agents to soon I'll have 332 00:17:59,680 --> 00:18:02,280 100 to soon I'll have 1000, right? 333 00:18:02,280 --> 00:18:03,960 These are the kinds of things that are going to help them get 334 00:18:03,960 --> 00:18:07,840 there. Have you seen this idea of 335 00:18:08,400 --> 00:18:14,800 rewarding AI and creating that culture of innovation, being 336 00:18:14,800 --> 00:18:18,760 successful across organizations, especially the larger 337 00:18:18,760 --> 00:18:20,480 organizations that I can imagine? 338 00:18:20,480 --> 00:18:24,000 You have folks that are set in their ways and they've been 339 00:18:24,000 --> 00:18:27,440 doing something for a while, and it's very hard to change. 340 00:18:27,800 --> 00:18:30,080 Yeah. It's a combination of things, I 341 00:18:30,080 --> 00:18:32,320 think, right? But I do think I see 342 00:18:32,320 --> 00:18:36,680 organizations seeking to reward, promote those who are adopting 343 00:18:36,680 --> 00:18:38,480 innovation. I've even started to see 344 00:18:38,480 --> 00:18:42,560 organizations who are measuring how often you use AI, right? 345 00:18:42,560 --> 00:18:44,640 If you're a regular consumer, you're doing your job. 346 00:18:44,920 --> 00:18:47,600 Organizations are starting to measure how often do you prompt 347 00:18:48,040 --> 00:18:50,880 your, your local LLM or your local authentic solution? 348 00:18:51,080 --> 00:18:53,880 How effectively are you actually using AI? 349 00:18:54,200 --> 00:18:57,120 And that's becoming one of the criterias for for your 350 00:18:57,120 --> 00:18:58,680 evaluation, for your performance. 351 00:18:59,440 --> 00:19:02,720 And I think adopting a culture of innovation and and a culture 352 00:19:02,720 --> 00:19:05,720 that pushes the boundary I think is a good thing to do. 353 00:19:06,920 --> 00:19:11,360 Yeah, I've seen that at certain companies that can remain 354 00:19:11,360 --> 00:19:14,760 unnamed. But it's also almost like the 355 00:19:14,760 --> 00:19:18,040 double edged sword is that it becomes toxic and then you have 356 00:19:18,040 --> 00:19:21,720 token maxing or just prompting to prompt. 357 00:19:22,120 --> 00:19:24,680 Yeah. And if you only are going off of 358 00:19:24,680 --> 00:19:26,640 that, then it can be dangerous too. 359 00:19:26,880 --> 00:19:29,560 So it it's like everything. It's quite nuanced in that 360 00:19:29,560 --> 00:19:30,840 regard. Yeah. 361 00:19:30,960 --> 00:19:33,280 Absolutely, absolutely. But I think, you know, if you've 362 00:19:33,280 --> 00:19:35,840 got I, I think the best thing you can do as a leader is 363 00:19:35,920 --> 00:19:37,720 embrace to change yourself, right? 364 00:19:37,720 --> 00:19:40,640 Be the example that you're seeking for your team to be use 365 00:19:40,720 --> 00:19:43,120 AI in your everyday use it for the right use cases. 366 00:19:43,120 --> 00:19:46,000 Be mindful of how much you're spending on it, but also, you 367 00:19:46,000 --> 00:19:49,200 know, don't hesitate to drive top line growth or bottom line 368 00:19:49,200 --> 00:19:51,960 growth with AI and be the best example of that. 369 00:19:51,960 --> 00:19:54,600 And if you can do that as a leader, then hopefully the rest 370 00:19:54,600 --> 00:19:56,720 of the organization can and use that and follow. 371 00:19:57,400 --> 00:20:00,640 Yeah, bring them with you. And yeah, where have you seen it 372 00:20:00,640 --> 00:20:04,840 Feels like you. This is a nice segue into where 373 00:20:04,840 --> 00:20:08,840 you can fail and where you've seen these efforts fall flat. 374 00:20:09,680 --> 00:20:12,360 Those who get it right versus those who stumble right is 375 00:20:12,360 --> 00:20:14,520 always a good question to kind of think about it. 376 00:20:14,520 --> 00:20:17,000 And it helps clients think about this too, because then they 377 00:20:17,000 --> 00:20:19,080 avoid those same mistakes that others have failed at. 378 00:20:19,520 --> 00:20:23,640 The first one I would tell you is the concept of AI everywhere. 379 00:20:24,080 --> 00:20:26,400 I don't think is the right concept for most clients to 380 00:20:26,400 --> 00:20:28,040 think about it. When you think about AI 381 00:20:28,040 --> 00:20:31,640 everywhere, you end up incurring a lot of cost for not a lot of 382 00:20:31,640 --> 00:20:33,560 value. So I the first kind of point of 383 00:20:33,560 --> 00:20:36,040 guidance I would give clients there is you don't want to think 384 00:20:36,040 --> 00:20:38,520 about it like AI everywhere. Of course we want AI to be 385 00:20:38,520 --> 00:20:40,720 pervasive. That's not what of course that 386 00:20:40,720 --> 00:20:44,320 we, we want that right. But we also want AI to be very 387 00:20:44,320 --> 00:20:47,960 focused on those high value use cases that you think are going 388 00:20:47,960 --> 00:20:50,440 to be game changing for your organization. 389 00:20:51,000 --> 00:20:53,680 You also might want to get AI focus and start with some low 390 00:20:53,680 --> 00:20:57,480 risk use cases that won't be super sensitive to the market or 391 00:20:57,480 --> 00:20:59,720 to your brand. So I think you the first, the 392 00:20:59,720 --> 00:21:02,320 first point of guidance I would share is don't think about 393 00:21:02,320 --> 00:21:05,560 applying AI everywhere. Instead think about it as I've 394 00:21:05,560 --> 00:21:08,880 got this fantastic tool. Of course, eventually it will be 395 00:21:08,880 --> 00:21:12,200 pervasive in my organization. But if I'm starting out or if 396 00:21:12,200 --> 00:21:15,080 I'm in those stages, going from 10 implementations to 100 397 00:21:15,080 --> 00:21:18,880 implementations, I want to be strategic about where I use it. 398 00:21:19,040 --> 00:21:21,440 I want to be prescriptive about where I use it. 399 00:21:21,440 --> 00:21:24,120 I want to measure the value that it creates, right? 400 00:21:24,120 --> 00:21:26,960 So the first thing I would tell you is it's not AI everywhere. 401 00:21:27,640 --> 00:21:31,720 It's AI applied and utilized in those workflows, in those use 402 00:21:31,720 --> 00:21:34,640 cases that are going to generate the biggest bang from my buck, 403 00:21:34,960 --> 00:21:36,200 right? So that's the first point, maybe 404 00:21:36,200 --> 00:21:38,680 I would share. Second point I would share is, 405 00:21:38,680 --> 00:21:41,440 look, you have to watch out for the compute costs, right? 406 00:21:41,440 --> 00:21:45,960 I think, I think consumption driven cost models sometimes 407 00:21:45,960 --> 00:21:47,320 great challenges. I think it's very, very 408 00:21:47,320 --> 00:21:49,960 important that organizations have the appropriate governance 409 00:21:49,960 --> 00:21:52,400 set up. So they're, so they're too, 410 00:21:52,400 --> 00:21:54,480 they've got some oversight and it's not a surprise. 411 00:21:54,480 --> 00:21:56,200 The surprise is what I worry about, right? 412 00:21:56,440 --> 00:21:59,440 A lot of clients have been enamoured with what AI can do. 413 00:22:00,080 --> 00:22:03,680 They haven't really focused on what do the cost look like as it 414 00:22:03,680 --> 00:22:05,400 starts to scale. And often they can get 415 00:22:05,400 --> 00:22:08,400 surprised, right. So I think there you've got a, 416 00:22:08,400 --> 00:22:12,280 first of all, you've got a architect for efficiency one, 417 00:22:12,360 --> 00:22:16,120 it's not one model that fits all use cases, right? 418 00:22:16,120 --> 00:22:18,720 So it's very important that you think about what am I trying to 419 00:22:18,720 --> 00:22:20,840 do? And I select the right model for 420 00:22:20,840 --> 00:22:22,840 that. If I'm doing image creation, I 421 00:22:22,840 --> 00:22:26,160 might pick a certain model. If I'm doing deep thinking, I 422 00:22:26,160 --> 00:22:28,800 might pick a different model. If I'm doing PowerPoint 423 00:22:28,800 --> 00:22:31,240 creation, then I might do a different model, right? 424 00:22:31,240 --> 00:22:34,680 So using the right model for the right use case will help clients 425 00:22:34,680 --> 00:22:39,880 mitigate cost. 2 the prompt engineering discipline is, is 426 00:22:39,880 --> 00:22:42,480 one that I, you know, I have to tell you is going to become 427 00:22:42,560 --> 00:22:45,840 increasingly important. The type of prompt you put into 428 00:22:45,840 --> 00:22:47,880 a model can generate different types of cost as well. 429 00:22:47,880 --> 00:22:50,800 So prompt engineering and making sure we've got the model 430 00:22:50,800 --> 00:22:54,280 architected, the workflow design architected to minimize token 431 00:22:54,280 --> 00:22:56,720 usage, I think is, is the second thing that I would kind of think 432 00:22:56,720 --> 00:23:00,120 about there. And then lastly, like you, you 433 00:23:00,120 --> 00:23:01,880 started the conversation with fin OPS, right? 434 00:23:01,880 --> 00:23:06,360 But having a financial way of tracking cost per workflow and 435 00:23:06,520 --> 00:23:10,040 enforcing that budget and forcing that governance, I would 436 00:23:10,040 --> 00:23:12,280 certainly say is a is a key thing for clients to do. 437 00:23:14,200 --> 00:23:18,680 So yeah, what I hit on, I hit on, I hit on not AI everywhere. 438 00:23:18,880 --> 00:23:20,720 The cost model is super important. 439 00:23:21,040 --> 00:23:26,200 And maybe the last point I'll still share here is as AI become 440 00:23:26,200 --> 00:23:30,040 smarter, as the reasoning capability of these models 441 00:23:30,040 --> 00:23:33,440 becomes better, we can apply them to more complex use cases. 442 00:23:34,160 --> 00:23:37,680 And as the autonomy gets better, as they're able to execute 443 00:23:37,680 --> 00:23:40,960 actions on their own behalf better, now I can have them make 444 00:23:40,960 --> 00:23:43,840 complex decisions and execute actions against those complex 445 00:23:43,840 --> 00:23:46,640 decisions. But what I also see happening as 446 00:23:46,640 --> 00:23:51,360 those two factors are improving is clients are seeking to use AI 447 00:23:51,360 --> 00:23:54,720 in more sensitive use cases. I want an AI, we talked about 448 00:23:54,720 --> 00:23:57,480 customer care. I want an AI agent handling my 449 00:23:57,480 --> 00:24:00,000 customer care call. Well, that's very important 450 00:24:00,000 --> 00:24:03,280 because your brand, your policies all need to be is how 451 00:24:03,280 --> 00:24:06,280 your AI agent needs to behave in a way that's consistent with 452 00:24:06,280 --> 00:24:08,720 your brand, in a way that's consistent with your policy, so 453 00:24:08,720 --> 00:24:11,160 your customer has a consistent experience. 454 00:24:11,640 --> 00:24:14,080 I want an AI agent approving purchase orders. 455 00:24:14,840 --> 00:24:18,080 And now in this case, I've got an AI agent making a financial 456 00:24:18,080 --> 00:24:20,680 decision on your behalf and approving purchase orders. 457 00:24:20,680 --> 00:24:23,520 And money is actually going out the door based on a decision 458 00:24:23,520 --> 00:24:27,520 that an AI agent has made. So as I think organizations seek 459 00:24:27,520 --> 00:24:31,360 to use AI in that way, seek to put AI in more sensitive use 460 00:24:31,360 --> 00:24:37,160 cases, really having a robust framework and methodology for 461 00:24:37,160 --> 00:24:41,160 testing accuracy, for testing hallucination, for testing 462 00:24:41,160 --> 00:24:45,560 responsible behavior, right, for essentially auditing agentic 463 00:24:45,560 --> 00:24:48,360 behavior is going to become super important for our clients. 464 00:24:48,680 --> 00:24:50,920 That's something at PwC we're pretty excited about because 465 00:24:50,920 --> 00:24:53,800 obviously we do a lot of risk and auditing and, and you know, 466 00:24:53,800 --> 00:24:56,360 we'd like to extend that kind of thinking to your AI world as 467 00:24:56,360 --> 00:24:58,880 well, right? And I think in the workforce of 468 00:24:58,880 --> 00:25:03,560 tomorrow, if I imagine it to be humans and AI working together, 469 00:25:03,800 --> 00:25:08,000 we've got to be able to really review, validate, audit that 470 00:25:08,000 --> 00:25:10,720 work product to make sure it's consistent with what we're 471 00:25:10,720 --> 00:25:13,880 doing. And as we gain more trust there, 472 00:25:14,360 --> 00:25:17,680 that'll be the unlock of allowing AI to participate and 473 00:25:17,680 --> 00:25:20,600 really make decisions in sensitive use cases. 474 00:25:21,040 --> 00:25:24,480 And just like, just like the cost point, I think the testing 475 00:25:24,480 --> 00:25:27,600 point, the responsibility point is still an area that a lot of 476 00:25:27,600 --> 00:25:30,800 our clients haven't explored just because it's relatively 477 00:25:30,800 --> 00:25:32,440 new. And I think that will be a key 478 00:25:32,440 --> 00:25:34,080 unlock for the for the industry as well. 479 00:25:34,800 --> 00:25:39,200 Have you thought through how that could potentially look we? 480 00:25:39,560 --> 00:25:40,880 Absolutely. We're starting to build 481 00:25:40,880 --> 00:25:45,840 capabilities now that can test, audit, validate results that are 482 00:25:45,840 --> 00:25:48,280 coming out of an LLM model. As you know, that's not easy to 483 00:25:48,280 --> 00:25:51,520 do. In the past, systems we designed 484 00:25:51,520 --> 00:25:55,480 was one input has a series of finite outputs, Let's say it has 485 00:25:55,480 --> 00:25:57,640 five. I can test for those outputs 486 00:25:57,640 --> 00:26:00,120 relatively simple. I can put the input in, I can 487 00:26:00,120 --> 00:26:03,040 validate that I got the one of the five outputs, and if I did, 488 00:26:03,240 --> 00:26:06,360 then the test succeeds. Much, much harder to do in the 489 00:26:06,360 --> 00:26:08,960 agentic world. In the LLM world, one input 490 00:26:08,960 --> 00:26:10,320 could have million different outputs. 491 00:26:10,760 --> 00:26:12,880 So we are now building frameworks and testing 492 00:26:12,880 --> 00:26:15,880 methodologies that essentially use AI to test AI. 493 00:26:16,520 --> 00:26:18,440 But to be able to do that at scale, that's really the only 494 00:26:18,440 --> 00:26:20,440 way to do it, right. So yes, we are absolutely 495 00:26:20,440 --> 00:26:23,680 starting to starting to build frameworks and capabilities that 496 00:26:23,880 --> 00:26:26,640 we can take the clients and offer them some help in being 497 00:26:26,640 --> 00:26:29,320 able to test and validate AI results. 498 00:26:29,800 --> 00:26:32,200 I do agree with you. However, like today, when you go 499 00:26:32,200 --> 00:26:35,000 buy a car, there's AJD power associate certification. 500 00:26:35,280 --> 00:26:37,360 So you know you're buying a car that you can trust. 501 00:26:37,480 --> 00:26:41,040 I do agree with you the, the I don't know what the answer will 502 00:26:41,040 --> 00:26:42,320 be yet. I don't think the marketer 503 00:26:42,320 --> 00:26:44,760 industry has has aligned on what the answer will be yet. 504 00:26:44,760 --> 00:26:47,960 But I do agree, I think clients and organizations are going to 505 00:26:47,960 --> 00:26:52,080 want a way to essentially certify or some, some sort of 506 00:26:52,080 --> 00:26:54,720 validate that this agent is good to go. 507 00:26:54,720 --> 00:26:56,680 I can use it in a sensitive use case. 508 00:26:56,680 --> 00:26:58,280 I can put it in front of my customers. 509 00:26:58,640 --> 00:27:01,360 I've got a way to mitigate the risk to my brand and to my 510 00:27:01,360 --> 00:27:04,120 policy. And I certainly think that once 511 00:27:04,120 --> 00:27:06,840 once the industry has aligned on the right solution, that'll 512 00:27:06,840 --> 00:27:10,120 allow us to take a leap forward. In the meantime, I think some of 513 00:27:10,120 --> 00:27:12,480 the frameworks in the agentic work that we're doing is exactly 514 00:27:12,480 --> 00:27:14,000 how class should be thinking about that answer. 515 00:27:16,240 --> 00:27:19,080 Yeah, I really wonder how it would look and what it would 516 00:27:19,080 --> 00:27:22,600 shape out to be. I've joked around about how you 517 00:27:22,600 --> 00:27:28,080 potentially would have verified MCP servers, but it's not 518 00:27:28,840 --> 00:27:31,400 necessarily. What's so weird about this is 519 00:27:31,400 --> 00:27:37,280 that you can't quite say like, well, maybe you can. 520 00:27:37,440 --> 00:27:40,680 Companies are Sock 2 certified, right? 521 00:27:40,680 --> 00:27:45,000 And that means that they've done a bunch of stuff and they've 522 00:27:45,000 --> 00:27:49,280 gotten that checklist of things and they've passed their Sock 2 523 00:27:49,280 --> 00:27:51,760 certifications. And there's more stringent 524 00:27:51,760 --> 00:27:55,640 versions of that and less stringent potentially every 525 00:27:55,640 --> 00:27:59,680 agent that's out there. You can ask what you can just 526 00:27:59,680 --> 00:28:04,200 like have a sock 2 type certification for the agents 527 00:28:04,200 --> 00:28:08,680 that are out in the wild, but it feels much harder to police. 528 00:28:08,680 --> 00:28:11,640 And it also feels like that might not be the right 529 00:28:11,640 --> 00:28:15,240 abstraction. So I'm a little bit at a loss 530 00:28:15,240 --> 00:28:18,600 for what and how you would audit it. 531 00:28:18,600 --> 00:28:23,200 What I do know is that the idea of just evaling the output, like 532 00:28:23,200 --> 00:28:26,120 you were saying, that's not going to get you anywhere. 533 00:28:26,640 --> 00:28:29,480 Yeah. And I, you know, I, I, I 534 00:28:29,480 --> 00:28:31,840 completely agree. I think we, I think the answer 535 00:28:31,840 --> 00:28:34,040 to exactly how it's going to be done is still being formulated. 536 00:28:34,040 --> 00:28:36,560 And it, and it is very nuanced, though it's still very use case 537 00:28:36,560 --> 00:28:39,960 dependent, right? But I do think responsible AI, 538 00:28:39,960 --> 00:28:43,240 trusting AI is still, it's still quite a challenge, right? 539 00:28:43,240 --> 00:28:45,360 When I speak with organizations and we're talking about a 540 00:28:45,360 --> 00:28:49,040 complex use case, I usually get a mix of opinion of executives 541 00:28:49,040 --> 00:28:50,840 on the table. There are some executives that 542 00:28:50,840 --> 00:28:52,560 are like, yeah, good to go, love it. 543 00:28:52,560 --> 00:28:54,520 Why wouldn't we do it? Let's go right. 544 00:28:54,520 --> 00:28:57,000 There are other executives who are focused on the financial 545 00:28:57,000 --> 00:28:59,680 picture and they can't ignore the benefits that AI can drive. 546 00:29:00,160 --> 00:29:02,440 But then there's always a, you know, executive or two that has 547 00:29:02,440 --> 00:29:05,480 a higher trust quotient. That feels like we've got to be 548 00:29:05,480 --> 00:29:09,160 able to demonstrate the the clarity, the accuracy, the 549 00:29:09,160 --> 00:29:10,720 compliance in order to move forward. 550 00:29:10,720 --> 00:29:13,000 And I think that's going to be an important lock for the 551 00:29:13,000 --> 00:29:14,120 industry. Absolutely. 552 00:29:14,800 --> 00:29:20,400 Another piece that I want to hit on that you mentioned and I tend 553 00:29:20,400 --> 00:29:25,160 to ponder about quite a bit is in these workflow type 554 00:29:25,160 --> 00:29:32,120 scenarios, how some of the optimization efforts can be just 555 00:29:32,120 --> 00:29:37,640 figuring out where to have that human sit in the loop, if at 556 00:29:37,640 --> 00:29:40,840 all. Because maybe the ultimate 557 00:29:40,840 --> 00:29:45,000 optimization is that you can let the agent just go wild and 558 00:29:45,160 --> 00:29:47,720 depending on the use case, like you were saying, if it is this 559 00:29:47,720 --> 00:29:53,320 low stakes kind of high value use case, well, let it run wild 560 00:29:53,320 --> 00:29:56,920 and see if it works. But then if an, if a human's 561 00:29:56,920 --> 00:30:03,480 getting pinged consistently about something that an agent is 562 00:30:03,480 --> 00:30:08,240 doing, is the value really there? 563 00:30:08,240 --> 00:30:10,280 I don't know. I guess if, if I don't have to 564 00:30:10,280 --> 00:30:12,520 do it, I'm going to be happy to have a computer do it. 565 00:30:12,520 --> 00:30:17,120 I'm not doing math by hand, right, even if it so I do 566 00:30:17,120 --> 00:30:21,280 understand that. But the whole idea, if you look 567 00:30:21,280 --> 00:30:24,680 at a workflow and you have these graphs and you have these nodes 568 00:30:25,040 --> 00:30:28,080 and you're thinking, well, an agent can take over this process 569 00:30:28,080 --> 00:30:32,120 and then we're going to have a human in here when it comes to 570 00:30:32,360 --> 00:30:36,360 approving invoices or anything that has to do with money. 571 00:30:36,760 --> 00:30:40,080 What happens with me, I don't know if you've had this happen, 572 00:30:40,120 --> 00:30:45,640 is that when I'm asked to verify stuff that the agent will throw 573 00:30:45,640 --> 00:30:50,480 at me, I just like automatically say, yeah, except let's see what 574 00:30:50,480 --> 00:30:54,160 happens type thing. And so I'm worried that that 575 00:30:54,160 --> 00:30:59,360 type of a habit will get instilled in us as we start 576 00:30:59,360 --> 00:31:02,960 trusting the AI more and more. And next thing you know, we 577 00:31:02,960 --> 00:31:07,400 don't even look at it and we're there as a box that you have to 578 00:31:07,400 --> 00:31:09,920 check. Well, I mean, I will tell you 579 00:31:09,920 --> 00:31:13,240 it's an evolving scale as, as the way I see it, most clients 580 00:31:13,240 --> 00:31:17,600 who are implementing use cases, implementing workflows with AI 581 00:31:17,600 --> 00:31:20,640 today certainly are having a human in the loop, right? 582 00:31:20,640 --> 00:31:23,040 Especially for, for the ones that I would classify anything 583 00:31:23,040 --> 00:31:26,000 as medium or high sensitivity, they're certainly keeping a 584 00:31:26,000 --> 00:31:29,000 human in the loop, but it's also with an evolving scale, right? 585 00:31:29,000 --> 00:31:31,600 The intention is these models get smarter, they learn over 586 00:31:31,600 --> 00:31:33,480 time, they get more effective over time. 587 00:31:33,800 --> 00:31:36,320 So the intention is that, hey, when you first start with that 588 00:31:36,320 --> 00:31:39,960 automation, maybe you have the human look at 50% of what the 589 00:31:39,960 --> 00:31:42,960 models performing, right? And as you start to gain more 590 00:31:42,960 --> 00:31:47,040 trust, as you start to gain more confidence that the matter model 591 00:31:47,040 --> 00:31:50,440 is behaving accurately, and as the model becomes smarter and 592 00:31:50,440 --> 00:31:52,920 more efficient at what it's trying to do, maybe you can get 593 00:31:52,920 --> 00:31:56,040 that 50% down to 20%, down to 10%, right? 594 00:31:56,040 --> 00:31:58,520 And that's how you kind of move up the chain eventually. 595 00:31:58,520 --> 00:32:00,760 I do think just like when you punch in a number to your 596 00:32:00,760 --> 00:32:02,400 calculator today, you trust the answer. 597 00:32:02,400 --> 00:32:04,640 You're not double checking that work today, right? 598 00:32:04,640 --> 00:32:05,960 But eventually, I do think we get there. 599 00:32:05,960 --> 00:32:09,440 But I would advise clients and organizations to think about 600 00:32:09,440 --> 00:32:12,440 that as an evolving scale for the if you've got a sensitive 601 00:32:12,440 --> 00:32:15,000 use case, for a medium sensitive use case, you want to keep a 602 00:32:15,000 --> 00:32:17,160 human in the loop. You want to validate that you're 603 00:32:17,160 --> 00:32:19,560 getting the right answer. And once you've really start to 604 00:32:19,560 --> 00:32:22,680 prove yourself that over a few quarters, then you can start to 605 00:32:22,680 --> 00:32:24,480 reduce the amount of human in the loop that you need. 606 00:32:25,840 --> 00:32:28,240 OK. So I wanted to also talk to you 607 00:32:28,240 --> 00:32:36,240 about this idea of now you have AI working within your company, 608 00:32:36,240 --> 00:32:39,360 you're having success. You can do more with less is 609 00:32:39,360 --> 00:32:40,880 what we're hearing a lot of folks say. 610 00:32:41,520 --> 00:32:45,600 And I was literally just talking to a friend this morning about 611 00:32:45,600 --> 00:32:52,800 how I don't buy into the idea of, oh, we're all going to be 612 00:32:52,800 --> 00:32:56,480 able to do more with less. That means that we're going to 613 00:32:56,760 --> 00:33:02,880 not need as many people to work because if we have all of this 614 00:33:03,640 --> 00:33:07,520 free cash flow that's being spit off now, yeah, we're assuming 615 00:33:07,520 --> 00:33:10,680 that companies are not going to reinvest that into their 616 00:33:10,680 --> 00:33:15,000 companies and either hire more people or potentially lower 617 00:33:15,000 --> 00:33:17,280 prices. Yeah, right. 618 00:33:17,920 --> 00:33:20,080 I agree with you. I think our research indicates 619 00:33:20,160 --> 00:33:23,600 the same, right? We don't we, we certainly see, I 620 00:33:23,600 --> 00:33:28,160 think more than 75% of jobs will require reskilling or 621 00:33:28,160 --> 00:33:31,240 upskilling, but that doesn't mean those jobs are going away 622 00:33:31,240 --> 00:33:33,840 or that hiring will stop. It just means they get reskilled 623 00:33:33,840 --> 00:33:36,120 and upskilled, right? So we certainly see the trend 624 00:33:36,120 --> 00:33:39,200 heading that way. Look, I think about it as the 625 00:33:39,200 --> 00:33:42,360 future of the workforce is humans plus agents working 626 00:33:42,360 --> 00:33:44,320 together, right? The benefits are too great to 627 00:33:44,320 --> 00:33:45,800 ignore. I think we will see that. 628 00:33:46,040 --> 00:33:49,520 But largely I, I, my hypothesis is more closely aligned to 629 00:33:49,520 --> 00:33:53,680 yours. I think the job reduction, the 630 00:33:53,680 --> 00:33:56,600 job displacement is relatively modest at this point. 631 00:33:56,600 --> 00:33:59,600 I think the job reskilling, upskilling is much more 632 00:33:59,600 --> 00:34:01,520 relevant. But I always tell folks that, 633 00:34:01,520 --> 00:34:05,800 look, this might be a controversial thing to say, but 634 00:34:05,800 --> 00:34:08,560 I will, I will say that if you're in a job function, 635 00:34:08,560 --> 00:34:10,639 regardless of kind of what your function is, if you're an 636 00:34:10,639 --> 00:34:13,760 accountant or if you're a salesperson or if you're a 637 00:34:13,760 --> 00:34:17,560 supply chain person. And if you, if, if you're, if 638 00:34:17,560 --> 00:34:21,560 you're kind of at the bottom 25% of other accountants, near the 639 00:34:21,560 --> 00:34:25,440 bottom 25% of other sales leaders, then you probably will 640 00:34:25,440 --> 00:34:27,600 be challenged with AI. Because if you're, if you're, if 641 00:34:27,600 --> 00:34:30,400 you're a low performer in that way, you're likely doing a 642 00:34:30,400 --> 00:34:32,719 series of mundane tasks that AI can replace. 643 00:34:33,480 --> 00:34:36,320 But if you're in the top 50% of those roles, if you're the top 644 00:34:36,320 --> 00:34:39,719 50% of accountants, if you're in the top 50% of sales leaders, 645 00:34:39,719 --> 00:34:43,560 then AI is going to assist you in being able to do a lot more. 646 00:34:44,040 --> 00:34:46,520 So I kind of think about it. As for those who are good at 647 00:34:46,520 --> 00:34:50,520 their jobs, they're going to think about AI as a massive tool 648 00:34:50,520 --> 00:34:53,040 set that they've just gone and now they can go from being good 649 00:34:53,040 --> 00:34:56,360 to great at their jobs and and that will allow them to create 650 00:34:56,360 --> 00:34:58,680 more value. Organizations will hire more for 651 00:34:58,680 --> 00:35:00,640 those skills. We might see a little bit of 652 00:35:00,640 --> 00:35:03,320 rotation from back office to front office, just like you're 653 00:35:03,320 --> 00:35:04,720 talking about. As the back office gets 654 00:35:04,720 --> 00:35:07,560 automated, I can dedicate more time to to revenue course. 655 00:35:07,560 --> 00:35:08,840 So yeah, I think we'll see all of those. 656 00:35:08,840 --> 00:35:12,920 But I think if I'm a, if I'm a person in the labor force, I 657 00:35:12,920 --> 00:35:15,400 always say this to, I don't think AI is going to take your 658 00:35:15,400 --> 00:35:19,680 job, but I think another human that's using AI, if you're not 659 00:35:19,680 --> 00:35:21,880 using AI might take your job, right? 660 00:35:21,880 --> 00:35:24,320 So I kind of the way I kind of think about it as whatever your 661 00:35:24,320 --> 00:35:27,880 function is, you need to be thinking about how can I do more 662 00:35:27,880 --> 00:35:29,800 in my role? How can I do more in my 663 00:35:29,800 --> 00:35:34,680 function, what with using AI? And that's going to not mean 664 00:35:34,680 --> 00:35:36,560 more hours. That's not going to mean more 665 00:35:36,560 --> 00:35:38,000 work. It's actually might mean less 666 00:35:38,000 --> 00:35:40,600 hours and less work, but it's going to mean more output. 667 00:35:40,600 --> 00:35:43,040 And that's how I think. That's how I think humans should 668 00:35:43,040 --> 00:35:47,320 be thinking about that. Yeah, in every job, I have yet 669 00:35:47,320 --> 00:35:52,440 to come across a job that doesn't have just an absurd 670 00:35:52,440 --> 00:35:58,760 amount of rote work that I would happily automate away any day of 671 00:35:58,760 --> 00:36:02,680 the week. And I guess I'm very privileged 672 00:36:02,680 --> 00:36:05,920 to be talking about like this kind of blue collar work. 673 00:36:07,200 --> 00:36:11,960 It's a much different story than if we're or blue collar is a 674 00:36:11,960 --> 00:36:14,760 white collar. I'm getting confused here on 675 00:36:15,360 --> 00:36:18,560 basically I'm sitting behind a keyboard and doing it. 676 00:36:18,800 --> 00:36:22,480 So I'm much more privileged to be able to sit behind a keyboard 677 00:36:22,480 --> 00:36:25,840 and do my work in that way. And I know that there's a lot of 678 00:36:25,840 --> 00:36:28,560 clicking around that I can automate away. 679 00:36:29,960 --> 00:36:39,000 So I will say that I am fully on board with a ton of rote work 680 00:36:39,240 --> 00:36:45,320 that can just get out of what I need to do and allow me to focus 681 00:36:45,320 --> 00:36:51,320 on what is important. And the real key here is this 682 00:36:51,400 --> 00:36:56,920 discernment of knowing what's important and knowing where I 683 00:36:56,920 --> 00:37:02,880 can add that extra value. But I still, I still will like 684 00:37:03,440 --> 00:37:10,440 say this until I'm blue in the face that what I see with 685 00:37:10,440 --> 00:37:14,800 companies is that they're not going to just say, all right, 686 00:37:14,800 --> 00:37:19,920 cool, we now can do the work of 100 people with 50 people. 687 00:37:20,200 --> 00:37:24,400 We're only going to have 50 people now, right? 688 00:37:24,400 --> 00:37:26,840 And then just continue to do the work of 100 people. 689 00:37:26,840 --> 00:37:33,080 No, what I'm seeing is they're going to take 100 people, use 690 00:37:33,080 --> 00:37:38,360 the 100 people, but just have the output now of double that. 691 00:37:38,360 --> 00:37:40,680 So now they have maybe the output of 200 people. 692 00:37:41,320 --> 00:37:45,000 And it's a, it's one of those ones. 693 00:37:45,000 --> 00:37:49,800 Actually, I was just seeing a friend sent me something on one 694 00:37:49,800 --> 00:37:52,560 of these big corporations that did exactly that, where they 695 00:37:53,200 --> 00:37:59,200 realized, oh, because of AI, we are now going to not need these 696 00:37:59,200 --> 00:38:02,720 positions. But instead of this trope that 697 00:38:02,720 --> 00:38:05,760 you're hearing thrown around a lot of, all right, we got rid of 698 00:38:05,960 --> 00:38:09,000 20% of our workforce because we're able to do more with less. 699 00:38:09,000 --> 00:38:12,920 And AI is the reason for these firings. 700 00:38:13,320 --> 00:38:15,720 They said no. What we're going to do is we're 701 00:38:15,720 --> 00:38:19,400 going to purposefully reskill these folks like you're talking 702 00:38:19,400 --> 00:38:20,440 about. Yeah. 703 00:38:21,040 --> 00:38:23,000 I mean, I it's spot on, spot on, right. 704 00:38:23,000 --> 00:38:27,720 Like I, I think about it as AI is going to reset your operating 705 00:38:27,720 --> 00:38:30,320 model, that's how you should think about as an organization. 706 00:38:30,320 --> 00:38:32,920 It is going to reset your operating model where as we 707 00:38:32,920 --> 00:38:35,800 automate the workflows, that means the operating model 708 00:38:35,800 --> 00:38:37,520 changes. And when the operating model 709 00:38:37,520 --> 00:38:41,760 changes, that means people's jobs change, not that those jobs 710 00:38:41,760 --> 00:38:43,720 go away. Those jobs change. 711 00:38:43,720 --> 00:38:47,400 The operating model has changed. I might need less people doing 712 00:38:47,400 --> 00:38:49,680 one function, but I need more people doing a different 713 00:38:49,680 --> 00:38:51,760 function. And that other function creates 714 00:38:51,760 --> 00:38:53,560 more value. And the people that I have in 715 00:38:53,560 --> 00:38:56,600 that function can use a bunch of tools for the automated mundane 716 00:38:56,600 --> 00:38:58,880 tasks, right? So I, I completely agree. 717 00:38:58,880 --> 00:39:02,320 I think you've got to think about the operating model is 718 00:39:02,320 --> 00:39:04,040 changing. I'm essentially now an 719 00:39:04,040 --> 00:39:07,880 organization that has humans and agents working together. 720 00:39:08,400 --> 00:39:10,640 And when I think about that equation, when I think about 721 00:39:10,640 --> 00:39:14,320 that peanut butter and Jelly sandwich, the operating model of 722 00:39:14,320 --> 00:39:16,520 that should be different than the operating model today. 723 00:39:16,520 --> 00:39:20,000 And that will that will create the job change the job 724 00:39:20,000 --> 00:39:21,960 reskilling that I think we're going to see in the market. 725 00:39:22,960 --> 00:39:26,920 Yeah, I was just on a podcast a few weeks ago and the title of 726 00:39:26,920 --> 00:39:31,360 the podcast was we're all managers now, because we've all, 727 00:39:31,520 --> 00:39:36,480 we now have these, the agent that we can go and spin off and 728 00:39:36,480 --> 00:39:40,000 if we manage them correctly, hopefully it makes our lives 10 729 00:39:40,000 --> 00:39:44,160 times easier and we're still able to get everything done and 730 00:39:44,160 --> 00:39:51,800 more Exactly, exactly right. So I mean anything that I should 731 00:39:52,000 --> 00:39:56,080 be asking you that I haven't? No, I think, I think we've hit 732 00:39:56,080 --> 00:39:57,800 out some. I think we've hit on the right 733 00:39:57,800 --> 00:40:01,120 topics, right. I I guess I would just part with 734 00:40:01,440 --> 00:40:04,800 transformation. Over experimentation, right? 735 00:40:05,560 --> 00:40:07,680 I think the first topic we talked about how to get really 736 00:40:07,680 --> 00:40:12,080 good value from AI, the best examples of that, the most value 737 00:40:12,080 --> 00:40:14,760 I'm seeing clients generate is when they're applying it to an 738 00:40:14,760 --> 00:40:16,880 end to end workflow transformation. 739 00:40:17,200 --> 00:40:20,760 Not an experiment here, not a use case here, and a POC there. 740 00:40:21,120 --> 00:40:23,840 So that I would kind of wrap us up, but that would be point 741 00:40:23,840 --> 00:40:28,960 number one point #2 I would say governance, cost governance, 742 00:40:29,040 --> 00:40:32,960 risk governance, all those those governance is, is a key lever of 743 00:40:32,960 --> 00:40:35,760 success, right? So and it's separating leaders 744 00:40:35,760 --> 00:40:38,280 from laggards, right? So if you've got the right 745 00:40:38,280 --> 00:40:41,280 governance set up, fin op set up it, then then I think you're 746 00:40:41,280 --> 00:40:44,680 going to have some success. And then, you know, lastly, I 747 00:40:44,680 --> 00:40:49,400 think mindset matters, especially early on, a lot of 748 00:40:49,400 --> 00:40:51,200 clients are seeking to make a shift. 749 00:40:51,200 --> 00:40:53,880 They feel like it's going to be an organizational shift and 750 00:40:53,880 --> 00:40:57,040 operating model shift. And it is, I think it's very 751 00:40:57,040 --> 00:41:00,080 important for leaders, but all the way up and down the 752 00:41:00,120 --> 00:41:04,360 corporate work pyramid to really adopt this mindset of of course 753 00:41:04,360 --> 00:41:06,400 we're going to innovate, Of course we're going to relearn. 754 00:41:06,760 --> 00:41:10,480 You know, I think the, the, this is the age of continuous 755 00:41:10,480 --> 00:41:13,320 learning, right? If you went to school one year 756 00:41:13,320 --> 00:41:16,320 ago or if you went to school 20 years ago, all good. 757 00:41:16,840 --> 00:41:19,240 But this is the age of continuous learning. 758 00:41:19,280 --> 00:41:21,920 Everyone, whether you're 20 years in the game or one years 759 00:41:21,920 --> 00:41:25,320 in the game, is going to need to find a way to continue to learn 760 00:41:25,560 --> 00:41:28,640 new capabilities, new tools, new work automations that are 761 00:41:28,640 --> 00:41:30,880 coming. Being able to learn those 762 00:41:30,880 --> 00:41:34,400 things, apply those things to my job and and make my work output 763 00:41:34,400 --> 00:41:37,520 even stronger and better. That's the secret to success 764 00:41:37,520 --> 00:41:40,720 now. And the beauty is that you can 765 00:41:40,720 --> 00:41:44,400 use AI to learn about AI. And how to better. 766 00:41:45,400 --> 00:41:47,520 Yeah, Yeah. Look, I joke around about this 767 00:41:47,520 --> 00:41:49,400 up. It has never been easier to 768 00:41:49,400 --> 00:41:51,720 learn. This is the age of information 769 00:41:51,720 --> 00:41:54,160 and the past. People had to go travel and and 770 00:41:54,160 --> 00:41:57,040 walk to the top of a mountain to sit down with a guru and learn 771 00:41:57,040 --> 00:41:59,320 something. Then people had to pay thousands 772 00:41:59,320 --> 00:42:01,880 of thousands of dollars to attend a university to go learn 773 00:42:01,880 --> 00:42:04,160 something. It has never been easier than 774 00:42:04,160 --> 00:42:07,160 today to learn, right. There's so much information 775 00:42:07,160 --> 00:42:10,840 available so it's never been easier to learn, but the onus is 776 00:42:10,840 --> 00:42:15,080 on all of us to always be learning and always be applying 777 00:42:15,280 --> 00:42:17,720 and if we can do that then then we're going to be successful in 778 00:42:17,720 --> 00:42:20,800 the workforce. All right, Manish, I appreciate 779 00:42:20,800 --> 00:42:24,240 you man. So great talking to you, so much 780 00:42:24,240 --> 00:42:25,160 fun. Thank you for having me.