1 00:00:00,040 --> 00:00:05,480 What are AI agents and will they really change the world as some 2 00:00:05,480 --> 00:00:11,560 folks claim? Today on episode 868, we're 3 00:00:11,560 --> 00:00:17,600 taking an advanced look at agentic AI with a prominent VC 4 00:00:17,600 --> 00:00:20,960 investor. Praveen Akiraju is managing 5 00:00:20,960 --> 00:00:25,080 director at the venture capital firm Insight Partners, where 6 00:00:25,080 --> 00:00:29,600 he's focused on AI agents. This is his second appearance on 7 00:00:29,600 --> 00:00:33,320 CX, so talk to discuss Agentic AI Praveen. 8 00:00:33,680 --> 00:00:35,640 Welcome back. Thank you, Michael. 9 00:00:35,640 --> 00:00:38,720 Excited to be here. What are AI agents? 10 00:00:38,880 --> 00:00:44,280 You can think of AI agents as application software that is 11 00:00:44,280 --> 00:00:49,040 able to understand a user's intent and be able to reason on 12 00:00:49,040 --> 00:00:53,240 that user's intent, come up with a plan and execute the plan. 13 00:00:53,720 --> 00:00:56,080 At a broad level, that's really what an AI agent is. 14 00:00:56,360 --> 00:01:00,120 So today, if you kind of look at how that task is done, you 15 00:01:00,120 --> 00:01:03,280 typically have an application, you have a human who comes up 16 00:01:03,280 --> 00:01:05,640 with the plan and executes the plan, right? 17 00:01:05,880 --> 00:01:08,760 So the, the transition that we are in the process of making 18 00:01:08,760 --> 00:01:12,160 with AI is that we're now offloading a lot of that 19 00:01:12,160 --> 00:01:15,880 planning process and the execution process to these 20 00:01:15,880 --> 00:01:20,000 agents and allowing them to use applications as humans would, 21 00:01:20,400 --> 00:01:21,840 right? So that's really the paradigm 22 00:01:21,840 --> 00:01:24,000 shift that we are in the process of making. 23 00:01:24,000 --> 00:01:26,120 That's why AI agents are so exciting. 24 00:01:26,600 --> 00:01:32,080 What is the role of LL miss large language models on this 25 00:01:32,080 --> 00:01:37,640 whole agentic AI world? It's important to understand 26 00:01:37,640 --> 00:01:42,000 that AI agents are not just all about large language models, 27 00:01:42,040 --> 00:01:43,760 right? You have to think about AI 28 00:01:43,760 --> 00:01:50,040 agents as essentially another software application which now 29 00:01:50,240 --> 00:01:54,680 incorporates large language models in in areas where they 30 00:01:54,680 --> 00:01:57,680 add the most value. So let's take a step back and 31 00:01:57,680 --> 00:02:00,720 think about sort of how applications were previously 32 00:02:00,720 --> 00:02:03,760 defined. You used to have a database, a 33 00:02:03,760 --> 00:02:07,200 system of record. You used to have a software 34 00:02:07,600 --> 00:02:09,639 workflow that was built on top of this. 35 00:02:09,639 --> 00:02:10,840 And then you had a user interface. 36 00:02:10,840 --> 00:02:14,200 That's typically what your classical application looks like 37 00:02:14,200 --> 00:02:19,400 today in the SAS context. So the power of large language 38 00:02:19,400 --> 00:02:24,280 models is that they are able to insert into the stack at various 39 00:02:24,280 --> 00:02:28,640 different points and be able to kind of dramatically improve the 40 00:02:28,640 --> 00:02:33,360 productivity and the capability of these application software 41 00:02:33,360 --> 00:02:35,280 models. So you can think of large 42 00:02:35,280 --> 00:02:38,440 language models as playing different roles in in in the 43 00:02:38,440 --> 00:02:40,440 sort of new paradigm we call AI agents. 44 00:02:40,880 --> 00:02:43,480 The first layer would be the user interface itself. 45 00:02:44,360 --> 00:02:47,080 You know, we are all now used to the ChatGPT interface where we 46 00:02:47,080 --> 00:02:50,200 go in and you essentially type in something or you can even 47 00:02:50,200 --> 00:02:53,720 speak to it and you know, it's able to kind of understand that 48 00:02:53,720 --> 00:02:57,920 context and comes back with a an answer as opposed to, you know, 49 00:02:57,920 --> 00:03:01,040 set a reference points or the 10 blue links, right. 50 00:03:01,360 --> 00:03:04,200 So similarly, in the application context, you have a user 51 00:03:04,200 --> 00:03:08,040 interface now where the user can interact and basically say, Hey, 52 00:03:08,040 --> 00:03:12,000 show me the data for a particular region for a 53 00:03:12,000 --> 00:03:16,520 particular product and it and that LLM is then is able to take 54 00:03:16,520 --> 00:03:20,200 that input and synthesize that and understand the context of it 55 00:03:20,560 --> 00:03:23,440 as opposed to the user having to go and figure that out, right. 56 00:03:23,440 --> 00:03:25,760 So that's one layer. The second layer, which is 57 00:03:25,760 --> 00:03:28,640 really important layer and it's kind of in some ways sort of 58 00:03:28,640 --> 00:03:32,840 almost the core of an AI agentic architecture is a reasoning 59 00:03:32,840 --> 00:03:37,360 layer, right, which is the ability to take the task given 60 00:03:37,360 --> 00:03:41,120 to the agent and break it down into a set of sub tasks. 61 00:03:41,240 --> 00:03:45,800 So for example, if you're saying let's go build a research report 62 00:03:45,800 --> 00:03:50,200 on a particular stock, right? So if that is the the, the 63 00:03:50,200 --> 00:03:54,320 prompt that the user provides, the AI agent then takes it down 64 00:03:54,320 --> 00:03:56,920 and says, OK, how do I do build a research report on this 65 00:03:56,920 --> 00:03:59,560 particular stock? I have to go to, you know, Yahoo 66 00:04:00,280 --> 00:04:02,960 Finance or one of these public websites, get all the 67 00:04:02,960 --> 00:04:06,760 information about that stock, be able to kind of synthesize all 68 00:04:06,760 --> 00:04:10,120 of that information, create a report and then publish a 69 00:04:10,120 --> 00:04:11,960 report. So it breaks that down to 70 00:04:11,960 --> 00:04:14,080 specific tasks. That's the reasoning layer, 71 00:04:14,080 --> 00:04:16,760 right? The next layer where the LMS 72 00:04:16,760 --> 00:04:21,880 play a role is the ability to go out and dynamically query 73 00:04:21,880 --> 00:04:24,400 information. So in the past sort of 74 00:04:24,400 --> 00:04:27,400 application context where things were static, the application 75 00:04:27,480 --> 00:04:31,200 could only look at things where it had an API called built in. 76 00:04:31,200 --> 00:04:34,120 So if it had access to a particular data store and had an 77 00:04:34,120 --> 00:04:36,760 API call, it would go pick that API call. 78 00:04:37,280 --> 00:04:40,560 Here we have an LLM that can reason and say, wait a minute, 79 00:04:40,680 --> 00:04:43,600 this information is available to me externally. 80 00:04:43,840 --> 00:04:46,440 This information is available to me through this API call 81 00:04:46,440 --> 00:04:48,640 internally. And I need to get both of these 82 00:04:48,640 --> 00:04:51,440 pieces of information in order for me to execute my task which 83 00:04:51,440 --> 00:04:52,880 is to build this research report. 84 00:04:53,160 --> 00:04:57,840 So let me launch a a web search API. 85 00:04:58,040 --> 00:05:04,000 Let me also launch an API to my internal research store, put 86 00:05:04,040 --> 00:05:06,120 those data points together, right? 87 00:05:06,120 --> 00:05:10,320 So it's, it's the LLM has the ability to understand what 88 00:05:10,320 --> 00:05:14,840 sources of data needs to go get the information from for it to 89 00:05:14,840 --> 00:05:18,280 execute on the task. Now once it does that right, 90 00:05:18,360 --> 00:05:21,840 they, the application software synthesizes it and then you come 91 00:05:21,840 --> 00:05:22,760 to the next step. It is. 92 00:05:22,760 --> 00:05:25,800 You have the output. A lot of really well designed 93 00:05:25,800 --> 00:05:29,840 agents have strong evaluation loops and this is really, really 94 00:05:29,840 --> 00:05:32,480 important, right? Once the agent actually comes up 95 00:05:32,480 --> 00:05:36,600 with the output, you test that output against, you know what 96 00:05:36,600 --> 00:05:39,120 essentially is the gold standard that you set up and say this is 97 00:05:39,120 --> 00:05:40,520 how a research report looks like. 98 00:05:40,760 --> 00:05:45,320 So the agent compares the output it got with the gold standard 99 00:05:45,320 --> 00:05:48,280 and says, does this match it? And then it corrects itself. 100 00:05:48,280 --> 00:05:50,600 Did it matches that standard, right. 101 00:05:50,960 --> 00:05:54,840 So as you can see, you inserting the large language model at 102 00:05:54,840 --> 00:05:59,760 different layers of the AI agent in order for it to be able to 103 00:06:00,080 --> 00:06:04,280 make the entire process a lot more dynamic and interactive. 104 00:06:04,560 --> 00:06:07,880 And that's basically what is different between a classic 105 00:06:07,880 --> 00:06:11,280 application flow and AI agent application flow. 106 00:06:11,760 --> 00:06:19,000 So we have this reasoning, we have access to this broad body 107 00:06:19,000 --> 00:06:23,360 of human knowledge, and of course that is different from 108 00:06:23,520 --> 00:06:29,960 traditional applications. How accurate are these agents 109 00:06:30,240 --> 00:06:32,400 today? What is the state-of-the-art? 110 00:06:32,400 --> 00:06:35,920 How useful are these agents in practice? 111 00:06:36,080 --> 00:06:38,400 We are quite early in this journey. 112 00:06:38,480 --> 00:06:41,600 You know, it feels like when you read the press you have, you 113 00:06:41,600 --> 00:06:44,040 know, AI agents are everywhere, right? 114 00:06:44,720 --> 00:06:47,400 Or if you look go look at a website today for any software 115 00:06:47,400 --> 00:06:52,040 company, essentially AI agents are the way they are now 116 00:06:52,040 --> 00:06:56,240 expressing their products. However, I think in in terms of 117 00:06:56,360 --> 00:07:01,520 the maturity of the technology, both from the the important 118 00:07:01,520 --> 00:07:04,360 question you talked about how reliable they are, but also in 119 00:07:04,360 --> 00:07:09,080 terms of on the customer end user side, the experience of the 120 00:07:09,080 --> 00:07:13,200 end users in terms of the deployments and the scalability 121 00:07:13,200 --> 00:07:16,120 and reliability of these. We're still in, in the early 122 00:07:16,120 --> 00:07:19,360 days now. I will say that there's a 123 00:07:19,360 --> 00:07:23,040 spectrum of your agents, right? So in, in, in the spectrum 124 00:07:23,040 --> 00:07:25,400 essentially, you know, at one end of the spectrum, you could 125 00:07:25,400 --> 00:07:28,760 have these consumer agents which are really focused on 126 00:07:28,760 --> 00:07:31,840 individuals. You know, we were talking about 127 00:07:31,840 --> 00:07:34,680 this before the show started opening. 128 00:07:34,680 --> 00:07:38,480 I just launched their consumer agent called Operator. 129 00:07:38,560 --> 00:07:40,720 Very interesting. You know, it's for us as 130 00:07:40,720 --> 00:07:43,760 individuals part of the pro plan and the other end of the 131 00:07:43,760 --> 00:07:49,160 spectrum are essentially the next phase of evolution of RPA 132 00:07:49,440 --> 00:07:52,480 or robotic process automation right in the enterprise. 133 00:07:52,480 --> 00:07:56,560 So agents that take on complex enterprise work flows, right. 134 00:07:57,040 --> 00:08:02,360 So we have tremendous amount of activity all along the spectrum 135 00:08:02,360 --> 00:08:06,840 in terms of, you know, start-ups as well as in comments advancing 136 00:08:06,840 --> 00:08:09,560 the state-of-the-art right experimenting the I agents. 137 00:08:09,880 --> 00:08:14,640 So we are in the early days primarily because there are a 138 00:08:14,640 --> 00:08:17,520 few things that we are still trying to figure out right now. 139 00:08:17,520 --> 00:08:20,960 Large language models by themselves are non 140 00:08:20,960 --> 00:08:23,720 deterministic. And, and I think what I mean by 141 00:08:23,720 --> 00:08:27,560 that is that and we all, we again, we've all experienced 142 00:08:27,560 --> 00:08:29,640 this. When you ask ChatGPT a question 143 00:08:29,840 --> 00:08:32,480 a certain way and you ask the same question in a slightly 144 00:08:32,480 --> 00:08:34,559 different way, you may get a different answer. 145 00:08:34,720 --> 00:08:38,159 Right now that's getting better with some of the newer models, 146 00:08:38,159 --> 00:08:41,240 particularly some of the the reasoning models that are able 147 00:08:41,240 --> 00:08:44,560 to correct themselves. But the fact of the matter is 148 00:08:44,560 --> 00:08:47,480 that the core of the large language model is this sort of 149 00:08:47,480 --> 00:08:51,920 non determinism. And so a lot of the AI agentic 150 00:08:52,280 --> 00:08:55,800 designers today, builders today are working on in what we call 151 00:08:55,800 --> 00:08:59,120 scaffolding in order for us to essentially take the power of 152 00:08:59,120 --> 00:09:01,600 these large language models and harness them. 153 00:09:01,600 --> 00:09:04,800 At the same time, making sure that we're able to understand 154 00:09:04,800 --> 00:09:08,760 that the non determinism exists and we have the right 155 00:09:08,800 --> 00:09:11,200 architecture to be able to handle that. 156 00:09:11,600 --> 00:09:15,640 So you get a reliable, consistent and most importantly, 157 00:09:15,640 --> 00:09:19,320 scalable AI agent. Where are we there? 158 00:09:19,560 --> 00:09:26,840 Because of course that non determinism, if you press enter, 159 00:09:26,840 --> 00:09:30,320 submit again on a prompt, it's going to give you a different 160 00:09:30,320 --> 00:09:33,040 answer. Is great if you're wanting help 161 00:09:33,040 --> 00:09:37,320 writing an outline or some type of summary of a document because 162 00:09:37,320 --> 00:09:38,960 there's different perspectives you can take. 163 00:09:39,320 --> 00:09:44,440 But if you want it to run a task like book me an airline ticket, 164 00:09:44,720 --> 00:09:48,000 you don't want a lot of stuff all over the map. 165 00:09:48,000 --> 00:09:50,440 You just want one thing done. You want that ticket to be 166 00:09:50,440 --> 00:09:53,240 booked in the right place with the most efficiency and so 167 00:09:53,240 --> 00:09:54,720 forth. So it becomes a big problem I 168 00:09:54,720 --> 00:09:57,680 would think. Yeah, and I think this is again, 169 00:09:57,720 --> 00:10:01,320 a central design consideration, right? 170 00:10:01,320 --> 00:10:05,680 When you're building AI agents. And, and I think we can maybe 171 00:10:05,680 --> 00:10:07,680 break this down into like 3 parts. 172 00:10:07,720 --> 00:10:09,920 So the first part is the task itself. 173 00:10:10,840 --> 00:10:13,080 And you gave an example of a task, right? 174 00:10:13,240 --> 00:10:17,480 So how, how important is it for you to get the task? 175 00:10:17,480 --> 00:10:20,440 Absolutely. Right Now in ChatGPT, you know, 176 00:10:20,440 --> 00:10:22,520 if it gives you a slightly different answer, it's like 177 00:10:22,520 --> 00:10:25,840 search, right is you're getting information, you're not 178 00:10:25,840 --> 00:10:29,760 essentially making a decision. So you're OK with some level of 179 00:10:29,760 --> 00:10:33,720 non determinism because the human mind is able to correct 180 00:10:33,720 --> 00:10:35,960 for that, right? The other end of the spectrum, 181 00:10:35,960 --> 00:10:40,600 if you're essentially betting on this AI agent to execute a task 182 00:10:40,600 --> 00:10:46,680 consistently, it could be, you know, a enterprise workflow and 183 00:10:46,840 --> 00:10:51,240 a back end workflow, right? It could be code generation or, 184 00:10:51,800 --> 00:10:54,800 you know, you're essentially a customer service interaction 185 00:10:54,800 --> 00:10:57,520 agent. You cannot have that level of 186 00:10:57,520 --> 00:11:00,360 non determinism. So, so that's one thing like how 187 00:11:00,360 --> 00:11:03,240 do you define the task? And I think that's a, a key 188 00:11:03,240 --> 00:11:06,120 question to ask. The second aspect that you want 189 00:11:06,120 --> 00:11:08,680 to look at is, OK, now that you say, let's say you, you have a 190 00:11:08,680 --> 00:11:10,360 task that you need to be accurate. 191 00:11:10,480 --> 00:11:13,400 And it's also again, important to understand that AI agents are 192 00:11:13,400 --> 00:11:17,600 not all about just LLMS. There's a lot of existing 193 00:11:17,600 --> 00:11:21,600 software, there's a lot of reusing, you know, machine 194 00:11:21,600 --> 00:11:24,040 learning models, predictive models, which are deterministic, 195 00:11:24,320 --> 00:11:28,160 right, as well as the classic things that you as a software 196 00:11:28,160 --> 00:11:31,400 engineer do to build applications that go into making 197 00:11:31,400 --> 00:11:36,880 an AI agent, right? So I'll say this again, AILLMS 198 00:11:37,080 --> 00:11:39,520 are a tool. They're not the product, right? 199 00:11:39,520 --> 00:11:41,040 They're not the agent, they're a tool. 200 00:11:41,240 --> 00:11:43,840 So you have to understand, as with any tool, what the 201 00:11:43,840 --> 00:11:46,800 capabilities are. So how do you so that's, that's 202 00:11:46,800 --> 00:11:48,120 the second piece. It's like when you think about 203 00:11:48,120 --> 00:11:51,080 the AI agent it you have to think about leveraging the 204 00:11:51,080 --> 00:11:52,760 right, the LM in the right places. 205 00:11:53,040 --> 00:11:57,080 Then the third piece is OK, so now you've figured out, OK, my 206 00:11:57,080 --> 00:12:01,080 large language model is going to help me with building a plant, 207 00:12:01,080 --> 00:12:03,360 for example, the reasoning layer that we talked about earlier. 208 00:12:03,880 --> 00:12:09,520 So most AI agents today essentially propose a plan and 209 00:12:09,520 --> 00:12:11,840 you have a human in the loop that then approves the plan. 210 00:12:12,000 --> 00:12:15,720 So a good example of this is a lot of the coding, you know, Co 211 00:12:15,720 --> 00:12:18,040 pilots and coding agents that you have today. 212 00:12:18,280 --> 00:12:20,760 They're very again, and particularly with the the the 213 00:12:20,760 --> 00:12:24,640 cloud Sonnet 3.5, that was like, I think a step function jump in 214 00:12:24,640 --> 00:12:27,360 the ability of large language models to accurately generate 215 00:12:27,360 --> 00:12:29,880 code, right? The way they work is 216 00:12:29,880 --> 00:12:32,720 essentially, you know, the programmers interacting with the 217 00:12:32,720 --> 00:12:35,280 large language model. It proposes a plan which is 218 00:12:35,280 --> 00:12:39,440 approved and edited by the programmer before it actually 219 00:12:39,440 --> 00:12:42,760 goes out and executes it, right. So the so-called sort of user in 220 00:12:42,760 --> 00:12:46,560 the loop, human in the loop is a very critical design component 221 00:12:46,560 --> 00:12:50,400 today in AI agents, particularly in that planning layer right 222 00:12:50,840 --> 00:12:52,720 now, there are other things you can do. 223 00:12:52,760 --> 00:12:54,960 And we talked about part of this, which is, you know, what 224 00:12:54,960 --> 00:12:57,360 we call like a, a reflection loop. 225 00:12:57,360 --> 00:13:01,320 So once the output of the agent comes out, you tested against 226 00:13:01,320 --> 00:13:05,200 another large language model which is trained with the right 227 00:13:05,200 --> 00:13:07,720 output. So the model, the agent 228 00:13:07,720 --> 00:13:10,920 essentially tests itself to say, did I get this right? 229 00:13:11,720 --> 00:13:15,480 And it's able to kind of think on whether the output is correct 230 00:13:15,520 --> 00:13:18,720 and then make those changes and go back and iterate again, 231 00:13:18,720 --> 00:13:20,840 right? So these kind of reflection 232 00:13:20,840 --> 00:13:24,400 loops, the way you build evaluations, which is how do you 233 00:13:24,400 --> 00:13:28,360 determine the output is correct? And using that data to 234 00:13:28,360 --> 00:13:31,720 continuously improve the AI agent is again part of the 235 00:13:31,720 --> 00:13:33,960 design process. So I gave you like 3 different 236 00:13:33,960 --> 00:13:35,480 things where you have to consider. 237 00:13:35,480 --> 00:13:38,400 There's a lot more we can go into in terms of depth, but at a 238 00:13:38,400 --> 00:13:42,760 broad level, you know, the take away is that you can and design 239 00:13:43,000 --> 00:13:47,800 AI agents to be deterministic. Assuming that you understand the 240 00:13:47,800 --> 00:13:51,640 task right, you provided the right scaffolding, the 241 00:13:51,640 --> 00:13:56,280 evaluation loops, and you essentially involve the human in 242 00:13:56,280 --> 00:13:58,880 the loop. Today, AI agents work well when 243 00:13:58,880 --> 00:14:01,040 you have a human in the loop, and I think that's going to be 244 00:14:01,040 --> 00:14:04,880 the case particularly on this end of the spectrum where output 245 00:14:04,880 --> 00:14:08,480 needs to be, decisions need to be a lot more accurate. 246 00:14:09,280 --> 00:14:13,680 Subscribe to the CXO Talk newsletter so you can join our 247 00:14:13,680 --> 00:14:18,280 community and we can tell you about our upcoming shows, which 248 00:14:18,280 --> 00:14:22,320 we have great ones. We have questions that are 249 00:14:22,320 --> 00:14:27,280 stacking up on LinkedIn, So let's jump to a few questions 250 00:14:27,280 --> 00:14:31,360 right now. And the first question is from 251 00:14:31,800 --> 00:14:35,720 Ravi Karkara. And he says, where do you see 252 00:14:35,720 --> 00:14:40,360 American universities on creating a skills workforce for 253 00:14:40,360 --> 00:14:46,960 the AI driven world economy? How will they learn to work with 254 00:14:46,960 --> 00:14:53,720 and deploy AI agents? I consider AI large language 255 00:14:53,720 --> 00:14:59,720 models to be a tool, right? Just as you know, we had cloud, 256 00:14:59,720 --> 00:15:03,840 mobile, all these fundamental platform shifts, the AI and 257 00:15:03,840 --> 00:15:07,240 large language models are a tool to help us accomplish our task. 258 00:15:07,240 --> 00:15:12,560 So what I mean by that is it still is important for you to 259 00:15:12,560 --> 00:15:16,640 deeply understand what is the problem you're trying to solve 260 00:15:16,680 --> 00:15:19,360 with this particular tool. Are you trying to book an 261 00:15:19,360 --> 00:15:21,960 airline ticket like Michael you had mentioned earlier? 262 00:15:22,400 --> 00:15:26,360 Are you trying to execute a payroll function in in an 263 00:15:26,360 --> 00:15:30,080 enterprise? Are you trying to respond to a 264 00:15:30,080 --> 00:15:34,960 customer support question? So understanding the actual 265 00:15:34,960 --> 00:15:39,000 task, which is what, you know, essentially good product 266 00:15:39,000 --> 00:15:43,960 management is, is foundational to understanding how this tool, 267 00:15:43,960 --> 00:15:48,240 this new tool, much more powerful, obviously dramatically 268 00:15:48,360 --> 00:15:51,400 more impactful than anything that we've seen in the past, is 269 00:15:51,400 --> 00:15:58,000 going to change how we as a we as basically educators, as 270 00:15:58,000 --> 00:16:01,600 knowledge workers or as consumers leverage AI. 271 00:16:01,720 --> 00:16:05,600 So in terms of, you know, how universities approach this, it 272 00:16:05,600 --> 00:16:07,800 depends on sort of how you participate in this. 273 00:16:07,800 --> 00:16:11,800 I mean, to one end, the spectrum, you know, the, the 274 00:16:11,800 --> 00:16:15,320 education in, in sciences and math and, and good grounding in 275 00:16:15,320 --> 00:16:18,320 that helps you be part of the design process of this. 276 00:16:18,520 --> 00:16:20,680 On the other end of the spectrum, you know, if you're 277 00:16:20,680 --> 00:16:25,760 more business oriented, you know, a deep understanding of 278 00:16:25,760 --> 00:16:29,880 your problem, how and essentially how the tool works. 279 00:16:29,880 --> 00:16:32,600 I mean, we all are experimenting with GBT today. 280 00:16:33,240 --> 00:16:35,480 I use it differently, My daughter uses it differently. 281 00:16:36,200 --> 00:16:38,680 You know, I hope hopefully when none of her teachers are 282 00:16:38,680 --> 00:16:41,880 listening, but you know, she uses it sometimes to do help her 283 00:16:41,880 --> 00:16:44,880 with her homework, right? And so she's learning as part of 284 00:16:44,880 --> 00:16:48,120 that process just as I'm learning to use it for my use 285 00:16:48,120 --> 00:16:49,920 cases. And so are all of us, right? 286 00:16:50,200 --> 00:16:53,440 So I think it's a tool that we experiment with, but a deep 287 00:16:53,440 --> 00:16:57,160 understanding the problem and figuring out how to understand 288 00:16:57,520 --> 00:16:59,720 how to use this tool is, is foundational. 289 00:16:59,720 --> 00:17:03,360 So, you know, product thinking is I think is another key 290 00:17:03,360 --> 00:17:06,640 aspect, which I think is something we should emphasize. 291 00:17:07,079 --> 00:17:10,760 You know, if you're not deep in the algorithms in math, you 292 00:17:10,760 --> 00:17:13,599 still have a tremendous role to play by understanding sort of 293 00:17:13,720 --> 00:17:16,160 how do you build products, how do you solve customer problems, 294 00:17:16,440 --> 00:17:18,920 right? And I think the third aspect of 295 00:17:18,920 --> 00:17:21,280 this is there's a huge human element to this. 296 00:17:21,599 --> 00:17:24,599 We just talked about sort of how do AI agents function today? 297 00:17:24,599 --> 00:17:27,480 And one of the things I said is human in the loop, right? 298 00:17:28,480 --> 00:17:32,800 And you know, real AI agents get better at reasoning probably. 299 00:17:32,800 --> 00:17:34,800 I mean, the O3 model is amazing, right? 300 00:17:34,800 --> 00:17:37,600 And you've seen huge advances just in the last, you know, few 301 00:17:37,600 --> 00:17:40,480 weeks last, you know, later part, later part of last year in 302 00:17:40,480 --> 00:17:42,240 terms of the reasoning capability. 303 00:17:42,440 --> 00:17:45,920 However, throughout the steps that you have to think about 304 00:17:47,120 --> 00:17:51,320 where the role of the human is and at is it an evaluation 305 00:17:51,320 --> 00:17:52,760 function? Is it a reasoning function? 306 00:17:53,160 --> 00:17:56,680 And so humans are always going to be involved and being able to 307 00:17:56,680 --> 00:18:00,240 sort of understand and engage with the technology is 308 00:18:00,240 --> 00:18:02,080 effectively the most important thing, right? 309 00:18:02,720 --> 00:18:06,880 There is a massive human side of this as much as there's a 310 00:18:06,880 --> 00:18:10,000 technology side of this, right? So those are some of the areas. 311 00:18:10,280 --> 00:18:11,880 And, you know, I'm not an educator. 312 00:18:11,880 --> 00:18:14,640 So as I approach this like, that's the way I think about it. 313 00:18:15,080 --> 00:18:20,440 This is from Suresh Babu Madala and he says do we need to train 314 00:18:20,440 --> 00:18:26,400 agents or do agents interpret the question as a step by step 315 00:18:26,400 --> 00:18:27,960 task? That's a great question. 316 00:18:28,400 --> 00:18:32,160 You do need to train agents. Again, there's a spectrum of 317 00:18:32,160 --> 00:18:34,840 what these agents can do in terms of simple tasks, which is 318 00:18:34,840 --> 00:18:37,560 really complex tasks. And the level of training 319 00:18:37,560 --> 00:18:41,640 obviously will depend on what you're expecting the agents to 320 00:18:41,640 --> 00:18:45,120 do. You know, the first phase, as 321 00:18:45,120 --> 00:18:47,800 you're inserting the agent into a particular task, there's a 322 00:18:47,800 --> 00:18:49,840 certain amount of grounding that needs to happen. 323 00:18:50,000 --> 00:18:53,440 And the grounding typically is, you know, connecting it to the 324 00:18:53,440 --> 00:18:55,680 right data sources, connecting it to the right application 325 00:18:55,680 --> 00:18:59,280 sources, connecting it and, and understanding, making sure that 326 00:18:59,280 --> 00:19:02,920 understand and grounded in the policies of your particular use 327 00:19:02,920 --> 00:19:05,160 case or your particular enterprise. 328 00:19:05,160 --> 00:19:09,560 So that's the initial part, however training, you know, just 329 00:19:09,560 --> 00:19:12,560 like we as human beings, right, we're constantly learning, 330 00:19:12,560 --> 00:19:13,840 right? And if you, let's say you 331 00:19:13,840 --> 00:19:17,720 onboard a new employee, you know, fresh college grad, they 332 00:19:17,720 --> 00:19:20,080 are in a constant training process, right? 333 00:19:20,080 --> 00:19:23,840 They learn you have somebody who watches their output and you 334 00:19:23,840 --> 00:19:26,560 give them feedback, right? And you hopefully you observe 335 00:19:26,560 --> 00:19:30,040 them continuing to improve. It's the same exact thing for AI 336 00:19:30,040 --> 00:19:32,120 agents. That's why these evaluation 337 00:19:32,120 --> 00:19:36,080 loops are so important aspect of designing an AI agent. 338 00:19:36,080 --> 00:19:38,440 I can't say that can't stress that enough, right? 339 00:19:38,840 --> 00:19:42,560 You have to be able to kind of constantly understand the output 340 00:19:42,560 --> 00:19:45,400 of the agent, figure out where you can correct it and continue 341 00:19:45,400 --> 00:19:48,600 to improve. And I think the the last part of 342 00:19:48,600 --> 00:19:51,720 this is the observability of how these agents are functioning, 343 00:19:51,920 --> 00:19:54,880 right, where are sort of the broader efficiencies and 344 00:19:54,880 --> 00:19:58,320 inefficiencies of what they are doing and not doing is a 345 00:19:58,320 --> 00:20:00,560 constant part of sort of your architecture. 346 00:20:00,960 --> 00:20:04,760 This would be an excellent time for everybody listening to 347 00:20:04,760 --> 00:20:08,120 subscribe to the CXO Talk newsletter. 348 00:20:08,520 --> 00:20:11,320 Go to cxotalk.com. If you're watching on LinkedIn, 349 00:20:11,320 --> 00:20:15,240 take a look at the address on your screen and subscribe to our 350 00:20:15,240 --> 00:20:18,440 newsletter so we can notify you of upcoming shows and you can be 351 00:20:18,440 --> 00:20:23,120 part of this amazing, amazing CXO Talk community. 352 00:20:23,920 --> 00:20:27,720 Our next question is from Greg Walters. 353 00:20:27,720 --> 00:20:29,760 And you kind of address this a little bit. 354 00:20:29,760 --> 00:20:34,920 But, he says, can't non determinism be prompted into 355 00:20:34,920 --> 00:20:37,680 existence? The first aspect of this is to 356 00:20:37,680 --> 00:20:43,040 really understand, you know, the role of the LLM in your task, 357 00:20:43,320 --> 00:20:46,440 right? And what is the level of 358 00:20:46,720 --> 00:20:49,360 reliability that you expect from an LLMS task? 359 00:20:49,360 --> 00:20:55,280 Now there's certain things where the capability of the LLMS is 360 00:20:55,280 --> 00:20:58,040 getting constantly better and that you know that there's a 361 00:20:58,040 --> 00:21:00,440 reason why a lot of large language models are 362 00:21:00,440 --> 00:21:03,320 fundamentally trained on math and on coding tasks, because 363 00:21:03,320 --> 00:21:06,080 there's a lot of transfer learning as they get good at 364 00:21:06,080 --> 00:21:08,800 coding, as they get good at math, they're also able to get 365 00:21:08,800 --> 00:21:12,400 good at reasoning tasks, right, which are much more broadly 366 00:21:12,400 --> 00:21:13,800 applied. That's why there's a lot of 367 00:21:13,800 --> 00:21:16,920 focus on those tasks. So, so the first point I'll I'll 368 00:21:16,920 --> 00:21:20,480 say is these models will progressively get better at 369 00:21:20,600 --> 00:21:23,680 managing the hallucinations, whether it is through reasoning 370 00:21:23,680 --> 00:21:27,560 loops, right, or whether it is through better post training of 371 00:21:27,560 --> 00:21:31,080 the models in your deployments or whether it's inference time 372 00:21:31,080 --> 00:21:34,160 reasoning, right, which is sort of another, you know, scaling 373 00:21:34,160 --> 00:21:37,120 level that we now have. There are different techniques 374 00:21:37,200 --> 00:21:41,240 right in the model itself, which allow you to decrease the 375 00:21:41,240 --> 00:21:43,760 aperture of the non determinism, right? 376 00:21:43,840 --> 00:21:47,400 So that is one vector. The other vector is, as I 377 00:21:47,400 --> 00:21:49,640 mentioned earlier, building a scaffolding around it, 378 00:21:49,800 --> 00:21:52,520 understanding that you're going to get an output from the large 379 00:21:52,520 --> 00:21:55,400 language model that needs to be synthesized into something 380 00:21:55,400 --> 00:21:58,440 that's a little bit more reliable, right, and gets to the 381 00:21:58,440 --> 00:22:01,040 level of output that is acceptable for you. 382 00:22:01,040 --> 00:22:03,960 So that's a reflection loop. That's your evaluations and 383 00:22:03,960 --> 00:22:05,640 that's, you know, and sometimes you may just have a 384 00:22:05,640 --> 00:22:09,520 deterministic runtime that you need, right to design to the AI 385 00:22:09,520 --> 00:22:13,720 agentic workflow. And Gus Speckdash says it's 386 00:22:13,720 --> 00:22:19,000 interesting that Agentic AI goes around the ridiculously 387 00:22:19,000 --> 00:22:23,040 frustrating prompt user interface that is not integrated 388 00:22:23,040 --> 00:22:26,360 with any workflow. This is huge. 389 00:22:26,800 --> 00:22:29,760 What are your thoughts about Agentic AI? 390 00:22:29,760 --> 00:22:34,960 Simply around the user interface and integration with workflows? 391 00:22:35,280 --> 00:22:39,160 This is the quantum leap in my opinion, right, In terms of the 392 00:22:39,160 --> 00:22:41,680 user interface and user engagement. 393 00:22:42,000 --> 00:22:45,560 In a lot of ways, I think we've had some form of natural 394 00:22:45,560 --> 00:22:48,800 language processing, NLP, right type interfaces. 395 00:22:48,800 --> 00:22:50,360 You know, you can think of chat bots today. 396 00:22:51,000 --> 00:22:53,040 It's hard to escape them. You know, if you're trying to 397 00:22:53,040 --> 00:22:55,360 book a ticket or anything, like the first thing that pops up is 398 00:22:55,360 --> 00:22:57,360 a chat bot that's trying to get your information right. 399 00:22:57,680 --> 00:22:59,400 So that's natural language processing it. 400 00:22:59,440 --> 00:23:01,600 You know, it's able to understand voice, translates it 401 00:23:01,600 --> 00:23:04,840 into text, you know, does a search and responds back to you. 402 00:23:05,440 --> 00:23:09,080 I think the opportunity with large language models is the 403 00:23:09,080 --> 00:23:13,040 ability to infer context, right? What we had in the previous 404 00:23:13,040 --> 00:23:16,280 generation of chat bots was a literal translation, right? 405 00:23:16,520 --> 00:23:20,440 And a static sort of rules based interpretation of that 406 00:23:20,440 --> 00:23:22,720 translation. So what large language models 407 00:23:22,720 --> 00:23:25,040 have now, because they've trained on like the entire 408 00:23:25,040 --> 00:23:29,240 corpus of human language data is they can infer context, they can 409 00:23:29,240 --> 00:23:33,040 infer tone, they can infer, you know, the particular sort of 410 00:23:33,280 --> 00:23:38,160 intent and they're able to then appropriately translate that 411 00:23:38,160 --> 00:23:41,960 into into their query, right, and get you back a response. 412 00:23:41,960 --> 00:23:45,040 So I think it's game changing that you would have a large 413 00:23:45,040 --> 00:23:47,560 language model in a user interface perspective. 414 00:23:47,840 --> 00:23:51,000 Now remember, like we're still today, still very text based, 415 00:23:51,120 --> 00:23:53,320 right? Most of ChatGPT interactions, 416 00:23:53,320 --> 00:23:55,840 though, they have voice mode, which is amazing, right, are 417 00:23:55,840 --> 00:23:59,520 still text based. But think about the ability for 418 00:23:59,520 --> 00:24:02,680 us to be multimodal, right? Our ability to do voice which 419 00:24:02,680 --> 00:24:06,600 were there today, ability to input images and right, which 420 00:24:06,600 --> 00:24:08,880 we're going to get to right. And over a period of time these 421 00:24:08,880 --> 00:24:12,120 models will effectively get to this point where we call they 422 00:24:12,120 --> 00:24:14,680 have a world understanding, understanding of the world 423 00:24:14,680 --> 00:24:16,520 model. And I think that could be game 424 00:24:16,520 --> 00:24:19,360 changing in terms of how we interface with this technology. 425 00:24:19,720 --> 00:24:23,120 And this is from Arsalan Khan, who asks a very interesting 426 00:24:23,120 --> 00:24:26,360 question. He says we want bias to be 427 00:24:26,360 --> 00:24:29,760 removed from data when it comes to AI. 428 00:24:30,400 --> 00:24:34,600 How do you remove human bias if humans are in the loop? 429 00:24:34,840 --> 00:24:37,680 And who decides what's a bias or not? 430 00:24:38,240 --> 00:24:42,080 It's a really interesting point. If our vision eventually is that 431 00:24:42,680 --> 00:24:45,280 as some companies have articulated that every company 432 00:24:45,280 --> 00:24:48,720 has an AI agent and that's sort of the first point of interface 433 00:24:48,720 --> 00:24:52,320 for a customer interacting with the company, right? 434 00:24:52,320 --> 00:24:57,960 Let's say you're an insurance company or you are a, you know, 435 00:24:57,960 --> 00:25:01,640 you're even like a government service, like maybe the DMV, 436 00:25:01,920 --> 00:25:05,560 right? At some point that interface is 437 00:25:05,560 --> 00:25:08,440 really important. So I, I think look, they 438 00:25:08,840 --> 00:25:11,600 obviously the model companies are doing a lot of great work in 439 00:25:11,600 --> 00:25:14,920 making sure that we are conscious about bias. 440 00:25:16,120 --> 00:25:20,760 Now it is a fact that that is not a perfect solution yet. 441 00:25:20,760 --> 00:25:24,920 We're not quite there yet in in certain instances, you know, 442 00:25:24,960 --> 00:25:27,400 you're not getting these perfect answers. 443 00:25:27,640 --> 00:25:32,200 So part of this is the context that I talked about in terms of 444 00:25:32,200 --> 00:25:34,280 how you ground the agent is really important. 445 00:25:34,600 --> 00:25:36,840 And So what does context really mean? 446 00:25:36,840 --> 00:25:38,480 Right? Context means examples. 447 00:25:38,840 --> 00:25:42,880 And so if it's a customer agent, for example, you could train it 448 00:25:42,920 --> 00:25:46,440 on the policies that you have. You know, you probably have a 449 00:25:46,440 --> 00:25:50,200 lot of voice recordings of existing agent calls, right, 450 00:25:50,200 --> 00:25:53,160 that are great examples of how to handle situations in bias or 451 00:25:53,160 --> 00:25:57,640 confrontational situations. So the training of this agent, 452 00:25:57,840 --> 00:26:03,760 grounding it in the policies, right, and the rules of the 453 00:26:03,760 --> 00:26:07,600 particular use case is a particularly important task. 454 00:26:08,000 --> 00:26:10,840 Now, I think look, the human in the loop is really about sort of 455 00:26:11,000 --> 00:26:15,720 how you reason things. And obviously look, the way a 456 00:26:15,720 --> 00:26:19,520 human interacts ideally is a positive in terms of our ability 457 00:26:19,520 --> 00:26:22,160 to eliminate bias, right? By inserting human in the loop, 458 00:26:22,360 --> 00:26:26,160 you're actually adding a step that improves the ability of the 459 00:26:26,600 --> 00:26:30,000 agent overall to be, you know, to correct it's biases. 460 00:26:30,000 --> 00:26:32,840 But I would say, you know, good training, grounding and 461 00:26:32,840 --> 00:26:35,440 policies, right? And obviously, you know, having 462 00:26:35,440 --> 00:26:39,440 responsible humans in the loop are the things that will help us 463 00:26:39,440 --> 00:26:41,760 get there. But it's an imperfect process, 464 00:26:41,760 --> 00:26:43,320 and that's why I say it's early days yet. 465 00:26:43,840 --> 00:26:49,080 Arsalan Khan comes back and he says if subject matter experts 466 00:26:49,080 --> 00:26:54,840 train AI to create AI agents, would we really need the subject 467 00:26:54,840 --> 00:26:58,680 matter experts? What happens when AI agents come 468 00:26:58,680 --> 00:27:02,080 across a scenario that they haven't encountered yet? 469 00:27:02,560 --> 00:27:06,560 Training is a constant process for AI agents, right? 470 00:27:07,120 --> 00:27:10,680 What you get when you initially start this process with a 471 00:27:10,680 --> 00:27:13,800 subject matter expert helping you ground the model, ground the 472 00:27:14,280 --> 00:27:17,440 AI agent is you're giving it a certain rubric. 473 00:27:17,440 --> 00:27:19,280 You're basically saying like, hey, here's basically what is 474 00:27:19,280 --> 00:27:21,560 expected, right? Here's a basic task. 475 00:27:21,560 --> 00:27:24,800 Here's how you perform the task right now. 476 00:27:24,880 --> 00:27:27,560 The task evolves. We are in a dynamic world, 477 00:27:27,560 --> 00:27:29,120 right? Let's say you're in a company, 478 00:27:29,520 --> 00:27:32,280 maybe you launch a new product, maybe you expand into a new 479 00:27:32,280 --> 00:27:35,200 region. You know, maybe you have you, 480 00:27:35,200 --> 00:27:37,720 you make an acquisition or you have new employees on boarded, 481 00:27:37,960 --> 00:27:39,920 right? It's a, it's, it's a process of 482 00:27:39,920 --> 00:27:43,560 constant change. So to some extent, I think you 483 00:27:43,560 --> 00:27:46,440 can design the AI agent to say, OK, I can expand into a new 484 00:27:46,440 --> 00:27:48,240 geography. This is how I understand it, 485 00:27:48,360 --> 00:27:50,200 right? But they may be different 486 00:27:50,200 --> 00:27:52,160 policies. I mean, as it's often the case, 487 00:27:52,320 --> 00:27:55,280 if you're operating in a different country, they may have 488 00:27:55,280 --> 00:27:58,760 their own, you know, regular rules and regulations that you 489 00:27:58,760 --> 00:28:01,840 need to, you need to now kind of ground the model in and such 490 00:28:01,840 --> 00:28:03,760 like. So I think the subject matter 491 00:28:03,760 --> 00:28:08,560 expert is really, you know, as, and we all do this, right? 492 00:28:08,560 --> 00:28:09,800 We're all subject matter experts. 493 00:28:10,040 --> 00:28:12,120 We're not static, right? We're constantly sort of 494 00:28:12,120 --> 00:28:14,280 learning, evolving, understanding, right, new 495 00:28:14,280 --> 00:28:16,680 technologies, new pieces. And it's the same thing for the 496 00:28:16,680 --> 00:28:20,640 model. So yes, the idea of the AI agent 497 00:28:20,640 --> 00:28:23,880 is to take away these sort of mundane things, OK, go get this 498 00:28:23,880 --> 00:28:26,160 document, put 10 documents together, create a research 499 00:28:26,160 --> 00:28:29,160 report, right? So that yes, you don't need to 500 00:28:29,160 --> 00:28:31,640 retrain that thing. But being able to say like, OK, 501 00:28:31,760 --> 00:28:34,960 how do I operate in the European Union, which may have a 502 00:28:34,960 --> 00:28:39,720 different set of rules, or in in Asia, in, in Japan or in China 503 00:28:39,720 --> 00:28:42,520 or in India, right, which may have a different set of rules. 504 00:28:42,720 --> 00:28:46,200 Those are things that there's a certain amount of requirement of 505 00:28:46,200 --> 00:28:48,800 understanding those rules and regulations that need to, again, 506 00:28:48,880 --> 00:28:50,640 you need to kind of train the agent on, right. 507 00:28:51,080 --> 00:28:54,320 Let's start talking about business. 508 00:28:54,680 --> 00:29:00,680 And Mario Garcia asks. He says it's inspiring to see 509 00:29:00,800 --> 00:29:04,360 the impact of AI in Fortune 500 companies. 510 00:29:05,000 --> 00:29:09,120 What insights about this stand out to you the most? 511 00:29:09,600 --> 00:29:13,560 A lot of these large companies today, I mean, it's been amazing 512 00:29:13,560 --> 00:29:19,960 to watch how rapidly, you know, both sort of large companies as 513 00:29:19,960 --> 00:29:24,480 well as incumbents and start-ups have really embraced AI and 514 00:29:24,480 --> 00:29:27,800 large language models. And it's largely, I think the 515 00:29:27,800 --> 00:29:32,320 ChatGPT moment which unleashed this because it, you know, it 516 00:29:32,320 --> 00:29:34,640 was so accessible. So you take a step back. 517 00:29:34,640 --> 00:29:37,280 You know, AI has been around for a long time, right? 518 00:29:37,960 --> 00:29:40,240 You know, I, I did a course in neural networks back in my, you 519 00:29:40,480 --> 00:29:43,680 know, grad student days. What changed I think in this 520 00:29:43,680 --> 00:29:49,920 generation is that today really powerful complex AI models are 521 00:29:50,000 --> 00:29:53,880 available on the other end of an API call, right? 522 00:29:54,120 --> 00:29:58,560 So that level of simplicity in terms of access to really 523 00:29:58,560 --> 00:30:01,280 powerful technology is essentially what enabled us to 524 00:30:01,280 --> 00:30:07,040 unleash large AI as you see it in in a lot of these use cases. 525 00:30:07,040 --> 00:30:11,040 Now I will again caution you to say that we are still very 526 00:30:11,040 --> 00:30:13,080 early. If you talk to a lot, a lot of 527 00:30:13,080 --> 00:30:17,640 these large corporations, they do have large language models 528 00:30:17,640 --> 00:30:19,040 integrated. Most of them have rolled out, 529 00:30:19,040 --> 00:30:21,800 for example, some form of a coding copilot. 530 00:30:21,960 --> 00:30:24,400 They've rolled out some form of, you know, customer support 531 00:30:24,400 --> 00:30:25,840 function. They've rolled out some form of 532 00:30:25,840 --> 00:30:31,800 an analytics sort of use case with large language models, but 533 00:30:31,800 --> 00:30:35,120 we're still not deployed at scale, primarily because we're 534 00:30:35,320 --> 00:30:40,160 kind of still tuning, tweaking, right, learning in terms of how 535 00:30:40,160 --> 00:30:44,360 do you manage these AI agents? How do you manage biases? 536 00:30:44,360 --> 00:30:47,560 As one of your audience members just asked, how do you make sure 537 00:30:47,560 --> 00:30:50,000 that they are current, right? How do you make sure they don't 538 00:30:50,000 --> 00:30:52,240 hallucinate? The most important thing of the, 539 00:30:52,240 --> 00:30:53,960 and this is fundamental, right? We all know this. 540 00:30:54,400 --> 00:30:57,000 It doesn't matter. There's a lot of really, you 541 00:30:57,000 --> 00:31:02,240 know, fun, exciting, interesting demos on, on, on X, right? 542 00:31:02,840 --> 00:31:04,120 And that's, those are like, great. 543 00:31:04,160 --> 00:31:06,040 You can say, oh, wow, this agent can do this. 544 00:31:06,360 --> 00:31:10,480 What's really important is can it do it consistently and can it 545 00:31:10,480 --> 00:31:13,400 do it at scale? And so those are the questions 546 00:31:13,400 --> 00:31:15,160 we'll answer this year hopefully, right? 547 00:31:15,160 --> 00:31:18,800 And that's why we're so excited in 2025 about the trajectory of 548 00:31:18,800 --> 00:31:22,000 these AI agents. Praveen, you and your team put 549 00:31:22,000 --> 00:31:27,440 together what you call a market map of companies involved with 550 00:31:27,840 --> 00:31:30,960 AI agents. Can you tell us about that? 551 00:31:30,960 --> 00:31:34,240 And I can bring it up on the screen so everybody can see. 552 00:31:34,440 --> 00:31:39,240 And there it is. Praveen, can you talk about this 553 00:31:39,240 --> 00:31:40,960 market map that you've put together? 554 00:31:41,400 --> 00:31:47,240 The market map is a dynamic living thing in the sense that 555 00:31:47,760 --> 00:31:50,800 it will evolve constantly, primarily because we're seeing 556 00:31:50,800 --> 00:31:57,720 so much activity, right, and energy around, you know, the AI 557 00:31:57,720 --> 00:32:00,720 agentic space. So what we tried to do is to 558 00:32:00,720 --> 00:32:03,920 basically construct this in, in layers, right? 559 00:32:03,920 --> 00:32:07,160 So there's, there's like a foundational layer where there's 560 00:32:07,160 --> 00:32:10,920 a lot of these kind of data sources, integrations and such. 561 00:32:10,920 --> 00:32:14,640 Like there's this new sort of bucket we, you know, we call 562 00:32:14,640 --> 00:32:19,000 sort of the agent computer interface, which is the ability 563 00:32:19,000 --> 00:32:24,720 for AI agents to use computer, you know, tools, right? 564 00:32:24,920 --> 00:32:28,240 And you know, it could be integrations, it could be, you 565 00:32:28,240 --> 00:32:30,800 know, web tools, some of the stuff is integrated in models, 566 00:32:30,800 --> 00:32:33,280 some of these are, you know, you have interesting kind of 567 00:32:33,280 --> 00:32:35,000 platforms that are created for this. 568 00:32:35,280 --> 00:32:38,360 So we try to kind of construct the model as, you know, layer by 569 00:32:38,360 --> 00:32:40,200 layer. What's the bottom layer, OK, 570 00:32:41,040 --> 00:32:42,800 where all the data platforms, right? 571 00:32:42,960 --> 00:32:45,200 What is what is this sort of middleware layer, if you will, 572 00:32:45,200 --> 00:32:47,840 which is the agentic computer interface as well as a lot of 573 00:32:47,840 --> 00:32:51,400 these agent frameworks, right? You know, we are investors in a 574 00:32:51,400 --> 00:32:55,200 company called Crew AI, for example, and I'm sure anybody 575 00:32:55,200 --> 00:32:57,480 who's talked about AI agents probably knows about crew AI. 576 00:32:57,480 --> 00:33:00,240 It's one of the most popular open source frameworks out 577 00:33:00,240 --> 00:33:02,280 there. You can use crew AI to build 578 00:33:02,400 --> 00:33:05,240 agents, say others like LAN chain, which are also do 579 00:33:05,240 --> 00:33:08,240 something similar. And about that, then what we 580 00:33:08,240 --> 00:33:11,120 tried to do is to say like, wow, let's try and kind of map out 581 00:33:11,120 --> 00:33:14,240 sort of where is the energy in the AI agent space? 582 00:33:14,240 --> 00:33:17,000 And it's important to understand, I think it's on the 583 00:33:17,000 --> 00:33:20,800 left side of the of the of the market map. 584 00:33:21,040 --> 00:33:26,480 There's a lot of AI agentic products and offerings from 585 00:33:26,480 --> 00:33:29,480 incumbents. So obviously Salesforce, you 586 00:33:29,800 --> 00:33:34,400 know, launch their own sort of agentic agentic workforce agent 587 00:33:34,400 --> 00:33:37,480 force as they call it, right. Microsoft has Co pilots. 588 00:33:37,600 --> 00:33:40,440 We just saw open AI launch operator, which is more consumer 589 00:33:40,440 --> 00:33:45,120 oriented agent. And so, you know, effectively 590 00:33:45,120 --> 00:33:47,280 all the incumbents have said like, hey, we've got these great 591 00:33:47,280 --> 00:33:49,160 customers, we've got all these great use cases. 592 00:33:49,400 --> 00:33:53,240 Is there a way for us to improve our customer experience or 593 00:33:53,360 --> 00:33:58,440 productivity by creating an agentic workflow on top of our 594 00:33:58,440 --> 00:34:01,960 existing software? On the rest of the market map, 595 00:34:02,200 --> 00:34:05,800 you can see sort of tremendous amount of energy in in different 596 00:34:05,800 --> 00:34:07,440 verticals. You know, particularly in 597 00:34:07,440 --> 00:34:10,520 coding, for example, there's been huge amount of a great 598 00:34:10,520 --> 00:34:14,040 Workman cursor by far seems to be the most popular among 599 00:34:14,040 --> 00:34:16,920 developers today. But you also see very specific 600 00:34:16,920 --> 00:34:22,679 vertical agents, right, sales, marketing, legal, right? 601 00:34:23,840 --> 00:34:26,480 You, you see finance, right? So you can take almost each of 602 00:34:26,480 --> 00:34:30,639 these different functions and you can see companies building 603 00:34:30,920 --> 00:34:35,080 agents which are customized to that particular vertical use 604 00:34:35,080 --> 00:34:37,440 case, right? This is really interesting 605 00:34:37,440 --> 00:34:40,159 company called Samaya, for example, that's building doing 606 00:34:40,159 --> 00:34:43,960 some amazing work focusing on building agents for the finance 607 00:34:43,960 --> 00:34:48,880 workflow, right? So you, you, you see that the 608 00:34:48,880 --> 00:34:53,000 idea was a market map was not to be precise, right? 609 00:34:53,000 --> 00:34:56,639 And capture the entire view. And you know, I do apologize to 610 00:34:56,760 --> 00:35:00,520 a lot of the the builders out there, some of whom we missed in 611 00:35:00,520 --> 00:35:04,320 the market map clearly, right. The idea really was to kind of 612 00:35:04,480 --> 00:35:08,320 give you a perspective of, you know, what this is landscape 613 00:35:08,320 --> 00:35:09,600 look like, right? Where? 614 00:35:09,600 --> 00:35:11,320 Where is the activity? Like where? 615 00:35:11,440 --> 00:35:15,240 How are builders approaching the AI agentic space? 616 00:35:15,640 --> 00:35:21,440 What are the opportunities and the use cases, the predominant 617 00:35:21,440 --> 00:35:24,120 or the most important use cases for AI agents right now? 618 00:35:24,640 --> 00:35:29,240 They're probably like 3 or 4 buckets and you know, the first 619 00:35:29,240 --> 00:35:33,560 one obviously that everybody knows and understands very well. 620 00:35:33,560 --> 00:35:36,800 Are these coding agents, coding Co pilots, coding platforms, 621 00:35:36,800 --> 00:35:39,040 What do you want to call them? Right. 622 00:35:39,320 --> 00:35:43,160 And depending on, you know, the particular style, you know, 623 00:35:43,280 --> 00:35:47,040 cursor has a particular way of, of working. 624 00:35:47,040 --> 00:35:50,000 It's more like a copilot. If you take something like Devon 625 00:35:50,000 --> 00:35:53,200 has a different sort of way it, it engages, you know, it fires 626 00:35:53,200 --> 00:35:56,920 off a bunch of agents that you know, execute your plan and such 627 00:35:56,920 --> 00:35:58,320 like. But there's a lot of energy 628 00:35:58,560 --> 00:36:03,920 around developer facing AI agentic work, right? 629 00:36:03,920 --> 00:36:06,120 And so you can see that in the market map as well. 630 00:36:06,280 --> 00:36:10,200 The second area is in the customer experience section. 631 00:36:10,200 --> 00:36:12,840 So customer experience, obviously everything from like 632 00:36:13,080 --> 00:36:16,720 customer support agents, which obviously is the biggest use 633 00:36:16,720 --> 00:36:18,440 case. We, we, as you, as we were 634 00:36:18,440 --> 00:36:21,760 talking about earlier, chat bots are already a fact of life. 635 00:36:22,160 --> 00:36:25,720 Can we make that experience much more realistic, much more sort 636 00:36:25,720 --> 00:36:28,680 of, you know, engaging? So, you know, like me, you're 637 00:36:28,680 --> 00:36:33,120 not basically saying agent as as soon as you as soon as you get a 638 00:36:33,120 --> 00:36:35,080 a chat bot, right? So there's a ton of energy in 639 00:36:35,080 --> 00:36:38,480 that space, lots of lots of great companies building 640 00:36:38,480 --> 00:36:41,320 interesting products. The other area of it's just kind 641 00:36:41,320 --> 00:36:43,920 of interesting is in the operation space, right? 642 00:36:43,920 --> 00:36:46,560 So if you think about operations, broadly speaking, it 643 00:36:46,560 --> 00:36:49,560 could be IT operations, it could be security operations. 644 00:36:50,080 --> 00:36:52,200 You have sort of this needle in the haystack problem. 645 00:36:52,200 --> 00:36:56,200 You have a lot of data, you have a lot of like alerts that come 646 00:36:56,200 --> 00:36:58,320 in and you're trying to figure out like, OK, which ones do I 647 00:36:58,320 --> 00:37:01,880 pay attention to, right? So this is actually a perfect 648 00:37:01,880 --> 00:37:04,560 use case for AI agents, the ability to synthesize all of 649 00:37:04,560 --> 00:37:07,600 that information. And if it's grounded in your 650 00:37:07,600 --> 00:37:13,080 policies and in sort of in in a company sort of particular way 651 00:37:13,080 --> 00:37:15,440 of doing things, it's able to like, say, like, hey, here's 652 00:37:15,440 --> 00:37:18,240 maybe the top three to five alerts you need to pay attention 653 00:37:18,240 --> 00:37:20,320 to. This is the problem that they're 654 00:37:20,320 --> 00:37:22,480 articulating. And here's a few ways to 655 00:37:22,480 --> 00:37:24,080 remediate this. So, you know, we were talking 656 00:37:24,080 --> 00:37:26,480 some interesting startups that are actually focused in this 657 00:37:26,480 --> 00:37:29,840 particular space. So I think those are the three. 658 00:37:29,840 --> 00:37:31,920 And, you know, there's a lot more, but I'll just, you know, 659 00:37:31,920 --> 00:37:34,040 in the interest of time, maybe I'll just pause there. 660 00:37:34,520 --> 00:37:39,720 We have a question from Elizabeth Shaw, and this is on 661 00:37:39,840 --> 00:37:44,440 Twitter. Who asks how are organizations 662 00:37:44,440 --> 00:37:48,400 using agentic AI in their business and their ecosystem? 663 00:37:48,840 --> 00:37:51,800 Let's just take for example, a customer service AI agent, 664 00:37:52,040 --> 00:37:55,400 right? So that's that's a use case that 665 00:37:55,400 --> 00:37:58,560 we're seeing a lot of customers experimenting with. 666 00:37:59,360 --> 00:38:01,600 So what is what is this customer service agent do? 667 00:38:01,600 --> 00:38:06,320 So you effectively again, you think of a chatbot today, the 668 00:38:06,320 --> 00:38:10,880 customer service agent is able to first of all, sort of be 669 00:38:10,880 --> 00:38:12,800 grounded in all of your data, your FAQs. 670 00:38:12,800 --> 00:38:18,160 You know, how do you, I mistype my password, how do I recover my 671 00:38:18,160 --> 00:38:20,560 password? Or, you know, how do I, you 672 00:38:20,560 --> 00:38:25,760 know, whatever, add, add my child to the, to the insurance 673 00:38:25,760 --> 00:38:28,640 policy or whatever. You have these things continuous 674 00:38:28,640 --> 00:38:31,920 task, you know that, that you were able to then interact with 675 00:38:31,920 --> 00:38:33,400 an agent. The agent understands what 676 00:38:33,400 --> 00:38:35,520 you're trying to do. Maybe you're looking for a form, 677 00:38:35,520 --> 00:38:37,400 maybe you're looking for a website, maybe you're looking 678 00:38:37,400 --> 00:38:40,360 for a particular sort of quote or something like that. 679 00:38:40,520 --> 00:38:43,240 So I think that is a use case that is getting a lot of 680 00:38:43,240 --> 00:38:46,400 traction. We're seeing a sort of a lot of 681 00:38:46,400 --> 00:38:50,440 our customers looking at sort of, you know, rolling that out 682 00:38:50,640 --> 00:38:53,440 into production. The other use case that I 683 00:38:53,440 --> 00:38:56,120 mentioned is on the coding side, right? 684 00:38:56,320 --> 00:39:00,040 And no, we're not just talking about like an IDE like like 685 00:39:00,040 --> 00:39:02,840 Cursor, which obviously has a lot of broad adoption, but 686 00:39:02,840 --> 00:39:05,440 things like automated testing and provisioning, right? 687 00:39:05,680 --> 00:39:09,040 So you have software you need to roll this out. 688 00:39:09,160 --> 00:39:12,600 Testing is a really critical part of that process of rolling 689 00:39:12,600 --> 00:39:16,000 things out. You, you're able to actually use 690 00:39:16,080 --> 00:39:19,960 AI agents, you know, very effectively in sort of that 691 00:39:19,960 --> 00:39:25,920 testing and sort of, you know, red, red team kind of use cases 692 00:39:25,920 --> 00:39:28,160 where you can like see if you can break it, right? 693 00:39:28,360 --> 00:39:31,440 That's a very critical function where you're seeing some level 694 00:39:31,440 --> 00:39:34,960 of, of deployments happening. The other one, as I said, is 695 00:39:34,960 --> 00:39:40,280 like in IT operations, right? And this, this is very exciting 696 00:39:40,280 --> 00:39:44,880 because again, if you, you know, these are mission critical, 697 00:39:44,960 --> 00:39:46,880 right? You need to be constantly up, 698 00:39:47,400 --> 00:39:50,360 you know, most of these IT teams, you know, you always have 699 00:39:50,360 --> 00:39:53,640 a 24 by 7 coverage because you cannot have, you know, critical 700 00:39:53,640 --> 00:39:56,840 systems going down, right? And so an AI agent is perfect 701 00:39:56,840 --> 00:39:59,600 because it's it, it has the ability to synthesize large 702 00:39:59,600 --> 00:40:04,560 amounts of data. It has the ability to basically, 703 00:40:04,640 --> 00:40:06,760 you know. Needle in the haystack, right 704 00:40:06,920 --> 00:40:09,240 problem now again, work in progress. 705 00:40:09,560 --> 00:40:13,040 I wouldn't say all of these things are at perfection, but we 706 00:40:13,040 --> 00:40:15,120 are definitely seeing these. And then I mentioned this 707 00:40:15,120 --> 00:40:17,200 company Samaya, which is very interesting. 708 00:40:18,080 --> 00:40:22,600 They are actively building like a finance analyst kind of agent, 709 00:40:22,840 --> 00:40:25,920 which is, you know, which is pretty accurate in terms of 710 00:40:25,920 --> 00:40:30,040 being able to extract context out of research and and provide 711 00:40:30,040 --> 00:40:35,520 you really focused information. So you say that it's really 712 00:40:35,520 --> 00:40:38,440 accurate. I'm assuming that what you also 713 00:40:38,440 --> 00:40:43,680 mean is that it is consistently reliable and predictable. 714 00:40:44,080 --> 00:40:47,320 Exactly. And it's much harder to do that 715 00:40:48,120 --> 00:40:50,240 if just straight out-of-the-box, right. 716 00:40:50,240 --> 00:40:53,760 I think a lot of times there's a lot of confusion in the market 717 00:40:53,760 --> 00:40:55,960 about like, hey, wait a minute, the large language models are 718 00:40:55,960 --> 00:40:57,800 just going to keep getting better and better and better. 719 00:40:57,800 --> 00:41:00,720 And, you know, ultimately they'll just they'll be like one 720 00:41:00,720 --> 00:41:05,280 model that solves all, right. And, and I think in it is true 721 00:41:05,280 --> 00:41:08,120 in certain simple use cases. I mean, absolutely the models 722 00:41:08,120 --> 00:41:09,760 are getting better, their reasoning better. 723 00:41:09,960 --> 00:41:13,840 Maybe we will get to a point of artificial general intelligence, 724 00:41:13,840 --> 00:41:18,040 right where, you know, these models can just use, you know, 725 00:41:18,040 --> 00:41:20,640 computers like we do. And, and, and maybe that's the 726 00:41:20,640 --> 00:41:22,920 bar, right? But I think when you're looking 727 00:41:22,920 --> 00:41:27,680 at complex enterprise workflows, as I mentioned, the ability for 728 00:41:28,280 --> 00:41:34,240 the agent to be grounded right and to be accurate and to 729 00:41:34,240 --> 00:41:40,040 present data in the way the end user expects will require some 730 00:41:40,040 --> 00:41:43,960 amount of post training, some amount of inference time, right 731 00:41:44,440 --> 00:41:47,440 reasoning, as well as some amount of scaffolding in order 732 00:41:47,440 --> 00:41:49,280 for you to build the perfect agent. 733 00:41:49,840 --> 00:41:53,960 Can you talk about the economics of agents? 734 00:41:53,960 --> 00:41:56,720 And then we'll jump back because we do have additional questions 735 00:41:57,080 --> 00:41:59,640 that have come in, but the economics are really important. 736 00:41:59,640 --> 00:42:05,640 So what are the aspects that Dr. economics and what do enterprise 737 00:42:05,640 --> 00:42:08,920 buyers need to think about when it comes to the economics of 738 00:42:08,920 --> 00:42:11,240 agents? You've seen sort of the whole 739 00:42:11,240 --> 00:42:13,320 spectrum of conversations, right? 740 00:42:14,480 --> 00:42:17,760 Everything from like with AI agents, we're just going to go 741 00:42:17,760 --> 00:42:20,880 and tally to outcome based pricing to like, well, you know, 742 00:42:20,880 --> 00:42:24,160 it's it's still software. So we have to kind of figure out 743 00:42:24,440 --> 00:42:28,080 how do you make sure that you're able to charge for it 744 00:42:28,080 --> 00:42:30,760 appropriately. The fundamental question to ask 745 00:42:30,880 --> 00:42:35,680 right when you think about pricing is can you measure the 746 00:42:35,680 --> 00:42:40,400 value of the AI agent output accurately, right? 747 00:42:40,640 --> 00:42:43,040 So what do I mean by that? Let's say you have a customer 748 00:42:43,040 --> 00:42:46,080 support agent. You can basically say, hey, the 749 00:42:46,080 --> 00:42:49,840 customer support agent handled 100 calls, right? 750 00:42:50,360 --> 00:42:53,760 And you'd have taken me X amount of dollars to handle those 100 751 00:42:53,760 --> 00:42:55,720 calls. The customer agent handled those 752 00:42:55,720 --> 00:42:58,520 calls. So I can attribute directly a 753 00:42:58,520 --> 00:43:01,600 value right to the output of that agent. 754 00:43:01,600 --> 00:43:04,440 It handed 100 calls. Each call is worth X. 755 00:43:04,440 --> 00:43:08,040 So there's basically a hundred X is basically the value of that 756 00:43:08,040 --> 00:43:10,920 particular agents task. In the other end of the 757 00:43:10,920 --> 00:43:15,600 spectrum, let's say you're doing, you know, a research 758 00:43:15,600 --> 00:43:18,200 workflow, right? So you've generated research 759 00:43:18,200 --> 00:43:21,760 report or you're helping an analyst basically with the 760 00:43:21,760 --> 00:43:23,440 research and you improve their productivity. 761 00:43:23,680 --> 00:43:25,240 How do you measure the value of that? 762 00:43:25,440 --> 00:43:27,920 Right? You measure it by, you know, 763 00:43:27,920 --> 00:43:31,240 individually, like how, you know, asking the the analyst, 764 00:43:31,240 --> 00:43:33,760 like how, how much more productive were you right? 765 00:43:33,920 --> 00:43:38,640 And you know, it's it's much harder to quantify certain tasks 766 00:43:38,640 --> 00:43:41,640 versus certain other tasks. So I think the first question in 767 00:43:41,640 --> 00:43:45,520 the understanding economics of agencies, can you attribute 768 00:43:45,760 --> 00:43:50,280 value in a reasonably accurate way to the output of the agent. 769 00:43:50,440 --> 00:43:53,760 So based on that, if you can, then outcome based pricing is 770 00:43:53,760 --> 00:43:56,160 essentially where we're eventually going to go to, 771 00:43:56,360 --> 00:43:58,720 right. If it's a lot more nebulous, 772 00:43:58,720 --> 00:44:02,080 then I think what we're going to see is some form of an evolution 773 00:44:02,400 --> 00:44:04,640 of the existing SAS pricing model. 774 00:44:04,640 --> 00:44:08,320 So you might pay a platform fee like for the agent thing and 775 00:44:08,320 --> 00:44:11,040 then maybe you hire an agent. So you pay on the number of 776 00:44:11,040 --> 00:44:14,240 times you run the agent, right? So it's some combination of 777 00:44:14,240 --> 00:44:16,080 that. So we see this again as a 778 00:44:16,080 --> 00:44:18,120 spectrum. There's no like absolute here, 779 00:44:18,360 --> 00:44:20,280 right? There's a lot of experimentation 780 00:44:20,280 --> 00:44:23,720 today in some ways, like companies are still trying to 781 00:44:23,720 --> 00:44:25,880 figure out like, you know, how's the customer getting value 782 00:44:25,880 --> 00:44:28,400 customers trying to figure out. And if you're ACFO, right, 783 00:44:28,480 --> 00:44:30,760 you're used to paying subscription software, you know, 784 00:44:30,760 --> 00:44:34,680 like, OK, I'm I'm I've got X amount of licenses for one year 785 00:44:34,680 --> 00:44:37,520 and I can budget that right now. If you go to sort of this 786 00:44:37,520 --> 00:44:40,680 outcome based pricing, again, if you don't have an accurate sense 787 00:44:40,680 --> 00:44:44,560 of value, how would you as ACFO budget right for these AI 788 00:44:44,560 --> 00:44:46,160 agents? So I think there's a lot of 789 00:44:46,160 --> 00:44:50,000 these things that need to be, you know, we're kind of 790 00:44:50,000 --> 00:44:55,280 experimenting and understanding eventually where this direct 791 00:44:55,280 --> 00:44:59,280 attribution of value, I think we will end up in the outcome based 792 00:44:59,280 --> 00:45:01,200 pricing bucket. But there's also going to be a 793 00:45:01,200 --> 00:45:05,360 lot of these intermediary models where, you know, you want to 794 00:45:05,360 --> 00:45:08,560 make sure that the developers are getting a fair value for the 795 00:45:08,560 --> 00:45:10,840 product that they're building and the customers are paying a 796 00:45:10,840 --> 00:45:14,160 fair price for it. So ultimately when we reach the 797 00:45:14,160 --> 00:45:21,440 point where agents have discrete measurable output results, then 798 00:45:21,440 --> 00:45:26,520 we can move towards performance based pricing and until then 799 00:45:26,520 --> 00:45:28,680 it's essentially usage. Right. 800 00:45:28,920 --> 00:45:30,560 Yeah, I think that's a good way to put it. 801 00:45:31,000 --> 00:45:37,080 We have an important point now raised on LinkedIn by Naresh 802 00:45:37,080 --> 00:45:42,440 Kumar, who is VP and General Manager of Product Management at 803 00:45:42,440 --> 00:45:46,320 Z Scaler. And he raises the question, what 804 00:45:46,320 --> 00:45:50,680 about security and agentic AI? And we haven't talked about 805 00:45:50,680 --> 00:45:52,200 that. So I'm glad you brought this up. 806 00:45:52,640 --> 00:45:55,080 Large language models help with this sort of needle in the 807 00:45:55,080 --> 00:45:58,320 haystack problem, which is inherent to sort of diagnosing 808 00:45:58,320 --> 00:46:00,880 security problems. I kind of grew up in the 809 00:46:01,000 --> 00:46:06,080 networking world and be used to, you know, build these large 810 00:46:06,440 --> 00:46:09,160 global scale Internet scale networks. 811 00:46:09,160 --> 00:46:12,400 And a big part of the task was like, you know, if if there was 812 00:46:12,400 --> 00:46:15,960 an outage somewhere to, to debug, that would essentially 813 00:46:15,960 --> 00:46:18,760 mean like we synthesize tons of data, right, and figure out 814 00:46:19,280 --> 00:46:22,040 where we need to focus our efforts, right. 815 00:46:22,040 --> 00:46:27,000 And security is the same way. You have a large aperture of 816 00:46:27,000 --> 00:46:30,120 exposure depending on, you know, the type of company you are, 817 00:46:30,600 --> 00:46:33,200 everything from your network to your applications, your devices 818 00:46:33,200 --> 00:46:35,280 to your individuals to identity, right? 819 00:46:35,280 --> 00:46:37,600 There's like multiple layers, right when you think about 820 00:46:37,600 --> 00:46:42,520 security and so, and it's been a tough challenge, right in the 821 00:46:42,520 --> 00:46:45,960 security industry, we've had, you know, these platforms called 822 00:46:45,960 --> 00:46:49,560 CMS which try to bring all this together and be able to give you 823 00:46:49,560 --> 00:46:52,320 like a unified view where you're able to manage this. 824 00:46:52,320 --> 00:46:56,640 But you know, a security OPS centre, essentially the nerve 825 00:46:56,640 --> 00:46:59,760 centre of how most companies run their security operations. 826 00:46:59,760 --> 00:47:03,040 So I would look at the role of LLMS in security in three ways. 827 00:47:03,280 --> 00:47:07,840 The first I think is from a operations perspective, I think 828 00:47:07,920 --> 00:47:11,240 it could be a very useful tool because he has the ability to 829 00:47:11,240 --> 00:47:14,320 synthesize large amounts of data and help in that needle in the 830 00:47:14,320 --> 00:47:16,120 haystack problem or prioritize it. 831 00:47:16,280 --> 00:47:20,920 I think it's a great use case. The second one is, you know, LMS 832 00:47:20,960 --> 00:47:23,920 integrated into the security products, right? 833 00:47:24,680 --> 00:47:28,320 Will essentially, you know, you talk about again, a security 834 00:47:28,320 --> 00:47:32,640 agent, existing security software, right, being able to 835 00:47:32,640 --> 00:47:35,800 dynamically understand policy, right, dynamically able to 836 00:47:35,800 --> 00:47:40,360 respond to, you know, you're adding more users etcetera. 837 00:47:40,600 --> 00:47:42,680 I think you're able to sort of build those. 838 00:47:42,680 --> 00:47:46,120 We're seeing companies starting to build large language models 839 00:47:46,200 --> 00:47:49,680 into their software stack. Just as we talked about earlier, 840 00:47:49,680 --> 00:47:51,680 it's a tool right, where where it's useful. 841 00:47:52,040 --> 00:47:56,080 The third I'd say is look, you know, LLMS do represent a new, 842 00:47:56,240 --> 00:47:58,080 you know, threat aperture, right? 843 00:47:58,240 --> 00:48:02,680 Particularly if the models essentially hallucinate or for 844 00:48:02,680 --> 00:48:06,320 example, in that psych OPS use case, you know, ignore or 845 00:48:06,360 --> 00:48:09,640 highlight or or miss critical, critical kind of threat. 846 00:48:09,640 --> 00:48:12,960 So while you're designing, while we talk about all these agents 847 00:48:12,960 --> 00:48:16,360 being deployed in a customer support use case or, you know, a 848 00:48:16,360 --> 00:48:19,960 finance analyst use case, if those large language models are 849 00:48:19,960 --> 00:48:24,400 not sufficiently grounded, right, and they're not, you 850 00:48:24,400 --> 00:48:28,680 know, the data, the training set that they have is not protected 851 00:48:28,680 --> 00:48:32,920 appropriately, you risk not just hallucinations, but effectively, 852 00:48:33,160 --> 00:48:35,760 you know, a hijack of the entire AI agent. 853 00:48:35,760 --> 00:48:41,360 So it's early days in that I think we've had some, you know, 854 00:48:41,360 --> 00:48:44,160 some interesting conversations with, with founders who are 855 00:48:44,160 --> 00:48:46,680 thinking about this problem deep ways and building interesting 856 00:48:46,680 --> 00:48:49,800 things. But yeah, you know, it's, it's, 857 00:48:50,120 --> 00:48:52,000 it's an problem space at this point. 858 00:48:52,000 --> 00:48:55,640 It's we will learn more and we will evolve the security 859 00:48:55,640 --> 00:48:58,440 architecture just as revolving with the maturity of the AI 860 00:48:58,440 --> 00:49:00,200 agents itself. OK. 861 00:49:00,240 --> 00:49:03,320 And obviously Z Scaler is thinking about this because he 862 00:49:03,320 --> 00:49:05,800 asked that question. Yeah, they're an important 863 00:49:05,800 --> 00:49:08,880 player and they have a huge role to play right in in in our 864 00:49:08,880 --> 00:49:12,360 overall architecture. We have another question from 865 00:49:12,360 --> 00:49:14,120 Arsalan Khan. I'll ask you to answer this 866 00:49:14,120 --> 00:49:16,840 really fast because we're just going to run out of time now who 867 00:49:16,840 --> 00:49:23,000 says should we create a time and motion AI agent that assesses 868 00:49:23,040 --> 00:49:27,600 other agents if they have saved money in an organization, 869 00:49:27,600 --> 00:49:31,600 obviously referencing back to the pricing discussion we had 870 00:49:31,600 --> 00:49:33,640 earlier? If you can listen to some of the 871 00:49:33,800 --> 00:49:36,320 industry luminaries talk about like we all have some of this 872 00:49:36,800 --> 00:49:40,480 army of agents, or we will have, you know, human employees and 873 00:49:40,480 --> 00:49:44,600 agentic employees. There is a requirement to one 874 00:49:44,600 --> 00:49:47,480 like train all these agents, ground all these agents, right, 875 00:49:48,240 --> 00:49:50,360 as well as to evaluate all these agents. 876 00:49:51,120 --> 00:49:54,080 So we talked about reflection loops in, in, in terms of like 877 00:49:54,080 --> 00:49:57,120 the specific sort of output governing these output. 878 00:49:57,120 --> 00:50:01,280 So similarly at a higher level of abstraction, which is the 879 00:50:01,280 --> 00:50:03,120 value, right? Yeah, it is. 880 00:50:03,360 --> 00:50:06,160 You know, you could, it's a, it's an interesting idea to be 881 00:50:06,160 --> 00:50:09,160 able to say you have an agent that's constantly measuring the 882 00:50:09,160 --> 00:50:13,440 value of the output of other agents to ensure that they are 883 00:50:13,440 --> 00:50:15,640 meeting a particular mark. For example, if you're a 884 00:50:15,640 --> 00:50:19,320 customer support agent, right? If it's not deflecting whatever, 885 00:50:19,520 --> 00:50:21,960 you know, 100 calls a day or something like that, right? 886 00:50:22,160 --> 00:50:25,880 And the metric may vary, then maybe it's not performing 887 00:50:25,880 --> 00:50:29,080 appropriately. So and you could have an agent 888 00:50:29,080 --> 00:50:31,440 that's it's, it's more like an operational function. 889 00:50:31,440 --> 00:50:36,440 So there are ways to, I think in this sort of agentic future, 890 00:50:36,720 --> 00:50:40,600 they are the worker AI agents and they're these sort of, you 891 00:50:40,600 --> 00:50:45,360 know, evaluation agents and you potentially will have manager 892 00:50:45,360 --> 00:50:47,600 agents at some point, right? You can, you can think of a 893 00:50:47,600 --> 00:50:50,560 future where there is some level of sort of different levels of 894 00:50:50,560 --> 00:50:55,200 hierarchy where you are actively evaluating and governing these 895 00:50:55,200 --> 00:50:56,600 agents. But again, we're we're early 896 00:50:56,600 --> 00:50:58,160 days yet. We're a little bit sort of 897 00:50:58,400 --> 00:51:00,320 hypothesizing how this looks like. 898 00:51:00,880 --> 00:51:05,600 Gus Beck Dash comes back and he says is it really better to have 899 00:51:05,600 --> 00:51:10,440 the agency and the knowledge in one model or have them as 900 00:51:10,440 --> 00:51:13,240 separate, loosely integrated systems? 901 00:51:13,680 --> 00:51:17,160 It'll come back to the type of problems space that you're 902 00:51:17,160 --> 00:51:21,440 addressing, right? You know, in general, we know 903 00:51:21,720 --> 00:51:25,160 from just, you know, the things around us that there's no such 904 00:51:25,160 --> 00:51:29,320 thing as one unified body of knowledge, right? 905 00:51:29,320 --> 00:51:33,840 We as human beings, there's so much complexity in sort of our 906 00:51:33,840 --> 00:51:38,320 world, in our workplaces, in our sort of consumer oriented lives, 907 00:51:38,320 --> 00:51:43,800 that there's no such thing as like 1 mega intelligence that 908 00:51:43,840 --> 00:51:45,400 essentially does everything for you. 909 00:51:46,680 --> 00:51:51,120 So, you know, I think we always solve problems by breaking down, 910 00:51:51,160 --> 00:51:53,440 breaking them down into smaller problems, right? 911 00:51:53,680 --> 00:51:56,400 And then using different tools to solve those problems and put 912 00:51:56,400 --> 00:51:59,960 these things back together. That's sort of the, the way, you 913 00:51:59,960 --> 00:52:02,760 know, human workflow happens, you know, irrespective of 914 00:52:02,760 --> 00:52:06,360 whether it's a, you're building a chair right, as a project or 915 00:52:06,360 --> 00:52:10,240 you're building a complex set of application in an enterprise. 916 00:52:10,360 --> 00:52:14,880 So I will again hypothesize here and say that I don't believe in 917 00:52:14,880 --> 00:52:16,320 this sort of single unified agent. 918 00:52:16,440 --> 00:52:19,360 I do think agents will continue to get better. 919 00:52:19,800 --> 00:52:22,440 Maybe we will get to this threshold of artificial general 920 00:52:22,440 --> 00:52:23,880 intelligence. How will you define it? 921 00:52:24,120 --> 00:52:25,960 That's another at worst conversation. 922 00:52:26,280 --> 00:52:31,200 But I think you will always have this notion of taking a problem, 923 00:52:31,200 --> 00:52:33,960 breaking it down, using different tools to solve that 924 00:52:33,960 --> 00:52:36,200 problem, to put it together. And you can think of that same 925 00:52:36,200 --> 00:52:38,720 architecture applying in an agentic world as well. 926 00:52:39,280 --> 00:52:44,920 OK, And with that, this has been an action-packed hour. 927 00:52:45,440 --> 00:52:49,960 Praveen Akiraju, thank you so much for taking time to share 928 00:52:49,960 --> 00:52:51,880 your expertise and knowledge with us today. 929 00:52:51,880 --> 00:52:53,400 I really, really do appreciate you. 930 00:52:53,920 --> 00:52:55,280 Thank you. Thank you for having me. 931 00:52:55,720 --> 00:52:58,280 And a huge thank you to everybody who watch. 932 00:52:58,280 --> 00:53:03,960 Before you go, subscribe to the CXO Talk newsletter so you can 933 00:53:03,960 --> 00:53:08,720 join our community and we can tell you about our upcoming 934 00:53:08,720 --> 00:53:13,080 shows, which we have great ones, and you can ask your questions 935 00:53:13,080 --> 00:53:15,440 during the live show just as today. 936 00:53:15,800 --> 00:53:21,160 And with that, a huge thank you to everybody and to Praveen. 937 00:53:21,440 --> 00:53:25,080 And I wish everybody a great day and we'll see you again next 938 00:53:25,080 --> 00:53:25,720 time. Take care.