1 00:00:25,160 --> 00:00:39,240 The. Welcome to the UN Export 2 00:00:39,240 --> 00:00:41,560 Territory podcast. For today's episode I invited 3 00:00:41,560 --> 00:00:44,680 Frank Elementary Show to explain to us GPU resource management 4 00:00:44,680 --> 00:00:47,440 and the challenges customs are facing with AI workloads. 5 00:00:47,640 --> 00:00:49,680 Hey, welcome to the show, Frank. Hey, Duncan. 6 00:00:50,120 --> 00:00:52,720 Now, I would hope that anyone who's listening to the Unexport 7 00:00:52,720 --> 00:00:55,520 territory knows who you are because of course you're one of 8 00:00:55,520 --> 00:00:57,640 the Co hosts for a very long time. 9 00:00:58,040 --> 00:01:00,440 But for those who recently started listening, who's Frank 10 00:01:00,440 --> 00:01:02,920 Deneman and what do you do for Broadcom slash VM there? 11 00:01:03,360 --> 00:01:06,760 So my title is what's called a Distinguished Engineer. 12 00:01:06,880 --> 00:01:10,880 I work for the AI Advanced Services team and so we're 13 00:01:10,880 --> 00:01:16,840 responsible for AI services within the the VCF stack. 14 00:01:16,840 --> 00:01:19,480 Well, that's a little bit too broad, right? 15 00:01:19,480 --> 00:01:24,760 Because there's there are a lot of teams within VCF, look at 16 00:01:24,760 --> 00:01:28,120 storage for example. And so each have their own AI 17 00:01:28,120 --> 00:01:33,440 focus. I am tasked to do the cross team 18 00:01:33,440 --> 00:01:39,240 team coordination of things, so from AAI knowledge centre pretty 19 00:01:39,240 --> 00:01:43,520 much to the rest of the organization to help the VCF 20 00:01:43,520 --> 00:01:46,160 platform grow towards an AI platform. 21 00:01:46,600 --> 00:01:50,080 So if our team starts talking about a A the storage guys, 22 00:01:50,480 --> 00:01:52,360 they've probably spoken with you recently. 23 00:01:52,880 --> 00:01:55,400 Yeah, well, yeah, actually the rest of my afternoon I have 24 00:01:55,400 --> 00:01:57,720 storage people on my calendar, so yeah, correct. 25 00:01:58,000 --> 00:02:00,440 I kind of had that feeling already because I'm going to be 26 00:02:00,440 --> 00:02:04,040 at the the tech Advisory Board next week and there's a lot of 27 00:02:04,120 --> 00:02:08,400 storage and AI related topics on the agenda I noticed. 28 00:02:08,720 --> 00:02:10,160 Yeah. Now, of course, but I've got to 29 00:02:10,160 --> 00:02:12,400 discuss that because all of that is still secret. 30 00:02:12,400 --> 00:02:15,160 But what are the things that I did briefly want to discuss 31 00:02:15,160 --> 00:02:18,920 because we just both came from a a event called Vima Connect and 32 00:02:18,920 --> 00:02:20,840 Amsterdam was helped for the very first time. 33 00:02:21,120 --> 00:02:25,360 I did the keynote yesterday and we had about roughly 500 people, 34 00:02:25,360 --> 00:02:28,560 slightly more in the room from 20 different countries. 35 00:02:28,560 --> 00:02:31,840 So I think that was really cool. You were there as well. 36 00:02:31,840 --> 00:02:33,920 You were there on Tuesday and of course you were there today as 37 00:02:33,920 --> 00:02:34,920 well. How was the event for you? 38 00:02:35,280 --> 00:02:38,040 Good. It was the first for three days 39 00:02:38,280 --> 00:02:40,440 Vmark event. One of the key things like you 40 00:02:40,440 --> 00:02:43,520 said, well like 2020 customers. I, I spoke with Brad, he said he 41 00:02:43,520 --> 00:02:46,440 talked about 2628 sorry countries. 42 00:02:46,800 --> 00:02:50,200 So I think it's a very, it became a very international 43 00:02:50,520 --> 00:02:53,080 event. The only thing that I, I, I 44 00:02:53,080 --> 00:02:54,800 didn't see that much were Dutch people. 45 00:02:55,160 --> 00:03:01,320 And so it was in Amsterdam, yet the majority of people were non 46 00:03:01,320 --> 00:03:04,760 Dutch speaking persons. Because I'm on Tuesday, I had 47 00:03:04,760 --> 00:03:09,320 this workshop for two hours and I asked like, can I do this in 48 00:03:09,320 --> 00:03:10,840 Dutch or should I talk in English? 49 00:03:10,840 --> 00:03:12,920 Like who's who speak, who doesn't speak Dutch? 50 00:03:12,920 --> 00:03:16,200 And 80% of the hands basically went up like, yeah, no Dutch 51 00:03:16,200 --> 00:03:18,800 here. So I had of course, luckily I 52 00:03:18,800 --> 00:03:22,880 had to do it in English, but yeah, so it was really 53 00:03:22,880 --> 00:03:25,240 interesting. And so I did the two hours or 54 00:03:25,480 --> 00:03:29,480 roughly 2 hours on Tuesday and I had a bake out session today, 55 00:03:29,480 --> 00:03:32,160 which was 45 minutes. And I basically took most of the 56 00:03:32,160 --> 00:03:35,120 content for the two hours and I crammed it in into the 45 57 00:03:35,120 --> 00:03:37,560 minutes. So these people were well, well, 58 00:03:38,000 --> 00:03:40,800 I, I have some pity for that, that, that I basically blasted 59 00:03:40,800 --> 00:03:43,000 through that content in, in 45 minutes. 60 00:03:44,080 --> 00:03:48,400 Yeah, they were probably cooked. And were you speaking at what 61 00:03:48,400 --> 00:03:51,240 level Duncan Epping fast or William Lamb fast? 62 00:03:51,240 --> 00:03:52,680 Because. That's a completely different 63 00:03:52,680 --> 00:03:54,640 level. Like like like Usain Bolt, 64 00:03:55,440 --> 00:03:57,120 William Lamb. Sprinting. 65 00:03:57,400 --> 00:03:58,800 Sprinting will stop the finish now. 66 00:03:59,040 --> 00:04:00,560 Yeah, it's something that I noticed as well. 67 00:04:00,560 --> 00:04:03,560 A lot what A lot of different or a lot of people from different 68 00:04:03,560 --> 00:04:05,320 countries. You know, I did meet a lot of 69 00:04:05,600 --> 00:04:08,520 Dutch people, but I was there on Wednesday, so maybe some of them 70 00:04:08,520 --> 00:04:09,960 actually came out for just a day. 71 00:04:09,960 --> 00:04:13,400 I'm not sure about that. I didn't really ask people if 72 00:04:13,400 --> 00:04:15,920 they were there for multiple days or not, so I didn't really 73 00:04:15,920 --> 00:04:17,920 notice that. But I would say at least the 74 00:04:17,920 --> 00:04:21,600 majority of sessions that I went to and the conversations that I 75 00:04:21,600 --> 00:04:24,400 had, people were very positive. Most of the people that came 76 00:04:24,400 --> 00:04:26,640 from abroad, for them, of course, going to Amsterdam is a 77 00:04:26,640 --> 00:04:30,760 treat for most Dutch people. Well, except for you because, 78 00:04:30,960 --> 00:04:32,640 yeah, except for you because you live close to it. 79 00:04:32,640 --> 00:04:35,560 But for people like myself, we have to travel for 2 1/2 hours. 80 00:04:36,080 --> 00:04:38,320 Yeah. I mean, I, I could do without 81 00:04:38,320 --> 00:04:42,320 it, but it is what it is. I, I, I just told Brad as well, 82 00:04:42,480 --> 00:04:45,280 the organizer of course, of the event that I thought it was a 83 00:04:45,280 --> 00:04:48,280 fantastic event and especially for the very first edition, 84 00:04:48,280 --> 00:04:50,200 right? If you look at the setup of the 85 00:04:50,800 --> 00:04:55,040 the solutions exchange, the session rooms, just the general 86 00:04:55,040 --> 00:04:58,840 general vibe it all felt, you know, like a a remark on 87 00:04:58,840 --> 00:05:01,880 steroids basically. Yeah, it was, it was a good 88 00:05:01,880 --> 00:05:03,800 event. And and like you said, right, I 89 00:05:03,800 --> 00:05:06,160 live close by. So for me it was a 20 minute, a 90 00:05:06,160 --> 00:05:09,600 20 minute drive. So if I compare the last debock 91 00:05:09,600 --> 00:05:13,920 where I had to fly for, well, I traveled for 13 hours, I did a 92 00:05:13,920 --> 00:05:16,560 keynote for 45 minutes and then basically went on the plane back 93 00:05:17,080 --> 00:05:21,000 the same amount of hours. This was the light and put that 94 00:05:21,000 --> 00:05:22,880 way. So they're also doing another 95 00:05:23,160 --> 00:05:27,200 four or five events in the USI don't know if you want to do any 96 00:05:27,200 --> 00:05:29,680 of those, but if you want. To I wish them all good luck. 97 00:05:31,240 --> 00:05:33,880 So you have no desire to fly to Minneapolis again? 98 00:05:34,080 --> 00:05:36,720 No. And, and to be honest, look, if, 99 00:05:36,720 --> 00:05:40,800 if I think like Hawk, it won't make any financial sense. 100 00:05:40,920 --> 00:05:44,720 There are a lot of people inside the US who can, who can cover 101 00:05:44,720 --> 00:05:46,520 the, the, the, the functionality. 102 00:05:46,520 --> 00:05:50,720 There are many different persons within the AI teams at at field 103 00:05:50,720 --> 00:05:54,960 level, at sales level. And so they should spare hat 104 00:05:54,960 --> 00:05:58,480 this, this effort in on the continent of the, of the US for 105 00:05:58,480 --> 00:06:01,040 sure. If it's South America, I think 106 00:06:01,040 --> 00:06:03,960 that's a different discussion, which we already had in in the 107 00:06:03,960 --> 00:06:06,800 past anyways, right? Yeah, of course, I wouldn't mind 108 00:06:06,800 --> 00:06:09,080 visiting, you know, Argentina, Brazil. 109 00:06:09,600 --> 00:06:11,920 Yeah, I would make a exception for that one. 110 00:06:12,120 --> 00:06:14,920 I'll also make an exception for V Mug Japan, by the way, so if 111 00:06:14,920 --> 00:06:18,000 you guys are listening, V Mug Japan, invite me. 112 00:06:18,240 --> 00:06:21,720 Yes, and and and of course V mug Huawei, of course that's. 113 00:06:22,360 --> 00:06:26,120 That's the best one for sure Now we're not going to talk about V 114 00:06:26,120 --> 00:06:29,320 mugs today. We're going to talk about GP us 115 00:06:29,320 --> 00:06:33,560 and how GP us are consumed by AI workloads and the complexity cut 116 00:06:33,560 --> 00:06:35,960 that comes with that now that's probably a good starting point. 117 00:06:37,000 --> 00:06:39,960 Even at the event, I had a whole bunch of conversations because 118 00:06:39,960 --> 00:06:43,400 of course in my slide deck, AI popped up a few times as well. 119 00:06:43,960 --> 00:06:48,080 And I was wondering, it's probably, you know, as I said, a 120 00:06:48,080 --> 00:06:50,560 good starting point first to explain what some of the 121 00:06:50,560 --> 00:06:53,760 problems are the customers are facing today when it comes to 122 00:06:53,760 --> 00:06:57,760 these GPU attached or GPU related workloads like AI. 123 00:06:58,360 --> 00:07:02,800 So we can start infrastructure up or we can start or we can 124 00:07:02,800 --> 00:07:05,920 look application down. And so if we look application 125 00:07:05,920 --> 00:07:11,360 down, it's finding the first use case like everybody wants to do 126 00:07:11,400 --> 00:07:17,120 AI, everybody sees the value of AI, OK, but how can you actually 127 00:07:17,160 --> 00:07:22,080 create something that shows value to the business that that 128 00:07:22,080 --> 00:07:24,640 makes sense also from an investment perspective, because 129 00:07:24,640 --> 00:07:26,960 these things are not cheap. The whole infrastructure is not 130 00:07:26,960 --> 00:07:31,280 cheap and is, does your data center actually have the 131 00:07:31,280 --> 00:07:34,760 capability from from a power perspective, from a cooling 132 00:07:34,760 --> 00:07:36,440 perspective, all of these things, right? 133 00:07:36,680 --> 00:07:39,720 And then of course, this is the big elephant in the room. 134 00:07:40,520 --> 00:07:42,560 What are you going to do with your data, right? 135 00:07:43,040 --> 00:07:46,840 First of all, it's typically price with data that you want to 136 00:07:46,840 --> 00:07:52,240 use for your, for your first POC, for your experiment. 137 00:07:52,880 --> 00:07:55,200 And so you need to think about that as well. 138 00:07:55,560 --> 00:07:57,880 And I was going to dig a little bit more deeper into that as 139 00:07:57,880 --> 00:08:06,240 well, is that data is stored, I don't have to tell you this in 140 00:08:06,240 --> 00:08:11,200 many different ways in inside the organization, we always had 141 00:08:11,200 --> 00:08:14,400 trouble finding data. We always had trouble than 142 00:08:14,400 --> 00:08:18,280 accessing the data. And now the third angle is the 143 00:08:18,280 --> 00:08:22,760 quality of the data. It can be very good for a human. 144 00:08:23,200 --> 00:08:28,280 It might not be as good to do this at A at a automated level 145 00:08:28,560 --> 00:08:32,960 where you can actually derive answers or make conclusions out 146 00:08:32,960 --> 00:08:36,039 of the data that is presented. So there's a data call of the 147 00:08:36,120 --> 00:08:40,039 issue as well right now, if you would basically, if you have 148 00:08:40,039 --> 00:08:43,760 covered that, then we start to dive into the infrastructure. 149 00:08:44,400 --> 00:08:47,560 We start looking at the data center design, of course, with 150 00:08:47,560 --> 00:08:50,280 all what I said about the power consumption and about the 151 00:08:50,280 --> 00:08:55,400 energy, about the, the cooling, what type of server do you need? 152 00:08:55,480 --> 00:08:58,680 What is the configuration? And then the, the big one for, 153 00:08:58,680 --> 00:09:01,560 for, for the, the audience is how do you actually manage all 154 00:09:01,560 --> 00:09:03,680 of this? Because that's the topic by 155 00:09:03,680 --> 00:09:06,560 itself. The funny thing is that we've 156 00:09:06,560 --> 00:09:11,480 seen similar challenges in the past when it came to, you know, 157 00:09:11,920 --> 00:09:14,840 when virtual machine, virtual machines were kind of new or 158 00:09:14,840 --> 00:09:16,800 virtualization was just brought into the market. 159 00:09:16,800 --> 00:09:19,040 And of course, we saw some challenges around that as well, 160 00:09:19,520 --> 00:09:22,560 especially from ACPU and memory perspective. 161 00:09:23,760 --> 00:09:26,440 They probably are. The challenges were probably 162 00:09:26,440 --> 00:09:29,920 similar to what we are facing with GPUs today. 163 00:09:30,120 --> 00:09:33,080 But of course the GPU is a completely different beast. 164 00:09:33,080 --> 00:09:36,680 Actually yesterday at the Devima Connect event, I was talking to 165 00:09:36,880 --> 00:09:41,520 to Intel and one of the the guys also was mentioned or they 166 00:09:41,520 --> 00:09:44,720 actually had a talk which more or less was on that on the 167 00:09:44,720 --> 00:09:46,760 topic. Now for those who are listening, 168 00:09:46,800 --> 00:09:49,640 it's, it's probably also good to explain, you know, what is so 169 00:09:49,640 --> 00:09:54,360 different about managing GPU resource resources versus GPU 170 00:09:54,360 --> 00:09:57,280 resources? Because if you look at CPU 171 00:09:57,280 --> 00:10:01,080 resources, well, most of us have been doing that for 20 years 172 00:10:01,080 --> 00:10:04,120 already. But GPUI, you know, I had a 173 00:10:04,120 --> 00:10:07,520 couple of systems with the GPU in there for 12 to 18 months, 174 00:10:08,080 --> 00:10:10,520 but but that's it. I don't even have any experience 175 00:10:10,520 --> 00:10:12,360 with it. So why is it that different? 176 00:10:13,040 --> 00:10:14,400 Yeah. You didn't do it at scale and 177 00:10:14,400 --> 00:10:17,600 you already had some, some issues right at at at the guest 178 00:10:17,600 --> 00:10:19,320 level and then at the lower level. 179 00:10:19,640 --> 00:10:22,640 OK, so let's let's start off with the basics first, right? 180 00:10:22,640 --> 00:10:27,640 So you and me have been in this industry for 20 years, right? 181 00:10:28,040 --> 00:10:32,480 So we've been, we've been basically used to thinking about 182 00:10:32,480 --> 00:10:37,280 CPU and memory as there are these resources and these 183 00:10:37,280 --> 00:10:41,520 resources can be oversubscribed these resources even if you 184 00:10:41,520 --> 00:10:44,640 don't design for it. Let's say you say I won't ever 185 00:10:44,640 --> 00:10:48,160 oversubscribe my memory, then that's your choice. 186 00:10:48,240 --> 00:10:51,080 But there's still the capability in the system itself, in the 187 00:10:51,080 --> 00:10:53,000 platform itself. Think about maintenance mode, 188 00:10:53,000 --> 00:10:56,840 think about HA events, think about affinity rule going wrong. 189 00:10:56,840 --> 00:10:59,480 I don't know, right? But the reality is, even if you 190 00:10:59,480 --> 00:11:03,080 design for it that you're not going to use oversubscription, 191 00:11:03,760 --> 00:11:08,240 you still have the the ability to do this, to do so, and the 192 00:11:08,240 --> 00:11:11,640 system has the ability to do so. So you're not locked out, you 193 00:11:11,640 --> 00:11:15,760 don't have placement issues, you don't have all of these these 194 00:11:15,760 --> 00:11:18,640 these problems that will interfere with your day-to-day 195 00:11:18,640 --> 00:11:19,920 operations. Right. 196 00:11:20,360 --> 00:11:22,600 And here's the key, and this is the most important thing. 197 00:11:23,240 --> 00:11:26,240 If you have free capacity, if you see free capacity in your 198 00:11:26,240 --> 00:11:29,320 dashboard, that typically means it's useful. 199 00:11:29,840 --> 00:11:32,560 Or we could talk about HA and admission control analysis. 200 00:11:32,560 --> 00:11:39,040 But in general, meaning if I see the system has 70% is 70% 201 00:11:39,040 --> 00:11:43,120 utilized, I can still use 20 or 30% easily, right? 202 00:11:43,120 --> 00:11:47,240 And I can overscribe and I can basically trust a particular 203 00:11:47,240 --> 00:11:51,760 functionalities that will allow to to do prioritization and to 204 00:11:51,760 --> 00:11:53,680 do some form of quality of service. 205 00:11:54,320 --> 00:11:57,440 It's not the case with GPS. First of all, there is no open 206 00:11:57,440 --> 00:12:01,640 subscription in GPUs. If you use the three different 207 00:12:01,960 --> 00:12:05,200 methods, you have a pass through, meaning you go to 208 00:12:05,200 --> 00:12:07,960 expose a physical device into a virtual machine and you allow 209 00:12:07,960 --> 00:12:09,640 the virtual machine to manage that device. 210 00:12:10,040 --> 00:12:11,360 That's a one to one relationship. 211 00:12:12,000 --> 00:12:16,240 If you then go to VGPU, for example, you have what's called 212 00:12:16,240 --> 00:12:21,120 fractional GPUs where you take a portion of a full GPU or you can 213 00:12:21,120 --> 00:12:24,800 have full GPUs or multi GPUs functionality. 214 00:12:25,760 --> 00:12:30,440 Each of these components, they will they will have some 215 00:12:30,680 --> 00:12:34,720 placement restrictions, some some topology awareness that you 216 00:12:34,720 --> 00:12:37,080 need to take into account. And especially when you go to 217 00:12:37,080 --> 00:12:42,640 fractional where you can say I want to use this GPU and assign 218 00:12:42,640 --> 00:12:47,440 it to a multiple VMS and I'm going to use a VGPU profile 219 00:12:47,600 --> 00:12:50,040 which has a little bit of compute and a little bit of 220 00:12:50,040 --> 00:12:53,400 memory. And I will basically make use of 221 00:12:53,400 --> 00:12:59,120 the the different workload patterns in the GPU to basically 222 00:12:59,120 --> 00:13:03,880 get a good performance. But if you have 80 gigs of 223 00:13:03,880 --> 00:13:08,880 memory inside the GPU, there is no way to allocate 100 gigs or 224 00:13:08,880 --> 00:13:12,920 provision 100 gigs. It is literally what's on the 225 00:13:12,920 --> 00:13:18,200 card can be provisioned once or allocated once and that's it. 226 00:13:18,280 --> 00:13:22,640 Even if those virtual machines are not using it, now you can 227 00:13:22,640 --> 00:13:25,880 say if people understand time slicing and make you can say, 228 00:13:25,880 --> 00:13:28,800 no, that's not entirely true, correct, right. 229 00:13:28,800 --> 00:13:32,120 Because if we do time slice, we do time slicing on the compute 230 00:13:32,120 --> 00:13:34,440 level. So we have the 80 gigs again, 231 00:13:35,160 --> 00:13:41,720 I'm going to use a profile with 20 gigs of resources that that 232 00:13:41,720 --> 00:13:46,040 portion will be allocated to that virtual machine from the 233 00:13:46,040 --> 00:13:49,120 moment it starts to the moment it powers down. 234 00:13:50,440 --> 00:13:53,480 Even if it doesn't use it, it's there for that particular 235 00:13:53,480 --> 00:13:56,240 virtual machine, right? Where that's something that the 236 00:13:56,240 --> 00:14:01,320 VGPU Dr. will will decide. But then if it needs to use GPU 237 00:14:01,320 --> 00:14:06,200 for its computations with time slice, it will wait its turn. 238 00:14:06,440 --> 00:14:11,240 Then it basically can use the entire compute and its memory 239 00:14:11,240 --> 00:14:13,720 portion will feed that compute part. 240 00:14:14,240 --> 00:14:18,520 Now the moment some other VM wants to run, they need to 241 00:14:18,520 --> 00:14:21,120 schedule right? And so it's time slicing. 242 00:14:21,120 --> 00:14:24,440 So the next slice will be for the other virtual machine and it 243 00:14:24,440 --> 00:14:29,200 can compute based on the memory slice it has on that on that 244 00:14:29,280 --> 00:14:32,080 device. Now when it comes to make multi 245 00:14:32,080 --> 00:14:36,960 instance DPU, it's a hardware isolation pattern, meaning I'm 246 00:14:36,960 --> 00:14:41,920 going to create multiple slices inside my DPU and that is 247 00:14:41,920 --> 00:14:47,880 assigned both compute and memory 24/7 or as long as the lifecycle 248 00:14:47,880 --> 00:14:51,160 of the virtual machine lasts to that particular virtual machine. 249 00:14:51,520 --> 00:14:54,680 If it's idle, nobody else can use it. 250 00:14:55,000 --> 00:14:59,800 If it is active, it's the only virtual machine in that area of 251 00:14:59,800 --> 00:15:03,360 computer memory that can use it. So it's a very deterministic 252 00:15:03,360 --> 00:15:08,640 way, but it's a very strict way of chopping up a car, right? 253 00:15:08,920 --> 00:15:12,520 So depending on your workload, you either choose one or the 254 00:15:12,520 --> 00:15:15,360 other. Now, if you then take one step 255 00:15:15,360 --> 00:15:19,920 further, you need to think about placement, placement on a on a 256 00:15:19,920 --> 00:15:25,320 GPU placement on of workload in, in a host with multiple GPUs and 257 00:15:25,320 --> 00:15:30,200 placement on on for a virtual machine in the cluster. 258 00:15:30,560 --> 00:15:33,440 And there are all these profiles, all these policies 259 00:15:33,800 --> 00:15:38,640 that will have an impact on where it's placed, where the, 260 00:15:39,280 --> 00:15:42,880 where the GPU is compatible with the workload requirement. 261 00:15:43,560 --> 00:15:50,440 And so in short, what you see from a free capacity perspective 262 00:15:50,800 --> 00:15:53,880 does not always mean that it's useful, right? 263 00:15:53,880 --> 00:15:57,200 So there is this mismatch where people are building dashboards 264 00:15:57,200 --> 00:16:02,480 and they'll say, OK, let's say I've got a, a TB of GPU memory 265 00:16:02,480 --> 00:16:08,840 in my in my cluster, but I only see 700 gigs being utilized. 266 00:16:09,920 --> 00:16:13,240 It can mean in certain situation that this is a very worst case 267 00:16:13,240 --> 00:16:15,640 scenario. But just to prove my point, it 268 00:16:15,640 --> 00:16:18,960 can be that you cannot power on any virtual teams anymore, even 269 00:16:18,960 --> 00:16:23,400 though you're staring at 300 gigs of underutilized space due 270 00:16:23,400 --> 00:16:26,360 to the way things need to be placed due to the way patterns 271 00:16:26,640 --> 00:16:28,680 appear from a policy perspective. 272 00:16:29,160 --> 00:16:33,080 And that's, I believe one of the most, most difficult things, 273 00:16:33,280 --> 00:16:38,280 understanding how your decision of profile choice will 274 00:16:38,280 --> 00:16:41,160 eventually drive to a lower utilization in your cluster. 275 00:16:41,480 --> 00:16:45,760 You've spoken about these, the option to have things configured 276 00:16:45,760 --> 00:16:48,800 in a pass through manner in a fractional manner as well. 277 00:16:48,800 --> 00:16:52,560 You talked about MIC, I don't know if that's similar to VGPU. 278 00:16:52,560 --> 00:16:55,840 We also have the dynamic model. Can you mix and match those 279 00:16:55,840 --> 00:16:58,960 models or do you need to select one and you're stuck to that 280 00:16:58,960 --> 00:17:04,440 particular one as well? It depends on typically super 281 00:17:04,440 --> 00:17:09,640 GPU decision. So I can have a host with eight 282 00:17:09,680 --> 00:17:14,960 GPUs and some of these are are set to direct paths, meaning I 283 00:17:14,960 --> 00:17:18,800 will basically allocate a full physical device to a a virtual 284 00:17:18,800 --> 00:17:21,599 machine. I can have some virtual machines 285 00:17:21,640 --> 00:17:26,160 doing fractional GPUs do make that's also a fractional GPU. 286 00:17:26,160 --> 00:17:31,120 Both are a VGPU construct and I can assign some of them that 287 00:17:31,120 --> 00:17:34,200 they are open for multi GPU functionality. 288 00:17:34,760 --> 00:17:40,440 It's a per GPU decision whether that makes sense from managing 289 00:17:40,440 --> 00:17:45,880 things at scale because we are always, we're so trained to, to 290 00:17:45,880 --> 00:17:48,800 think in homogeneous configurations inside 291 00:17:48,800 --> 00:17:53,000 virtualization that if you do this, then you typically need to 292 00:17:53,000 --> 00:17:55,960 dive in and drill into particular configurations per 293 00:17:55,960 --> 00:18:01,160 host that will, will, will, will have an impact on operational 294 00:18:01,160 --> 00:18:07,440 aspects, on visibility aspects. So because you could doesn't 295 00:18:07,440 --> 00:18:09,680 mean you should, right? But it's possible. 296 00:18:09,680 --> 00:18:13,760 If you have a use case that that fits that fine, it's supported. 297 00:18:14,480 --> 00:18:18,000 I think that's it's very valid to say that if you can doesn't 298 00:18:18,000 --> 00:18:19,280 necessarily mean you need to do it. 299 00:18:19,840 --> 00:18:22,200 I just had one of those discussions with a VCENT 300 00:18:22,200 --> 00:18:24,560 customer as well. They were planning on adding 301 00:18:24,560 --> 00:18:28,160 additional devices to their hosts and the the new device 302 00:18:28,160 --> 00:18:30,320 that we're planning on introducing with 10 times bigger 303 00:18:30,320 --> 00:18:32,840 than the existing devices. And of course we fully support 304 00:18:32,840 --> 00:18:34,160 that. But from an operational 305 00:18:34,160 --> 00:18:37,920 perspective, from a fairly domain standpoint it may not be 306 00:18:37,920 --> 00:18:40,000 be smart. So I think that's a very valid. 307 00:18:40,440 --> 00:18:44,400 Now if you look at the stack itself, we also have of course 308 00:18:44,400 --> 00:18:47,360 solutions that help with scheduling and placement, like 309 00:18:47,360 --> 00:18:51,080 for instance DRS. So does DRS actually understand 310 00:18:51,080 --> 00:18:54,400 the whole GPU concept and does it help you with the placement 311 00:18:54,400 --> 00:18:57,880 itself or are you responsible and you need to do it manually? 312 00:18:58,080 --> 00:18:59,760 How does that work? Yeah. 313 00:18:59,960 --> 00:19:04,760 So DRS is what we call GPU aware, meaning that we 314 00:19:04,760 --> 00:19:10,760 understand or we as the royal we DRS understands GPU, the 315 00:19:10,760 --> 00:19:17,080 existence of GPU's, but it won't use it primarily for load 316 00:19:17,080 --> 00:19:21,160 balancing. It's not a a access where it's 317 00:19:21,160 --> 00:19:24,920 going to figure out the load balancing pass perspective, but 318 00:19:25,160 --> 00:19:28,520 what it does do displacement and it does that very well. 319 00:19:28,640 --> 00:19:31,560 Now does it do it all by itself? No. 320 00:19:31,960 --> 00:19:35,520 There's a construct called assignable hardware, and 321 00:19:35,520 --> 00:19:39,880 assignable hardware creates a list of compatible devices in 322 00:19:39,880 --> 00:19:43,360 its Esxi host and it will basically bubble that 323 00:19:43,360 --> 00:19:46,600 information up to DRS. So each host will do that. 324 00:19:47,080 --> 00:19:50,960 And so DRS will get a good sense of what's available or not. 325 00:19:51,400 --> 00:19:54,520 And then depending on what type of GPU you have, the moment 326 00:19:54,520 --> 00:19:57,960 you're trying to place that virtual machine, the arrest 327 00:19:57,960 --> 00:20:00,440 together with assignable hardware will create a 328 00:20:00,440 --> 00:20:04,120 compatibility list for that particular virtual machine. 329 00:20:04,560 --> 00:20:07,440 And then it will say, OK, these are these are the lists that are 330 00:20:07,440 --> 00:20:11,680 from a GPU perspective, these are compatible. 331 00:20:11,800 --> 00:20:15,480 Now let me do my own magic. Let me figure out CPU, let me 332 00:20:15,480 --> 00:20:18,880 figure out memory in certain cases, networking as well. 333 00:20:19,400 --> 00:20:23,120 And then say OK, this is the best host possible. 334 00:20:23,680 --> 00:20:27,840 For placing the virtual machine while it has the GPU resources 335 00:20:28,200 --> 00:20:32,920 and I'm also aligning to the VM happiness of the VM and also the 336 00:20:32,920 --> 00:20:37,320 cluster load off, right? So it's a combination of things. 337 00:20:37,440 --> 00:20:41,160 Now if we then think about like the private AI services with the 338 00:20:41,160 --> 00:20:45,480 model runtime, we basically use that as the orchestration layer. 339 00:20:45,720 --> 00:20:49,560 And so when a data scientist says I want to deploy this model 340 00:20:49,560 --> 00:20:52,640 endpoint, we basically work with DRS. 341 00:20:52,720 --> 00:20:58,360 So in VTS worker notice build a, a VM class is used to basically 342 00:20:58,800 --> 00:21:01,760 allocate the GPUs that's being sent to DRS. 343 00:21:02,000 --> 00:21:06,360 Give me a compatible host for this, this, this, these GPUs, 344 00:21:07,000 --> 00:21:08,840 and then the rest we start to orchestrate. 345 00:21:08,840 --> 00:21:11,960 We basically load the model from the model gallery into the 346 00:21:11,960 --> 00:21:14,640 virtual machine. We basically load it into GPU. 347 00:21:14,640 --> 00:21:18,440 We basically create an insurance engine and we send the model 348 00:21:18,440 --> 00:21:20,160 endpoint back to the to the user. 349 00:21:20,400 --> 00:21:23,400 And so you got this full orchestration by just providing 350 00:21:23,400 --> 00:21:25,920 you a desired state. I just want to run a virtual 351 00:21:25,920 --> 00:21:29,320 machine with some GPUs and this is my model endpoint that I want 352 00:21:29,320 --> 00:21:33,120 to consume to deal with it. And so DRS and available 353 00:21:33,120 --> 00:21:37,120 hardware and all the constructs in in the VCF cluster will start 354 00:21:37,120 --> 00:21:39,440 to work for. You now you said something 355 00:21:39,440 --> 00:21:44,120 interesting because you've mentioned VKS and of course for 356 00:21:44,120 --> 00:21:48,360 those that are not aware that is our Kubernetes based platform, 357 00:21:48,560 --> 00:21:51,560 but that also has its own scheduling mechanism as you 358 00:21:51,560 --> 00:21:54,400 said. So is there any potential for 359 00:21:54,400 --> 00:21:59,480 DRS and that scheduling mechanism to start interfering 360 00:21:59,480 --> 00:22:01,360 or start interrupting each other? 361 00:22:01,920 --> 00:22:03,400 Could there be any conflicts or whatever? 362 00:22:03,400 --> 00:22:06,680 Because typically or normally you wouldn't see two placement 363 00:22:06,680 --> 00:22:08,600 mechanisms stacked on top of each other. 364 00:22:08,600 --> 00:22:10,760 Or do they deal with something different? 365 00:22:11,280 --> 00:22:13,280 No. And that's the beauty of of 366 00:22:13,280 --> 00:22:17,480 virtualized Kubernetes. So yeah, you have a schedule 367 00:22:17,480 --> 00:22:21,920 inside your guest OS, inside your working out, typically as 368 00:22:21,920 --> 00:22:27,360 Ubuntu, and then you have your your normal case schedulers. 369 00:22:27,800 --> 00:22:30,320 But the reality of a virtual machine is that it's always a 370 00:22:30,320 --> 00:22:34,960 confined world of what the administrator exposes. 371 00:22:35,080 --> 00:22:38,640 Meaning if I create a virtual machine and I use AVM class and 372 00:22:38,640 --> 00:22:42,920 AVM class like a, a template pattern, it's not a, a template 373 00:22:42,920 --> 00:22:45,360 in the sense of I'm using these bits and bytes. 374 00:22:45,560 --> 00:22:47,600 It's more template, a resource template. 375 00:22:47,600 --> 00:22:52,360 Like I want to deploy a virtual machine with X amount of CPUs, X 376 00:22:52,360 --> 00:22:55,920 amount of memory, this storage policy, this amount of storage 377 00:22:56,160 --> 00:22:58,840 and this GPU or these GPUs, right? 378 00:22:58,840 --> 00:23:02,320 Or combination of GPUs and, and NICs and and so on. 379 00:23:02,960 --> 00:23:07,240 Now that is that becomes the source of truth that virtual 380 00:23:07,240 --> 00:23:10,720 machine 4 pits for the Kubernetes scheduler. 381 00:23:11,080 --> 00:23:13,560 They don't understand that the host has eight GPUs. 382 00:23:13,560 --> 00:23:17,240 What they what it sees is the under the underlying layer of 383 00:23:17,240 --> 00:23:19,280 the virtual machine. And let's say you basically 384 00:23:19,280 --> 00:23:23,040 selected the VM class with four GPUs, it will see those four 385 00:23:23,040 --> 00:23:25,760 GPUs and that's this only settling. 386 00:23:26,040 --> 00:23:29,000 We already allocated those GPUs to that particular virtual 387 00:23:29,000 --> 00:23:30,680 machine. Knock yourself out. 388 00:23:31,000 --> 00:23:33,640 Now here's the key, and this is very, very interesting. 389 00:23:34,200 --> 00:23:40,640 If you think about this construct, we are relying on a 390 00:23:40,680 --> 00:23:44,400 situation where the administrator needs to predict 391 00:23:45,240 --> 00:23:48,960 the actual sizing. Right now, that's one of the key 392 00:23:48,960 --> 00:23:52,480 things where we set with base. Let us expose a self-service 393 00:23:52,480 --> 00:23:57,520 portal where we can allow the customer to to basically select 394 00:23:57,520 --> 00:24:00,800 the right VM class. And now they have to, they can 395 00:24:00,800 --> 00:24:04,800 basically deploy the VTS process themselves. 396 00:24:05,600 --> 00:24:08,440 But what we're also doing, and this is very important, this is 397 00:24:08,440 --> 00:24:12,480 basically the scheduling part that I want to zoom in to is 398 00:24:12,480 --> 00:24:17,640 that in the previous versions, there was a very rudimentary 399 00:24:17,800 --> 00:24:22,560 VDPU scheduler, meaning I could, they might deployment YAML file. 400 00:24:22,800 --> 00:24:27,320 I could say I want to have one GPU or two GPU's, right? 401 00:24:28,240 --> 00:24:33,720 That's a little bit too coarse because I might want to have a 402 00:24:33,720 --> 00:24:37,000 particular virtual machine, sorry, a particular container on 403 00:24:37,000 --> 00:24:40,880 a particular GPU or I want to schedule something that has 404 00:24:41,200 --> 00:24:44,160 topology awareness. Now, we already have topology 405 00:24:44,160 --> 00:24:46,160 awareness with assignable hardware. 406 00:24:46,600 --> 00:24:51,120 If you have a, if you have GPUs inside your Esxi host that are 407 00:24:51,120 --> 00:24:54,040 directly connected to each other through an NV link, which is 408 00:24:54,040 --> 00:24:57,000 typically a hard cable, or through an NV switch, which is a 409 00:24:57,000 --> 00:24:59,520 whole mesh, then we understand that. 410 00:24:59,680 --> 00:25:04,320 And then we actually schedule the virtual machine aligned with 411 00:25:04,320 --> 00:25:06,920 the requirements of an NV link or an NV switch. 412 00:25:07,280 --> 00:25:10,320 We isolate things, we make sure everything is correct, and we 413 00:25:10,320 --> 00:25:13,720 give you the best apology. Think about this, if you run a 414 00:25:13,720 --> 00:25:17,080 model, it has to communicate between multi GPUs if you span 415 00:25:17,080 --> 00:25:20,000 it across multi GPUs and it has to do this on a continuous 416 00:25:20,000 --> 00:25:22,920 basis. So the topology matters. 417 00:25:23,160 --> 00:25:28,720 If you communicate through PCI E 5, you get 64 gigs of bandwidth. 418 00:25:28,760 --> 00:25:32,840 If you go through an ND link or an ND switch that's in modern 419 00:25:33,760 --> 00:25:38,080 architectures is 900 gigabytes to 1.8 terabytes of bandwidth 420 00:25:39,000 --> 00:25:42,800 significantly faster right now you want to use that. 421 00:25:42,800 --> 00:25:45,560 So you need to have the platform to understand that the policy. 422 00:25:45,800 --> 00:25:50,200 Now what we're doing with the newer levels of ETS, we're the 423 00:25:50,200 --> 00:25:56,240 first gates distribution that is AI compliant, meaning from this 424 00:25:57,320 --> 00:26:05,880 CNCF organization they created like an AI compliance program. 425 00:26:06,360 --> 00:26:10,080 And one of these things is what's called DRA dynamic 426 00:26:10,080 --> 00:26:13,160 resource allocation. And that allows the container to 427 00:26:13,200 --> 00:26:17,840 actually specify the required topology and we will match that 428 00:26:18,080 --> 00:26:23,520 inside the VM as well. So we will expose the topology 429 00:26:23,520 --> 00:26:28,200 from a device group and then it allows with the DRA driver 430 00:26:28,200 --> 00:26:31,920 installed inside the guest OS. We will allow a container to 431 00:26:31,920 --> 00:26:37,040 actually use that topology instead of letting Linux guess 432 00:26:37,040 --> 00:26:40,120 where to place it. Well, that's a a nice new 433 00:26:40,120 --> 00:26:43,120 acronym DRA and then combined with DRS. 434 00:26:43,440 --> 00:26:46,080 So it's got a cause of confusion I guess. 435 00:26:46,680 --> 00:26:48,960 Well, that the problem. I, I don't think even think 436 00:26:48,960 --> 00:26:52,000 DRA's problem, it's the actual, the, the word metric. 437 00:26:52,000 --> 00:26:55,440 It's called driver. If if you talk to people who are 438 00:26:55,440 --> 00:26:58,720 into hardware and all of that stuff, driver is not the right 439 00:26:58,720 --> 00:27:01,400 term, right? Because it's not in the critical 440 00:27:01,400 --> 00:27:03,640 IO path. It's not a menacing aspect. 441 00:27:03,640 --> 00:27:08,360 It's just an orchestration thing, so the controller would 442 00:27:08,360 --> 00:27:10,040 have been a better word, but hey. 443 00:27:10,800 --> 00:27:14,720 Yeah, it's, it's not a driver, but it's driving something more 444 00:27:14,720 --> 00:27:17,680 or less. Yeah, Yeah, it's, you know, when 445 00:27:17,680 --> 00:27:19,360 you were explaining it, it actually made sense. 446 00:27:19,360 --> 00:27:22,360 But if I would see the acronym for the first time, I probably 447 00:27:22,360 --> 00:27:24,720 wouldn't understand it. Now talking about things I 448 00:27:24,720 --> 00:27:27,840 didn't fully understand either, where I was actually reading 449 00:27:27,840 --> 00:27:29,520 some of your blog posts that you have a whole list. 450 00:27:29,520 --> 00:27:32,160 So I'm going to Add all of them to the to the show now because 451 00:27:32,160 --> 00:27:35,280 they are very valuable. But one of the things that stood 452 00:27:35,280 --> 00:27:39,880 out to me is that you were talking about different VGPU 453 00:27:39,880 --> 00:27:43,080 modes at some point of time, and you were talking about same size 454 00:27:43,080 --> 00:27:46,440 and mixed sizes. Now, I played around with VGPU a 455 00:27:46,440 --> 00:27:50,520 while back, but I can't recall having seen those different 456 00:27:50,520 --> 00:27:52,080 types. So maybe you can explain what it 457 00:27:52,080 --> 00:27:54,600 is and what it does and where that actually fits into this 458 00:27:54,600 --> 00:27:56,040 whole picture. Yes. 459 00:27:56,680 --> 00:28:02,440 So same size and and mixed size are for time slice VGPU 460 00:28:02,440 --> 00:28:05,840 profiles. It doesn't apply for full, it 461 00:28:05,840 --> 00:28:09,440 doesn't apply for mid, it doesn't apply for multi GPU. 462 00:28:09,960 --> 00:28:13,120 What it means is if you're going to deploy virtual machines with 463 00:28:13,120 --> 00:28:20,000 a fractional VGPU profile, we need to set a particular 464 00:28:20,000 --> 00:28:24,920 acceptance mode on the physical GPU inside the ESXI host. 465 00:28:25,120 --> 00:28:31,240 So per GPU, you can specify whether it will accept VGP 466 00:28:31,240 --> 00:28:38,240 profiles with the same size or VGP profiles that are not in the 467 00:28:38,240 --> 00:28:40,600 same size. So we can mix these things, 468 00:28:41,280 --> 00:28:43,200 right? So the reason why you didn't see 469 00:28:43,200 --> 00:28:47,440 it is didn't click through that, that, that that layer inside 470 00:28:47,440 --> 00:28:51,160 the, the, the, the hypervisor or inside the center, I guess, 471 00:28:51,280 --> 00:28:53,280 right? It's not that obvious. 472 00:28:53,760 --> 00:28:59,760 So, but what it means is that it will impact what you can 473 00:28:59,760 --> 00:29:04,160 schedule and what you can place on the the actual host and the 474 00:29:04,160 --> 00:29:07,280 actual GPU. So let's let's use an example of 475 00:29:07,280 --> 00:29:11,240 a host with two GPUs. And now I have 3 virtual 476 00:29:11,240 --> 00:29:15,400 machines. I've got a 10 gig, I got a 20 477 00:29:15,400 --> 00:29:19,080 gig and I got an 8 gig. Now if I place these, if I put 478 00:29:19,080 --> 00:29:22,480 the the both the GPUs in same size and I start the first one 479 00:29:22,480 --> 00:29:25,560 with 10 gig, it will land on one of these two, right? 480 00:29:26,200 --> 00:29:27,880 Doesn't matter. Let's use the first one. 481 00:29:28,440 --> 00:29:29,800 Now the second one is the 20 gig. 482 00:29:30,480 --> 00:29:33,080 It will be placed on the other one, 38 gig. 483 00:29:33,400 --> 00:29:37,120 There's enough room because let's say they are both 80 gigs, 484 00:29:37,120 --> 00:29:40,000 right? So I've got 160 gigs in total of 485 00:29:40,000 --> 00:29:42,920 free or of capacity and I use 30. 486 00:29:42,920 --> 00:29:48,760 So I have 150 gigs in capacity. But because the profile is not 487 00:29:48,760 --> 00:29:53,280 aligned with the same size requirement, it cannot fit the 488 00:29:53,280 --> 00:29:55,640 10 gig because it's not compatible with the 10 gig, 489 00:29:56,120 --> 00:29:58,880 doesn't fit the 20 gig. So you'll get a failed placement 490 00:29:59,640 --> 00:30:05,640 right now with mix size, we say does matter as long as it fits 491 00:30:05,920 --> 00:30:10,720 the GPU. So the 10 gig will be placed on 492 00:30:10,720 --> 00:30:14,800 one, the 20 gigs will be placed probably on the second one and 493 00:30:14,800 --> 00:30:18,600 the third one will be placed either on both, so either on one 494 00:30:18,600 --> 00:30:21,280 or two. And so we can now start to mix 495 00:30:21,280 --> 00:30:23,360 and match. Is this perfect? 496 00:30:23,600 --> 00:30:25,880 No, it's not. Because in order to actually 497 00:30:25,880 --> 00:30:30,360 place make sizes, we need to adhere to the actual logic 498 00:30:30,360 --> 00:30:33,640 inside the GPU. And So what we have are like 499 00:30:33,640 --> 00:30:38,080 slots, like HA slots, but then what we call it placement IDs. 500 00:30:38,920 --> 00:30:42,880 And every profile has a range of placement IDs. 501 00:30:43,040 --> 00:30:47,160 So the 10 gig has placement IDs, 20 gigs, 40 gigs, 8 gig, right? 502 00:30:47,560 --> 00:30:51,240 Now if you place a 10 gig, it will probably start at at 0, so 503 00:30:51,240 --> 00:30:54,600 you get 0 to 9. And so the next placement ID for 504 00:30:54,600 --> 00:30:59,720 10 gigs is 10 to 19. So on until you get reached to 505 00:30:59,720 --> 00:31:02,160 79, right? Because zero to 79 are 80 506 00:31:02,160 --> 00:31:06,720 placement IDs. If I place 10 gigs and 20 gigs 507 00:31:06,720 --> 00:31:11,600 on the same GPU, my first placement ID, my first valid 508 00:31:11,600 --> 00:31:16,400 placement ID for the 20 is not ten because placement ID for 20 509 00:31:16,680 --> 00:31:22,560 is 0 to 1920 to 39 until you get to 79 right? 510 00:31:22,960 --> 00:31:25,400 So my placement starts now at 20. 511 00:31:25,520 --> 00:31:27,800 So I actually have a gap of 10 in between. 512 00:31:28,720 --> 00:31:33,520 If I then basically deploy the 8 gig, it won't fit in between the 513 00:31:33,520 --> 00:31:39,360 10 and the 20 because placement I DS start from zero to seven, 8 514 00:31:39,440 --> 00:31:42,440 to 15. Both are taken by the 10 gigs 515 00:31:42,440 --> 00:31:47,720 and the 50 and the 20 gigs. So my 8 gig will be placed on 516 00:31:47,920 --> 00:31:53,400 behind the the the the the first one after 39 right of the 20 gig 517 00:31:53,680 --> 00:31:56,760 of the 20 gig still follow. Makes sense what I'm saying. 518 00:31:57,600 --> 00:32:02,000 OK, now if I now play some 40 gigs won't be available anymore. 519 00:32:02,440 --> 00:32:05,160 Why? Because that 8 gig actually 520 00:32:05,160 --> 00:32:09,240 started to creep in that for in that valid 40 gig placement ID, 521 00:32:09,440 --> 00:32:12,720 right? So that is one of the challenges 522 00:32:12,720 --> 00:32:16,320 and that's what I meant with because you see free capacity 523 00:32:16,320 --> 00:32:22,480 doesn't mean it's useful, right? So what I did, because I speak 524 00:32:22,480 --> 00:32:28,560 with a lot of customers and I, we have a lab to develop things. 525 00:32:29,160 --> 00:32:31,840 And then I started to think, OK, I need to visualize all of this. 526 00:32:31,840 --> 00:32:36,520 So I actually created 2 tools that you can, that you can see 527 00:32:36,760 --> 00:32:40,160 what the impact is of your choice. 528 00:32:40,320 --> 00:32:45,520 Because it's easy to say it's all the fault of the profile. 529 00:32:45,560 --> 00:32:48,320 No, it's not. It's actually, if you understand 530 00:32:48,320 --> 00:32:52,560 placement IDs, you can also think about your, your profile 531 00:32:52,560 --> 00:32:55,360 catalog. You can say, OK, I got this 8 532 00:32:55,360 --> 00:32:57,440 gig profile and I got this 10 gig profile. 533 00:32:57,440 --> 00:33:02,560 If I have like a workload that requires for example 7 gigs in 534 00:33:02,560 --> 00:33:07,320 our right size mind, which we have talked about for 20 years, 535 00:33:07,560 --> 00:33:10,880 it would make sense say I'll give you an 8 gig profile 536 00:33:11,120 --> 00:33:15,320 because I can then squeeze more workloads into the GPU because I 537 00:33:15,320 --> 00:33:20,800 have got more potentially I've got 10-8 gig profiles in an 80 538 00:33:20,800 --> 00:33:23,520 gig. But because you are mixing and 539 00:33:23,520 --> 00:33:26,560 you're not using a single profile, that doesn't add up 540 00:33:26,560 --> 00:33:28,600 anymore. And so now you need to think 541 00:33:28,600 --> 00:33:31,760 about, OK, maybe 8 gig is not the right choice. 542 00:33:31,760 --> 00:33:34,960 Yeah, it will. I will basically slim fit my 543 00:33:34,960 --> 00:33:38,920 workload into the profile. But in order to right size at 544 00:33:38,920 --> 00:33:42,000 the cluster level I actually need to think about the place my 545 00:33:42,000 --> 00:33:46,920 ID's and where they all line up. So a 10 gig start to make way 546 00:33:46,920 --> 00:33:51,120 more sense because if I had a 10 gig and a 20 gig and I had 547 00:33:51,400 --> 00:33:55,480 instead of the 8 gig, I used the 10 gig, it would fall in between 548 00:33:55,640 --> 00:33:58,480 the 10 and the 20 again. And now I have room for the 40, 549 00:33:59,080 --> 00:34:01,120 right? So I've made these visualizer 550 00:34:01,120 --> 00:34:04,560 where you can basically say these are my this is my catalog 551 00:34:04,560 --> 00:34:06,360 that I want to use, this is my GPU. 552 00:34:06,600 --> 00:34:11,040 Show me what it means. Same size, mix size in mix size. 553 00:34:11,199 --> 00:34:15,080 Show me what it means from a population perspective, whether 554 00:34:15,080 --> 00:34:17,639 I get silo capacity, because that's the problem, right? 555 00:34:17,639 --> 00:34:21,360 The silo capacity, you can see it, it's open, it's not 556 00:34:21,440 --> 00:34:26,199 available for everybody else. So yeah, it's you slim feed it, 557 00:34:26,440 --> 00:34:30,000 but you actually are wasting a lot of actual resources. 558 00:34:30,760 --> 00:34:34,360 Yeah, that's interesting because when I started reading some of 559 00:34:34,360 --> 00:34:38,639 the articles and you spoke about fragmentation, you spoke about 560 00:34:38,639 --> 00:34:41,800 these type of slot sizes, the placement. 561 00:34:41,800 --> 00:34:44,400 It really reminded me of the early days of HA. 562 00:34:44,400 --> 00:34:47,520 And I still recall some of the discussions that we had, some of 563 00:34:47,520 --> 00:34:49,880 the diagrams that we created, of course. 564 00:34:49,880 --> 00:34:53,080 Yeah, it's all in the the various books that we published. 565 00:34:53,080 --> 00:34:58,120 Now then when I went through the list of the articles and I read 566 00:34:58,120 --> 00:35:01,320 the next one and then the other one, of course the tool popped 567 00:35:01,320 --> 00:35:03,960 up. Now if I look at the tool 568 00:35:03,960 --> 00:35:07,000 itself, it really visualises indeed what happens. 569 00:35:07,040 --> 00:35:11,800 And I think it also clearly demonstrates why a GPU is 570 00:35:11,800 --> 00:35:16,600 completely different than ACPU, that it really visualises that. 571 00:35:16,600 --> 00:35:19,560 But if you look at the tool, I can imagine that people use it 572 00:35:19,560 --> 00:35:24,880 to figure out, OK, will it actually work in my environment? 573 00:35:24,960 --> 00:35:26,960 What should the profiles look like? 574 00:35:26,960 --> 00:35:30,880 But can they also use it, for instance, to start sizing their 575 00:35:30,880 --> 00:35:32,440 environment? Because that's probably also a 576 00:35:32,440 --> 00:35:35,480 typical request you get. OK, I've got these workloads. 577 00:35:36,440 --> 00:35:40,160 I'd probably end up buying these GPUs, but will that actually 578 00:35:40,160 --> 00:35:42,360 work? Can you use the tool for that as 579 00:35:42,360 --> 00:35:44,080 well? OK, you even say these are the 580 00:35:44,080 --> 00:35:45,680 workloads. What should I buy? 581 00:35:46,600 --> 00:35:49,520 Yeah, that's the that's, that's the billion dollar question 582 00:35:49,520 --> 00:35:54,040 right now in the market because AI workload is extremely hard to 583 00:35:54,040 --> 00:35:58,160 predict. It is it's it's most of the it's 584 00:35:58,160 --> 00:36:01,600 so difficult, especially when you talk about language, right. 585 00:36:01,600 --> 00:36:06,640 So if you, if you dive into the the resource requirements of of 586 00:36:07,040 --> 00:36:11,560 a large language model and let's say you use it in a chat 587 00:36:11,800 --> 00:36:15,320 functionality. So if I ask a question like 588 00:36:15,760 --> 00:36:19,600 what's the capital of France, it's, it's a very simple 589 00:36:19,600 --> 00:36:21,720 question. It's a very straightforward 590 00:36:21,720 --> 00:36:24,880 answer, right? It's a short prompt and it's a 591 00:36:24,880 --> 00:36:29,200 short generation of an answer. But if I ask, can you, can you 592 00:36:29,200 --> 00:36:31,920 explain the relative theory to me like I'm a 5 year old? 593 00:36:32,800 --> 00:36:36,520 It's quite a short question. It's an extremely long answer 594 00:36:36,960 --> 00:36:40,440 right now if if these two questions are popped, are 595 00:36:40,440 --> 00:36:43,960 basically sent to the large language models at the same time 596 00:36:44,160 --> 00:36:45,840 we have what's called concurrency. 597 00:36:45,840 --> 00:36:50,200 I've got 2 sessions going at once and now the first one is 598 00:36:50,200 --> 00:36:52,480 very fast, the second one is is is longer. 599 00:36:52,480 --> 00:36:55,200 Now if you then take another example, which is one of the 600 00:36:55,200 --> 00:36:59,720 most common use cases right now in the enterprise, it's called 601 00:36:59,720 --> 00:37:01,640 RAG retrieval augmented generation. 602 00:37:01,920 --> 00:37:05,040 So instead of instead of relying on the information or the 603 00:37:05,040 --> 00:37:07,960 knowledge inside the model itself, you're going to create a 604 00:37:07,960 --> 00:37:12,040 database where you're going to store your own documents. 605 00:37:12,280 --> 00:37:15,360 And with every question that you ask, we will first start to 606 00:37:15,360 --> 00:37:19,480 retrieve answers or documentation out of the 607 00:37:19,480 --> 00:37:22,480 database, which is the retrieval part. 608 00:37:22,840 --> 00:37:27,560 And then we're going to augment that information with the prompt 609 00:37:27,560 --> 00:37:30,160 with your question. And then we will send it to the 610 00:37:30,160 --> 00:37:32,160 large language model and we'll say, hey, I've got this 611 00:37:32,160 --> 00:37:34,480 question. I've got all these resources you 612 00:37:34,480 --> 00:37:37,600 can use to generate the answer. Go figure it out. 613 00:37:37,960 --> 00:37:40,960 It's a very long prompt text right now. 614 00:37:41,560 --> 00:37:46,400 If you look at what actually is on the GPU, there are two main 615 00:37:46,480 --> 00:37:49,640 constructs. There's the static construct, 616 00:37:49,720 --> 00:37:52,080 meaning the weights of the the model. 617 00:37:52,440 --> 00:37:55,680 That's that's a resident thing. That's that's always there. 618 00:37:55,680 --> 00:37:59,120 It's the first thing that loads and that will consume a lot of 619 00:37:59,360 --> 00:38:02,400 out of the the resource. So what you typically hear is a 620 00:38:02,400 --> 00:38:06,000 parameter count like this model is 80 parameter, 80 million 621 00:38:06,000 --> 00:38:08,000 parameters or 120 billion parameters. 622 00:38:08,000 --> 00:38:12,320 And and depending on the position, it will it consume a 623 00:38:12,320 --> 00:38:17,520 few bytes per weight or can use a half a byte per weight. 624 00:38:17,840 --> 00:38:21,640 And so typically the most common one right now, which will change 625 00:38:21,640 --> 00:38:26,080 in the future, but is FP162 bytes per per parameter. 626 00:38:26,400 --> 00:38:33,240 Meaning if I load an 80 billion model, I need 140 gigs of static 627 00:38:34,440 --> 00:38:37,640 memory. It will always be there. 628 00:38:37,840 --> 00:38:40,720 Now look at look at models or look at the GPUs. 629 00:38:41,040 --> 00:38:43,840 You typically need multiple GPUs and actually to load that thing, 630 00:38:43,840 --> 00:38:48,280 right, Because the biggest one is 141 gig, so you have 1 gig 631 00:38:48,280 --> 00:38:51,040 less. That means that the dynamic part 632 00:38:51,720 --> 00:38:55,760 is actually the amount of memory available for use it. 633 00:38:55,760 --> 00:39:00,880 So the static part will load the capability of the brain of the 634 00:39:00,880 --> 00:39:03,240 model. The rest is what you can do with 635 00:39:03,240 --> 00:39:06,520 it, like how many users can use it, what type of questions and 636 00:39:06,520 --> 00:39:09,920 that's where the that's where the sizing typically goes goes 637 00:39:09,920 --> 00:39:13,800 wrong, right. What is all of this is what you 638 00:39:13,800 --> 00:39:15,680 need to load. What are the questions? 639 00:39:15,680 --> 00:39:22,000 What are the answers all of that now to file one on even more is 640 00:39:22,000 --> 00:39:26,760 it we're now entering a durable agent era. 641 00:39:27,160 --> 00:39:30,800 So we first got ChatGPT where everybody understood like this 642 00:39:30,800 --> 00:39:33,160 is this is tremendous. This is great. 643 00:39:34,040 --> 00:39:37,360 And now we're saying I don't want to deal with loss and 644 00:39:37,360 --> 00:39:39,000 responsible on a day-to-day basis. 645 00:39:39,040 --> 00:39:44,080 I will use an agent something that runs in the background and 646 00:39:44,080 --> 00:39:50,360 will figure out things for me. I The most popular thing today 647 00:39:50,360 --> 00:39:54,240 is called open claw. And what open claw does is 648 00:39:54,240 --> 00:39:58,560 nothing more than you create an agent for a particular task that 649 00:39:58,560 --> 00:40:02,280 you want to do. And it will continue to learn 650 00:40:02,400 --> 00:40:04,960 about your behaviors. Like what do you want? 651 00:40:04,960 --> 00:40:08,360 Like if you are, let's say you, you want to travel to the United 652 00:40:08,360 --> 00:40:12,120 States and you want to book that through your own agent, it will 653 00:40:12,120 --> 00:40:16,920 start to understand that which airport you want to fly out of, 654 00:40:17,280 --> 00:40:20,160 which time you want to fly out of because you need to travel to 655 00:40:20,160 --> 00:40:23,320 the airport. Do you prefer aisle seat? 656 00:40:23,520 --> 00:40:30,320 Do you prefer window seat, or do you have particular dietary 657 00:40:30,440 --> 00:40:33,000 restrictions? And it will, you can book your 658 00:40:33,000 --> 00:40:35,600 ticket based on all of that. That's a, that's a, that's a 659 00:40:35,600 --> 00:40:39,680 very, that's a, that's an interesting example, but there 660 00:40:39,680 --> 00:40:42,360 are many others that you can, you can use agent for. 661 00:40:42,640 --> 00:40:45,360 So we're moving into a world where there's always a 662 00:40:45,360 --> 00:40:49,520 background process that helps your world, helps your, your, 663 00:40:49,520 --> 00:40:53,840 your job or whatever. Right now, if you tie that back 664 00:40:53,840 --> 00:40:59,800 to your platform, that means I need to have a context that is 665 00:41:00,120 --> 00:41:05,200 always present and that will start to grow and grow and grow. 666 00:41:06,040 --> 00:41:09,080 The GPUs won't grow from a memory perspective that fast. 667 00:41:09,440 --> 00:41:11,880 So now we need to think about where do I store it? 668 00:41:12,160 --> 00:41:14,760 Where do I store my long running context? 669 00:41:14,880 --> 00:41:17,080 So I start to use steering again. 670 00:41:17,240 --> 00:41:23,560 So good one, CPUCPU fast responding storage and now for 671 00:41:23,560 --> 00:41:27,320 the first time storage. From an inference perspective. 672 00:41:27,360 --> 00:41:30,840 By the way, storage becomes a part of the critical path again, 673 00:41:31,120 --> 00:41:36,200 where with chat it's a, it's a web request going into a large 674 00:41:36,200 --> 00:41:38,720 language model which is loaded inside the GPU memory. 675 00:41:39,000 --> 00:41:41,160 So there is no critical data path for storage. 676 00:41:41,640 --> 00:41:45,960 Now we start to score context at different layers of the 677 00:41:45,960 --> 00:41:48,880 platform, right? And that's where we see that 678 00:41:48,880 --> 00:41:51,040 there's what's called a durable state. 679 00:41:51,280 --> 00:41:53,600 It sits between stateless and stateful. 680 00:41:53,840 --> 00:42:00,040 It needs to stay resident as long as the session is working. 681 00:42:00,520 --> 00:42:04,360 Now, what's the end, right? So we can, we can argue that if 682 00:42:04,360 --> 00:42:09,080 I do a task that takes 8 hours and once it's completed, I'm 683 00:42:09,080 --> 00:42:11,160 done. But there's always this history 684 00:42:11,160 --> 00:42:13,000 about your preferences and all that thing. 685 00:42:13,280 --> 00:42:16,680 So that, that what's going to be the duration of the durable 686 00:42:16,680 --> 00:42:18,480 agent? That's that's the that's the 687 00:42:18,480 --> 00:42:20,400 significant question for the next three years. 688 00:42:20,760 --> 00:42:24,640 And how is the rest of the platform start to operate and 689 00:42:24,640 --> 00:42:26,920 focus on these long context windows? 690 00:42:27,520 --> 00:42:28,960 Yeah. And I guess the next question 691 00:42:28,960 --> 00:42:34,160 will be how can you get the data to that platform in a efficient 692 00:42:34,160 --> 00:42:36,240 and fast manner? I can imagine that that is going 693 00:42:36,240 --> 00:42:39,160 to be, well, somewhat problematic for some of the 694 00:42:39,160 --> 00:42:43,280 systems out there. Yeah, and also think about peak, 695 00:42:43,640 --> 00:42:47,920 like if I have a model and at one point a lot of agent, then a 696 00:42:47,920 --> 00:42:50,680 lot of users are hitting that same model and I need to scale 697 00:42:50,680 --> 00:42:53,800 out. OK, now I need to start a new 698 00:42:53,800 --> 00:42:56,040 worker node and I need to load the model. 699 00:42:56,520 --> 00:43:00,600 I want to get that model into GPU memory as fast as possible. 700 00:43:00,600 --> 00:43:05,240 How can I basically expose this storage element as fast in the 701 00:43:05,240 --> 00:43:09,600 most fast way possible without the least amount of detours? 702 00:43:09,600 --> 00:43:14,360 Right, So your fast storage becomes also a critical element 703 00:43:14,560 --> 00:43:19,400 for scaling out at the GPU side, because I can really load that. 704 00:43:19,400 --> 00:43:23,520 Like like I said about the bytes in memory, it's the same thing 705 00:43:23,520 --> 00:43:26,560 for storage. A 70 billion model is roughly 706 00:43:26,560 --> 00:43:30,120 around 200 gigs of of storage footprint. 707 00:43:30,760 --> 00:43:33,640 How fast can you transfer 200 gigs to a GPU? 708 00:43:33,880 --> 00:43:35,800 Right? That's that's the interesting 709 00:43:35,800 --> 00:43:38,000 part, what we're going to see in the next few years. 710 00:43:38,560 --> 00:43:40,880 And it's interesting because when we started out, we said 711 00:43:40,880 --> 00:43:43,360 that it all sounds very similar to the early days of 712 00:43:43,360 --> 00:43:45,800 virtualization and some of the challenges we had with VMS. 713 00:43:45,800 --> 00:43:47,800 And of course, storage was a big part of that as well. 714 00:43:48,200 --> 00:43:51,040 Everyone had a shared storage platform, but those shared 715 00:43:51,040 --> 00:43:53,760 storage platforms back in the days were not designed for 716 00:43:53,760 --> 00:43:56,800 virtual machine workloads. So we started hitting all of 717 00:43:56,800 --> 00:43:59,240 these performance issues. And I can imagine, especially as 718 00:43:59,240 --> 00:44:03,600 you said, how fast can you move 100 gigs, 200 gigs worth of data 719 00:44:03,920 --> 00:44:07,560 that will start seeing similar problems again, where that peak 720 00:44:07,800 --> 00:44:10,360 is going to be the issue, not the sustainable load, but the 721 00:44:10,360 --> 00:44:13,560 peak will be the issue. Now the other thing that I also 722 00:44:13,560 --> 00:44:18,680 wanted to bring up, and this is just me thinking out loud, you 723 00:44:19,120 --> 00:44:22,400 previously gave the example of those different more or less 724 00:44:22,400 --> 00:44:25,960 slot sizes. So you had 10 gig versus 20 gig 725 00:44:25,960 --> 00:44:29,320 versus 8 gig, etcetera. Now one of the things that I 726 00:44:29,320 --> 00:44:32,800 could also imagine, especially with VGPUS, you can have 727 00:44:32,800 --> 00:44:36,840 multiple VGPUS allocated to a single virtual machine. 728 00:44:37,040 --> 00:44:41,040 So in that perspective, from a allocation standpoint and from a 729 00:44:41,040 --> 00:44:43,880 scheduling standpoint, would it make more sense for customers 730 00:44:43,880 --> 00:44:47,720 that would have, let's say you know, allocations of 10 gig and 731 00:44:47,720 --> 00:44:52,120 allocations of 20 gigs to just use do 10 gigs and then allocate 732 00:44:52,120 --> 00:44:56,520 to VGPUS? Or doesn't that make any sense? 733 00:44:56,520 --> 00:45:00,040 It's just me thinking out loud and being weird as hell. 734 00:45:00,640 --> 00:45:02,760 You're not the first one who who asked this. 735 00:45:02,760 --> 00:45:07,080 Right, but it's not supported to do fractional multi fractional 736 00:45:07,080 --> 00:45:10,200 GPUs at the moment. OK, well that ends the 737 00:45:10,200 --> 00:45:12,200 conversation then. I guess I got a feature request 738 00:45:12,200 --> 00:45:14,440 for you. But because it feels like you 739 00:45:14,440 --> 00:45:16,880 know it's all some of the problems, at least if you can 740 00:45:16,880 --> 00:45:20,600 find a common size, then at least don't don't end up with a 741 00:45:20,600 --> 00:45:24,160 lot of fragmentation in your environment, which is, 742 00:45:24,360 --> 00:45:27,000 especially with the GPU, very expensive. 743 00:45:27,560 --> 00:45:29,240 Yeah. And and that can be the driving 744 00:45:29,240 --> 00:45:32,560 force behind that feature request that is, is not 745 00:45:32,560 --> 00:45:37,360 necessarily that you want to add more complexity because the 746 00:45:37,400 --> 00:45:42,680 distributing across multi multi fractional GP us is is going to 747 00:45:42,680 --> 00:45:44,560 be a challenge from an orchestration perspective. 748 00:45:45,040 --> 00:45:47,880 But from a from a placement perspective, it actually 749 00:45:47,880 --> 00:45:51,280 simplifies things or at least it can, right. 750 00:45:51,280 --> 00:45:55,400 So it does make sense. However, from a from a 751 00:45:57,960 --> 00:46:01,400 technological perspective, that might be a problem because you 752 00:46:01,400 --> 00:46:04,080 need to access that envy link to a certain way, right? 753 00:46:04,080 --> 00:46:07,160 So. But let's discuss this offline 754 00:46:07,520 --> 00:46:09,280 for that way. Yeah, I'm not saying it's an 755 00:46:09,280 --> 00:46:11,880 easy problem to solve. That's what I typically tell our 756 00:46:11,880 --> 00:46:13,760 engineers as well as part of the VSAN team. 757 00:46:13,760 --> 00:46:17,560 Hey, I come up with crazy IDs if it's difficult or easy to solve, 758 00:46:17,560 --> 00:46:18,960 I don't know. You need to figure that out 759 00:46:18,960 --> 00:46:21,560 right? Have it explained to PM team why 760 00:46:21,560 --> 00:46:23,960 you're spending all of these hours on a random project. 761 00:46:24,400 --> 00:46:27,320 Yeah, but I don't want to burn all my bridges for this one. 762 00:46:28,160 --> 00:46:29,720 I hope Chris Wolfe is not listening. 763 00:46:31,680 --> 00:46:34,480 Hey, thanks Frank. Before I let you go, any famous 764 00:46:34,480 --> 00:46:37,160 last words or any final thoughts you would like to share with the 765 00:46:37,160 --> 00:46:39,960 audience? Just try it, just figure out a 766 00:46:39,960 --> 00:46:43,800 way to start trying things. Look, we're part of VCF 767 00:46:43,800 --> 00:46:46,480 division, right? So we'll always say look, try to 768 00:46:46,480 --> 00:46:51,120 to do this in VCF. But if you can and if you want 769 00:46:51,120 --> 00:46:56,320 to start to investigate the AI workload, how it will will do 770 00:46:56,320 --> 00:47:00,120 things, there's always these resources in cloud or with your, 771 00:47:00,400 --> 00:47:02,640 your, your cloud partner as well, right? 772 00:47:02,640 --> 00:47:07,480 So you don't have to think about the, the entire configuration, 773 00:47:07,640 --> 00:47:12,160 but you can start to try and then you can start to, to bring 774 00:47:12,160 --> 00:47:15,600 back that, that workload into the into your environment. 775 00:47:15,800 --> 00:47:18,400 This is one of the key things right today, the frontier 776 00:47:18,400 --> 00:47:21,920 models, the models like cloud or an open AI, they are the AWS 777 00:47:22,760 --> 00:47:28,240 moment for the entire industry where AWS was a, a competitor 778 00:47:28,240 --> 00:47:31,880 for every retail organization out there. 779 00:47:32,160 --> 00:47:36,080 Frontier models will be for everything because has a lot of 780 00:47:36,080 --> 00:47:39,880 knowledge in that environment. And so we can basically impact 781 00:47:40,080 --> 00:47:42,120 every type of organization out there. 782 00:47:42,200 --> 00:47:44,440 And So what you will see is that most customers will start to 783 00:47:44,440 --> 00:47:48,040 figure that one out. They will say, I'm going to not 784 00:47:48,040 --> 00:47:51,640 train that beast with my private data, so I'm going to run this 785 00:47:51,640 --> 00:47:53,920 on Prem. And so in order to do so, you 786 00:47:53,920 --> 00:47:57,040 need to be prepared in order to to to bring that, bring that 787 00:47:57,040 --> 00:47:59,720 home. So yeah, try it somewhere. 788 00:48:00,160 --> 00:48:02,520 Yeah, and make sure to to read the articles as well that Frank 789 00:48:02,520 --> 00:48:04,720 has written. It's a, it's a, they're all 790 00:48:04,720 --> 00:48:07,960 fairly lengthy, but they, you know, contain an insane amount 791 00:48:07,960 --> 00:48:09,680 of detail and I think they are very useful. 792 00:48:09,680 --> 00:48:10,880 And make sure to look at the tool. 793 00:48:10,880 --> 00:48:13,640 And probably to add a disclaimer, because that was the 794 00:48:13,640 --> 00:48:17,560 first thing I noticed, is that you specified that this tool was 795 00:48:17,560 --> 00:48:21,120 designed by you and it has no relationship to specifically 796 00:48:21,120 --> 00:48:24,600 NVIDIA or Broadcom in any shape or form, Correct. 797 00:48:24,880 --> 00:48:26,160 Awesome. Thank you very much, Frank. 798 00:48:26,280 --> 00:48:27,160 Thank you. And that's it. 799 00:48:27,360 --> 00:48:29,680 Thanks for tuning into the Unexplored Territory podcast. 800 00:48:30,160 --> 00:48:32,440 You enjoyed this episode. Don't forget to scribe and leave 801 00:48:32,440 --> 00:48:33,880 a reviewer rating wherever possible. 802 00:48:33,880 --> 00:48:36,520 And please join us again next time as we cover more insights 803 00:48:36,520 --> 00:48:38,600 into cutting edge solutions shaping the world of IT. 804 00:48:39,080 --> 00:48:41,560 Until then, staying comfortable, keep exploring.