The. Welcome to the Adamsport
Territory Podcast. For this episode, I invited John
Nixon to the show to talk us through the various
considerations with recurring hardware for visa based VCF
environment. Stewart.
Welcome to the show, John. Howdy, happy to be here.
Hey, this is a a a different one for the change.
Normally I'm a guest on your podcast, but today you're a
guest on my podcast. Now, for those who have never
listened to virtual speaking, who is John Nicholson and what
do you do for the company? You know, every time people say
that I just have flashbacks to Office Space and being like I, I
talk to customers for the engineers, but my name is John
Nicholson. I work broadly in the storage
space, primarily on VCN, but other cover other things.
I've been in tech marketing, which if I'm not mistaken was
Duncan's old job a long time ago.
And you actually have a great blog, I think out there still on
what is tech marketing. But I've owned the VCN design
and sizing guide for the better part of a decade now.
And I, you know, work on the how and why with VCN and new
features and storage. Yeah, I think if you look at the
content, there's typically, you know, only two people who get
involved at this point of time. So it's either, of course, Pit
Killer, who we had on the podcast multiple times, where
it's yourself. Now, today I'm going to talk
about something that is near and dear to your heart.
I think some of the people may have seen some of your rants on
Twitter about hardware configurations, what you should
be buying, what you shouldn't be buying.
And there are a couple of things that I just wanted to go over
because I think, you know, some people have some serious issues
when it comes to figuring out what their building material
should look like. But before we actually dive into
it, I gotta ask first, how do you feel about TPMS trusted
platform modules? Because that's something that
comes up regularly. It's, it's one of these things
that, and I think this has spawned some rants, but it's
what a TPM is there, there are kind of software TPMS in your,
your desktop or your laptop type stuff, but hardware TPMS, which
are commonly used in servers. It's this little like $50 thing
and people forget them and it causes them unimaginable pain
and suffering. And I, I think it's, it's a
symptom of a larger thing that I think the rest of this episode
will probably touch on is that I think a lot of people on
procurement of servers, they've historically either, you know,
in larger shops, they've had a procurement team or they've had
a server OEM who's like, Hey, I've got this quick buy, you
should buy it. And they don't think about the
little details on that bill of materials.
Their eyes just kind of glaze over when they see 12 items,
including that TPM as an example, and it's very
accidentally leaving something off or trying to save 50 bucks
can cause you a lot of pain and suffering and and money.
Yeah. And especially if you look at
the configurations for the average server that we typically
require right for V SAN or just for V Sphere in general, we're
typically talking about thousands of dollars, closer to
10s of thousands of dollars than then then thousand, so.
Sorry Duncan, it's been a week. It's now 20,000 instead of a
$10,000 server. So.
Yeah, it's true. Yeah, the price of memory is
going up fairly fast now. Besides TPM is one of the other
things you also comment on fairly regularly, of course is
network interface guards the Nic and I still see people for
instance buying 10 gig NICs. So in your opinion, does that
even make sense? What's your take on this SO?
One thing to keep in mind with NICs is a lot of people they,
they, they work backwards from, well, what, what switch am I
connecting it to? Oh, I haven't, I have a, a nine
year old 10 gig switch and I promised my CIO that I was going
to get 10 years out of it. So I should just buy another 10
gigs. I, you know, I have existing
switch I'm going to plug into that or maybe I'm already using
10 gig and I, I get happy with that.
So there's a couple of things to think about. 1, you can use 25
gig NICs with 10 gig. And if you look at the price
difference between a 10 gig Nic and a 25 gig Nic, it's like $80
or something. Or it's, it's, it's a rounding
error. And that, that interface,
assuming you're using optical or, or twin ax, that SFP plus
that's 10 gig, those will work in a what's called SFP 28, which
is the 25 gig standard. And so there is nothing stopping
you if you have a 10 gig switch from just going ahead and
buying. Because I see people getting
this chicken egg problem where they say, well, because I have
10 gig switches, I'm only going to buy 10 gig NICs.
Well, I'm refreshing my switches.
I'm going to buy 10 gig switches because I have 25 gig.
And it's like you're, you're going to be stuck at 10 gig and
you're just going to be circular buying for 30 years at this
point. So we need to kind of snap out
of that. We need to look at, OK, what is
the cost increment if I go and buy, you know, a, a 25 gig top
of rack data center class switch from, you know, an HP or a Dell,
someone who's, you know, just getting commodity silicon,
they're getting a probably Broadcom, all Hill Broadcom, you
know, Trident 3X switch or something.
And, and in many cases, what's actually funny is if you could
try to buy a 10 gig switch, you're really actually buying.
If you go look at the ASIC, it's actually that ASIC is actually a
25 gig ASIC. You're just getting like a, a
15% cost reduction for getting a 10 gig switch.
It's, it's more of like AI won't call it a software limit.
I'm sure there's some hardware limitations, but you're saving a
little bit of money to have a significantly slower host.
And you also need to think about what is the network impact in my
environment? Well, historically when we were
refreshing environments, we got bigger, our VMS get bigger.
Over time, the amount of VMS that we have get bigger and we
get bigger hosts and we do larger consolidation ratios or
maybe we just cut our fleets in half if if our that
application's not growing. Well, if you choke the network,
there's a couple problems. So storage obviously near and
dear to my heart, you're restricting your storage speeds,
you're potentially having to deploy more nodes, you're
potentially having application degradation.
But also think about V motion. Think about V motion for a
second here. How many you know, how quickly
can I evacuate a host when I try to patch a host put into
maintenance mode if I have 10 gig?
When the 10 gig first came out, it was common.
You might have had a host with 128 gigabytes of RAM.
Now you're buying a host with a TB of RAM.
If you're trying to evacuate that over the same 10 gig
network, it's going to take 10 times as long.
And you know, I know some of you, like myself, just Yolo OPS.
You patch things, you hit, you know, upgrade everything and you
wander off and go get eat a slop bowl for lunch.
But there's some of you who actually have to be eyes on
glass and do your patch cycles at night or you just want
maintenance activities to not take, you know, forever.
And for those that small upgrade and network performance going to
25 or even going to hundreds, say, look, we have 10 times as
much RAM. We should have 10 times as much
emotion bandwidth that sorts of impacts and there's other
decisions that people make backwards from that of I talked
to a service provider said well we can't have a cluster larger
than 12 nodes because we can't patch that in a 2 hour patch
window. And that is a downstream
decision of because they were still using 10 gig.
It was, it was setting the maximum they're clusters.
Like, you know, you ever have that conversation about the,
there's this like saying about like the width of train tracks
is just working backwards from roads and like the width of a
horse or something in or wagons in Rome or something that we've
just zombied on this, this tech debt or this architectural
decision. That is what's happening with
Nix. And you may say, well, I don't
use that much bandwidth. Well, you know what, it's a
marginal cost. It has second order effects.
You might not need it in production, but during
maintenance or other activities it's choking that and and the
same thing with TPMS. You might not think you need it
for secure boot today, but it now encrypts configs.
It's now used for cashing V SAN keys.
And when someone from compliance says, hey, banking regulations
change, you have to do this. And then you have to figure out
how do I reinstall the Sxi on 7000 hosts to because I save $50
per host and I have to go one by one or the same with NICs, I
have to go swap that. Think of that labor cost to
that. So yeah, that was a lot of
words, but there's a lot of downstream decisions of you
saving money or just being on autopilot and saying, yeah, I
just need the same thing as before.
Yeah, I think that makes sense. And I actually had a
conversation with a customer last week and they were talking
about virtual machines with four terabytes of memory.
So you can imagine, you know how long it would take to migrate
those and those are active virtual machines.
So they actually have a lot of active memory.
So that would take a significant amount of time with 10 gigs.
So I think those are very valid points.
Now the other thing that you also already more or less
alluded to and I think it's one of the things I also wanted to
briefly discuss is the switch itself because it's one of those
components from AV send perspective which has been often
overlooked. And I think you've probably, you
know, commented on this on our internal chat channels probably
a billion times that people should more look into the
network from a switching perspective because some of
those switches may lack, you know, the correct buffers
etcetera. So maybe you could talk us
through some of those decisions around the the switches itself?
So coming from the VM Ware world, I always joke I'm like
the worst network engineer. I dropped out a net cab halfway
through my CCNA. But as AVM wareness has said
men, you should know what you're connecting to, even if that's a
separate team. Because it's a bit like you're
you're buying cars and you don't know what Rd. you're going to
drive on or what highways those connect to.
To extend this analogy further out from the top of rack to the
leaf. So some terminology, you know,
we're going to do some networking for for VM Ware
admins here, modern networking, you basically have what's called
leaf and spine topology. You have a leaf.
This is what's sometimes called a Tor.
Top of rack is what that is, is what your networking people are
meeting there. You typically have two of these.
Now these are generally going to connect back to something called
a spine or several spines. Specifically you want redundant
spines and some things to think about. 1 is the speed that you
connect to and be aware that you can, just because the top of
rack is a specific type doesn't mean that that actually forces
your your hand on Nick's selection entirely.
So like let's say your top of racks right now are, are 25 gig,
but you know that you're going to refresh those in two years or
maybe you just want to future proof yourself.
You can actually go buy 100. There's some 100 gig NICs on the
market you can buy today and you can get a breakout cable.
And so that way down the road, if you ever do to site upgrade,
you'll be ready. But you can actually get what's
called a QSFP, which is what your 100 gigs typically run on Q
stands for quad. You can get a four way break out
to 420 fives on Broadcom on the single Nic parts Intel, the dual
Nic parts, the 800 series do that Mellanox stone for some
reason, I I believe, But you can also think about you can mix and
these speeds can be and you can also go the other way.
You could be, you might already have servers that are 25, but
you're in the middle of a network refresh and you know
what the marginal costs go to hundreds, not that much.
You can go ahead and get 100 to 25 breakouts and be backward.
You can be backwards before it's compatible.
Now you asked about buffers and some of these other things to
think about. One thing is do look at when you
start running really heavy database workloads, you've got
heavy write throughput. You've got lots of cluster to
cluster chatter, which HCI by definition is that way, but also
V motion is you can run into some of these problems called
TCP in CAST where you have multiple ports trying to talk to
the same port at the same time. And normally that would cause in
TCP you would have a retransmit. If you want to Google something,
Google TCP in cast. That's what this problem is
known as. And some of the ways you can
mitigate that, particularly for short bursts of traffic, often
called microburst, which is how a lot of applications have is
you can get deeper buffered switches.
Now this is kind of come in and out of fashion with 10 gig for a
while when they first came out, you know, 12 years ago, we just
had shallow buffer. They were like 8 megabytes of
buffer, which is not for the whole switch.
And then people came out with, oh, let's come out with a 1 gig
buffer. But then 25 gig shallow buffers
came out. And you know, and the thing to
think about is it's imagine you've got, you know, multiple,
you know, bottles or cans on top of each other.
And if you have that buffer, it's a place that you can store
packets rather than them just falling on the ground or
spilling over temporarily for that short burst and not having
to pay that, that latency hit, that CPU hit that other things
of having to go through the whole TCP retransmit cycle.
So these ultra deep buffer switches when you hear that,
that means you typically have a buffer in the multi gigabytes.
And so if you've got bursty transactional workloads, things
like that, these can help for your reliefs.
Now your spines, you know, when we when we had multi layered,
when we have a multi layer design, which we're going to
have the thing to look for there is not actually buffers.
The thing to look for there is the subscription ratio or
oversubscription ratio. So if I have two switches at my
top of rack that are 32 ports, 100 gig and I'm using sixteen of
those for hosts, you know, then that and that leaves me 16 to
connect to a spine that's a one to one.
That means I'm not going to ever have contention between that
leaf and that those spines that are aggregating that traffic
together. Now if I oversubscribe that,
maybe I'm using all but four of those ports and only have 4
ports for that. Now I have an oversubscription
ratio. Maybe I'm 4 to 1.
And at 4:00 to 1:00, you got it when you start oversubscribing
that leaf to spine. Now you need your networking
team to do something that they they hate doing.
You need them to monitor that. You need them to set alarms.
And really what we're seeing in the industry as it used to be,
you know, when I started 10 years ago, we said 3 to 1 was a
good oversubscription ratio. Then we came two to one.
When you went to your 48 port type switches with four uplinks
and things like that, really what we're seeing is actually
people going 1 to 1. And this sounds crazy because
you're saying, look, half of my ports on my top of rack are
going to be devoted to not servicing my house.
They're going to a spine, but really what you're building
there with that one to one model is you remember the chassis
switches back in the day, like a 6509 or something really, you
know, had those line cards and they had a backplane and they
could all communicate. It all acted like one giant
switch. What you're building by not
oversubscribing is you're basically building a chassis
that's distributed. And actually, by the way,
chassis still exist and they're really still expensive.
This leaf spine is actually cheaper, so you get the benefits
of that. You also don't have a single
point of failure 'cause that that backplane could fail.
Very rare, but they could warp. Weird things could happen, but
those multiple redundant spines can cover you and you can
actually be in a situation where you no longer have to worry so
much about monitoring. Your networking team can go off
and solve other Wan problems or things like that.
And one of our largest customers out there has written, they have
a lot of blogs and things about this.
They've written this and I've talked to them privately and
they said it's great. Like the amount of overhead we
had on monitoring and troubleshooting and, and ghosts
in the machine and things like that just went away when we went
to 1:00 to 1:00. And I encourage, you may say,
wow, that sounds really expensive because if I'm buying,
you know, 100 gig top of racks, I'm going to need 400 gig spine
switches. But you start looking at the
prices on these things, particularly when you're, you're
not buying the most expensive proprietary ASICS, you're just
buying, you know, the vendors like a rest and people use
merchant silicon. If you compare that to the, all
the money you're putting in to fill up a rack full of servers
and RAM, it's maybe 15% of your bill image.
It's not that much. So this is something I also
encourage you when you look at cost, don't say, oh, wow, that
spine switch is going to cost me $40,000.
I want you to look at that and go, OK, that's $40,000, but
that's servicing half a million in RAM.
OK, that's probably worth it. Or, you know, maybe you are
smaller. It's a different world.
But look at these as a percentage of the total cost.
And this is something how to think about is don't be that
guy. Like remember that, you know, if
you see somebody at the gym, the guy who skips leg day and has
like giant arms, you know, and, and beefy stuff, don't be that
guy in the data center who skips networking day.
And I know we're not on the networking team.
We're all VM Ware storage admins here.
And we hate the networking team. They are the worst.
It is correct. But we do need to actually
encourage them to properly design and help, because
otherwise we're going to have problems.
I figured you were making a reference to Pete Fletcher when
you were talking about the guy who skips leg day, but we're not
going to go into that one. Now.
The reason I actually invited you today was to specifically
also talk about something which have been coming up more
frequently and that is the emulated ready note and
especially for V San ESA. Now before we start talking
about some of the specifics, maybe you can explain what an
emulated ready node is. So when ready, when we came out
with ready nodes for ESA, we initially launched where they
have to be ready nodes, they have to be these exact, they
have to be purchased as a ready node BOM and things like that.
And we were trying to, we did reduce a little bit of
flexibility there, but we're also trying to be very
prescriptive and avoid accidental ordering mistakes and
things like that. So if you look, if you actually
Google V SAN emulated ready node, you'll actually find a
blog that was written by Pete Killer and he he goes through
and talks about this that there is we have flexibility with the
program. So let's say you already have
some servers that meet the requirements of the ready node.
You can use those so you don't have to go buy the chassis, you
know explicitly that has the dash RN on under that code, you
can buy an equivalent server that has the same, you know,
backplane and Dr. connectivity. Now, one thing I will caution
you on is note, you know a deal. When is the DL380 or 380?
Well, the reality is the DL380 or an R650 or whatever server it
is you're buying. You may see that model number
and think they're all the same, but in reality it may be 1 of 12
different completely different chassis codes in the back end
that it has different, you know, maybe it has different drive
sizes, It has 3 1/2 inch drives or maybe it has 2 1/2 inch, you
know, universal form factor. Or maybe it uses the the you're
one of the cool kids. You have the new EE3, the little
ruler form factor drives that are really cool.
You can go real dense on those are different.
It's some of them. They make cable the drives to
connect to a RAID controller, which on the V San world, we do
not want. We do not like, please don't do
that by by all means go buy RAID controllers for other use cases.
You know, it's there's a probably Broadcom ASICS again,
you know, Broadcom is all the things simultaneously.
But this is the case where I'm going to tell you not to buy a
Broadcom product. So we want to see Nvme direct
connection and this stuff specified out there.
But the other thing to, to pair with that, that guidance on
emulated ready node of OK, I can I have a server, can I get it to
look like a ready node, so to speak?
Can I get certified drives? Is also look at there's a,
there's a KB and there's a blog called what can I change on a
ready node? And there are some things you
can change. So people sometimes say, hey,
look, the ready node has 22 cores.
I want to buy the 24 quarts CPU. That is fine.
You can absolutely change that. Do note, you know, changing the
number of cores and frequency in them, it's going to have some
performance impacts. They're, you know, and and
again, don't skip leg day, don't don't go, you know, buy 128
cores and have one drive and like 2 dims of RAM and you know,
things like that. But there there are things you
can change and that KB we actually step by That's a great.
Idea. Now it's funny because that blog
post has been out for a while and I barely heard anyone
talking about it. But recently more and more
people started talking about this specific concept.
So why is that? Why all of a sudden is this
relevant? So if you haven't gone to buy
servers in the past three months, I'm sorry.
It's going to, it's going to be fun.
Historically, the price of, you know, memory and I feel like
flash storage especially went down about 10% year over year.
We're currently because of the AI super cycle and memory super
cycle prices in some cases have quadrupled.
And customers are taking a real hard look at procurement.
There are a lot of customers who didn't try to right size, you
know, they, they just bought more hardware whenever they
needed something. And people are trying to say,
OK, what do I have and how do I get the most value out of it?
Because hardware prices are basically going in reverse.
We're basically unspooling all the savings in the past several
years. And this is, this is wild to the
point to where, you know, I think kind of our original plan
of record was sort of like, yeah, maybe we'll drop Cascade
Lake from PCF 9. Now we're like, Oh yeah, yeah,
we're, you're starting to see search show up on it.
And I'm even hearing, you know, I think if you, there's some
people even asking for RPQS on, on Skylake, which is wild 'cause
it's like 6 generations ago CPUs.
So people are trying to make do with what they have.
Orders right now are incredibly painful.
I mean, I've seen signed, executed, POS basically
cancelled and the OEM come back and be like, hey, about that
order, we're cancelling it and if you want it, we're just going
to double it. I'm sorry, somebody else ordered
$8 billion of, you know, AI hardware and that's not actually
a joke. That's probably under selling
how big that order was. Like this is the kind of
situation we're in. And so some things people are
looking at us saying, I've got some servers, I've got VCF
licensing that has AVCN entitlement.
How do I make, you know, lemonade from the, you know,
sugar and lemons and ice that I have laying around.
And that's, that's really driving a lot of these kind of
discussions and that stuff. And it's, it's kind of fun
because if you go back to the early days of virtualization,
Duncan, you remember how you people really tries to squeeze
the most out of VCP over subscription ratios.
And they learned every best practice on DRS and, and memory,
like we learned, you know, TPS and we learned about the balloon
driver and we learned, you know, all the things you could do and
VDI people would run 2 to one and things are three to one on
RAM and crazy stuff. And, and we kind of got away
from that because hardware got cheaper and we got lazy and we
didn't want to deal with the occasional angry user who had an
SLA violation. And I, I think we're kind of
going backwards. We're getting back to the basics
of how do we squeeze the most out of this hardware.
And I mean, that's what VM Ware does.
Well, that's our, that's our core competency.
If people already own hardware, whether it's Dell, HPIBM,
Lenovo, whatever it ends up being, and it could potentially
be used for VSA and ESA, how would they go about validating
that? How does that actually work?
Because I've also unfortunately seen some people have already
mentioned it, right? They try to create a
configuration for themselves and then they ended up with this
Dawn try mode controller and end up in a situation which is not
supported. So how would they go about doing
this? So some kind of quick like
rules, rules of thumb that I see, for instance, if I see a
RAID controller on that bill of materials, especially with Dell,
like if I see PERC, I know that they've cabled all those drives
to that. So I'm really going to I'm not
going to be able to do ESA. I'm going to be doing OSA.
Even if I have universal drive based.
There are some configurations, some servers out there, like
Cisco has specific sub chassis where if you have this code, the
46NVM E drives got, you know, slaughtered a different
direction or things like that. But for the most part, if you
see a HP, a smart array, if you see a, a RAID controller on a
Cisco, you see these things, that generally means, OK, this
is going to be an OSA, This is cabled to where everything's
going to connect to a central point.
I don't have direct NVME. Now if you go back and you look
and go back and find your original bill of materials.
And if you find that and it says something, it's like NVME
direct. OK, now we're, you know, we're,
we're cooking. That means I've got PCIe raw
lanes and, and some of the challenges, you may say, hey,
look, we used to run OSA, you know, with drives, connect to
the ray control. It wasn't that big of a deal.
Well, some of the challenges are when you put NVME drives kind of
RAID controller, some of them emulate them into SCSI drives,
which is weird. And we, we, we just see
performance regressions. The other fun thing is most Nvme
drives are looking for four PCIe lanes.
That's what they're expecting. And some of the, some of these
RAID controllers, when you start to try to cable a lot of Nvme
drives to them, somebody decided to save some money in the wiring
harness or there were just limitations.
And so they started just putting a single PCIe lane connection
and the performance it's just it goes, it goes and then it just
slams and so real. So and keep in mind this isn't
just for ESA for OSA also, if you have one of those
controllers, one, the controller has to be supported, but two,
you're only going to be getting OSA.
There is no ESA and there's no NVME.
We don't support NVME behind Tri mode for OSA or ESA.
Weird stuff happens. Maybe in the future someone
might come out with the Tri mode controller doesn't have this
problem, but you're also kind of contending with physics of that
Tri mode controller is still sitting on an 8 or 16 PCIe
lanes. And if you're trying to put 16
drives on that, that's there's not enough lanes, it's
problematic. There were some older
configurations where people at PCIe switches sometimes referred
to as a PLX and there are some of those on the already nodes.
So you, you did have servers, particularly early Ice Lake and
stuff where they would have this thing to get more in VME ports.
But another thing to look for is you're looking for it with Intel
Ice Lake or newer. Cascade Lake is not certified
with ESA. There weren't enough Nvme lanes.
It was Gen. 3 SSDs. They really weren't.
The Nvme wasn't as fast. We were really wanting that Gen.
4 and beyond capability. So we waited till Ice Lake.
I'm guessing you're the Broadcom Compatibility guide would be
your best friend at that point in time, right?
Is that something that you would go over and over and over again?
Yeah, so go to the Broadcom compatibility guide, find the
matching ready node and look and pay attention as an OSA or an
ESA. I talked to someone just today.
It was like, oh, I got a bunch of eight 40s.
I'm like, great, those are Cascade Lake, those are OSA and
they're like, oh, I was like, well, you know is what it is.
But these are things to look for.
Look at the bill of materials. Compare your bill of materials
against that. Pay attention to those
controllers. Also, if you have really old
hosts, you know, this is slightly off script here, but
watch the boot device. If you've still got something
booting from SD cards, you'll buy that little PC IE card with
the m.two riser, the boss or whatever.
You're just gonna have a happier life in general.
And as we go forward, we need larger, we need larger endurance
boot devices. They also boot faster.
Like this isn't just a there's some nice to have to the also,
this is the direction the world's moving.
Yeah, I fully agree. It's something that actually
does come up regularly still people using SD drives.
Now the other thing that also comes up fairly frequently and
something that of course people should be using the the Broadcom
compatibility guide for are the NV me devices.
Now one thing I've noticed lately and this seems to be a
trend is that more customers are planning on procuring a lower
number of devices with a higher amount of capacity.
So what is your, what is your take or your opinion on that?
Because you know, in my opinion, it, it's kind of a strange
trend. So how do you feel about it?
So there there's some things we've done to make it better,
but there's still gotchas you got to watch out for.
So one of these issues are if you, when you start going down
to like one or two drives, you run into the situation to where
if that drive fails, you basically could potentially be
losing a fault domain. And that has like higher level
rate issues. So I really like trying to get
to at least you know, three drives or more.
The other thing also is when you start going with real high small
VM counts, there are some object limits that you're going to
potentially run into if you're trying to run really, really
dense, not a lot of capacity, but just you know, hundreds of
tiny VMS on like sub six drives, you can get into some object
limit considerations that you have to be aware of.
The other kind of of gotcha with that is going to the opposite
direction of just using a lot of small drives, you're potentially
preventing yourself from having in place to capacity to expand.
I think that's what drives is people say, you know, I want to
only I want to fill in, you know, three of my 8 drives
today. So that way I have lots of room
for expansion. But you're potentially hitting
either performance limits or object limits or you're, you
know, potential fault doing limits if you go too small.
So don't go to, don't go too high.
Leave yourself a little room to expand or potentially leave room
for memory tearing. You know, those also use drives
that uses a drive slot or slots potentially.
But the other thing here is, is, you know, don't undershoot this.
Don't don't Again, it starts to look top heavy and weird when
people try to run one or two or, you know, very few drives,
especially in denser environments.
And I know we've got, you know, dedupe and things like that and
compression and we've gotten more efficient on this, but this
is something to watch out for. And I, I will say the Gen. 5
drives are pretty fast. So it used to be, if we go back
to the early days of Esan, you remember we needed to use all 24
drive slots. It felt like to get the most
performance. You, you don't need to run 24
drives to hit the, you know, you're going to the, the you'll,
you'll even with all the threads that ESA can go, you're
probably, you know, going to hit that, you know, 300,000 IOPS,
you know, whatever per node wall it'll be based on the CPU.
You'll probably hit that long before you can saturate 24
drives where the capability. But there is there is kind of a
sweet spot of trying. I try to shoot for at least 6
and one thing to look at is if you think, well I've, you know,
I'm using one U pizza boxes, I only have a drive base.
Look at that E3 form factor because you can get 12 or 16
drives to those in one U and at that point you really have no
excuse not to. You're still leaving yourself
room to grow without going too low on those minimums.
Oh, it's funny, you mentioned the component limit on a per
device basis. It is something that has never
come up with with V San in the past two years, but in the last
three weeks we had two customers actually complaining about this.
So I think it's a very valid point to take that into
consideration. Now the other thing when it
comes to NVMES, which is also something that probably people
should be taking into consideration are the types of
device that they use. And of course, these days we see
read intensive, write intensive, mixed and we support basically,
you know, fairly large variety of drives.
Some of those who have, you know, one drive right per day,
some of them more. What's the consideration when it
comes to buying those types of devices, and what do you
typically recommend for customers?
So this was fun when we first came with the ESA, we just
supported the three drive right per day mixed-use.
And then we we added read and what are called read intensive
drives and read intensive. It's a weird name because it's
it's like it's it's like saying my tires are parking optimized.
It's not that they're better at reads than the write intensive
drives. It just means they're not as
good at writes. They don't have as much
endurance. They have a third the endurance
and and let's kind of unpack why.
Like let's look under the hood. So you'll notice these these
sizing increments of like 1.6 and 3.2 for the mixed-use, the
three drive write per day. And then you'll see right next
to it a 1.94 and a 3.843 point, whatever that is.
And it's about a 2022% difference in capacity.
And it's like, huh, that's funny.
And I notice it's like this lock step.
So what it is, is where those, those extra two drive rights per
day is coming is that drive is basically they're functionally
in many cases the exact same Dr. effectively it's same controller
or very similar controller. It's it's the same NAND.
It's the same actual flash die underneath, but they're actually
just reserving about 2022% of that capacity, keeping it in
reserve. So as cells die, then they can
reallocate and rebalance things. So that's actually where the
endurance have no one as far as more has come out with a
firmware to like re flash it one way or the other.
But that's that's kind of what that is the and and they're
priced often about the same. So it's about a 2022% price
difference. So people say, hey, cheaper
capacity, let's go this way. Now, realistically, most people
are not that right heavy and if you are, you probably should
know who you are. You've got some VRA environment
from hell that's repaving 500 times over.
You're doing full clone VDI and you're doing like 100 cycles a
day. Go look at your change reports
from backups. Go look at your in OPS at your
rights. You know if it if you're seeing
really crazy stuff that makes you there.
The other one is you're trying to get the highest consistent
performance, particularly on high sustained rights.
Those mixed-use drives are still going to perform better.
Also the mixed-use are probably going to be you.
Even if you're going to read intensive, you may still want to
put one or two of those mixed-use in there for memory
tearing because memory tearing is going to be a little more
sensitive on performance. And that may, it's probably
worth it, especially because you're not really using that
extra capacity in many cases either way.
So speaking to people who work in the industry on the flash
side, 80% of the market in aggregate, not just the same,
but in aggregate is kind of moved to read the read intensive
drives. They are becoming kind of the de
facto standard, but for high endurance environments, memory
cheering use cases, stuff that's just really, really you're not
as focused on capacity costs, you're really focused on right
latency consistency under heavy load.
You're still going to go for that mixed-use drive.
But the other thing also you're going to, you're going to see is
we can talk about all this in theory and then you're going to
go talk to your distributor, you're going to talk to your OEM
because that's no matter what they have in stock.
And so I'm seeing some of the purchasing decisions being made
based off of what's available as as prices are going up, as
supply is getting crunched, people are often just having to
make do. And this is also waking people
up to decisions, which is good. But this is another thing to
consider. And actually one of the most
crazy like second order effects of purchasing decisions I ran
into recently was someone was doing a very large order and
they went to they wanted 256 gigabyte Dems for RAM and
they're like, great, we're going to buy all this RAM.
And they came back and their OE Ms. like, yeah, we can't get
those to you at a reasonable cost.
All those Dems are going to the AAI weirdos.
But you know, we can just sell you twice as many 120 eights and
they go, OK, we'll do twice as many 120.
Wait, that's a lot more power. We're maxed on power.
OK, can we double the size of our V SAN drives and cut them in
half so we can save power to pay for the extra the more Dems so
we don't have to go to deploy more generators and more cooling
in this data set? Like this is the kind of like,
you know, 3D chess that's going on here on these decision
logics. So I know we always just like
look at these in AB decisions of oh, read in terms of write in
terms of, but this is stuff that's kind of coming down from
other angles. And if you're doing fewer
drives, OK, there are bigger. What's the endurance than that?
OK, it's scaling. These are the things you need to
be thinking about. Now you've mentioned it a few
times already, so I probably should ask the question.
When it comes to memory tearing, is it something that comes up
with customers fairly regularly? Are people actually adopting it?
And of course, what is the reason for it?
But I guess we already know. So yes, so one of the largest
customers that I've talked to recently, you know, if we go
back 18 months ago, people were buying RAM, small, small dams
for like $5 a gig, maybe $10 a gig on the denser stuff. 1
customer I talked to recently and these guys have, I'll say,
some purchasing power. Like we're talking like, you
know, 910 figure orders here of RAM.
I think they were paying like $40 a GB.
And when we first came out with the feature memory tie ring
memory was like half the cost of a host or something.
When you get to these RAM dense builds, memory is like 90% of
the host cost. Like Intel could come out with
another CPU that's like, hey, this is an extra 20% clock
speed. We're going to charge you an
extra 3 grand. And that would just be a
rounding error on some of these server builds.
It's, it's, it's crazy where we're at.
And so memory is basically driving everything and it's,
it's not getting, you know, people say, well, it's just
going to get better tomorrow. Well, there's only three memory
vendors left. I mean, you got SK Hynix, you
got Samsung, you got Micron. There used to be 17.
And as you know, one of the Intel CEO's said in the 80s when
he was leaving memory like only the paranoid survive.
And it's, it's very true like these guys, it takes, it's
multiple years to spin up, you know, their ASML orders to get
fabs, to get the downstream supplies.
This is something that takes, you know, they, they have been
through many boom bust cycles and not everybody wants
everybody looks at the other, you know, 14 vendors that aren't
there and has a memory. So this isn't going away
quickly. This is RAM is getting very
expensive. And so memory tie ring is
basically in every customer conversation I have.
If you, you know, if you were, if you're a partner or you're an
SE or someone, it should be in every customer conversation you
have. If you're a customer, you should
be asking about it because if you can cut that RAM bill in
half on that dense order from 40,000 to 20,000 or even from
20,000 to 10,000, that's gonna that's gonna help offset some of
that sting. And you may say, well, isn't
NVMU a lot slower than ramp? Yeah, it is.
But we have a really good hypervisor at identifying which
pages are active and which pages are idle.
And I got a hold of a phone home data set with a couple million
workloads and actually saw an aggregate average memory page
activity. And the median environment, it's
about 20%. So even tiearing 1 to 150%,
that's actually conservative. Like let's start there, please.
Let's start there and let's start with your tier, you know,
1-2 apps. Let's start with your
applications, your VDI, it will work our way up.
But I've talked to people, Brandon Frost spoke at VM World
and he was using the tech preview and he was running 1 to
one with with sequel server and he's like, yeah, at a 10%
performance head. And I cut my RAM bill in half.
Like that's, that's absurdly compelling.
You know, that is that is selling free money, so to speak.
And we, you know, how do we do it?
We have all this IP around V motion and DRS.
We've been tracking memory page activity better than anybody for
longer. This is why your V motions don't
stun, you know, and they and they they also don't stun a lot
less when you buy better Knicks going full circle at the
beginning of this podcast. So.
Yeah, I actually spoke with Brendan at the event and he had
a fantastic use case and a great story.
So I was very happy for him to be up on stage and explaining
how how we actually use the product.
And I'll make sure to to add a link to that particular session
in the in the show notes. Now, before I actually let let
you go, one of the things I also briefly wanted to go over is for
people that actually can still procure new hardware, how do
they actually go about doing that?
I mean, we know there are different ready note profiles,
but what is your take on that? I I always have a very simple
approach, but some customers take a very complicated approach
where they look very specific to which ready note they need and
then they get confused about what they can change and what
they cannot change. So how do you go about that
whole process when you talk to customers?
There's kind of two ways to buy there is the like, I'm going to
do this the nerdiest way possible, which I'm inherently
drawn to because I, I love staring at these stats, which is
like, OK, what are your resources?
What are your constraints? How many nodes do we have?
OK, how many racks, how many ports do I have?
Like sometimes people work backwards on cluster size from
top of rack ports and subscription ratios.
This all just works. This is why you have to learn a
little bit of everything, unfortunately.
And then I go, OK, how do I, you know, fit those puzzle pieces
together with this many megahertz, this many gigabytes
of RAM, this much capacity? What's my constraint, you know,
in the VCN sizer and which is evolving in the VCF sizer?
By the way, I was on design calls for that today.
And, and these are the things to where it's like, OK, how do I
pack this in the most efficient way possible?
Because I don't want to be in a situation where I decided that I
was always going to buy nodes with two terabytes of RAM and
then that causes me to double the size of my order, which now
is going to cost me 3 quadrillion dollars.
The other way people buy which people when when I feel like
hardware was cheaper, people had kind of fallen into was OK,
we're going to have our procurement department, we're
going to figure out one or two nodes we're just going to run
our entire fleet on. We're going to pick one or two
Lego bricks, you know, and yeah, we're just going to order like a
bazillion of those. And we'll just take our
workloads and like, you know, play Tetris into those bricks.
And that latter 1 used to be more popular, but I, I, you
know, one, don't overthink and like, try to have a bespoke for
every single node or node in a cluster or cluster.
Like do try to find some kind of ratios, but then also pick
outlier workloads and build separate different clusters to
them. But I, I caution people in that
ladder to where people are like, well, we've always bought ADL
whatever with this number. And we do this.
And often times when I ask people why they're like, well,
our procurement apartment goes to, you know, they do an RFP
once every six years and then we get a, a discount percentage and
then we just do that. And I'm like, you're probably
potentially wasting a lot of money having this situation
where you're just trying to shove everything into that.
And I encourage you if you are doing that, look at OPS.
Look at, you know, how much CPU or memory, look at that ratio
because they used to be something hardware's cheap.
I don't care that hardware's getting really expensive,
particularly in memory. And I would often look at memory
and how that is bottlenecking me because the other stuff you can
kind of adjust. You can adjust drive sizes, you
can adjust cores per node, you can adjust down sockets.
Actually single socket servers, by the way, we used to all like
snuff our nose and be like, oh, those are for poor people or
something. Why would I have a single
socket? Single socket servers are
actually awesome. Now you can actually get all
memory channels, like all the weird things people crippled
sockets on. So I'd encourage you to to
revisit some of your priors. Look at this stuff, re evaluate.
Don't just cargo cult and say we're going to buy the same
thing because it always worked before.
And look at this and, and, and try to tune this stuff and
figure out what your primary cost is.
Is that memory is it could be cores.
Maybe you're buying an article rack and it costs 40,000 a core,
but figure out what those business inputs are and work to
that logic. So I know there's a long
rambling answer, but there's a lot of inputs and you kind of
need to learn them all and and put them together.
Before I let you go, any famous last words or any final thoughts
you would like to share with the audience?
Please put TPMS in your servers before you install ESXI.
It's, it's, it's going to make your life easier in the future.
But, but, but also just, you know, the I, I wasn't always a
networking guy. You know how I learned about
networking? I went to lunch with those guys.
Go to lunch with your peers. Go if you're not a storage guy,
go hang out. The storage guys learn, learn
what an IOP is, learn what your ratios are.
Be curious, it'll help you in your career.
It'll help you be a better view, more admin.
And if you're trying to get approvals for budget right now,
which is really hard, it's hard for me and everyone you know,
I'm ordering new lab hardware for my house and got to get that
through the the local budget process.
By the way, you can buy used opt in cards for cheap from memory
tearing on eBay. But these are the things like be
curious, learn about these other domains adjacent to you and the
resources and learn about how you're, you know, the the trades
and the arbitrages and things between them to learn about your
applications, figure out what's driving those workloads.
It's going to make you a better architect and it's going to make
it's going to make you learn some things.
It'll be fun. And that's it.
Thanks for tuning in to the Unexplored Territory podcast.
If you enjoyed this episode, don't forget to subscribe and
leave a review rating wherever possible.
And please join us again next time as we cover more insights
into cutting edge solutions shaping the world of IT.
Until then, staying comfortable, keep exploring.