None. Welcome to the Young Sport
Territor podcast. With this episode I invited our
cash back to the show to talk us through the VCF Native S3 object
storage announcement. Welcome back to the show,
Rakesh. Thank you, Duncan, as always,
excited to be here and have a great conversation with you.
Yeah, the last time you were here, we were actually talking
about VCENT over Fibre Channel. Today we're going to be talking
about the native S3 solution that we just announced.
But before we do, not everyone may have listened to the
previous episode, so maybe you can first introduce yourself.
Who is Rakesh and what do you do?
Sure. At VM Ware Broadcom, my current
role is Senior Director of Product Management.
I'm the product head for storage data and data protection.
I've been with VM Ware for 12 years now.
I came in through a startup that we sold to VM Ware.
I was the first PM to bootstrap VSAN, which as you know is the
flagship software defined storage offering from VMware.
We grew VSAN together along with you from zero to a billion
dollar product line and at one point it became the fastest
growing product in VM Ware. These days I am driving new
initiatives in VCF such as Storage for AI, Cyber Recovery
and Database as a service. I think that's a nice one to
have on your LinkedIn, right? I grew the business to a billion
dollars. So yes, absolutely.
I did a podcast with Pete Keeler not too long ago when we spoke
about the 9.1 release and specifically of course about V
San and were really cool features that were part of the
launch. Now, besides Native S3, because
I know you you are really excited about that one, what
else excited you about the VCF 9.1 launch?
Yeah, super question. So there are two things that
excite me about VCF 9.1 from a storage and data standpoint. 1
is we actually G8V send global deduplication, right.
And I think I've mentioned this in other podcasts.
It is truly global compared to an array or you know, we, we we
have a much larger deduplication domain which results in very
high PCO savings, right. So, so that is a significant
capability that we have introduced especially in these
times when you know there's so many hardware, there are
hardware shortage and and the cost of hardware is so high,
this is going to help our customers reduce their TCO
significantly. So that's one and the other one
is our flagship VCF cyber recovery feature, which we are
launching basically providing an end to end fully automated
workflow to recover from a cyber, a cyber attack or a
ransomware attack with, you know, complete automation built
in end to end. I know you've done several demos
and you've spoken about this in many forums as well.
To me, these are the two big capabilities I'm excited about
with VCF 9.1. Yeah, and I actually had jotting
on the podcast a few times to talk about this.
And I've also just invited Velina to the show because she
hasn't been on the show yet and I think she's a fantastic PM
show. I'm getting her on to talk about
the evolution of Dr. to ransomware recovery.
Now today, of course, as I mentioned, we'll be going to be
talking about the native S3 solution that we've just
announced. But there's one question that I
got to ask first, because this is a question that has been
asked a lot the last couple of weeks because people are
downloading 9.1 and they cannot find the S3 solution.
So when can they expect it? When does it actually ship?
Do you know and can you share it?
Yes, I can share it. So you know, although
technically native S 3 was launched as part of nine One,
it'll be available in the product in nine dot 1.1, which
is the 1st update release of Nine one plan for around August
of this year. It's going to be a tech preview
for non production workload workloads.
We plan to tech preview. We have a bunch of design
partner customers. We're going to tech preview this
for three months and then we plan to G8 in 912, which is 3
months after, likely in the November time frame.
You got to add a caveat there because you never know, right?
With these types of things, we know how it goes with
engineering and. Absolutely, Yeah.
This, this is all planned and no commitment here.
This all planned, planned releases and timelines, Yeah.
Yeah, I'll make sure to add a disclaimer to this particular
episode just in case. Now, for those who are not
familiar with S3 Object storage, and I'm guessing most people
are, but for those not familiar with it, maybe you can briefly
explain what it is before we go into the next bar, which is what
we just brought to market or what we just announced.
So what is S3 Object Storage? OK, so as you know, historically
we SAN has supported block and file as the primary data data
types for storage, right? So, but what we've seen is that
with the advent of Kubernetes, you know, a lot of Kubernetes
and dev OPS workloads landing on the private cloud and also a lot
of AI workloads landing on the private cloud.
These applications almost always need an S3 compatible interface
for storage. So S3 object storage is nothing
but you know, our native offering of V SAN in in the S3
format compatible with Amazon S3.
So now applications that require S3, they require the ability to
store data, access the data through, you know traditional S3
get input commands. We allow that, but but what it
really is is it's a scale out storage offering.
It's a scale out St. storage offering.
It brings all the benefits of V SAN and all the benefits of the
storage policy based management and things like that.
So extending that from block in file to scale out object
storage. Now, one of the things we always
emphasize in the early days of V San was that it's not a
distributed file system, but it's distributed storage.
And to be more correct, it's distributed objects storage.
You know, when I hear S3 object storage and I hear V SAN object
storage, the question of course obviously that then arises is,
is that the same thing? Will developers get direct
access to V SAN or is it different?
No, it's different. So yeah, I know, I know there's
a bit of confusion with that word object.
So see AV SAN object, right? Historically refers to the
logical software defined unit of storage created by VMS,
hyperconverged infrastructure, right?
And historically it's primary function was a block storage at
the hardware or hypervisor level.
Each V SAN object is a collection of components like a
VMDK, snapshots or namespaces designed specifically to run
virtual machines and applications.
That's what we send object to us right now when we talk about S3
object or this native S3 object that we are launching that is
nothing. But you know, strictly if you
look at traditional objects, storage data is flattened into
an unstructured pool where files in are and their accompanying
metadata are assigned a unique flat identifier rather than
being placed in a nested directory tree.
So this is when we talk about we send native S3 object.
This is completely different. It is the traditional S3 format
that the industry is used to. It is an additional data type to
block and file, and we'll shortly talk about all the use
cases we plan to support. Yeah.
So I guess it's, we're offering a different interface and yeah,
I mean, I like the mechanism and I know there's a lot of
customers excited about this. Now.
I when I looked into it, one of the things that surprised me was
the way you had to enable and configure it.
It's not something that I expected, so you probably should
be talking about that as well. So how do people configure
native S3 object storage services?
How does that actually work? Yeah.
So see, first of all, we have to recognize as you know, you know,
I've been trying to get objects storage prioritized in VM Ware
for a long time, right? And we, we have been able to do
that and deliver this primarily because the, because of the
advent of new use cases such as dev OPS and AI, where the
application developer needs access to programmatic storage,
right? Without the need to go and file
an IT ticket if they want to actually they want to be able to
directly interact with the storage.
So, so this is the context in which we are building it and
it's very Kubernetes centric. Therefore, as you know, our
flagship product for all of Kubernetes of automation is what
we call VCFA or VCF automation. So we have all the workflows
built into VCFA for a the provider admin or the
infrastructure admin to set up the object storage service
right? And then create supervisor with
the VSAN cluster, assign zones and regions and things like
that. Then we have a set of workflows
for the tenant admin to provision that storage and
manage users. So create an object store and
add tenant user access to object store like other services.
And three finally the the main user that we are targeting the
app dev, they just come in once it's all done, they come in into
their tenant org and they consume that object store.
They create buckets, manage up buckets, objects and so on and
so forth. So all of this is built into VCF
automation natively and it takes advantage of all the
self-service and multi tenant capabilities as well.
So all of that is offered in VCF automation.
It's something that came up during a meeting I had two weeks
ago. It was probably the first
question that they had. It's because they were all about
multi tenancy. Now the interesting thing is
that the next meeting that they had, the customer said, well,
you know, VCFA looks nice, but I've got my own tools that I use
for automation. So for those types of customers,
will they also be able to, for instance, you know, configure it
in a different way through the VC vsphere interface or how does
that work? What are the options for them?
Absolutely, yes, right. So we realized that, you know,
VCF automation is something that we have just, you know, recently
launched and we we realized customers have their own
tooling, some of them may not be comfortable moving to VCF
automation anytime now. So therefore, you know, going by
our API first strategy, we've ensured that every piece of S3
code is available through an API.
So I talked about how the provider admin sets up the
infrastructure. So we have APIs from VC, VC APIs
that customers can use to set up the object store and then all
the other activities like the tenant admin, you know, managing
users and the users actually consuming the storage assigned
to them in a multi tenant fashion.
All of that can also be done through the APIs.
So in fact, you know, you bring up a good point, I've talked to
customers, 50% of the customers I've talked to have their own
tooling and their own automation and they would just like to do
this by consuming APIs. So we saw this well in advance
and we made sure that this works for even customers who are not
planning to adopt PCF automation.
Yeah, it's it's interesting because I went through the
exercise of configuring it and if I look at how you do it
through the API or you do it through VCFAVCFA, feels much
easier to me and especially the multi tenancy aspect.
Is that your idea as well or how do you see that?
Absolutely. Like if if you take the APIs in
general right. If you look at the APIs and it's
always more complicated to set up a multi tenant infrastructure
with APIs and make sure that everything works fine and you
put in all the guardrails correctly, even the self-service
aspects, right. In many cases for many customers
who are think about customers who are not as savvy on with
containers and DevOps as you know, you know, many of these
traditional VM customers moving to that, you know, they the, the
automation that we provide with VCFA specifically around
self-service and multi tenancy makes their life much easier.
You know, rather than building all these automation through
scripts and AP is now those who have AP is and scripts and
they're very familiar with this. Yes, this might work, might be
incremental, but strictly speaking I strongly believe that
VCFA provides a lot of value and a lot of simplicity for things
like self self-service and multi tenancy.
And I think the multi tenancy aspect is really important.
And one of the things that of course came up during the
discussions I had with customers as well is that they've tried to
use, for instance, Visa and file services.
Because when I talk about the solution that we have, you bring
up object storage, which is the block interface.
And then of course, and we have a file interface as well.
And of course immediately customers say, well, if I look
at file services, I can't really do multi tenancy from a
networking perspective. I only can enable it using a
single authentication domain, I can only use a single DNS
server, there are no namespaces, etcetera.
So how is that ever going to work with native estuary object
storage? So what would you say to them?
Yeah, it's a great question, right.
So first of all, before I answer this question, we have to
understand where we came from historically, what was fees and
file services meant to be right? It it was not, you know, in the
initial days when we launched it and we sent files has been
around for a while. It was not targeted for this
these DevOps kind of use cases where, you know, multi tenancy
becomes super critical, right? It was your traditional file
shares and things like that. So yes, it had limitations
around networking, authentication, you know,
configuring those things and stuff because the goal was never
to provide direct access to the developer.
OK. So in some sense when we looked
at native S3, which is all as I said focused around the
developer, the networking in S3 for example is VPC based, right,
which is required to enable multi tenancy with object, you
get bucket level isolation so that even if the object stores
are stored within AVPC, they're completely secure and isolated,
right? Coke will not know about Pepsi,
HR will not know about finance. So we built all that into native
S3. So now what we're doing is we
are also planning to introduce file services as a first, as a,
as a first class citizen in, in VCFA and provided the same level
of multi tenant capabilities, simplifying all of that
networking and everything as we are doing for native S3.
So, so to to answer your question, the limitations,
current limitations of VSAN file services does not apply to S3
because you know, it's fully, we provide full self-service multi
tenancy through VCFA and we take it a step further, even file
services, we are providing the same level of isolation and
multi tenancy and things like that in VCFA.
So we're actually improving VSAN file services to address the
same use case because there are several applications in this dev
OPS space which require file and not object.
So we need to bring both file and object to the same level in
terms of capabilities and in terms of simplicity of usage
with VCFA. Yeah, I think that's fantastic.
I'm pretty sure that the customers and partners that I
spoke with, because as you can imagine, some of those are the
cloud partners as well, will be very interested in having a
solution like that. Now one other thing that I
noticed, which was also interesting is when I set it up,
I actually had the ability to create one or multiple S3
instances. And then within those instances,
I could have one or multiple buckets.
Now at first when I saw that, I wasn't really sure what that was
for. And I think I'm, I'm guessing
that the listeners will also kind of get confused when they
go through that effort. So maybe you can explain why you
would have multiple instances versus just one instance.
Because to me, when I work for an enterprise, I would probably
just create 1 instance and a whole bunch of buckets and
assign access to it. So why did we introduce that
capability? Yeah, great question.
See the use case is essentially to provide a less level of
isolation required for multiple tenants.
So you know, there is a concept of tenant even within the
private cloud, even within large and enterprises, there's a
concept of you know, engineering can be a tenant, HR marketing,
so on. And so there's an organization.
So we support multiple S3 instances within an object store
service, OK, each instance represents a single tenant or an
independent organization. Now within each org you can have
multiple projects. Like I said, engineering, HR,
marketing, each project can have multiple buckets and then the
set of users assigned within that tenant org can access one
or multiple buckets based on their access control privileges.
So it's really the multiple instances is to be able to
provide a level of isolation required at a tenant level.
Yeah. I think that's going to be very
useful and I think it's going to be very appreciated, especially
by the cloud providers because it adds an additional level of
security or layer of security. Now the other thing of course
that is also asked fairly frequently is when it comes to
the S3 object storage implementation.
Most people of course immediately make a comparison to
AWS because they created it. But of course there's many
different implementations out there, at least different
solutions out there. Now, when it comes to our
specific implementation, will we offer all the functionality that
AWS offers from day one? What will that look like?
Yeah. You know, so this conversation
can get a little misleading. If I talk about how many APIs of
S3 do I support, that is less relevant.
I would like to talk about this in terms of the functionality.
So we support all the at MBP, we support all the CRUD
functionality which is create, read, update, delete,
authentication, access controls, etcetera.
We even support some advanced S3 capabilities such as multi part
which is for large object optimization.
You know when you have really large objects, how do you
transfer them across the pipe in an efficient manner.
You know, we support things like sync to and sync from which you
can use to copy or replicate the data across across sites.
Even so, the, the initial set of capabilities in terms of the
APIs that we support for S3 is supports all those foundational
use cases that required crowd authentication, access control,
multipath, moving the data from one place to the other.
So, so that that's what that's what the focus was to provide a
fully functional S3 object store interface that addresses all
these use cases which are critical for DevOps as well As
for AI. OK, now in the future we have
plans to extend this implementation to things like S3
select, which is a more optimized way of, you know,
finding the data that you need and transferring, reduce the
amount of transfer on the pipe, object versioning, object lock,
life cycle policies. Those will come in the next
release, but I think the the the the capabilities that I referred
to on MVP should be more than good enough for all customers to
start off with our object storage solution.
Yeah. And I guess as you mentioned,
the first release is a MVP order or even the tech preview in the
next one is an MVP. So it'll make sense to expand on
that. Now we're looking at thinking
about those customers. What are the use cases that you
have in mind when it comes to these, you know, first couple of
releases? What are you thinking about?
Yeah. So you know, I look at it from
2, two different perspectives. And one is I look at the use
cases, as you know, from an infrastructure standpoint, what
are the, you know, what are, what infrastructure are we
providing? And then there's the application
centric use cases. So from an infrastructure
standpoint, you know, the main, one of the main use cases, as I
mentioned already is primary storage for dev OPS.
That's one. OK.
And then I'll talk about this a little bit more in detail.
The second big use case is S3 for AI, right, training,
inferencing and analytics. So that's the second use case.
And the third is S3 as a secondary storage where it's a
cyber vault or an archive or something like that, right.
So that's the infrastructure pieces.
Those are three use cases. Then on the application centric
use cases, we have many customers who want S3 object for
a CICD pipeline, right? We support service accounts
which can be pre created and fed into the CICD pipeline with
tools like Jenkins. And so that's fully automated.
So we support that, right? And S3 can be a artifact
repository for that CICD pipeline.
That's one. And the second is like a backup
repository for Kubernetes applications where you can
backup the entire Kubernetes cluster or selected
applications, namespaces, labels, resources and so on and
so forth. So those are some of the
application centric use cases that we are targeting with MVP.
About knowing you and I know the way that your brain works, so
I'm pretty sure that this is correct.
You probably already have ideas in terms of how the platform can
evolve and what are the use case we can potentially support in
the future. And there could be 12 or 18
months out, even longer than that.
So what are your ideas around those, if you can share?
Because I know not everything can be shared.
So maybe, you know, you can give a hint of what you're thinking
about for the future. Absolutely.
See, here's the thing, right, Ideas are always a function of
some customer actually trying out something that we didn't
expect them to try out. If you remember, Duncan, in the
initial days of VSAN, right, 12 years back, we advertised VSAN
as a Tier 2 storage, you know, don't put Tier 1 workloads and
stuff. And we had hundreds of
customers. They ended up putting Tier 1
workloads anyways, and they said, hey, look, vsan supports
all this already. And then we just went and
claimed support, right? So we learn new things from
customers. So I was very, very, very
surprised when I found out because, you know, now with VCF,
we offer one terabyte per core of VSAN.
That's a lot of vsan, right? So with 100,000 cores, that's
100 petabytes. So customers got creative,
right? And they already started using
vsan for AI use cases. OK.
So right now they're using the vsan file service like a read
write many volume. So 1 is that they're using vsan
for VLLM, right? A model store that you want to
store their models which can be loaded to run all the agentic
workflows, right? They, they, they do that with
files and customers came back and said, you know what, I'd
like to do the same thing with object, right?
And then, you know, they use files for an unstructured
database for RAG for RAG workflows.
Again, they feel that S3 object is better suited for the
unstructured database. So customers are asking me for
that. And then, you know, we have a
big partnership with our data Tanzu data intelligence team
where green plum runs on top of V San and it does time based
tearing. OK, so by that what I mean is
that any data that is less than one year old, you know, hot data
that lands on that can land on V San block, right, with high
performance storage. And then you know, they they
automatically tear that data, you know, 1 to 9 years, it gets
teared to an S3 storage, right with QLC.
Now guess what, now we have S3 support now we support QLC.
So this becomes up a very fundamental use case, right?
So these are three things, three use cases, the unstructured
database for RAG, the model store and the time based tearing
with green Plum that I just learned from customers that
they're either they're already using some for sort of we sent
for that or they're considering our object storage for that,
right. So these are like in the near
term, as soon as our S 3 is out, I expect us to support all three
use cases right now. OK.
Now in terms of road map, in terms of more advanced use cases
on VSAN object store in the future, we are working on a
number of things, right, like MCP readiness, which is model
context protocol. It enables AI agents and LLMS to
natively access VSAN buckets as context sources.
You don't need a custom middleware.
So that's one semantic tagging. We are natural language
processing, you know it automatically we go and classify
and tag objects at ingest. And so this enables semantic
search and intelligent data governance.
You know, combined with tons of data intelligence, this is going
to be even more kick ass right when we bring this technology
together. Then other things like, you
know, integrating with vector DB, you know, packaging a native
vector DB with V SAN object store, you know, I talked about
S3 select earlier, which is like a more efficient way to reduce
the data transfer for green plum queries.
You know, as the scale of S3 increases, we will start looking
at things like S3 over RDMA because RDMA transport for S3 IO
eliminates CPU bottlenecks on high throughput AI training
workloads, you know, so that's something that we're looking for
in the future. And then, you know, GPU direct
storage, right, enabling direct GPU to S3 storage, direct DMA
paths, bypassing CPU RAM for AI model loading and checkpointing
at scale. So those are like some of the
real forward-looking things that we are working on.
Super excited as you can see that the entire team is now
focused on delivering all this AI innovation road map for V SAN
object storage going forward. Yeah, I'm already excited about
the the session that we're going to be delivering at Explore
hopefully because I know we're going to reveal a lot of the the
future functionality that you're now working on.
Now, before I let you go, Rakesh, any famous last words or
any final thoughts we'd like to share with the audience?
Yeah, you know, it's, it's funny, my the PM on my team who
is actually developing A3 object store, he actually has he
carries buckets with him into a car wash.
Okay. And his bucket recently went
missing. Okay, so he was stuck in the car
wash he had No, it's one of those car washes do your own car
wash kind of a thing. And he had nothing to actually
clean his car. But it turns out that in his
boot he had another bucket, OK, which is his wife had left that
in the boot. So he took that and he was able
to clean his car. All right.
So I so I joked with him that look, you need replication, you
need, you need 2 copies, you need multiple sites.
So you were lucky that you had another S like, you know,
analogous to an S3 bucket. You had another bucket there.
And so, you know, at that point we said, you know, S3 to S3
replication is a super critical feature because how do you
really protect your S3? How do you really, you know,
there's no real way to backup S3 or protect it unless you provide
a replication? So I love this S3 bucket analogy
and it happened with my S 3:00 PM.
And so, so yeah, that's a big thing I wanted to mention is
that we will be providing native replication with S3 in the
product also in the coming in, in the near future.
Fantastic. Thank you very much, Prakash.
Awesome, thank you Duncan. Hope this is useful.
I look forward to having more conversations with you on other
topics in the future. And that's it.
Thanks for tuning in to the Unexplored Territory podcast.
If you enjoyed this episode, don't forget to subscribe and
leave a reviewer rating wherever possible.
And please join us again next time as we cover more insights
into cutting edge solutions shaping the world of IT.
Until then, staying comfortable, keep exploring.