Before we start our episode today.
I have an exciting news to share with all of you as we are moving
towards the end of 2020 and the upcoming Tech lead Journal. 20th
episode. I would like to share some Joy
by having a massive free giveaway of jetbrains.
All products pack, personal licenses.
Each personal license is worth two hundred forty nine US
dollar. It's 100% completely free to
enter and I will choose five lucky winners to win those
licenses for more information on how How to participate.
Please check out this URL. Aztec, Legend, o.f / giveaway,
here's the link one more time. Technology, you know, the dev
slash giveaway. Please make sure to participate.
And I wish you all good luck. What I come to realize is that
technology doesn't move that fast.
The fundamentals are roughly the same.
It's the fact that we don't necessarily teach fundamentals.
When you start to focus on the fundamentals, then you don't
mentally get attached to one particular implementation.
Hey everyone. My name is Henry sura, one.
And you're listening to the tekhelet journal.
The show will be bringing you the greatest technical leaders
practitioners and thought leaders in the industry to
discuss about their Journey ideas and practices that we all
can learn and apply to build a highly performing technical team
and to make an impact in your personal work.
So let's dive into our Journal. Hi everyone.
My name is Alyssa from Singapore and I'm currently an
undergraduate studying information systems at Singapore
management University. I've had a great honor to meet
Harry through a mentoring program at school and he
graciously shared with me about this podcast that he's working
on. And I'm so glad he did.
I've definitely gained a lot of insight by listening to all the
experts here. Techne.
General is an eye opener and a great initiative.
There has never been anything quite like it before.
The content and gues brought in by Henry are specially created
to provide a wide diversity. Understanding what tech
leadership and good Tech practices are what I appreciate
most about. This podcast is a mono effort
and attention to detail in the podcast notes, which can be
assessed on Techni journals website.
The transfer was openly share independent films are nicely
structured s even a breakdown on the conversation, highlights are
easy digestible information and not to mention.
What I was brought up during a podcast, be thank God jargon
terminologies or references. I especially enjoyed a podcast
that feature Neha Stephanie Crystal as well as Richard.
And googly eye find a once with women in Tech, especially
inspiring because it motivates me to Aspire to be just like
them, work hard, and to consider practical and meaningful
projects and initiatives to make a text-based about the police.
They're sharing insights to their personal Journey
challenges and how they overcome those obstacles to get to where
they are and advise, the three techniques Kristen definitely
value at the conversation. Thank you, general, practice
self in building and growing Their audience based
organically. So, any form of support would be
greatly. If you like or find benefit in
the work being done, please consider subscribing and
becoming a patron to support writing a podcast.
Like I did. I cannot wait to see the
pockets, grow and flourish Henry, definitely deserves it
all for the amount of work. He has footed.
So yeah, that's all for me. And thank you so much for having
me. Hello, my friend.
Thanks for tuning in. It's really great to be back
here with another new episode of the tech lead, you know, with
me, Ojos Henry Surya with Robin. We just heard from Alyssa one of
my early. Is of this podcast.
Thank you so much for sharing your story with us a Lisa.
I really really appreciate your support and I'm also very happy
to hear that you have benefited so much from the episodes and
the show notes. If a Lisa story is resonating
with you and you would also like to make a contribution to the
show by becoming a patron. Please check out for more
information at technology. You know that Dev slash Patron,
your support will tremendously help me towards achieving a goal
that I'm currently running on the page.
If you haven't joined Any of the technology, you know, social
media channels, I would like to invite you to join us on
LinkedIn Twitter or Instagram and you can find the links to
those channels in the show notes, make sure to also,
subscribe and follow this show on your favorite podcast app in
today's episode. I am so excited to share with
you. My conversation with Kelsey
Hightower. Kelsey needs no introduction.
As he is one of the leading figures in open source, cloud,
computing, and kubernetes. And it's um, One that I look up
to for his thought leadership and contributions to the
community. We started the conversation by
having Kelsey. Share his inspiring career
Journey from where he started at the beginning to where he is now
today at Google Cloud. He then shared his invaluable
advice on how one should learn and develop knowledge in the
current fast. Changing technology, landscape
overcome impostor syndrome and thus be able to succeed in Tech.
We then continue our discussion to This latest technology
updates beat in cloud, sevilla's and communities including his
latest observation and views on micro Services versus monolith.
If you follow Kelsey for quite some time, you might have heard
about his guide, kubernetes the hard way, which you can find in
his GitHub repository, Kelsey shared with me.
The reason why he created such the hard way guide to learn
about a particular technology. And at the same time doing it
publicly. This is definitely an episode.
You don't want to miss, it's jam-packed with knowledge and
wisdom. And I personally learned so much
from this conversation with Kelsey.
I hope that you will enjoy this great episode.
Please. Consider helping the show in the
smallest possible way by leaving me a rating and review on Apple
podcast and other podcast apps that allow you to do.
So, those ratings and reviews are one of the best ways to get
these podcasts to reach more listeners and hopefully the show
gets featured on the podcast platform.
Also looking forward to hearing any comments and feedback on the
social media or you can also directly send to me at
technology. You know, that death / feedback.
So without further Ado, let's get started.
Welcome everyone. So today I have a very special
guest someone that I admire a long time especially for his
technical leadership and also experience in terms of the cloud
and also kubernetes is none other than Kelsey Hightower.
So Kelsey very pleased Have you in the show today?
Welcome to the technique Journal.
Awesome. Thanks for having me.
Kelsey. Maybe for a start.
What are you up to these days? These days.
I spend a lot of my time around the serverless thing.
So I work at Google Cloud as people may know.
And the area that I'm most interested in is around all
these service Technologies, primarily Cloud run.
Most people are familiar with kubernetes and running their
containers, but there's still a lot of friction in managing
clusters and all the complexity that comes with that.
So I'm very interested if we Close the gap on the server.
The side. Can we remove the infrastructure
that keep the majority of the flexibility for running people's
applications at scale. So it's my primary thing that
I'm doing internally at Google on the outside world.
I'm very interested in the security landscape.
So it's been a lot of time with things like spy or which means
one of the open source projects behind spiffy giving identity to
our applications in a way that's portable.
So for people that are interested in Cloud, you know
that there's I am you have an identity for Amazon.
You have an identity for Google Cloud.
You have an identity for Azure, and those things are not
necessarily super easy to make portable.
So this whole concept of spiffy is basically giving people an
identity. You can think of a domain like
example.org and then having your app live at calculator or Foo
and then those things could be passed around and TLS
certificates or drop tokens. So this is really interesting to
have Federated identity across VMS container service and even
other Cloud providers, so it's very interesting.
Haven't heard about that. So is it like a portable service
accounts in that sense in some ways?
I am is a complete package. When you think about I am, not
only do you get an identity. You also get the how it can be
used. And you also get a way to
enforce those things. This take is steel.
For example, most people may be familiar with service mesh.
And when you say service, mesh is steals, one of those projects
that come to mind. And what sto does is allows you
to have a control plane, but that control plane, you can do
things to say at a can talk to Abby and then you need a bit of
enforcement at the lower level. So Envoy is a modern proxy where
all your network traffic and come in and out.
So, what is spiffy fit in? So, we have to tell the system
who is at a, so imagine, you're running a container inside of
kubernetes, and kubernetes will give your container a service
account. We could think about that as
like the root of trust. If you are on a VM, you'll have
a metadata service where you can go and get a token.
Again, another way of ID in yourself, and these are two
different things, what tends to happen, inside of a Service mesh
is you take those trusted identities and something like
spires and open source component, where it can trust
kubernetes, or it can trust Google cloud and you can trade.
Those security accounts for another artifact.
These are called like security documents.
It could be a TLS certificate. So you can do TLS Mutual off
between app a and at b, or it could be a jet token that you
can pass around in the typical HTTP header.
So, now that you have this kind of universal identity, it can
work as head of sto it can work on.
Set another cloud provider or any tool that understand the
stiffy IDs and then you can start to do things like policy.
Now that I know who you are. Maybe you're using an SSL
certificate to authenticate, to me.
I can look in that SSL certificate.
And there's going to be a subject alternative name that
has your spiffy idea is just a string, something super fancy.
And that string will tell me what domain you're in
example.com., And then what your service ID is Foo, I can take
that and then go look up to see. Can you even call?
He's in points. So it's more of a way of
formalizing identity in a Federated way.
That's independent of things like, kubernetes service
accounts or Google cloud service accounts, very interesting
indeed. Thanks for sharing that.
So Kelsey, before we go deep into the technical stuff as
usual. I'd like to ask my guests to
share their courage, any highlighting maybe major
highlights or turning points that are interesting for the
listeners to learn about. So maybe, Kelsey.
Can you share your career Journey?
I've been very fortunate, maybe a couple of weeks ago.
Go. So when did a nice profile
piece, a guy named Tom in the protocol?
So this is the protocol website. If you go there, it's also
pinned to my Twitter profile, but it walks you through my
career trajectory from working at fast food in high school
McDonald's. At maybe the age of 15 all the
way to where I am now and Google for those that haven't read the
article. The biggest turning point for
me. Number one is like many people
listening. I'm self-taught.
I remember, just buying books, on Unix and learning python a
little bit on my own even though we say self-taught.
Our own actually, we were taught by the authors of those books
and doing a lot of self-study. So, no college background.
I really just got off the ground just doing certifications and I
tell most people that was my start into Tech just getting
certifications. There was a few turning points.
There was a time where I have my own small computer store in a
small city, right outside of Atlanta Georgia.
And from there. I've met a lot of customers did
a lot of Windows support built PCS inside of the computer
store. Maid service call.
Calls. And if we were to fast forward,
a lot of my early career is around system administration
change Windows deploying software watching monitors and
trying to resolve issues Zoom a little further out.
A lot of my experience starts to become more development.
Experience, writing code, automation tools and some apps
that would even go on to run in production.
And then there's this other Turning Point around, open
source and public speaking around 2012.
I ended up working at Puppet Labs, which is one of the first
open source container or configuration.
It tools for managing servers. This is right around the Arab
devops this idea that developers and operations folks with work a
little bit closer together and infrastructure as code was born.
And that was a right around the time.
I started doing a lot more public speaking at meetups and
larger conferences that kind of helped me identify.
My second set of skills, which is teaching people.
So instead of just Building Technology and shipping things
teaching other people how to do. So, I think those are the major
turning points in my career. That Kind of lands me where I am
today. Yeah, I read that piece of story
as well. It's really inspirational.
I would say, looking back reading your article.
How do you feel reaching your career so far?
It can be good to hear other people.
Tell your story through their eyes.
One thing that was nice about that particular post or article
was the fact that they took the time to interview people that
I've crossed paths with over my life and let them tell that part
of the story. I know how I viewed that story
but watching the other people even though I know them and some
people I've only met once. But just watching them, recount
those stories and Scenic put in that timeline with some context
around it. It was also like reading and
learning about someone that I've never met either.
It was flattering nonetheless, but it was also nice to see
someone recap my life in a way that I can digest as an
outsider. Yeah, interesting indeed Kelsey
like one of the major phrase in the article saying that there's
just no way for a guy like you be able to succeed in the tech
that represents a lot of effort and probably a lot of things
that it got involved into your career.
Or to be where you are at this moment.
And one of the things that I see, for example, is that you
are problem, the underrepresented groups in
technology or so these days, there are a lot of people from
those kind of groups. Also trying to break through in
the technology landscape in the technology communities in the
technology career from your point of view.
What are some of your maybe advices, or maybe even tips on
how to break those kind of barrier?
Especially for these under-represented groups?
Because we can learn probably from your journey.
Yeah, so think about tech it's warm.
Industries, I think on the positive note, where typically
skills are enough to get you to succeed typically or should be
because in many situations a lot of this technology is so new.
That if people only try to hire people with phds, it will be
hard for them to be relevant because there is no great Ph.D
program for things like kubernetes or some of the
software development practices. We use today in Industry.
So that's good news that you don't necessarily require
credentials to be successful in Tech.
Now some of the barriers Is that can be challenging in every
country is different, right? When you think about globally,
every country has local social issues that make it harder for
some groups to participate than others.
Maybe it's not appropriate to assign all of those problems to
the tech field, but they're exaggerated inside of tech
because of the social things that are around them.
So for me coming from an underrepresented group, I'm
considered black or African-American and there's
lots of biases in this country that I live in, right?
Some people may look at me and say there's no way you know what
you're I mean, there's no way you can be technical, maybe you
should be playing sports or something.
And so that presents a challenge meaning.
If people look at you that way, it may be harder to get a job or
even if you get the job. It may be hard to work on
complex projects. You need to skill up and level
up. So what happens?
Then you now have to take on a little bit more responsibility
of finding those opportunities holding onto those opportunities
and then being able to execute and then outside of that in
general. It's always amazing to me that
in technology, lots of copies, just water.
I hire people who know everything on day one, even for
a beginner role. You want someone that has five
years of Technology, x, five years of Technology, Y.
And we just don't do a good job in our field of teaching people
continuously, whether they're new or they've been around for a
while. It's always this friction of.
We expect you to know everything and we don't really have any
formalized training to continue to grow skills.
There's other Industries like being an electrician.
They have a whole progress. You go from like a journeyman to
It's new you pair up with someone who does know what
they're doing or have been doing it longer and allows you to
continue to improve over time. And there's an expectation that
you can pair and have something more official.
So those are some of the challenges in Tech that I think
a lot of people have to deal with but I also feel that to be
able to overcome those things by having access to the knowledge.
So that's one thing I think Tech has done well, which is making a
lot of this knowledge and tools accessible.
Whether that's free and open-source the knowledge that
we see in blog posts and in books, but The other thing we
haven't done. Well, I think is make the rolls
the opportunities to leverage those skills.
We haven't made those as accessible as we have the
information. It's very interesting that you
pointed out about these continuous training.
So I didn't think of it that way initially, but now that you said
it, I think it's true as well, and especially these days.
People want to hire people who know a lot of stuff.
Could it be also? Because of that technology,
moves so fast and they just can't probably spend enough time
to invest time in people. You upscale themselves and they
just want to have everything starting from the day.
They join, and then start running straight away.
Could it be? Because of the pace of the
technology itself, doesn't allow them to actually have the time
to invest for training. You know what, I used to think
that but what I've come to realize is that technology
doesn't move that fast. The fundamentals are roughly the
same. Like when people say, kubernetes
is new, all the fundamentals. I can probably recall. 10 years
ago. The workflows may be different.
Instead of copying a binary onto a Linux server and start.
Earn it by hand. There's a much bigger system
around kubernetes that does that for us.
So the concept of scheduling isn't new technology that dates
back, 40 to 50-plus years, this idea of scheduling.
It's just a how we're using the fundamental.
So for anyone that has fundamentals, you look at these
new systems and you'll say, okay.
They're more workflow engines. They're more about composing,
the fundamentals into a system with a design purpose.
So, I think when people look at technology shifts their bigger
in the minds of people who don't, The fundamentals and
there's nothing wrong with that. What I'm saying is it's the fact
that we don't necessarily teach fundamentals.
When you go to a job. You hear people say I'm a Linux
system administrator or I'm a Oracle DBA, not really that
you're a database engineer. If you are focused on databases,
then you would focus more on how data structures are held in
memory. How the sequel execution engine
works, how the storage is laid out on this because those
fundamentals apply to postgres. They apply to db2.
They also Apply to things like Cloud spanner, where you have a
multi-region, replicate a SQL database.
When you start to focus on the fundamentals, then you don't
mentally get attached to one particular implementation.
So, that's what makes things seem impossible is like, oh, I
have to learn this implementation that
implementation and that implementation and then people
feel like they need to start from scratch.
The way I look at new technologies just says, okay,
what fundamentals is this thing building on top of?
And once I can see those, it's okay.
Now, I know that I know 80% of what's going on here.
Now, let me go fill in a twenty percent which is their
configuration, their workflows. In your opinion.
What are some of the building blocks of the fundamentals that
people in the tech? Should know about?
Let's talk about networking. So you'll see people in Tech to
say, hey, we're going to be doing service mesh and I say why
or what is the service mesh doing for you?
And then you might see a long pause, because it's just, it's
Theo, is going to make it easy for me to secure my
microservices, and that's straight off the website.
Now, the fundamentals in is Is at the very bottom there still
networking. So IP addresses subnets routes
L3 versus L7 L3 is going to be in some ways to simplify this.
How does one packet get to another destination?
May be outside of the core Network and at L7 now we start
to get into the protocols. We're all used to like HTTP
where we think about posting a request to summoned Point.
That's great. So, when we think about securing
those layers, you're going to do different things for each layer.
So, at L7, we're going To be talking about things like TLS
certificates. SSL.
We're going to do things like we want security to help us with
deciding if you're trying to make a request that this path.
How do I identify you? And then how do I reject you in
a way that makes sense to are seeing you back up, 500 or do I
send you back a 403? Those are things that you just
know at the networking, tier based on their protocols that
you're dealing with? But guess what?
It's still IP addresses. They're still routes TLS is not
necessarily brand-new. So doing TLS Mutual off to
encrypt between your two. And then give them an identity
that you can actually apply policies on.
These are fundamentals that were true 20, 30 years ago.
I think in that case make sure you learn about authentication
and authorization and while those are different if you
understand those fundamentals, then when you look at something
like is Theo or service mesh, you can look at how they try to
represent those Concepts and their own systems.
Yeah. I think the yellow message is
definitely resonated with me as well.
So focusing on the fundamentals, they tend not to change
especially for a long years. We can go back, not just for
infrastructure but also application design like design
patterns and also integration patterns and things like that.
They tend not to change so much but the tools and the
implementations seemed a plenty these days with all the
Frameworks and the language is popping up here and there.
So another aspect of breaking through in Tech, I feel is that
there's this thing called impostor syndrome, especially as
a junior person coming to the tech industry with so many
technologies that are available these days either open source,
either cloud. Also, our pride three.
So there seems to be insurmountable amount of
technologies that someone needs to understand first of all, or
know the general principle and also understand about the
implementation of it. The details.
These things tend to put someone down in the first place in the
sense that, oh, there are so many things.
I'm probably not good enough to learn all these things.
So what are the lobby suggestion or advice from you of dealing
with this imposter syndrome? I think there's probably way
smarter people have written about this in general.
So I'll try to just speak from My own experience at some point
in my career. I realized that I didn't need to
be the best in every piece of technology.
I was going to be responsible with.
I knew I needed to understand it to the degree that my job
required. And if I thought it was going to
make sense. I could go a bit deeper than the
job required as an investment in myself.
That was the foundation of my career that I at some point
arrived to. So, what does that mean?
That means that I can decide what technologies are important
to me? My skill set that I want to be
able to put on the market or Leverett for the things I want
to do it. Mel is hot right.
Now. I have very little interest in
learning tensorflow right now. Someone can save your missing
out. That's the hottest job prospect
in the world Etc. But that's not for me.
Maybe I'll touch it every once in a while just to see what's
going on there. Maybe do a Hello World type of
tutorial and then I'm okay saying that it is not my area of
expertise. There's only so much time in the
day. Now the areas where I am
interested I take on the A of going as deep as I can.
For example, I like to go programming language, but I'm
familiar about how it parses a syntax unfamiliar about the
surrounding ecosystem. I understand how go routines
work, understand some of the trade-offs you make in the go
programming language when you're making system calls to the
kernel because I also deal with things like containers and
containers definitely get low level.
And so for me, I'm very patient with saying I'm investing in
myself and I'll pick and choose the right things to make that
investment in. And in terms of imposter.
Drum, I think there's two people who feel this.
There are people who are truly in postures.
Meaning you're trying to be something, you're not.
And whenever you do that, that can also make you feel very
uncomfortable because you're pretending to be something.
You're not around people who might be and that's not
necessarily A person who knows the most.
But if you try to force yourself to be one of the people who
knows everything, then you're going to trap yourself in this
feeling. I've resigned to say, you know
what, it's okay to learn in public since I don't know
everything. I'm comfortable with asking
questions. Ian's and I get it.
Sometimes you're going to get penalized for not knowing
everything. There's some companies who are
some organizations or teams, who practice a very unhealthy
practice of anyone who asks a question.
We're just going to believe that they're dumb and no longer ask
them to do anything. That is just unhealthy.
That's what I was talking about earlier.
We don't have a formalized set of training to say we are going
to invest in people continuously.
So for me I decided I'm not going to pretend, I know
everything. So therefore I'm not worried
about being an imposter and therefore I don't Worry about
trying to be the best at everything.
So I Stay Focus in the areas that I am interested in and take
the responsibility just to get slightly better day over day
year over year. So in the first place, knowing
what you want needs a certain kind of conviction for some
people, this tends to be easy while you got at.
For example, I'm good in application development.
So I know what I need to focus on, maybe programming language,
maybe some Frameworks, but for some people, especially those
who come from non Tech background, they might not have
such a conviction. They might just see from the
news or see from the job. Korea's what are some of the hot
things to do. So, is there any things?
Maybe how should you advise people to have that kind of
conviction? If I think back to high school?
I remember you had elective classes.
So outside of math science, may be English, you had to take
electives and this could be things like home Mac where you
learn how to cook and maybe so close.
I check on Mike by the way, there's technology class their
Sports. There's all kind of things you
can do in that. Elective in the point of the
elective is to Those kids two more things than they would
probably naturally pick on their own.
So it's about exposure. So, I think when people are
starting their career or making a career transition, I think
it's healthy to try a little of everything because there's no
way for you, to know what you're hearing from other people is
things that may have worked for them, but it may not necessarily
work from you. So I think one thing is hey,
maybe try tech support. Maybe you like helping people in
a way that allows us to support people go into and don't think
one role is inferior to another. I think that's another mistake
people make. I in my career and tech support
and even within tech support, you can go super deep and I
think tech support is a great path to site.
Reliability engineering as free or even software development,
especially if you can understand how to troubleshoot and debug
these systems. The next area.
I think people have to look at here is once you try a little
bit of everything and something resonates with you that might be
a period of your life. We always talk about the
t-shaped engineer. So you might say, hey, I'm
really liking working for some reason.
It motivates me to want to Deeper.
He know what? For the next 125 years.
Maybe you want to become a network engineer.
Go get all your certifications work.
In an industry where you can actually improve your skills
around networking. And guess what?
It's okay to switch. Maybe you go from network
engineer to someone who write code for networking systems and
now you're shifting to software engineering and maybe you like
python that's another opportunity to go super deep on
Python and then maybe after 10-15 years you step back you've
gone D and Several areas, and now you have the top of the T.
You have the horizontal set of skills, and maybe at that time
you're super deep, and maybe you go back to network engineering
where you combine all your skills and go deeper than you
were before. Thanks for sharing that Kelsey.
I think there are few things that I could pick up here.
So, first of all, is that? If you don't know what you're
good at all, you don't know what to pursue, try a little of
everything. And also don't think of one role
as inferior to the other. This applies not just to roll
but maybe also to Technologies. And also, when you Find
something that you are resonating with, you can go deep
and the last is that it's okay to switch.
You don't have to self identify yourself with a particular
technology or particular role. So at one point in time in your
life, if you feel like comfortable switching to another
role, I think it's okay to do so as well.
So Kelsey you have been working in the cloud landscape for the
past. I don't know how many years.
It's a long time. Now, what are some of the
possibilities in the future when you talk about Cloud?
Yeah. Crowded interesting cloud is
infrastructure for most people. And if you think about
infrastructure, what's possible, if you think about airports,
then it becomes possible to travel other parts of the world
in a more accessible way. And for a lot of people more
affordable. A lot of times infrastructure is
going to enable us to do things. We can't do by ourselves or with
limited set of resources. So when I think about cloud and
we talk about public Cloud, there's a lot of computing
tasks, where maybe you want to start an online e-commerce site
and these days, people expect that side to be available in
all. The country's High availability.
It's up all the time and they want to be able to use any form
of payment that they want. And the goal of the cloud is to
enable that as seamless as possible.
So, when we think about the cloud, I think we've got a good
lock on infrastructure, components, like databases and
compute networking, some security layers.
So when we think about the friction that's left, is we're
asking people to understand all of those things before, they can
go out and build something. And that's the opportunity for
the Cloud. So this is why I'm excited about
things like serverless, where we try to abstract away as much of
that underlying infrastructure as possible to get people closer
to, here's my idea. Here's the code that powers, it
run it for me. We have a lot of work to do in
terms of user experience there, right?
We still exposed to many configuration options, when most
people are just after the best practice.
So if you're going to create a database you may not understand
all of those hundred options that are available on cloud SQL,
what you want, is it? Where's the button that says,
best practice for what I'm doing?
That's what most people want to do.
So I think cloud should be something where you can bring
your ideas. And when you have an advanced
use case, of course, you can always drop down to the lower
level infrastructure and kind of build a platform that you need.
But as Cloud providers, we should be trying to make the 80%
use case as secure and easy to adopt as possible.
So when adopting Cloud these days, people also tend to talk
about being Cloud. Native.
So can you explain to us? What is Cloud native basically,
that's a fun one. Some people would say cloud,
native is 100% in the cloud and leveraging, the patterns that
were born in the cloud. I like to simplify this a little
bit. If you think about the patterns
around observability having structured logs in a way that
instead of logging that there's just an error, you're going to
log that. With some context, this
particular package or service is having this error.
Here's the client that called it.
Here's what I was doing and to give you more insight.
To go along with that log message.
We may even put in a request ID and that request ID could have
been generated from the client or the hdp load balancer that
will sit in front of me and I can then take that request
context and put it into HTTP traces so you can know how much
latency between the various Services because you're thinking
about these distributed systems. You're going to need a lot more
observability than you have before, right?
Because these things are a lot more complex.
Again. This is a pattern born in the
cloud where we've made it really easy to get machines.
Across the globe. So in order to make those things
easier to troubleshoot, we need things like observability, and
not just logging plaintext to some file.
We need a different kind of pattern.
And as you look around, the whole Cloud native set of
patterns, right? So when you say cloud native, I
think of patterns, think about health checks, instead of
running some bash script to see if application is healthy, that
won't necessarily scale to the cloud model.
And I'm not talking about just a bunch of machines.
I'm talking about having things across multiple zones or regions
or That could go away at any time because in early Cloud,
there was no guarantee your virtual machine or run forever.
So you need it better patterns around Health checking.
So you had some other tool that can come by and check the health
of all you're in points. And if one were to go away, it
would then be able to automatically provision another.
So to me, it's a collection of patterns that were born in the
cloud. I think, most people would do a
good job of deciding what your those patterns benefit them the
most, because you can actually apply some of those patterns
even outside of Of the cloud, maybe you're running a couple of
servers in your own data center. You might still be able to
benefit from things like standardized, help, checks and
metrics. And then the last thing I'll say
here is that a lot of these Cloud native patterns.
Now are no longer ideas that you find in our white paper there.
Now open source projects that you can find in this CN CF.
So the cloud native Computing Foundation is a home to a lot of
these open source projects with the maintainers and the
communities can come together and collaborate and push a lot
of these standards forward. I do hear some people.
Then say like, for example, Cloud native is if I use all the
particular Cloud providers products, so, then, I'm Cloud
native. Is that a misconception?
Or is that something that is true?
In what sense? Maybe you have a take on that.
If you use all the cloud providers products, I don't
think that automatically qualifies you for cloud native.
I'll give you an example. Let's say you have an app that
just runs on the set of virtual machines, behind a cloud load,
balancer. Well, you Coulda did that on
Prim, you could do that with being where you could do that
with openstack. You don't.
Need to adopt any of the cloud native patterns to make that
work and typically that's referred to as the lift and
Chip, take what you were doing 10, 20 years ago and just do it
in the cloud. And the cloud supports that via
infrastructure-as-a-service is that to me isn't necessarily
Cloud native. Now, there's a lot of value into
leveraging cloud services, and I think nothing's wrong with that.
But I think when we say cloud native, we're basically talking
about a world that allows you to effectively leverage those cloud
services in terms of resilience. See, reliability, observability
and to get all of those things. This is where we start to
delineate, or distinguish. What's a cloud native, set of
patterns that go along with that and in many ways, when I really
think about it, a lot of these Cloud native ideas or concepts
are really at the application tier.
So, for the first time, we're now focusing on the relationship
between the application and the infrastructure.
And that's where a lot of those patterns are to be found.
So you mention about patents and a lot of people associate
clowning. Beef with the 12 Factor
application design or principles.
Do you think that 12 factor is a good representation of all the
patterns? How, this might be an unpopular
opinion, but my answer is no, I think 12 Factor was a great
precursor to allow you to practice some of the patterns
that you find in Cloud native. So, a lot of people have come
from this world, the past, like, Heroku and Heroku.
Was this kind of infrastructure or pass?
That would say, look, giving your source code, but here's
some restrictions in Were to be able to run your application on
any of our servers and a way that's portable for us.
We're not going to allow you to do things like have a local
volume for storage. We're not going to be dealing
with configuration files and copying files all over the
place. So now you're going to have to
take things like in an environment variable and a lot
of times. These are really nice ways to
make an app a little bit more portable because you're no
longer requiring a file system. You're being very explicit about
State, you make sure that if you do have persistent data, you put
in a persistent data. Base explicitly and you also
build your apps in ways that are easier to start up, that can be
rebuilt. So, I think a lot of those
patterns or just really good software engineering patterns,
that allowed us to actually, run applications at scale and a
platform such as Heroku or app engine or Cloud Foundry.
So, I think it was great for those.
Remember those panels came out what 10 15 years ago, nothing
wrong with those, but then we get you a slice of some of these
Cloud native patterns. And the reason why I think they
apply just a little, is because we're talking about Out
documenting and formalizing practices over the years.
So 12 factors a good place to start but here's the thing, you
can have Cloud native applications that are not 12
Factor applications. Like Cloud native could be
things like Kafka or it could be something like, at CD at CD,
definitely uses storage. It definitely doesn't use
environment variable. So it probably violates maybe
four or five of the 12 Factor principles, but that doesn't
mean that it's not Cloud. Native Cloud native is not about
stateless, only. I think stateless applications
are a little bit easier to adopt some of the cloud native
practices, but I don't think we should exclude stateful
applications because they're not 12 factor that the totally makes
sense. Kelsey.
You mentioned that you have been dealing a lot with several has
Technologies these days. So what do you see the trends
coming in? The Civil has area?
Are there some technologies or maybe things that everyone
should know about? When I first learned about
surface, I want to through maybe AWS Lambda functions as a
service changing events together.
And these were So engines that we lay on top, that was a really
nice way to think about this problem.
It wasn't necessarily a brand new way of thinking about this
problem, because a lot of people have tried of have done
successfully, event-driven architectures in the past, but
when I hear serverless, I think about the operational model that
comes with it, which is, how do I reduce as much friction as
possible. And you think about the word
servlets, we're talking about a couple of things.
One. We're talking about the
application Level, the concept of a server, if you're going to
write a web app you typically in bed.
A web server that you get from your framework and that web
server has to buy into an IP address.
It has to think about logging. If you get to the lower levels,
you have to think about some of the connection details and
timeouts. There's a lot that goes into
running the server, even outside of your application logic.
And then when you add in the underlying server, that needs to
be patched, it needs to be updated and he's a network.
There's a lot of friction there. So if you look at Lambda, it
removes all of those things are attempts to and what you're left
with is a function. Business logic will be invoked
whenever there's a data packet or something that you need to
request or an event. And what I think we're trying to
do now is say, hey can we not give the same operational model
to normal containers where you do decide that you want to
package in your own web server or some other protocol?
That makes sense for you. So I think one thing we're
trying to do now is on the cloud provider side.
We're asking ourselves tougher questions and trying to meet a
higher bar of usability. So what I do, I want to do is
tell you to rewrite your application.
The perfect state would be keep your application as it is.
Now if you do certain things like enable health checks or
support metrics, it will definitely help you because
there will be no server for you to log into to troubleshoot.
So you may want to add those things.
But if you don't, I still want to give you the same operational
model. Meaning, I want to be able to
scale down to zero. So when you're not using this
application may be in, Deborah QA, then you don't pay for it.
If you want to run over multiple regions.
You don't have to learn about all of this multi-region.
Networking. And it gets very complex at
those layers and then I can also promise you a bit of security
because I can keep the server's patched underneath you.
If the contract becomes a little bit clearer.
So I think what people should be paying attention to is think
about all of your favorite tools and services whether they're
open source databases like postgres in MySQL, or if you're
writing containers, regardless of the framework Ruby on Rails
or spring boot in the Java World.
Imagine just being able to tell the cloud provider, you run
those systems for me. And now just give you the code
and configuration. I want to leverage with them.
What do you think the current state of civilization?
Is it good enough for people to use and bet there are
Technologies on it, or is it something that still needs a lot
of improvements? I think it's super mature
depending on what you want to do for certain workloads.
So if you're doing event-driven architectures Lambda from AWS
Cloud functions from Google Cloud, those are super mature,
and tons of people are already using those things in production
at pretty decent scales. So this means that you've
written code. Executes fast understands how to
deal with events and the ecosystem around.
That is also fairly mature. A lot of the data Brokers are
doing a good job of retries and giving you visibility into how
the execution flow is going. And then there's configuration
tools that help you to deploy those kind of systems.
So I think for event-driven architectures is fairly mature,
especially when you're being very explicit about State on the
container ecosystem side. I think our goal should be.
Can we provide the equivalent experience?
Is that you get with a virtual machine, for example, with a
virtual machine, you can mount a data volume.
You can do em FS. You can do machine learning with
the GPU. You can do all of these things.
When someone gives you a virtual machine and the serverless work
today. If I think about Google Cloud,
run we can do about 70% of the things.
I just talked about, we could do NFS, we can run any Docker
container. You can use any library that you
want. But there's some challenges
around doing things that maybe you want to run Docker inside of
that thing. Today, that won't be necessary.
Easy to do, bi-directional to your PC is also something that's
coming down. So you have to think about the
protocol that you use. We support websockets into your
PC, but we may not support every single protocol that you're used
to when you're dealing with a virtual machine.
We got to close that Gap. And once that Gap is closed,
then it becomes a trade-off. How much control do I want on
the underlying infrastructure and not about what you can and
can't run. So we got work to do this.
Some other Technology support from Outrun upcoming in the
civiles area. So when I say serverless, now,
I'm thinking about the operational model serve.
This can be applied to more of our data stores and databases.
Typically, they'll have a free tier, they'll scale, 20 pay for
use. We also have things around
workflows and for building things.
There's Cloud. Build a lot of people.
Like, I don't run Docker on my machine anymore, as it may be
two and a half years ago. I just don't run Docker locally.
I just use cloud build, so I still keep dr.
Files, but whenever I want to build I just have a build script
that just says g-cloud submit to the cloud build system.
And then there's a container that's living in some registry
in the cloud. And then I can actually complete
the rest of my workflow. That to me is another concept of
service. Just more managed Services where
I don't really have to do all of this work.
And we're trying to do that with control plane.
So, if you're thinking about service mesh today, instead of
your clusters, a lot of people will install sto and all of its
control plane components. That one thing we're trying to
do with tools, like traffic director is say, hey, what are
we were able to centralize This open source control plane, same
open protocol support standard sto and Envoy proxies, even
bring your own but then that just runs if a more managed
environment to me that can also be seen as a form of serverless.
So I think for us to goals going forward is Imagine taking as
much of our managed Services as we can and giving people the
option to leverage a cost model. That's paper, use today with a
VM you created in your pay by the hour, but we want a world
where you only pay when that thing is doing.
Something. So, that's a goal that we have,
and that could be Cloud SQL, that can be postgres, that can
be Cloud spanner. That could be Memory store.
I just want to see it everywhere.
Interesting, take indeed. So, Kelsey, let's switch to
another topic, which you also a passionate about, which is about
monolith fascist microservices. So I want to go back into a
tweet that you posted somewhere around late 2017.
You wrote something that in 2020 prediction.
The monolithic applications will be back in style.
What do you see about this? Action.
Yeah, that was more of an observation because people had
already been going back or some people have never left.
I have an example. I wouldn't say it's a good
example, but it's an example is to use this service that we
talked about a few times here and it had a few components, it
had what we call the mixer. This is where you would send all
your metrics and logs and the mixture would decide.
There's a good a data dog or stackdriver.
It was pluggable and configurable.
We had a thing called the pilot. It was an independent service
that will integrate with things like kubernetes.
We had Like the galley, there's just like all of these small
little Services, then when you start sto it was also
leveraging. Microservices itself to help you
manage your microservices, but guess what, the biggest
complaint we had was it's really hard to deploy all of these
components configure, all these components and most people
wanted something, a little easier to manage.
So guess what? The solution was the solution
was to take all of these components and make them to a
single service. So they went the other way.
They went from microservices to a Monolith and this is not a
knock against them. It was a good idea because you
still can deploy the individual components if you ever need to
scale, but the fact that they did that to simplify things.
I've seen this come up a lot. If you look at some of the
engineering blog post from, I won't mention all the companies
names, but they've done these blog posts with they say, hey,
we used to have 20,000 services and we noticed that we were
spending, a lot of our CPU serializing, Json or tracking
metrics or applying security policies instead of doing actual
work. So instead of doing that, we're
going to go from 20,000, back to 2000 and things are faster.
Things are easier to manage things are easier to understand.
And I think in those cases it's not that microservices are bad.
It's just that when some people were leveraging this pattern
unnecessarily or too far, it became unmaintained, Belen might
even make performance worse. It's the trade-off that you
should make when you're doing it for organizational purposes.
So if you're at Google, you might have a Gmail service.
And even within the Gmail team there might be.
Don't services, but that's where I see it.
Being super effective when it helps you align, your
organization around areas of specialty.
But even then in the monolithic world, you can achieve a lot of
these things with a monolith. If you're just a solo developer,
you're probably better off just writing the bottle.
If the start, if you're in a small team and maybe have some
good engineering practices, like writing modules that then get
compiled together as a. Deployable again, monoliths may
help you go very far in the last thing, I'll say on this is for
most Intents and purposes, a large majority of Facebook could
be considered a big model in the same is probably true of GitHub.
I'm not saying they don't have any microservices, I'm saying
that you can get real far without them interesting.
Take, actually, I also heard about the sto project how they
started as a micro services with all the components that you
mentioned. Just now, and with the recent
version release a, go back into a monolithic design.
So, certainly, there's some wisdom on why they're doing it
that way. Although some people now are
still trying to adopt, Services for their architectures.
So it seems almost like the de facto standard.
I would say in the industry, like oh, I want to build
something. Let's start with microservices.
So what's your take on that? Is there any rule of thumb for
you to decide when to use micro service or monolith?
Yeah. I think I would probably start
with the monolith whenever I can.
But remember we're not talking about a monolith with no
engineering discipline. I think when people say monolith
there, really describing in some cases, not all in some cases, a
lack of engineering discipline and standards.
A lot of times, if you have modular code based, meaning all
the services are in their own repositories, and they have
their own workflow and release schedules and integration tests
with the rest of the other services at a package level.
We're not talking about deployables yet.
We're still just talking about modular packages, just like the
standard library and maybe different libraries, you use to
create your own Services when those things are modular.
And there's clear ownership and good testing.
Then you can make a decision. Do you deploy?
All of those things individually?
Or do you import them all into the same main binary?
And then wire them up with some routes?
And that can actually be as maintainable or scalable from
Team concept as having independent, Deployable.
So, when you separate those two, I think you can say, hey, even
if we start with a monolith, we're still going to adopt some
of the practices that you find in a micro Services
architecture. We're going to split our
services into different repositories.
We're going to have clean interfaces for invoking.
Those I've seen Go as far as making a monolith and then
having all the services have to call each other service through
localhost. You can see that in some of the
old JBoss world or some of those big application Frameworks.
Everything is a web request even when you're deployed together in
a single binary. So you don't necessarily have to
get that kind of discipline in isolation, just from
microservices. And then if you develop this way
from the beginning, when the time comes to split one of those
components out, you'll be able to do it very naturally because
the boundaries will be very clear about how.
You do it. The last thing I'll say here is
for people that are struggling. Like I really want to do
microservices for my resume, or I want to make my LinkedIn
profile look really good. If you have a model of today,
one thing I always highlight to the team that's thinking about
making this transition. I say I guarantee you you're
already doing microservices today and I say no no Kelsey.
We're doing monologues. Trust me.
I wrote the code. I know my company better than
you do. That's okay.
How do you do DNS? They say my library calls the
DNS server based on this configuration.
Why are you asking that's your Service, right domain name
service. It runs over here.
So usually a separate binary and it does one particular thing and
all your apps call on that service to do name lookups and
they look around and say so you mean we've already been doing
microservices was like yeah. There you go.
You have Micro service architecture already.
Interesting story indeed in your definition is also is confusing
for people, right? What is actually micro service
to Cal CI Towers. If you don't compare the two
then I think it's easier to understand I worked at A company
that had a lot of stuff written in Python.
Everything is written in Python. That was a standard language and
then we had to do some work where there's only Java
libraries available. The thing we needed to integrate
with there was only a Java SDK. So we have to make a decision.
Do we write our own integration and maintain it in python or do
we leverage the Java mature SDK and we decided to do that
integration work. Using the Java SDK and naturally
that forced us into a separate Deployable and a separate
service. So now we have this big monolith
over here and we have this one new service over here that's
written in Java. But it exposes an HTTP API so we
can integrate with it with our python monolith.
That makes a lot of sense. And then someone says, Hey
coughs, we need to process some batch files again.
I revisit this decision. Should I really add batch
processing into the monolith and then have to deploy all of that,
monolith? Just to do, batch processing.
My answer would be no. So in that case, even if I
choose to stay with python, I might start a new binary, or a
new project or a new service that is written in Python, but
it can be deployed independently for dealing with the batch
process. Now, there might be code in the
monolith that I need to reuse. So then I have to make another
decision. Do I call the monolith or do I
import the same libraries at the monolith isn't using inside of
the thing doing batch processing.
And so when I think about microservices, I tend to think
of there is a pattern for deciding when to keep all the
logic. Gather and went to split it
apart and you can do both at the same time.
At the same company on the same team.
Yeah, I think it makes sense in that sense, but I would try to
also think of it like, it's a service-oriented architecture.
Anywhere. Is there any particular reason
why there's this micro in the prefix of the word.
So when I was first introduced to like so our service oriented
architecture, it was actually the Java ecosystem when I was
dealing with JBoss quite a bit. The Java.
Ecosystem to me did a decent job back.
Back then of saying, even if all this stuff is going to end up in
the same War ear file. There's a way to make sure that
we draw clean boundaries over each service.
So even though we may have deployed it all together.
It was a clean. Way to say this service is for
the user database will have this particular context this servers
for dealing with the data domain.
You had all these ways of deciding.
Okay, here's a various area. So that different people on the
team could work on the different service.
Maybe that's in the form of a jar or whatever.
And then at build time, we would take all these services and
we'll push. Together and we will create a
release and then that release, we can drop into a web
application server. And then that thing will deploy
all the services and then hook them up in a way that they can
either call each other directly or they could call each other
over HTTP. Depending on the framework, you
reuse them. So that kind of services
architecture, made a lot of sense.
Now, the drawbacks to that in some cases were when people
start reaching instead of a service inappropriately.
If you're not using HTTP as an interface, you may be going
behind the scenes, start calling private methods or you make the
private method public. So you can call it and now you
got the spaghetti code. It's things are so bad that you
just say, you know what, let's never allow this to happen
again. So what people say, look, if the
monolith was too big, if we regulate ourselves to make
things small, that might be one way of preventing those past
mistakes. So it's like going from a 10,000
square foot mansion. You live in a huge house with 20
bedrooms. And you say, you know what?
I have too much Furniture. The whole place is dirty.
I'm just going to move into a one-bedroom apartment, to make
sure that never happens. But guess what?
If you still have the same practices of not, cleaning up
after yourself, there's a huge chance that your one-bedroom
apartment will also be messy over time.
So, I think the idea with the micro service would be, let's
try to split things up in a way that we can be very clear about
Our intention. The user service, only does user
service stuff. And when you want to do
something else, you have to go to a whole different repository.
A whole different way of thinking about it and we can
then catch it. Things are getting too big.
If I start seeing the user service.
Is handling the data model for the web catalog and I can say,
hey. Well, well, that doesn't belong
over here. This thing is getting too big.
It has too many responsibilities.
That's how we got in trouble last time now.
All this is a good idea when you start to think about different
departments, where there's a team that only deals with user
management. It might be nice for them to
have a service that's coped. Well for their domain AKA micro.
But yes, to your point, the word micro is debatable.
Should it be maximum 1,000 lines of Load should only have five
HTTP routes. That's a debatable thing that I
don't know if it makes sense to try to regulate yourself to a
certain size, but I think it has to be around a certain domain.
And I think that's what we're after.
I like the story where you use the analogy of mansion.
And to one bedroom. I think habits never change.
And I think, yeah, the term micro is very confusing in the
world these days. Like sometimes I tend to speak
with fuel Technologies or sometimes even friends talking
about micro Services. They will say yeah, I'm doing
microservices. This but all I can see is okay.
You almost like just splitting your model it into just three
different services and call them micro services.
So to me, that's confusing and I like the responsibility of the
domain thing that you explain, maybe we should even call it
responsible service, but that's probably about another time.
So Kelsey, you also do a lot of things around kubernetes, right?
Including even in the beginning, where people haven't even heard
about kubernetes and sometimes the containers as well.
And also, you wrote a book about kubernetes up.
Running. We also did kubernetes the hard
way, which is your GitHub repository about learning
communities, the hard way. And I also heard that you are
doing the service mesh the hard way.
So, first of all, what are some of the lessons learned from
doing all these things? The hard way?
Yes. Remember we talked about going
deep. I remember when I first got
involved with the kubernetes project.
I was working at Coral West and we need this project was going
to be coming out red hat and Google or big partners and maybe
a few other companies. But court was wasn't really
contributing. Shooting at the time.
There's going to be a press release.
Those going to come out the next day and I remember getting early
access to the repository and I'm looking at this things.
Like, how do I even try it? I don't need to see the
documentation for running this on a system that I have.
At that time. It was my MacBook with VMware
Fusion running on it. So I have to spend that night
just really learning what the couplet is.
What does it do? What are these flags?
Even mean? What is the scheduler doing?
How does it work? Where should it run?
And going through that process? I really learned all the
individual components of kubernetes how to compile.
How to deploy them. And I remember just
experimenting with the configuration till I got
something that could actually run a container.
I remember a publishing that blog post with screenshots.
Here's how you run kubernetes on VMware fusion.
And I remember when I was going through the code base, I was
like, wow, a lot of this go code, looks like Java.
We should call it covid g eyes to contribute early improvements
to try to clean up the code base a little bit and also maybe add
a few missing pieces. So some earlier work that I was
doing, was thinking about how to automate node membership because
at that time you would have to bring NG a node in and then just
generate the config and connect manually to the API server, and
then register the node yourself. So, Otto node registration was
one of the first things that I worked on and added support for
coreos to automatically provision nodes and join them to
the API server. Little contributions like that
around and did a little bit of work on C and I but what I've
learned from then documenting was, when people would say
kubernetes is Hardware. I will go and do a lot of these
workshops. I'm super excited in those early
days. I'm like, who wants to learn
kubernetes? I could be walking down the
street. Hey, what are you?
What a last will cover Nettie's and then I would just sharing
all the knowledge I had with people because I was still
learning and this is what I call learning in public as I'm
getting excited. I had all of these Avenues to
share my excitement. Whether that was a keynote
stage, or a YouTube video, or my text editor, adding some new
functionality. And I remember when we were
asking ourselves, why are people saying kubernetes is hard?
My conclusion was, they don't understand it.
It's not about having the easiest bash scripts run, that
gives you a kubernetes cluster and five minutes.
It's that even if you ran the script, most people didn't know
how to maintain the cluster. They don't even know what
components are running there. So I decided early on maybe a
little later when I first joined Google right before that instead
of trying to build tools that make everything easy.
How about we make it easy to understand not necessarily easy
to provision and we ended up doing both soku badman came out
around the same time, but I also wrote kubernetes the hard way,
which was stepping people through all the underlying task
getting the binaries provisioning.
The VMS creating. SSL certificates, copying them
to the servers, and wiring everything up.
And then what we found was, lots of people around the world were
like, wow, I finally, never understand, all the moving
pieces, and that adopt gone through it.
Maybe once or twice, it's still complex, but it doesn't
necessarily need to be hard. Once you understand the
components. And remember, this is written
towards anyone that has an operations mindset or wants to
learn how it works at the infrastructure, level.
That was for them. It wasn't meant for developers.
Even if you're a developer and you're curious, it wasn't to try
to make your developer workflow. Oh easy, but for people
responsible for managing a cluster, it was from them.
So what I kind of learned from this is sometimes the easiest
way to make something easy is not necessarily make it easy to
use. But make it easy to understand
and is that also the motivation. Why you coming with the service,
smash the hard way. So anything that you want to
learn more about service mesh. Yeah, you're exactly right
again, learning and public. I started learning things about
issue, maybe two years ago, maybe two and a half.
I even gave a few key notes. Is the when I was looking at it
I first learned sto, like a black box, like a lot of people
I installed it. It has a config interface.
I can configure things like encryption between my services
rate-limiting, all kind of nice, high level networking policies.
And this is great. When I started to really look at
the code base, one of the first things that contributed was a
prototype for the sidecar injector.
So for people that use service mesh.
Typically, you will deploy Envoy next to your application and
then Envoy would do the heavy lifting in terms of making sure
that the policy that you put in STO would be enforced by the
psyche, our Envoy and Envoy all traffic.
Will go in Envoy and all traffic will come out of envoy, but you
need to configure your deployments to do that.
So the injector would basically rewrite your deployments
automatically before they got scheduled to a node.
And then when they start to really understand this TSA, wait
a minute. If I look at the history of
control plane and its high-level configuring, which what is it
actually doing? And looking underneath the
covers. I was like, wow, this thing is
basically just Dating a config for Envoy, including the SSL.
Certificates to do this encryption between the services
and just pushing everything down.
How's it? Pushing it down?
Go through the XDS protocol Envoy has a way of streaming
configuration. So as things change in the
infrastructure, you can keep the policies and the back ends up to
date. That's like, wow, this is
intriguing. But what about everything else?
Like how do you do off Z once you do get a service that
connects to, you has Envoy handle dealing with whether you
are? Allowed to do this thing or not.
And again, you need something like open policy agent or you
can say craft the set of policies that says this service
ID or this spiffy ID can call these paths.
And then how do you lock all these things?
But it's Prometheus fit in. So I look the best if that's a
lot to understand coming from the top down.
So I decided to start working on this, from the bottom up.
What I've done is to set the stage for mesh, the hard way is.
Okay. Let's start with just a set of
apps. Actually, I start with the
monolith where I do everything inside of a single.
Binary is a very simple calculator app with an API.
So you call the API, it will call the ad service subtract
service or multiplication service and it has a little bit
of authentication before you can.
And then I split that up with no visibility.
No logging, no tracing and no security and just have a bunch
of little services and then I posed, the question.
How would you lock this thing down?
And that's when I start introducing things like Envoy.
Start introducing things like open policy agent Prometheus
Zipkin for tracing and I start adding these Things one by one
so that people can understand the role of a service mesh and
how it all works under the covers as they build back up to
their own service mesh by hand. Wow, really interesting.
So, how far before we can see it in public, or is it already
available as available to me on our private GitHub repository?
All the code is written. All my configs are written, is
this thousands of envoy configs in there.
So now what I got to do is get the narrative if anyone's seen
any of my latest talk to. You see me tease a little bit.
I've been doing a little sections of it.
Little Live demos, I think couple of days ago.
I did one for the production identity day, for the Linux
foundation and the CN CF right before, Coop, Khan.
And there, I showed a little bit about how to create your own
spiffy IDs and jot tokens yourself from scratch.
All of that stuff will be covered.
So for me, it's going to probably take me about three or
four more months given that I'll probably release it in 2021.
Maybe I'll do it around my birthday, which is in February
as a present to myself to deliver it.
And this year, or this time. I want to add a lot of diagrams.
One thing that kubernetes. The heart of it doesn't have is
any diagrams and I think a lot of people would like me to go
and just say, hey, add a few visualisations and diagrams.
So we know what's going on each step of the way and I think I'm
going to do that this time. I'm looking forward for that.
For sure. So Kelsey, another Trend that I
see in communities world or people thinking about adopting
kubernetes is actually to use it almost for everything.
These days. We see people create their own.
See are these custom resource definitions or you see a lot of
kubernetes related tools and Frameworks even or something
workflows. And the more of it, actually, I
see, the more these tools will keep popping up.
So what is your take on? Doing everything the kubernetes
way? So, one thing that I'm really
happy is that the curvature is Project, really decided to draw
the line around, what's core functionality.
And what's not so core functionality, and kubernetes is
going to be some of the components, like the scheduler
things, like our back, so you can control, who can write what
configs, and who can read, what configs.
And configs would be things like your deployment objects, your
secrets, your config minutes. And then we have some objects
that are also considered core like a replication controller.
So when you say hey create me a deployment, those replication.
Controllers are the thing. That makes sure that you have
three or five pods depending on how you configure it.
But then we said, okay, outside of that core stuff.
What else do people want to do? And most people want to create
their own object type. So instead of just the standard
collection of deployments and config Maps Etc.
They want to create their own things and deploy their own
control loops. So, the nice thing about
kubernetes is if you step back from kubernetes, And you don't
install the agent for running containers, the couplet and you
don't do any of that. What are you left?
With your left? With this, basically control
playing framework? It has roles, and permissions.
It can generate routes for you. It can generate client libraries
for you. It can do all of these things
like what we call the API machinery.
And so an API Machinery does is it allows you to also create new
API endpoints without writing all the code to do all the other
things. And so the contract there is
given this universal control plane.
You can create your own API through a crd custom resource
definition and maybe that control.
Plane does. I don't know.
Let's call it fly an airplane. Some reason you might want to do
that. So you might create a resource
definition that says tell me the plane tell me it's starting
point and tell me this destination.
So with those three things you may design a crd that captures
that and also in your crd you also talk about how it should be
presented. So when I say Coupe CTL get
airplane controller, then you'll see them listed out.
So great. So you have all the Side server
side stuff done for you. So then what's left then you can
create your own control Loop. And here's the thing.
A lot of people get confused. You don't have to deploy the
thing that flies airplanes that control Loop inside of
kubernetes. You can deploy that on a VM, you
can deploy that on the service platform because kubernetes
doesn't need everything to be in a container.
The only thing you have to be able to do is communicate with
the API server. So you don't even need any
machines in your quote-unquote, kubernetes cluster.
You just need to API machinery and Then given that Machinery,
you can run that control Loop somewhere else.
So, at that point, then cabernets can actually be
helpful in almost any context where you want to declarative
control plane. As long as you're willing to
create this, crd the API and the thing that does the work.
So when you start to say, hey, let's use coupons for
everything. The truth is on one hand.
It can be helpful in any of these contexts as where you want
to control plane to drive some outcome, whether that's eicd.
So you see it used for tools like Argo or tecton, where you
Say, hey, here's a list of build steps.
And then, yeah, Cooper days was like, great give me.
That definition. I'm not going to do anything.
I'll just store it, secure it, and let things collaborate with
this object. But other than that, there's
nothing for me to do and then you can build your own control
Loop that actually runs the ICD jobs.
That's the beauty of kubernetes. So this is why you're seeing a
lot of people use kubernetes, but more appropriately, the
kubernetes API Machinery to back these other projects.
So in that sense, looking at these Trends and especially The
containers has becoming more and more popular.
Do you see that? The VM approach will be
Obsolete? And do you see more like
containers in the future, becoming more like a platforms
and there will be the infrastructure of choice for
people to build the application on.
So the funny thing about kubernetes.
Kinase is actually very node Centric, even though we talked
about containers. We've done a good job in the
container Community for the most part of separating the machines
from our applications. So that means we typically
package all of our Tendencies but we still need the colonel.
What could burn Eddie's does as actually in my opinion?
It's just a very thin layer on top of for most people virtual
machines. So kubernetes to me, makes
virtual machines easier to use, not necessarily makes them go
away. Now, of course, you can use
kubernetes with bare metal, but for most people they're using
kubernetes with virtual machines.
So does that make virtual machines go away?
I wouldn't say that. I would say kubernetes makes it
easier to Manage a group of resources, typically, since
we're talking about nodes, and the form of a virtual machine,
and this is not very different from like puppet Chef or
ansible. Right?
These are configuration management tools, where you can
take all of your machines and put them into some form of an
inventory and assign roles to them.
And then that configuration management tool will do the work
of making sure that those machines behave the way that
you've described turbinates. Does it slightly differently by
having higher level Concepts like a container and that will
be placed in run. So what it does for this step of
our journey. Makes the virtual machine have
to do less work. It basically says, Hey Lennox,
we don't need a package manager down there anymore because the
user is going to be bringing all the code and dependencies that
they need. We no longer need people to be
logging into the machine. We no longer need to be putting
a bunch of Agents on the server, because now we can include some
of those agents with the deployment object.
So it reduces the role a little bit of a virtual machine.
Now, serve this, a slightly different, depending on the
contract of your service platform for So if you look at
Lambda s Lambda is code-based, then Lambda can do things.
Interesting. Lambda can say, give me your
code. Here's the languages we support,
but it's up to me. If I want to compile your
deployment for Intel or arm. You may not even get to pick the
CPU architecture because they can compile it to something that
they want to do for their own efficiency, whereas with the
container and if you pre-compile the binary, then I'm going to be
a little bit closer to a particular machine architecture.
So this is the big battle we have.
So in the serverless world at the Treatment is typically
source code based because then we can control what the
underlying architecture is. If you give me a container then
I may have to expose and make you make a decision.
Is this a Windows? Is this Lennox or is this
entail? Or is this arm?
Those are the things that kind of keep us gravitated to a
particular set of machines. So for those people who haven't
been exposed to communities, what would be your advice for
them to start with, like we're to learn these kubernetes, ask
yourself what your real? Our goal is, it's okay.
If your goal is just to learn kubernetes just to be educated
in the know. And in those cases, there are
lots of great kubernetes books out their crewmates up and
running is going to be probably a much gentler introduction.
My co-authors are Brendan Burns and Joe beta, those two are two
of the founders, not the only Founders but two of the founders
of the kubernetes project. So all three of our voices are
in this book, there's a second edition out and we try to take
your from. Hey, here's where you probably
are. Now, here's how you build the
container using things like that.
Care and then try to progress you up the stack to leverage
more and more kubernetes over time and I encourage you that
there's other books out there as well, that will take a different
angle at this and then there's some people who's like, hey,
maybe I'm going to pass the basics.
I'm done trying things out. I also want to build kubernetes
from the ground up and that's where kubernetes is.
The hard way comes in to try to help you with all of those
things. Now, if you want to be a
professional in that, you may want to look at maybe the Kuban
a certification program that the scene CF run.
So I think it's like the cuckoo bird, a certified administrator
or NG. And in that world you might say,
look, I want to be able to prove my skills so I can start
managing the stuff at companies or take a job with it.
That's a lot of infrastructure, Focus paths.
Now, if you're a developer, the way I like to think about it is
as a developer, you can start in a different direction.
One is, you can start with your own application, is my
application portal Bowl, where it can actually be decoupled
from the machine in a way that kubernetes likes, for example,
in my building, my containers, in the way, where it can run on
a wide range of kernels. That's a good place to start.
Do I have helped checks and my leveraging some of those Cloud
native patterns that we talked about earlier.
And if you do all of those things, then if your
infrastructure team or maybe even yourself decide to use
something like kubernetes, you're going to have a great
head, start of being able to use all the features of kubernetes.
Because your application has a lot of those standard interfaces
that crewmates can use. For example, if you expose
metrics using something like Prometheus, and those standard
libraries, then those metrics can be used to Auto, scale your
application. Education and side of
kubernetes. If you log to standard out and
maybe even do some structured logging again, criminals will
automatically collect those logs for you and send them to
something like stackdriver logging.
You have so many choices. But if I was the developer, I
would probably start by making sure that my application can
take advantage of all those things.
And then learn more about how to do a deployment inside of
kubernetes. So again, if I was purely focus
on development, maybe I don't necessarily take a lot of time
to learn how to build a cluster from scratch, but I will try to
learn how to A deployment object.
How to expose my service, we have things like pod disruption
budgets and that's more of a little bit of an advanced
concept. The idea there.
If you say I want three copies of my app, you can use a pod
disruption budget to say. If someone is doing maybe
automatic upgrades for the cluster, I don't want to ever
have list into of my apps running for reasons.
And that way, when someone goes try to scale you down to one
that pod disruption configuration will be your way
of saying. Hey, you can go no lower than to
and it Will prevent the administrator from doing that.
So those are a lot of application Centric things that
you can learn inside of kubernetes in order to
articulate how your workload should run.
Thanks for sharing that. I'll make sure that to put all
these resources in the show notes.
So, Kelsey, it's been a pleasure talking to you.
I learned a lot about communities, Cloud native
civilization, and all the things that we have discussed so far as
my last question, normally, I would ask every guest that.
I have to share their three technical leadership.
Wisdom, Kelsey. Do you have any that you want to
share it all of us here here? So the I've learned over time is
to decouple your identity from the technology and that helps
you make decisions. I think in a better way, so
instead of being a Linux system administrator because there's
going to come a time where Windows is going to be the best
platform what you're doing. If your identity is so close to
Linux and Linux only you may not even consider anything else
because you're so walking to that the same is true for people
to ICD system. There are some people that will
only use Jenkins even when something like Spinnaker, might
be a better. A fit or something like
Spinnaker and Jenkins is a better fit.
So if you decouple yourself from the low-level Technologies and
you step back and maybe try to align yourself more with the
fundamentals. I think that's going to help you
be a little bit. Little bit easier to work with
because you won't be so dogmatic about things but it also helped
you open the door to New Perspectives versus getting
bogged down in your job title. The second one would be take
your time. I know everyone talks about high
productivity and going fast and all these things.
But what I've learned over time was I slow down and enjoy the
path to get there. So if I'm learning a new
programming language, I'm okay. Slowing down.
Do some Hello World. Try to rebuild things that I've
done before and then stop. And then next week, pick it up
some more and I know that maybe within a year, I'm going to be
really comfortable with building the things that I want to build.
That doesn't need to happen in two weeks.
It doesn't need to happen in three months.
I'm okay if it takes a little bit of time because it's an
investment. For the Long Haul.
So to me that level of patient. So what does that mean?
I don't look for how do I learn everything?
I need to know in five minutes or show me the easy way.
I'm okay saying it's going to take a while and then slowly
builds up my skills and continue to make progress.
I guess the third one. I found it easier to inspire
people and to boss them around and that's by bringing my whole
self to the job. So if you're in a situation
where you're someone's boss or manager, of course, you could
always try to tell them. Exactly what to do and then
maybe punish them if they don't do it correctly or praise them.
If they do, that's one way of doing things and it may work for
a lot of people but what I found most effective for me was to
inspire people into action because when that happens they
fill in the blanks in a way that I would have never asked them to
and then approach it typically with way more passion and energy
and they will bring their whole selves to the project and I
think inspiration, so leading by inspiration or persuasion
instead of authority. Even when you have authority
over, folks can change the outcome and dramatic ways and
allow the other person to grow. So I would think, for a lot of
people explore, what it would take to inspire people versus
forcing them to do something and you might be happy with the
results. Very insightful indeed.
So thank you so much, Kelsey for participating in the show.
I really, really enjoyed this conversation, and I'm looking
forward to have you in the future episodes.
So, thanks again. Kelsey and good luck with your
mesh, the hard way. Awesome.
Thanks for having me. Thank you for listening to this
episode and for staying right till the end.
If you highly enjoyed, please share it with your friends and
colleagues who you think would also benefit from listening to
this episode. And if you're new to the
podcast, make sure to subscribe and leave me your valuable
review and feedback. It really really helps me a lot
in order to grow this podcast better.
You can also find the full show notes of this conversation on
the episode page at technology. No, the death website.
Including the full transcript interesting quotes, and links to
the resources and mentions from the conversation.
And lastly make sure to subscribe to the show's mailing
list on technology. No, the deaf to get notified for
any future episodes. Stay tuned for the next
technique Journal episode. And until then.
Goodbye.