Definitey is the top International software
development conference, with an emphasis on coding architecture
and Tech leadership skills. The lineup for this year is
truly stellar and features many Legends in software development
names, such as Robert Uncle Bob, Martin can back Scott Hanselman,
Franca subramanyam Carolyn honey Alan.
Hello, Mary poppendieck and many other prominent names including
some of those who have also appeared in this podcast before
the conference. It's takes place online so you
can enjoy it from the comfort of your couch.
We spoke to the definitey organizers, and I'm happy to
share that technology. You know, has got the 10%
discount code for you. Enter the promo code, awsm
underscore tlj. When you purchase the ticket on
definite e.com, here's the promo code.
One more time awsm underscore, tlj.
Depending on the time when you purchase a ticket, early price
is still available. See you there?
And that's a totally reasonable way to get started.
It's all about just, again, embracing those service truths
that we talked about it first, right?
Rye, Billy's, most important thing.
Your users Define your liability not you.
So make sure you're measuring the right thing and 100% is out
of the question. So pick the right target.
You can Embrace those truths without real time monitoring and
advanced statistics and all the stuff that comes along with.
It just get started, even if it's in a spreadsheet even if
it's only just once a month, Hey everyone.
My name is Henry Surya with Robin.
And you're listening to the technology, you know, podcast
the show where I'll be bringing you the greatest technical
leaders practitioners and thought leaders in the industry
to discuss about their Journey ideas and practices that we all
can learn and apply to build a highly performing technical team
and to make an impact in your personal work.
So let's dive into our Journal. Hello to all of you, my friends
and my listeners, welcome to the technology.
Now, podcast the show where you can learn about technical
leadership and Excellence from my conversations, with great
thought, leaders out there. And today is the episode number
96. Thank you for tuning in and
listening to this episode. If this is your first time
listening to technology, you know, make sure to subscribe and
follow the show on your podcast app and social media on
LinkedIn. Twitter, Other and Instagram.
And for those of you who enjoy this podcast and wanting to
contribute to the creation of the future episodes support me
by subscribing as a patron at Tech Legend, l.f / Patron,
implementing a sorry Concepts and best practices can be
daunting. Although Google released a few a
sorry, books, including the famous site, reliability,
engineering book. Many of us still have some gaps
in terms of really understanding the essence of the concepts and
practices such as this. Service level indicators or SL
is service level objectives, as a Lowe's and error budgets.
And on top of that, how can we start building a good SRE
culture and avoiding some common pitfalls, especially when
communicating the benefits of these set of practices, for
example, to the business or stakeholders.
Also, do tools matter in implementing a sorry how
reliable should our service be and how should we measure it?
These are some of the common questions that I know.
People usually ask. When introduced with SRE
concept, and if you do have the same questions and thoughts in
your mind, then today's episode is definitely for you.
My guest for today's episode is Alex Hidalgo, Alex is the
principal reliability Advocate at Noble 9 and the author of
implementing service level objectives book.
Alex previously work at Google as a site, reliability engineer,
and also customer reliability engineer and also contributed
multiple chapters to the site reliability.
Look book. In this episode, we discuss the
Practical guide on how to implement a sorry practices and
service level objectives or S ellos Alex started by explaining
the basic concept of service reliability and the tree service
truths, he then explained the concept of reliability stack,
that includes the famous SRE Concepts as allies as a Lowe's,
and error budget. Alex, then shared his insights
on how we can define a service reliability Target.
Why a higher reliability Target? Is expensive and the risk of a
service of being too reliable towards the end, Alex shared his
tips on how we can start building, a sorry culture and
how we can use the error budget as a communication tool within
the organization. I very much enjoyed my
conversation with Alex and even though I have been learning the
asari concepts for quite some time.
Now, there are still a number of insights that are learned from
Alex in this episode. And if you also find this
episode useful, please share it with your friends.
Colleagues who can also benefit from listening to this episode?
Leave a rating and review on your podcast app and share your
comments or feedback about this episode on social media.
It is my ultimate mission to make this podcast available to
more people. And I need your help to support
me towards fulfilling my mission.
Before we continue to the conversation.
Let's hear some words from our sponsor.
Today's episode is proudly sponsored by skills matter.
The global community and events platform with more than 1000
software professionals here members can organize their
learning experiences around the technology topics.
They care about most you get on-demand access to their latest
content thought, leadership insights, as well as the
exciting schedule of tech events running across all time zones.
So where the devops our data science is your bus or you're a
fan of functional programming or all things Cloud, you can make
real connections with people who share your interests head-on.
The two scales method on cam to become part of the tech
community that matters most to you.
It's free to join and you will find it easy to keep up with the
latest tech Trends. Are you looking for a new cool
swag package? You know now offers you some
swags that you can purchase online these wax are printed on
demand based on your preference and will be delivered safely to
you all over the world where shipping is available, check out
all the cool tracks available by visiting technology, know that
death / shop, Oh, and don't forget to break yourself.
Once you receive any of those tracks.
Hi everybody. Welcome back to Tech lead, you
know, podcast today I have a guest with me named Alex
Hidalgo. He's the principal reliability
Advocate at Noble nine and he's the author of a book, titled,
implementing service level objectives and contributed to
another book which is part of the asari book from Google,
which is titled site reliability workbook.
So as you can tell Today we are going to talk a lot about SRE
and slos and things like that. So if you are wondering about
this practices SLO how to set it, right?
And things like that, we're going to cover it today.
So Alex really? Thank you so much for your time
today looking forward for this composition.
Thanks so much for having me. So Alex, I like to actually
start with my guests to tell his or her story.
Any career turning points any highlights?
So maybe if you can share yours. Sure.
My route to where I I am today has been an interesting one.
I grew up as a computer nerd. My dad helped teach me how to
program when I was eight or nine years old.
I was on the internet before the web existed back in was also
text. So through, most of my
education. I always assumed I wanted to
work with computers so I didn't go to college After High School.
I went and I got a job in the tech industry, but that first
job was doing network security work for the government of the
US. And I hated it.
I was pretty good at it. Actually got a promotion within
a year but it made me miserable. And so I thought I didn't want
to work with computers. I thought computers were just a
hobby for me so I quit my job and realize it was still time
for me to go back to school. So I started college a few
years. After most people I ended up
studying philosophy and history and I took a bunch of creative
writing courses. All the while computers were
still a hobby but I kind of decided Added computers weren't
for me as a career. Then I decided to move to New
York at the height of the 2008 recession.
So right when the global economy and especially here in the US
things are just absolutely tanking suddenly I'm in one of
the most expensive cities in the world and I can't find a job.
My money's running out whatever I had of savings at the time
because in my 20s I worked in the service industry.
I was a server. I was a cook, I worked in a
warehouse, my money was a starting to run out.
I kind of I can probably still do this computer thing even if
it's just for now, even if it's just to make some money and get
back on my feet, the very first tech job I applied for which was
an, IT desktop support kind of position.
I was hired right away. I actually had so much fun.
Is that the small company only about 10 people?
We were the IT department for companies that didn't have their
own. So, every day I'd travel all
over New York City, every Borough all over the place, just
helping people with whatever they need.
Needed help with and I really loved it.
I loved that human part that interacting with other people
and helping them learn how to use their computers fast
forward. A few years, I'm working for
this company called add meld now and add meld is purchased by
Google and suddenly I'm at Google as a site, reliability
engineer. I don't even know what that
meant at first. But as I learned more and more
about it, it spoke to me so much.
It was really what I had always meant to do.
It was every bit of My personality, like the things I
truly love, it was all kind of reflected in it.
The human aspects thinking about users, putting humans.
First blameless culture for incident response and incident
retrospectives, just everything about it.
I absolutely loved. So, I really fell into the role
really well fast forward. A few more eaters.
My last roll at Google before I left was on the customer
reliability engineering team or see reaching the Siri team was a
group of Fairly Variants azeri, we were tasked with teaching,
Google's, largest cloud, customers, howl, to a sorry, how
can people make their services more reliable?
We realize that the biggest thing that we needed because the
customers we were engaging and were not other tech companies,
they were not subdivisions, a Google, right?
They were retailers and they were industry manufacturing
companies and they were all over the place.
So we realized we needed a common vernacular, we needed a
shared language. And what we Sided was SOS
service level objectives would be that shared language so I
spent a good year or traveling all over and teaching all sorts
of people what s lows are and why they're so great and how to
use them. Because that was really the
building blocks that the Siri team wanted in order for us to
engage with people in the best possible way.
I had some fun doing that but it was my time to move on from
Google. I'd been there for a long time.
Everyone needs a change at some point so I moved over to
Squarespace. When I started at Squarespace, I
was asked, hey we want to do SL owes you know how to do is Loz,
can you help us? And I was like, sure, no
problem. I set this up with my manager, I
was going to spend like, 60% my time on teamwork and about 40
percent of my time teaching, the entire organization, including
my own team, how to do slos. I didn't realize how much work
that really was. When you're getting started from
scratch, when you are starting from just the absolute bottom,
People barely even understand what all the different terms
mean. It's a lot of work.
You need to build the right to Lang.
Chances are you're monitoring systems.
Don't even do a solo math. In the first place you need to
do education and workshops need to build document repositories.
It was a lot of work. We didn't Define our first SLO
at Squarespace for about six months.
After I started after about a year and a half of that.
I was running a workshop every Friday afternoon for Four hours
from noon until 4 p.m. and we had a break, but still, it was a
for our Workshop, every single Friday, and people really liked
them. They were popular, but I was
just tired of saying the same thing over and over again.
So, at some point, I was complaining to a friend of mine,
they're like a good co-worker. And I was just like, I wish
there was a book about this, an entire book, not just a chapter
and yessiree workbook, but a whole book because that way I
could Point people at it instead of just doing Workshops over and
over again. My friend said, well you should
write it and I said no an expert, you read it and he said
you are the expert. And I cursed like I straight up
just like said curse words because I knew he was right?
I just didn't realize it until he had said it, I suddenly knew
I was writing a book and I'd always heard how difficult that
is a few months later, I was working on a book for O'Reilly
implementing service level objectives.
That's directly led me to where I am today.
I'm now at Noble 9, which is a start-up based entirely around
how to do service level objectives, how to measure them.
We do all the tooling for you etc.
Etc. So I came to fall in love with s
lows at Google because I realized they made my life
better and they made my users lives better.
They made humans happier and that's always been the most
important thing to me. I've been very lucky to have
been able to focus primarily on us laws for last six.
Almost seven years of my career. Now, it's a great place to be
Thank you so much for sharing your story.
It is a very beautiful story, a lot of ups and downs, including
how you got to find your passion.
And hopefully today, you are not tired to speak about SLO one
more time. So at least today, we're going
to cover some of the basics. Thank you so much for sharing
this story. So Alex you wrote this book,
implementing a sailor, there are number of books, not many, but a
number of books about SRE SLO and all that.
But I still find a people, find it difficult to grasp the
concept. What do you think us?
Some of the challenges that people face in understanding
these practices. So I think some of the biggest
problems with both understanding, what a sorry is
site reliability engineering, as well as slos, what service level
objectives are is that no one's really ever fully defined.
It there is a course, the first Google as to rebook.
But you know what? It's like 30-something chapters
and not a single team at Google actually does all those things.
Anyway, their best practices, their And sure there's the
original definition that been trainer /, basically, the
inventor of site reliability, engineering.
He said, S3 is what happens when you ask software Engineers to
solve operational problems. That's great.
That's also very vague. And then, what is the difference
between a sari and devops is a devops engineer?
Even a thing or is devops erson approach.
And then marketing teams, got a hold of it, right?
Suddenly the SRE book was selling really well.
Well, so now Sony, oh, we're gonna have a sorry, companies,
or gonna have tooling. That is s retooling.
There isn't one definition and I would actually it doesn't really
matter as long as our end goals are kind of the same and I see
the same width as the lows. I have my own definitions.
I do think there are true kind of definition that everyone can
agree upon but you know in the first and second Google s re
books, they don't even Define what an SLI is in.
The same way. A service level indicator which
we can talk More about, of course, a very important part of
how to do is Loz the two different books, don't even
agree. So I think part of the confusion
is just that they're often aren't single resources to point
people at. We don't want degrees for this,
your purely a software developer, you can get a degree
in computer science, certain algorithms have certain names
have very strict definitions if you're writing code and a
certain language, you have to adhere to the syntax of that
language for certain languages very strictly Now we can very
clearly say this is a Java program versus this is a Python
program, but when it comes to more philosophical things like
site, reliability, engineering like service level objectives.
I think one of the reasons people sometimes struggle is
because it can be more difficult to start from scratch.
Like my story about Squarespace, where I realize, even though I
knew how to do these things, I had this 1200 person
organization that I had to teach from the ground up because there
aren't these strictly defined resources that people Can just
learn I think what you said speaks truth to my experience as
well. It's really hard to read those
as our ebooks. By the way, it's very dense.
Sometimes could be dry, sometimes could be Google
related. And there are not many available
experts out there, that can be set like certified.
SRE to actually tell you, this is how the better practice
should be and many tools vendors just came out and probably like
what you said, somehow polluted the term, they may be defined
their own definition, and things like that.
So I think that's one of the challenge.
Today, let's try to also discuss about these Concepts from the
basic since you have a lot of experience.
You write this book, implementing slos.
But first of all, what is the definition of service and
reliability? Because some people actually use
these terms interchangeably. So many different variations,
maybe let's start from there. Yeah, absolutely service is
actually not difficult to Define because you probably already
know it's what the word service just means.
One of my favorite examples is that I've I'm so much my
experience and so much of why I love ASL owes is because I used
to be a server like server in a restaurant.
I provided a service for people which was to take their orders
and bring them food. A computer server is not very
much different. It is a thing that takes your
requests and response to its correctly is very similar in
concept. That's what a computer service
is, Computer Services, something that listens to a request West
from something and response to it appropriately that I think is
the best way to think about it instead of trying to Define
exactly what it means as a technological level because I
don't think that's important to some teams.
It is important to think of their service as a pod running
in kubernetes or a series of PODS or a Docker container
somewhere or a binary running on a virtual machine or a piece of
Hardware even networking gear. Those I'd Services as well and
those don't fit any of those previous definitions.
For some people, the service they care about is though.
Retail website you go to a place to buy a pair of socks.
Some people at that company, just care about that entire
website, even though it might be composed of Ed individual tiny
micro Services can really be defined as anything that does
something for someone else, I don't think we have to say,
okay, that's a service because it Is a set of PODS deployed via
a single deployments on kubernetes.
Whatever this isn't a service because it's a user Journey.
No, no, no, a service. You can think of it
holistically, you can think of it philosophically.
It's something that does something for and that leads us
right into what I think the correct definition of
reliability is because people often conflated with
availability, but they're very different things because the
service can be available and not be reliable, Rye.
Billy's old term reliability. Frankie's and goes back to the
1940s. It's not a concept unique to SRE
or Google or check or computers. What reliability really means
is, is the system performing, how it was defined to perform
for computer services. That basically means is it doing
what it needs to be doing. If we are cool with the concept
of a service, being a thing that does something for someone else.
Then reliability, is, is that thing doing that thing, it's
supposed to be doing the reason I always always try to steer
people away from just thinking about it, as being like, all
availability, is because you can be very available and still be
doing a bad job, that example, of the retail website where you
just need to buy a pair of socks.
Well, maybe you can log into the website and it's very quick and
you can search and you get 10,000 results for socks and
they have every color and every size and everything you ever
wanted, but then when you go to check out, you can't that's not
being reliable. Even if that service is being
available to you at the time. So a service is Anything.
It's doing something for something else.
And reliabilities means. Are you doing that?
Well enough. Thanks for Expediting.
This is very, very simple. I think everyone here could
understand that really as part of the foundation, right?
When you understand what is service, what is reliability?
The next few things will become easier.
But before we go into mallis re reliability and all that, I saw
one section in your book where you mentioned about service
truths. I think this is an interesting
concept for me. Would you be able to share,
probably what do you You mean by this service truths and what are
those sure? So I personally believe that
there are three things that are true about any service one.
Is that reliability is its most important feature, if your
service is not being reliable, it's not doing much.
As we just said, if we're defining reliability, is are you
doing what? You're supposed to be doing?
Well, then it kind of follows. If you're not be reliable,
you're not doing what you're supposed to be doing.
So you can always be thinking about reliability, first, the
And Truth is that you don't get to decide what your reliability
is. I don't care what your
measurements say, I don't care what your log say, what your
metrics a, I don't care if you have a million healthy podzol
reporting up if your customers. If your users, anyone that
depends on you thinks you're being unreliable.
You are if you're not meeting the needs of your users, you're
not being reliable. So you need to take their
perspective into account, and then the third service truth, is
that nothing is ever 100%. Sent so don't aim for it.
This is just a truth of the world outside of pure
mathematical constructs nothing's ever 100% things fail
failures occur. It turns out that people are
actually fine with failure. As long as figure doesn't
happen, too often. The third truth is, just don't
aim for 100 percent, because that's a Fool's errand.
So instead, pick a more reasonable Target.
Understand what? Reliable means, understand that
someone else determines what reliable means.
Understand that right abilities. Most Part of your service.
And then make sure you're only trying to be reliable enough,
like an achievable mount, something that works for both
you and the people that depend on you as you can tell, these
are the fundamental understanding.
So let me try to reiterate the first is that reliabilities, the
most important attributes or characteristics of your system.
No matter how your system has so many features or functional
requirements but if it doesn't perform reliably that's probably
not a good service and the other one is you don't Define you.
Reliability, but users do it for you on behalf so when they use
your system they will tell you if your system is reliable or
not and the last one. Nothing is 100%.
Don't ever try to achieve that because people are okay with
failures as long as it's not failing frequently.
Let's go back to the mall. Deep dive definition about SL is
Loz and all that. So you have this concept of
reliability stack, maybe let's start from there.
What are in the stack? How do you define them and how
they work with each other? Sure.
So you have three primary components of what I call the
reliability stack. This is what people really often
refer to. When they're saying s ellos,
when they often use the term as slows, they really mean a few
different things. First is SL eyes or service
level, indicators service level, indicators are measurements.
There are bits of telemetry about your system that tell you,
is it doing what it's supposed to be doing.
And again, this should be from your users perspective as close
as you can get at least then Next you have slos or service
level objectives and service level.
Objectives are just targets for how often you want your SLI to
be true. So if your SLI is able to tell
you, yes, we're currently meeting the expectations of our
users or people to tell you. Oh, we're currently not meeting
the expectations of our users. You're now able to inform a
ratio, good events over total events equals a ratio equals
some kind of percentage. Your SLO is just a target for
what you want that percentage to be.
So you can Something more reasonable.
So we just talked about a hundred percent is impossible.
No one ever hits that anyway. So you can say something like,
well, go back to the sock, buying scenario, we measure user
all night. Why not?
If you can't check out when you try to buy socks, that's not
good, but it's okay. If that only happens one and 100
times because you try to check out it doesn't work, what are
you going to do? Probably is going to click again
and as long as it works the next time fine.
So maybe you only have to make sure that sock check Out works
99% of the time. It lets you pick a Target.
That's realistic that accommodates failure.
That make sure you're not spending too much money and
trying to aim for something you can.
And then finally at the top of the liability SEC you have error
budgets are budgets. Are just a way of thinking about
how your SLO has performed over a period of time, an error
budget, often takes into account a Time window.
That's often fairly large generally anywhere from a week
to 28 days to 30 days sometimes even a Quarter.
The idea is your error budget, is that other side of the
percentage. So if you say sock, check out
should work 99 percent of the time.
But you're also saying, is 1% of the time sock?
Check out is allowed to fail. Your error budgie is that 1%
your error budget? Measures are we feeling more
often or the right amount? And so you are budget.
Lets you think about things over periods of time over the last 30
days? What?
Percentage of the time have we failed?
And is that helping us meet our SLO targets?
Are we exceeding it, your error budget, it's worded that way
because of budget, something you can spend and an error budget is
exactly that. We're allowing ourself this one
percent or whatever you've defined for your own service
course. But in our continuing example,
you have this one percent of checkouts are not allowed to
work. What amount of that have you
spent over time? And that then helps you make
better decisions about where you need to focus your attention.
Thanks for mentioning about this budget term because it is that
consciously Define. It's not something that someone
just take, but it's actually something that can be spent
errors is expected sometimes, but it also can be spent.
So you mentioned about defining reliability.
Many people got stuck even here, I just like how reliable is your
service should be? You said, it's not 100% some
business people. It should be 100% how can we
actually Define our reliability our service?
How should we go about it? What's the approach?
Unfortunately, this is one of those times where I invoke my
very senior engineer, who's been doing this too long card and I
say, it depends. It's Unique.
It depends on your service. The important thing is to be
meaningful and to be thoughtful, the important thing is think
about what is my service who are my users real quick, tangent.
I say user a lot, but I don't just mean customers.
I mean anyone that relies on your service, it might be
another team. The hallway, a might be internal
people at your company. It might be another service.
Your service might be six layers removed from an actual human,
but that other service still a user of your service.
So that's why I say user a lot because it can meet any of those
things. There isn't a single answer, be
thoughtful about it, take your time.
Think what does my service do? What is it supposed to do?
What do you do the users expect? Can I go talk to those users?
If I can? Let's go talk to them.
Let's ask them directly. Is there a product?
Direct management team. Do they have a user Journeys
document? Maybe we should go look at what
did the find user Journeys? Are why is this software written
in the first place? Why was it spun up at this
company? What does it do?
It's not the most satisfying answer because I don't have like
an answer. I don't have a special formula.
People can use to instantly understand what it is that they
need to be thinking about when they pick how reliable they need
to be. But what I can say is in every
single case, if you're careful and your thoughtful, That will
lead you to the best answers. One thing that I really love
about that sorry Concepts and slos and all that, they actually
always put users in the first place.
It's not some random technical decisions, okay?
This is how the reliability should be and it's actually
always comes from the users perspective like to mention in
the beginning, 100% is not possible sometimes or even
things that relies on Internet by default.
They are not reliable because your packets could be loss or
you need to retry know that reliability and other Thing very
important to understand, maybe you can help to explain is that
as you go higher, if you aspire to go higher, it becomes more
difficult complex, and also expensive, tell us more why?
This is the case. Sure?
I mean, if you want to ensure that you are being more
reliable, you need to ensure you're having fewer failures
individual components fail more often.
So, if you add more and more, you're just going to have more
and more failures, right? You have something that feels
99% of the time and you have just that one thing.
But if you need to be redundant, if you want to make sure they
have this thing, fails thing, a then you also now need to have
thing be in place because nothing be needs to take over in
case thing a fails that's all good and well, but thing be my
also only be 99 percent reliable.
So now you need something to determine whether or not thing a
or thing, B is currently being reliable or not and where to
send that traffic. So now you have things, see we
can go on and on. On.
But, you know, you just end up having to build more and more
complex systems to ensure that you are, in fact, being more and
more reliable time. This is sometimes totally
reasonable for some systems in some Services.
You do need to hit very high Targets.
This means you need to be distributed.
You need to ensure your built for high availability, and
quick, failover, purrs and redundancy.
There's all sorts of engineering for reliability Concepts that we
could spend hours and hours talking about W once you
introduce so many things, you're also spending a lot more money
because, no matter if, this is running on your own Hardware, or
your own data centers, or everything is in the cloud,
you're now running more things so it costs more money, both
just in a per month, building situation as well as the fact
that now, you need more Engineers to take care of it.
If you have 100 components to the one, you need more Engineers
to take care of those other components.
So now we have to hire a whole bunch of Engineers and now we're
trying to hit a really high reliability.
Ility Target. Well, what a lot of people don't
always do the math about is what that really means in terms of
like time, if you want four nines reliability that gives you
only seconds per month to respond, you can barely have any
downtime at all ever. So now, what do the on-call
rotations look like? So, you now have like a hundred
different components of your service because you have to have
incredible redundancy in all sorts of extra availability
stuff. And now you have a whole bunch
of engineering team to take care of all those components.
Now, they have to be on call. But they also would be on call
on respond immediately. That means no longer can those
teams just be singly home, do you need a follow the, some
rotation? So you and I are exactly 12
hours apart. So we could have a team in
Singapore and again with team in New York.
Great. But now you need offices in
Singapore, and New York. Now, you don't just have X
number teams. You have to X number of teams
because every team now needs teens and Singapore and teams in
New York to ensure that someone's only being on call
during the day, because no one could always wake up at 3 a.m.
and And in time to try and defend a Target as stringent as
99.99%, this is goes on and on, and on the closer, you try to
get 100% the more and more that grows its exponential if we can.
All agree that just logically you cannot ever hit 100%.
Anyway, it turns into a limit, the limit approaches Infinity
forever. You'll never actually hit on
upset anyway and trying to get there.
We'll just cost you more and more money in terms of how many
services you need how much you're paying for your Providers
got, maybe you can even be on one Cloud.
Maybe gonna go multi-cloud, cause what if AWS goes down,
will be better, be running in gcp as well.
Do you know how difficult it is to write stuff that lives on top
of both of those things and ensures that it can route to gcp
or AWS at the right moments in time how to even detect whether
or not you're a devious or gcp instances are running.
Correct at that point in time he gets so complex to try to hit
these high Target that you're just gonna kill yourself trying
to spend too much money, turns out, you don't have to hit those
targets anyway. It's Yeah, I was about to say
Indian after all these complexities and effort, your
users actually don't need that kind of reliability.
So I think the best way is again to check with your users.
What is their expectations? And yes, sure some systems will
require this kind of high availability but yeah hopefully
people who listen to Alex explanation.
You could actually understand you need to really Define your
reliability. It's not just some nines that
you see okay 49 so let's just put it that way.
Yeah so often leadership Tucson We're going to hit this target.
They pick a whole bunch of nines all in a row because they think
that's reasonable or because they know that some of Google's
Services hit that. You know what?
You're probably not Gogol. It's like Google has all those
things. Google does have teams all over
the world. Google does have to eat, it
starts off with for LT valerii, right?
And you probably don't, as you said, just be reasonable, be
thoughtful. Think about the target you're
picking and make sure that they make sense for you.
So let's start with the fundamentals of reliability
spec, which is SLI. So you measure is like a matrix
that defines how your users experience in terms of
reliability. Probably if we can relate to the
asari best practice, commonly it advocates for different golden
signals for monitoring probably Define the metrics of your
system. Is this the right place to
start? How do you define your SLI?
I have mixed feelings about the golden signals.
I'm on record does not going to blow any ones.
And his know me for a while. I kind of wish we hadn't in the
first SRE book written about them the way we did.
Because I feel that too many people think that by having
labeled them, the golden signals that they're the only things
that matter. And I see a lot of people both
start and stop there. And there's nothing wrong with
measuring availability. There's nothing wrong with
measuring latency. There's nothing wrong with
measuring throughput. That's not my issue.
My issue is that people often start and Stop there they are.
In fact good starting points. Absolutely?
They are. So, if you're asking me, are
they good? Starting points.
Yes. If that's where you need to
start. Start there.
Almost everyone can measure those things.
It's a great place to start but those things rarely tell the
whole story, they rarely tell you what your users are actually
experiencing as return earlier, reliability is what your users
need from you. And we can just tell a quick
little story, right? So, is your service available?
Yes, it's responding to requests.
Is it? Latency low, sure.
It's responding to requests very quickly, is experiencing much
errors. No, it's available, and it's
responding to things very quickly.
And every response is HTTP, 200, everything's great, but if
you're sending them the wrong data, if the data you're sending
them is not what they're asking for, you not being reliable at
all, and that's not covered another golden signals at all.
So I think people can graduate. They can upgrade to a more user
Journey based Focus, which is again.
Is this doing what? Users Is need it to do which
includes many levels Beyond just what the golden signals tell
you. So yeah I have some mixed
feelings. I think they were exid ently
taken too seriously or as too much of an end goal the way they
were written about. But I do think they are good
starting points because almost everyone has that data already.
If you don't, it can be generally speaking.
Pretty easy to instrument. It can be much more difficult to
measure. Did we send the data that the
client asked for that? Can be a lot more.
It took a long time to get there but yeah this is my kind of
feelings. They're a really good SLI is
ever changing its ever adopting because the world is going to
change your user expectations are going to change what your
service is defined as even doing is going to change over time.
So make sure that you're looking at your SLI eyes.
You're looking at those measurements you're saying is
what we're measuring. Is it still telling us enough is
it? So telling us what our users are
experiencing? Is it still looking at things
from their perspective. So start with errors Art with
latency start with availability, but move from there.
Think about what else your users actually need.
You mentioned about checking like the correctness of data,
right? So this is sometimes where I am
confused as well. Personally, if you are always
checking, whether you send the correct data, that he gets like
infinite Loop, how do you check in the first place?
The data is correct. What about the testing phase of
your product? How do you actually cover this
correctness thing? Because do we really need to
measure the correctness of all requests?
Or is it only partially Oh maybe only some service types that
need to have this correctness attribute measured.
So maybe, tell us a little bit more about that.
I think is all the things that uses mentioned.
It depends again on your service, a favor, a mine is
using synthetic checks a good example, is I was once
responsible for a very large-scale log system, hundreds
of thousands of incoming logs every minute or maybe even four
S very very high volume. We wanted to make sure that we
were doing things correctly. What we realize is that if we
could craft a special log with a special tag on it and we insert
it and then we waited found out how long it took for it to go
through the entire pipeline, it get indexed and for us to be
able to retrieve it on the other side and verify that signature
we put in, was there that gave us availability latency ensuring
there. Weren't errors and data
correctness and even data freshness.
But the thing is they told a pretty complete story.
This is what the Users of this log service actually needed to
happen. They needed to insert logs and
then they needed his logs, be indexed, and then they need to
be able to retrieve those logs. So we just wrote a job that
would just continually do this and luckily this logs are some
more decently Estonia to happen a few times per minute.
Even at a few times per minute that gave us more than enough
data to set and a slow on top of.
So we're no longer dealing with the 100,000 requests per second
at the service actually dealt with.
It was only a few a minute now, but we were reasonably sure.
Or at least close enough to being sure that if any step
along the way broke for most people, we would find out
because our special crafted log, which are specially crafted but
otherwise went through the same workflow, the same pipelines.
Is everything else that gives a pretty good signal and that one
measurements told us 45 different things because it
covered five different Services, Glock producing Services, the
Kafka assessments at between the log stash service that pulled
off the Kafka topic. The elasticsearch cluster that
log session inserted the logs into, which were then index by
elasticsearch. And then the check talk to
Cabana that sat in front of elasticsearch to actually
retrieve the log. So all those services and the
availability of all of them and the latency of all of them and
the error rates, all, and we're all being covered by this one
synthetic that just ran over and over and over again.
So that's one thing. You don't always need as much
resolution as you think you do. Because even if we only had a
few of these per minute, As opposed to hundreds of thousands
per second, or we were covering so many components of the
system. He was just as good a data, but
you can also set a solos off of actual 100% user data.
Now, this requires against racing Solutions, this requires
a lot of work we do not have time really to get into all the
technology involved for you to be able to trace from your human
end-user clients or browser across the entire internet
through your Provider and CDN and load balancers into your
app, only back to the databases and whatever resources they need
to talk to an older backups like render time at the client or
whatever. But the fact is I've seen that
and it is absolutely possible and that is a lot of work but
you couldn't fact also your SLI could be set off of actual human
traffic, your actual user request could, in fact be used
for that as well or anything in between.
If there's anything, I want to drive home, it's be thoughtful,
be meaningful. Do what works for you.
You don't over spend resources, trying to do the real user
Journey tracing that's not reasonable.
Farewell, My Rana has the resources or the knowledge or
the time to do that, so maybe set up a synthetic instead or
you know what? Let's take another step back,
maybe for you right now, just latency and error rate.
Just a golden signals, maybe that's just fine.
Thank you for clarifying. That again it's a very
insightful for me. Thanks for explaining that.
So you mentioned a couple of times about User Journey.
Maybe some people also confused about this term.
So what do you mean by user journey?
Is it like the entire experience I want to buy socks?
Or is it like when you check out or is it when you load a page?
There are so many different ways to Define this maybe if you can
help. Also to Define what is a user
Journey, or some people call it critical usage in it sure.
So a critical user journey is generally something that's
defined by the product aspect of your organization telling you
what needs to happen with your product.
For you to be a successful company.
Generally in this case meaning making money, that's the most
General product management product owner definition of what
a user journey is, right? It is the expected way that a
feature will work and a feature, very rarely connects directly to
a single service in terms of a micro service in terms of a
single team owning it. So user Journeys.
Generally, span multiple different components of your
system that generally means Is that there are many different
teams, not just one responsible for all the components that user
Journey travels over, its kind of a high level product manager,
explanation to the user Journey. But I think they're just a good
analog for what a good SLI is a good service level indicator.
Basically is a user Journey, have user Journey basically is a
kpi or key performance indicator, which is even one
more step up, right? This is now the business side.
What does the business say? Your business operational?
It's not your computer operational team, what do they
say? We need to be measuring.
What do they say? Is important for our revenue for
a bottom line. What does the chief Revenue
officer or the CFO? What do they care about?
But they're all similar in the sense that they're all
measurements, likely having to do with your customers or users.
And none of them are ever going to be 100%.
So you can set targets and all of them.
I kind of like to say that an SLI is a user journey is a kpi.
They're all kind of the same thing, just different business
units. Have slightly different ways of
talking about them, I think the most important thing also again
is coming from the product rice, not randomly from some
Engineers, okay? This is our critical user
Journey. You mentioned that a user
Journey could typically involve a number of services, we could
be components database load balancers and all that.
It could also be multiple micro services that span across
multiple teams. One part of a confusion here.
How do you define the SL eyes? Is it per team?
Is it for whole user Journey itself?
How do you advise people to think about this?
I think you need both. I think individual teams need SL
is set on their own Services. They need SOS settle me room
Services, they need to understand how their services
are operating for the things that depend on them again.
Even if those things are only other services as well, but then
perhaps the director of your organization needs to be setting
SL eyes. It's not say, the director has
to be necessarily implementing them, but needs to own SLI is an
SLO. For the kind of user Journey,
stories that go across many different services and perhaps,
the VP of engineering needs to be oming, the concept of like
King you check out. So, the checkout microservice
team has their own SLO that measures, how often their micro
service is not throwing an error.
The Commerce Department, the director of that department owns
an SLO that says does the checkout workflow work, the VP
of engineering owns the slow that Says something like or
users able to use your website and send us money.
Something along those lines. But, yeah, I think these things
build on top of each other. I think every step of that way
needs its own a slow and you can use those individual as close to
inform an overarching. A slow the SLI for an es.
Lo could be the S low status of different slow you know like as
a big has no super hard fast rules here.
It's just are you measuring things?
Are you taking your users into Cal and are you trying to make
sure you're not try to be honest.
You mentioned about different departments.
Owning slos right now let's move to SLO part so you have defined
your SLI so you have maybe a latency error rate availability
and all that and then you set your slos.
First of all how many SLO should a service have?
Is it in hundreds of how many is good enough?
So I think there's some art here.
I think it's mentioned in the SRA book as well in your book as
well. What's the art?
How to define the number of slos for your service to be so boring
through the listeners because it's just, it depends.
Again right, it really is though.
It's solely depend on your service.
Make the right decisions, make sure it is enough that you are
covering all important aspects of your service.
So don't choose to few. Because if it's too few, you
can't understand what's going on.
And also make sure it's not too many because if you have too
many run in the multiple comparison problem, or you have
too much data and you don't know what to look at and you don't
know what's telling you what and you no longer have a good idea
of what these signals are anymore.
I think that Google has sorry. Book said five or six or service
that seems reasonable to me it really does.
Yeah. Sure why not.
But I also wouldn't again. Like so many things.
I don't like hard and fast rules.
I like approaches. I like philosophies.
So don't feel bad if you don't have five, don't feel bad.
If you have many more than five, just make sure it's the right
amount for you. There's also another thing that
is covered in the book, so when you set SLO Target, make sure
you are not setting it too much beyond what you see.
Expect because it could cause a little bit of complications.
Maybe you can explain, first of all, how do we know that we are
exceeding users expectations? Maybe we ask the users, like,
what you mentioned and secondly, then what should we do?
Do we lower the definition, the target itself?
Maybe if you can explain the bit, the problem of being too
reliable here. Yeah, so the main problem you
run into by being to reliables, that people will end up
expecting you to continue to run that way.
So if your users were previously, okay, With you being
only reliable 99% of the time, but then you proceed to spend
like a year, being 99.9%, reliable, their expectations.
May have now changed and now, they're going to hold you to
that 99.9%. Even though they were totally
happy with only 99% before, it's a lot of nights, but I think
everyone hopefully was able to follow what I was saying there.
So you paint yourself into a corner.
If you are two reliable too often because user expectations
will change, what you want to do is you want to make sure that
You aren't being so reliable, you may accidentally do this,
you may just accidentally have a few months where everything is
like super lucky, or maybe your whole team on a vacation.
He's didn't touch anything. So nothing broke for a while.
We can couple the bunch of different examples, but you
know, then you run into the problem of people are now going
to expect this moving forward. My favorite example, this is the
chubby team at Google. This is written about in the
first Google history book. You can read the whole story.
See there, the Quake versions, chubby is a global Lock Service
so it holds tiny bits of data that are useful for various
highly distributed services, to be able to read and understand
at certain points in their operation.
Chubby, just generally Ram pretty well.
When I tell the story and I say that my old chubby SRE friends
often, give me a dirty look, because apparently they're
large, not always that easy, but from a user perspective, Global
chubby, which was the global version of this that globally
available. Rain very well, I believe they
had 49 so a 99.99% Target every quarter.
Generally speaking chubby still had that because it ran pretty
well. So what they would do with the
end of every quarter is they would just shut chubby off.
They would just burn whatever budget they had remaining.
This would be communicated teams emails would be sent.
Alice this Thursday afternoon at 3:00 p.m., we're going to be
shutting chubby off for exactly 2 minutes and 17 seconds because
that's how much error budget we have left.
Even though people were told about this other services would
always crash because someone would have a dependency on
chubby that they didn't know about chubby being good
citizens. Being ran by an excellent SRE.
Team would say, we're going to make sure you find out because
we're only promising you for knots.
If you're expecting anything more than four nines, you're
going to have trouble. So we're going to ensure you
find out if you're gonna have trouble because we're giving you
exactly four nines per quarter, that's kind of a humorous story,
and it's kind of out there and there's not a lot of teams that
will ever get to the point. There's not a lot of
organizations, We'll get to that point, but it's a really good
example of ensuring that your users aren't getting too.
Used to you being too reliable, because once you paint yourself
into that corner, now you might be stuck with being held to a
level of reliability. That's otherwise, too expensive
and too difficult, another probably assumption that people
think of when the service being too, reliable is like Google
search. When an internet is down from
the first thing that they will test is Google down as well.
So I think when Google is Dimas reliable that much And so, I
think people expect that Google is like a benchmark for
internet, even the reliability of Internet.
So you mentioned about error budget.
This chubby story is really interesting.
How they use error budget to actually make sure that the
service doesn't perform to Reliable in the error budget,
sections of the book. One thing I find really
interesting, you mentioned an error budget is actually not a
technical term, right? It's a Communications framework
and this is probably between engineers and business on.
It may be other departments. So tell us more about this
community. Should aspect of error budget?
Yeah. So really what error budgets let
you do? Is they let you tell others,
here's how reliable we have been.
So your s low Target tells others.
He's how reliable we want to be and your error budget.
Let you tell other people. Here's how reliable we've
actually been. And the reason that's such a
good communication tool is, because it helps other people
figure out what their own SLO should be.
And it also helps other people just basically understand what
oldest actually boils down to because we've said the number 9
and On us like probably a hundred times already but what
does 99.9% even mean? Well when you translate it to
meaning 40 minutes per month, well that's something that
humans understand. So when you're able to go to
someone and say, all right, we were unreliable for 17 minutes
last quarter but that means we still had a budget left of 24
minutes. Although unreliability periods
are not always all in a row, they're not always about
downtime than always, but outages.
But the point is it gives you a more In friendly way of
communicating. These kind of historical
reports. What did q1 look like?
What did 2021 look like from our user perspective, from this
hypothetical All-Seeing user who never stopped watching us?
What did it look like in time, for example, it's just an
eventual output from an SLO based approach that helps you,
then go have conversations with people whether it's via the
time-based error, budget, definitions, which agains make
it. Easier for some human heads to
wrap themselves around it or just by being able to say like,
look, we exceeded our error budget every quarter last year.
We believe we're aiming for the right target, but we can't.
So we need more resources or we need more head count or it could
be the opposite. It could be like, hey we've been
exceeded. Our budget, a ton maybe we move
some of the staff over to this other project that's having some
problems or maybe we should be moving quicker.
Maybe we should be shipping features more often.
Because you know what? We're really awesome.
We're almost 100% all the time. Let's spend that budget.
A chip more features. Let's experiment.
Let's try things. Lets you chaos engineering.
There's so many different things that error budget, lady do, but
they almost all involve communicating to other people
saying. Hey, this data is told us this
thing over time. What can we do with that data?
I really like all this concept. They kind of like, built on top
of each other, like you mentioned, reliable this Tech.
So, the error budget, specifically, if you use it,
right? And if the people aligned in the
The organization that error budget is an important concept
for them, you can use this for communicating priorities as
well. Like, you mentioned to be
shipped more features. Should we hire more people?
Should we even fix the reliability, or should we do
experiments? And even like chubby, right?
Should we just spend it because we are doing good.
I think that's really interesting.
So I think many people are interested in this s.res.
Ellos es, el eyes and all that but implementing it.
Like you mentioned in your Squarespace experience is hot.
Maybe tell us a little bit more tips, how should people start
building this as a Culture or even defining how to get the
buy-in from the people within the company.
The best advice I can give or at least my favorite advice because
it's not always possible for everyone.
I want to be very upfront that I understand that not everyone is
in a situation where they can do what I'm about to say but if
possible just get started just do it.
Just pick a service and pick an SLO and measure it and maybe
it's the totally wrong target and maybe your SLI is a bad one.
That's fine. Pick a new one, they're not
agreements. Like their objectives, just get
started with it and start Gathering the data and see what
you can start doing with the data and maybe first.
It's just you and then maybe it's your team, or maybe you can
get your whole team on board right away.
And then, you can start showing the teams that you work closely
with. Hey, look at this.
Cool data were getting. Look at the Greek
decision-making, we've been able to make and how we've been able
to more effectively plan, our Sprint's because of our error
budget data. And then this other teams I call
that seems kind of cool. Maybe we should try that.
I really think as lows are From the bottom up thing, really?
Honestly, I think they're very often organically grown.
I think that people can be sold on them philosophically, but
they don't understand why they need to spend time implementing
them until someone's kind of give them the hard data.
So if possible just start, just go just start doing these
things. See what it gives you.
If it doesn't give you what you want, maybe pick different
targets, maybe pick different measurements or maybe you're not
quite ready for them. That's also totally possible but
every Industry out there already in some way understands failure
happens. It's not just Computer Services.
Embracing failure is always a good thing.
By ensuring, you're not feeling too often.
This is always going to lead eventually to happier engineers
and happier business and happier users and happier customers.
Therefore, so that's my favorite advice.
I don't know if it's the best advice because like I said, I
know some people are not in a situation where they can just go
do it. I'm sympathetic to that but
that's Generally what I tell people if possible just give it
a shot and see what happens. Yeah, sometimes people get stuck
into the tooling. You know, can we get the data?
I know that but I think we can always start simple, right?
Yeah, I've seen plenty of teams get started with s lows by
manually calculating them at the end of each quarter.
Like no joke, they didn't have real-time alerting on the rest
Lowe's or real-time error budget status.
They even have error budgets. They got started by at the end
of every quarter when they were getting ready to plan, What
should our priorities for next quarter?
Be, they would go run some queries from their monitoring
system, do math against it. Put it into a spreadsheet and
then calculate, okay? Last quarter, we were X percent
reliable. What does that mean for our next
quarter? And that's a totally reasonable
way to get started. It's all about just, again
embracing those service truths that we talked about at first,
right, Rye, Billy's, most important thing, your users
Define your reliability, not you.
So make sure you're measuring the right thing and 100% is out
of the question. So pick the right?
You can Embrace those truths without real time monitoring and
advanced statistics and all the stuff that comes along with.
It just get started even if it's in a spreadsheet you know, if
it's only just once a month I like that you men said this even
though you don't have the tooling to start because some
people think after reading the book again, it's philosophical,
right? We are not Google, we don't have
all the tools so we are stopped. So I think that's the key
message here. Just stuck and I think these
three services and likely discuss in the beginning is
really important. Once you get it right, you will
find ways to to actually measure your user happiness.
So Alex, thank you so much for spending your time is like the
crash course of SRE and SLO definitely.
But unfortunately, we need to wrap up pretty soon.
But before I let you go, I normally ask one last question
for all my guests which is to share your tree technical
leadership. Wisdom.
So this is maybe some kind of advice for you to give us.
Also may be based on your career Journey experience or maybe had
lessons, sometimes sure there's three things I have to share, be
kind. It goes a very long way.
You're dealing with other humans, no matter what you
think, your job is no matter what your computer services, it
exists at some level for other humans, whether they are your
customers or your end users or your co-workers or whatever.
But be kind just be nice to each other.
Don't be pompous. Try to always remember that.
Every decision you make impacts other people initiate those
decisions with kindness be thoughtful, I'd said that a lot
tonight but it really is. I think maybe my most meaningful
mantra At this stage in my career is think things over.
Sometimes, you need to react, sometimes you're in an
emergency. Sometimes you deal with an
incidence or immense business pressures, or your company is
under immense Financial strain, I'd been all those places but
always, you always have time to be thoughtful, you always a time
to take at least a few seconds be like, okay.
Is this the right thing I'm about to say is this the right
thing I'm about to do is the correct action?
I'm about to take And then finally, adopt blamelessness,
make sure that your organization is building an appropriate
culture, where we understand that humans don't make mistakes
on purpose, right? That's not the humans all make
mistakes. A course we do.
Of course every single one of us does every single day, but
generally speaking, unless you're a bad actor unless you're
literally trying to bring down the company from the inside,
unless that's the case people aren't doing it on purpose, and
always remember that. So adopt linguist list.
The combined data thoughtfulness combine that with the kindness
and be better to each other, I love the old wisdom because it
all touch the human aspect nothing.
He about technology or slos. Sorry.
So thank you so much for this beautiful message.
So Alex for people who want to follow you, or maybe look for
your product Noble 9 where they can find you online.
So you can find me on Twitter primarily a dog os3.
That's a hid, a lgo SRE on Twitter.
Also, my website, Alex Dash Hidalgo, Dot-com and definitely
go check out Noble mind. I believe anyone can get started
with s ellos, we exist to help you do that.
We exist to help you measure s ellos and calculate your are
budgets. The best possible weight no
matter where your data lives. So come check us out at Noble.
My not calm, that's and obl nine.com.
Thank you so much again. I really enjoy this composition
so it was a pleasure. Thanks Alex, thanks Henry.
I had a blast. Thanks so much for having me.
Thank you. Listening to this episode and
for staying, right? Until the end, if you're highly
enjoyed it, I would appreciate if you share it with your
friends and colleagues who you think would also benefit from
listening to this episode. And if you are new to the
podcast, make sure to subscribe and leave me your valuable
review and feedback. It helps me a lot in order to
grow this podcast better. You can also find the full show
notes of this conversation on the episode page at technology
node, death website, including the full transcript interesting
quotes and links To the resources mention from the
conversation. And lastly, make sure to
subscribe to the show's mailing list on pack leader.
No dot f to get notified for any future episodes.
Stay tuned for the next technology.
No episode. And until then goodbye.