The box is a style of working. It's about creating a
collaborative environment between the development team and
the operations team and aligning goals and incentives between
those two teams. Because when you think about it,
so many of the problems that we encounter in life not just even
in technology is a misalignment of goals.
Hey everyone. My name is Henry Surya be Robin.
And you're listening to the tekhelet Juno, the show will be
bringing you the greatest technical leaders practitioners
and thought leaders in the industry to discuss about their
Journey ideas and practices that we all can learn and apply to
build a highly performing technical team and to make an
impact in your personal work. So let's dive into our Journal.
Hello, everyone. This is Henry. 201.
Welcome to another episode of the technology, you know,
podcast. Thank you for tuning in and
spending your time with me today, listening to this
episode. If you haven't, please follow
technology, you know, on your podcast app and social media on
LinkedIn, Twitter. And Instagram also consider
supporting the show by subscribing as a patron at
technology, you know, dot, f / Patron, and support me to
continue producing, great content every week.
There were UPS as a culture and practice is one of the most
widely talked about in the current technology landscape
according to the state of devops report Elite and high devops
performing. Companies are leading the
industries in terms of organizational performance and
continue to outperform the companies that do not practice
devops optimally. And some of the reasons
companies do not practice devops optimally are due to the
misconception of the whole Dev, Ops concept as a culture and
also some anti patterns. That may get adopted
unconsciously to shed more lights on devops culture and
practices. For today's episode.
I'm happy to share my conversation with Jeffrey Smith.
Jeffrey is the author of a book titled operations and the
patterns, develop Solutions, and he's also the director of
production operations at Centro. In this episode Jeffrey,
describe devops Essentials and importantly a size.
What devops is not? He also broke down and explain
about cams. S-see AMS.
A framework that outlines the core components required for a
successful develops transformation.
We then discuss three anti patterns that are taken from his
book. The paternity syndrome alert
fatigue and wasting a perfectly good incident.
Jeffrey explained, how to recognize those anti patterns
and how we can avoid them on our devops journey.
And finally, Jeffrey, also talked about post-mortem and he
shared some tips on how we can cultivate a Post-mortem culture.
This is such a fun conversation with Jeffrey.
And I really, really enjoyed it. Especially discussing about
those three devops and the patterns that I could personally
relate from my own experience. And I believe that you would
highly enjoy and learn a lot from this episode as well.
And if you do consider helping the show by giving it a rating
and review on your podcast app or share some comments on the
social media channels, those reviews and comments are one of
the best ways to help me get this podcast to reach more.
Listeners. And hopefully, they will also
benefit from all the contents in this podcast.
So let's start our episode right after our short sponsor message.
Are you looking for a new cool swag taglit Journal.
Now offers you some swags that you can purchase online.
These wax are printed on demand based on your preference and
will be delivered safely to you all over the world where
shipping is available. Check out all the cool swag is
available by visiting technology, know the dev slash
shop, and don't forget. The bracket self once you
receive any of those tracks. Hey everyone, welcome back to
another 50 Journal podcast show. Today.
I have with me an author named Jeffrey.
Smith. Jeffrey is an author of the book
called operations and the patterns develop Solutions.
So I guess it's like using develop solutions to overcome
those anti patterns. So today we will be talking a
lot about what are some of the anti patterns, for operations,
people, or in the devops world. We're going to be talking about
Some of the things that maybe you should try to avoid in your
operations team and how we can actually do humor strategies
that actually can improve how you manage your systems, how you
operate your systems? And so that using the devops
solutions we can overcome those things.
So thank you so much for spending your time with me
today. Jeffrey.
Hope you have a good conversation with you today.
Awesome here. Thanks for having me and I'm
looking forward to it. So Jeffrey in the beginning,
maybe if you can introduce yourself telling us more about
your career, any highlights or turning points.
Sure. Yeah, so as you mentioned, my
name is Jeff Smith. I'm currently the director of
production operations on a company called Centro in
Chicago, Illinois here, the United States.
We are A rad Tech platform, which didn't sound.
Like the most exciting thing to me.
When I originally took the job ad Tech is like, the bane of the
internet, but in reality, it is, what fuels the internet and keep
so much of it, free. I'll talk a little bit about
more about that later. But prior to that, I was with a
company called GrubHub, and they're in the food.
Delivery space, my time to grow up, was really when I got first
introduced to devops and do It's Concepts.
It was the first place. I could really sink my teeth
into it and experiment in my career, really started.
Twenty odd years ago back in Upstate New York where I grew
up, I had been working doing data entry at a local health
insurance provider in the area that I'd always been interested
in computers, but I was a terrible high school student
gentleman, who was the manager of operations?
Who would later become my mentor walked by and saw me reading
Richard Stevens TCP IP illustrated book.
I don't know if you know that book, but it's like the
networking Bible, what? Didn't know at the time.
Was that book? Was way over my head.
I was reading it and I was getting some of it, but it was
pretty dense stuff. He saw.
And reached out was like, hey, what do you read in there?
We talked a little bit about it, and we would have conversations
every now and then eventually an operations position to opened up
his team and he said, hey, would you like to switch careers?
Stop doing data entry and do some computer stuff.
So I was like, yeah sure. So that's really what kicked off
this 20-year romance with tech. I'll say so I was at that
company for about 10 years. I worked my way up to operations
manager eventually. Seating him, you know, it's one
of those moments where you're like, wow is every place is
messed up. Is this one?
Maybe I need to Branch out and see what else is out there.
My wife, what? Girlfriend, at the time.
Now, wife said, why don't we open up a search?
Why don't we look anywhere instead of just looking in our
local area? So that's what brought us to
Chicago. I got a job in Chicago switch to
a couple different companies. Had some success doing
performance tuning for a company called Accenture and one of
their like lab environments, but that was right during the start
of the financial downturn. I'd been brought out as a
contract to hire and then as soon as I got brought on the
financial crisis hit so they weren't really hiring and they
kept extending me. Extending the extended me.
And then finally, I said, you know what?
I need health insurance, right? Like I'm walking around here.
I bumped it away from Financial ruin.
So I started looking for a new opportunity in the day.
I got a new job was the day. They told me that they were
going to be able to extend my contract anymore.
And I was like, oh that's funny because I was gonna have the
city meeting with you to tell you.
I was leaving. So everyone was of the but, you
know, work the couple Jobs before getting to the pub.
I think GrubHub was a huge turning point one.
It was the first time I'd work for a company that actually made
a product. I cared about and you never
really think about how important that is.
So often you're just stuck in this field.
If you think about it technologist, a really
mercenaries, they don't really care about the field that
they're in. They're just like what language
you're my coding in the field or whatever.
The company does is secondary to.
That was the first time I worked at a company that actually made
a product that I've cared about the really fueled my interest in
how things truly work, not just from a computer.
Active. But from a business perspective
as well. And that gave me all types of
insights that I was lying to before.
Because, you know, before that, it was like, I don't care about
health insurance, right? Your life, the inner workings of
health insurance. I think care about legal tax
software, which I was responsible for running in a
company called Wolters kluwer. I just didn't care about those
things. So I didn't have a vested
interest in understanding how they work.
Being a grub hub and being curious about how things work.
Open me up to this world of possibility when it comes to
operations, so I carried that experience with the into Centro
and First, I didn't care about how I attack worked.
I knew that it was important and could be career change and to
really understand the business. And I'm glad I did because that
took is way more fascinating as much as it is problematic from
the various different perspectives, but it is
definitely needed in the internet age because people
don't want to pay for Facebook. It's the trade-off.
So how do we do that and do that in a respectful manner for our
customers clients and users. So that's a quick recap of 20
plus years of experience, but it's been a fun ride.
Thanks for sharing. Your story is very interesting.
How you started your journey? So through this book, TCP IP.
I have to agree that. Yeah, that book is also way over
my head. It's a great book.
That is like absolutely Bible. But back then, at that time.
I was like a good prop was a really good problem.
So over these 20 years Journey. I'm sure you have seen a lot of
things right in the beginning. You also said that the messy
things of operations administration's, and now, in
this era of devops. Maybe you can If I first of all
the audience and listeners hear what is actually devops.
So the first thing I'm going to describe is what is not.
It is not a role. It is not a job title.
That's a major pet. Peeve of mine.
I think it might even be a little detrimental in my hiring
at first because people are searching for that.
Devops rolled pops is a style of working.
It's about creating a collaborative environment
between the development team and the operations team and aligning
goals and incentives between those two teams, you know, we
say, Dev and Ops and focus on those team.
So it really can be any grouping of Team.
It could be Devon Finance, option, finance, options,
security, and we keep coming up with all these acronyms, that's
a cop's Finn. It's just like, let's just call
the devops. We understand what the idea is.
I always say you would never post the position for an agile
engineer. That sounds insane.
You would never do that because actually the style of working,
same thing with that mops. How do we build cohesive joint
incentives? For teams to work towards a
common goal and putting that sort of front and center.
Shape a lot of your underlying decisions moving forward because
when you think about it, so many of the problems that we
encounter in life. Not just even in technology is a
misalignment of goals. You set that it's not a role,
right, but I agree from this part of the world.
I keep seeing also devops engineer role.
What do you think should be the name of the title?
Is it like a sorry, as some people actually chose to prefer?
I think that many system Engineers system administrators.
Whatever you want to call. I think there are enough people
that have been in these proles previously better doing the same
things that we're doing today. It's just with a different tool
set, right? When we made the switch from
Unix to Linux. We didn't really have to come up
with a new job description. It was the same deal.
It was just, you know, now there's a new tool when we make
the switch from python to go. We don't come up with a new job.
Title is just an asterisks under the software developer title.
I think devops is the same. I think we have plenty of titles
that could be used. I'm not naive to Could forces
though and I know that if you have a have Ops title, it's
probably an extra 10 or 15 thousand dollars a year.
So I don't begrudge anyone using that.
In fact, I tell my team that I manage if you want to call
yourself a devops engineer on LinkedIn, that's fine.
I'll pack you. But that's not what we're called
here. Because the minute you give
someone a title like that. It becomes their job.
The minute. You have a QA team.
Guess whose job quality is Right?
Language is a very Finicky thing because it does so much to the
way we perceive the world just by giving something a name or
title. That's why I try to avoid it so
that everyone knows is like this devops things were talking
about, is everybody's job and we need to think about that too.
From a times perspective, right? Because we can quickly fall into
a trap where we're like all we're building operations stuff.
Nobody else needs to see this. Well that kind of runs counter
to this collaborative environment that we're talking
about. So maybe we do need to make sure
that anyone and everyone can see that maybe You to make sure
we're keeping the bodies out of the Ops Code so that we don't
have to lock down the repo. Some configuration file that
they're like, we can't share that with anyone because there's
too much dirt in there. There's just way too many dead
bodies. We've got to keep that under
tight lock by putting that mindset forward.
Hopefully, to helps to prevent and eliminate those sorts of
traps. Thanks for giving that
clarification. So, in your view, I think you
mentioned is in the book as well.
There are few things that are very important.
When you want to adopt this devops culture, right?
This new style of working. So it's an acronym.
I'm called camps or some people actually call it calms.
See am SOC a LMS. So maybe you can give us a light
here. What is cams?
So current is really dislike framework around devops as a
concept to think about the core components that you need in
order to make this transformation C is for culture.
The A is for automation. The AL that you alluded to that.
Some people call. It is lean metrics and sharing
or the lasted. So culture is like one soul.
Much of devops is cultural and you need to build a cultural
environment in your organization where these sort of practices
and concepts are embraced, where people are free to experiment,
without fear of Retribution. If they get something wrong,
there are all these small. Insidious things that happen in
our culture. That especially us as engineers
and technologists might gloss over.
Not realizing what an outsized impact that they have.
Have you ever worked for a company where the culture?
ER, is if you don't use your budget by the end of the year,
you lose it, and you won't get it next year.
That's a cultural thing. It doesn't have to be that way.
There is no golden Finance rule. That's just the way people
operate. So what does that do?
That creates a culture? Where everyone is spending
aimlessly? They're spending a lessly at the
end of the year to make sure that they use that budget and
I'm sure the people in finance are like sweet.
That's exactly what we wanted to happen.
But you created that culture through a small rule change.
So culture is just something that we always have to keep
from. It's better when we're talking
about devops automation is really the thing that powers
this. One of the things is you're
always going to be asked to do more with less.
But the other thing that we need to do is the more automation
that you have. The more empowerment.
You can give to other people in the organization.
It's this thing that in the book.
I talk about this idea of exporting expertise.
So it is a technical dance to fail over a database.
I don't know if you've ever had to fail over production
database, but like, everyone's been in that company where it's
all, we got to feel the database over, get bob bobs.
The only one that can feel the database server.
The minute, Bob turns it into a script where someone just has to
execute fail underscore database.
Bob has transferred, a large chunk of his expertise into this
automation script and now he's empowered.
Tens of people that can do this thing.
So automation is key. I'll talk about lean briefly,
but I'm not a huge fan of adding the elk.
Lean is just about operating dynamically and quickly.
I don't know that it falls in the category of devops because
typically, you're adopting a All that is similar to the
organization. So like if you're forced to
adopt lean but during a waterfall shop that can be
problematic metrics, devops should be rooted in data as part
of that empowerment. People need to know if the thing
that they're doing is actually having an impact.
We always talk about it with systems and computers and
whatnot. But really extends everywhere.
Right? As we are running, our ticket
queue is actually performing the way we want to perform and what
metrics are? We judging that success or
failure by? We should be able to objectively
point to success. Or failure and we do that
through metrics and then sharing.
You know, how do we again back to that expertise exporting?
How do we share knowledge? How do we share access when
appropriate? How do we share responsibility?
How do we make sure that we're all sort of on the hook for the
same thing and we're all contributing to make that thing
better. Another thing that's sort of
front and center. So these things sort of collapse
together to build this framework and how we should think about
approaching these devops Transformations.
And when you look at, Nations that aren't doing well in their
transformation. You can usually find behaviors
or actions that tie to one of these five categories.
I keep leaving out lean. I'm sorry for all the lien fans.
I'm not, hey, let me pay you in a little bit.
So thanks for sharing. I really like the concept
exporting the experts. So these things about automation
sharing your knowledge. I think it's a great thing
especially in this devops culture.
Let's go into the anti patterns that you covered a lot in the
books. I think they are probably 10.
No, 12 kind of anti patterns so we can start with the favorite
one, probably. But in the beginning, let's
probably dive into why you wrote this book.
Why are you covering anti-patterns?
Sure. So it's a funny story.
The whole journey book was actually a long winding road.
So Manning had reached out to me years ago to write a book on
puppet. I was pretty active in the
public community at that time. At least from a user
perspective. You know, I thought about it was
like, yeah, sure. But then before we got things
started, my first kid was born. So wasn't the ideal time to
write a book. So then years later they reached
out and they were like, hey, curious, if you'd be interested
in writing a book on devops and I was like, I don't know, what a
book on devops would be like, but then when I thought about
it, I was like, you know what, I want a book that is practical
advice for people that aren't in glossy startups.
That take so much for granted when you read a lot of these
books either their companies that are firmly rooted in
technology. And the fact that everyone is
sort of bought in is a Ian and that's not the reality that most
people face, most people are in a company that's got a lot of
Legacy baggage. They've been around for 40, 50
years. They've got all this inertia
around bad practices that we don't really talk about in the
devops books. We just simply say stop doing
what you're doing and start doing this and it's like, okay,
easier said than done. The other thing was a lot of the
books seem like they were written for ctOS, right?
Because you would need either a CTO or CT 0 by in to be able to
do a lot of the things that they're talking about and I was
like, well, there's got to be stuff that and Visual
contributor or line manager or something can do because I feel
like I've done it a couple times and I'm not a CTO.
So that was really what sort of got me interested in writing
this book and it was originally titled devops for the rest of
us, but people felt that was an exclusionary Us Versus Them
thing and I hadn't thought about it from that perspective and I
was like, yeah kind of hard to talk about a book about, you
know, bringing everyone together and then immediately setting up
as an US versus them. So the first draft of the book
was not an anti patterns format. Actually.
This is my first first book, I was very academic and formal and
I got a piece of feedback in our review process.
So what happens is with Manning, you'll do a third of the book
and then you release it to a bunch of potential buyers that
will review it and give you feedback.
So we were doing that. With the first third of the
book, someone made a comment and they said I've seen Jeff speak
and I was really excited to see this book because he's such a
fun energetic speaker and that personality does not translate
into this book at all. And I was like, wow, okay, so
All right, and my editor was like, I don't know you well
because I'd never really met miters who's like, but it sounds
like this book, is you right now, so that's how we reworked
it into this anti-patterns thing.
So I could have a little fun with it, get more of my own
voice into it. That's how it everything sort of
took shape, but it was definitely not intended to be an
anti patterns book when it started.
That was the result of a pretty cold but honest piece of
feedback that I got. So I hope when we go through all
these and Tibetans, you can also use some of your font styles.
Going forward for that. So let's start with the first
anti patterns, which is called paternalist syndrome.
What is this actually, so I called paternalist syndrome.
It is when someone in a relationship assumes the role of
the parent, even though no true hierarchy exists.
And you know who I'm talking about, Ops people, raise your
hand. We're all guilty of this the
bof, for those that don't know. Go ahead and Google it, but
there's a reputation of operation seems being the team
of know and I Instead, I spent my entire career not hop.
So I totally have all of that, baggage from years of devs, just
throwing stuff into production, but I quickly was able to
identify that the reason again, was the fact that we had
misaligned incentives. So the paternalist syndrome, is
this behavior that operations people, anyone can do, if I'm
really talking to the office looks right now, this Behavior
where we assume everyone else is out to destroy the system, and
we have to protect it. That means nothing can go to
production. Action without it, coming
through us first. That means no one can have
access to production. No matter how scene of a request
it is you just can't do it because, you know, we can list a
whole bunch of reasons and will invoke security as like a red
herring as to why you can't have access.
Well invoke audit controls as a red herring as to why you can't
have access and sometimes those are true, but we're really
starting from the position of no and defending that instead of
starting from the position of. Yes, but yes, but how do we
address? The audit.
Yes, but how do we ensure that access doesn't spread?
And it's really a mindset. So what the paternalist syndrome
will do, is anytime that there's a problem.
The first tool we reach in the toolbox, is another gate,
another checkpoint. Oh, this didn't go through
change control or this went through change control, but Ops
wasn't on the Change Control process.
So now Ops has to approve every change in.
Every time we do that. We're adding another layer of
slowdowns, very little value. Add work.
And waste of time for Ops folks to there was a book that I read
rework by chasing freed from 37signals.
And it's one of my favorite quotes.
I use it all the time. Policies are organizational scar
tissue and every time that there's a policy that's enacted.
You could typically go back to some inciting incident where
that happened and the policy came out as a result of that,
almost to the point where it becomes lore and Legend, where
no one was even there for that anymore.
Right? It's like, oh, yeah.
There's some dude on life support in the lunchroom.
We keep him around just so he can explain to us.
Why we don't do stored procedures anymore.
There's always some story like that.
So the pattern listener on chapter talks about how we can
identify some of the common issues that we run into and
started the pattern of fear, honestly, because that's what it
is. Fear around particular
situations and how we can go about breaking that down using
cams. How do we do that through
automation? So that we can Empower someone
perfect example as we don't let developers restart background.
Select Services, right? So why not?
They know way better than I do. How the system is going to
behave because they wrote it. So why wouldn't they be able to
reset it if they're seeing something weird?
Maybe we don't want to give them SSH access to production.
That's fair. But there's a lot of different
ways that we can restart a service and then we can expose
those ways to the developer and Empower them to be able to do it
all but then we don't know. Okay, and your program can send
an email, right? That's not crazy.
So if you really want to know, if you're not approving and you
just want to be notified. Have your script send an email
saying, hey so-and-so just restarted the service through
this access point. Boom, done.
And the other thing with the paternal syndrome list, we often
insert ourselves as Gatekeepers when we're adding zero value to
the process. So I'll give you an example.
A real world example, from Central when I started at
Centro. There were a lot of requests for
ad hoc script execution from development.
Hey, I need this Ruby script run so we would have to get the
script copy it out to one of the boxes.
SSH into the box, run the script, and no commit mode share
that with the developer or he would look at it or she would
look at it and say, okay, it looks good and commit mode.
We'd run in commit mode and then I'd send them the output.
What value do we add to this process?
Other than being a middleman? What value are we addict?
And the truth is 0? Because even if we wanted to be
part of the approval process, I don't know anything about the
script. I didn't write any of this code.
Okay, you're updating flights to make sure the campaign IDs
match. I did.
You know what? That means?
You just said a bunch of words that I'm going to take at face
value that. This is important.
So, we said, what if we were to set this up via gyro, maybe we
could create a change process where someone attaches a script
to the jira ticket. The Jura ticket, has to be
approved by another Dev who is way more qualified to approve it
than I am. And once it approved, we have
some automation crab the script from the cheer, a ticket and
executed on the box and then attach the output to the ticket.
Now, you don't even need us now. There's not a Dev sitting around
waiting like, My goodness is Ops back from lunch yet.
I really need to run this script but no one's there.
They can just do that on their own.
They self-police it, but he self-govern it.
Now everything is not without trade-offs.
So now you have the issue of like, oh, we don't have to fix
this problem because any time it comes up with just run the
script and because there's no friction anymore.
A permanent fix is probably not as attractive to them.
Whereas before they had the pain of going through Ops to be able
to entice them to do that. But still all in all it's a
boom. It's a win with some
standardized scripts with they read and able to push that
functionality down to customer service.
So, the customer service is 0. When you see this problem, you
have to submit this gyro ticket and then once it's approved,
you'll be able to execute this script to clean it up.
So the paternalist Ingram is really about changing your
mindset getting out of the habit of just being no and shifting to
a guess. But yes, but how do we solve for
these problems? Because no one wants to feel
like they don't take their job. Seriously.
Everyone at work is doing their best with the I was at, they
have to do their job and this idea that there's a group that
instantly assumes you're an idiot is not a warm fuzzy
feeling for anyone. So, yeah, I assume mentioned all
these stories like operations, people assuming the role of a
parent like it's what they always say.
So the Ops job is to actually make the system safe, mix the
system secure, stable. Whatever that is while the other
parts of the company or the other parts of the team, like
death is actually out there to introduce changes in stability,
like what you mentioned. It is like they are there to
destroy the company. So I think when you mention all
this anti-patterns, it's resonate a lot with the
traditional world. Yeah, there's something else
that when you said make the system save, you know, that's an
interesting way to phrase it because I completely agree.
Think about the system that we're running this on as like a
huge tool box. Basically, what we're saying is
the only tool we have is a bunch of sharp knives.
We don't want to give you access to it because we're afraid
you're going to cut yourself and it's like, well, okay, maybe we
could throw some different tools and right.
And if you get through our Marin maybe we could throw a spoon,
something. That's a little safer that still
allows me to do the job. But yes, if you're asking me to
hammer a nail with a machete, this probably unsafe and that's
essentially what we're doing. Like, wow, because we've only
got machetes. I can't give anyone access.
So, yeah, for all the listeners, who listen to this.
I hope you notice this patterns or empty patents in your team.
Make sure that you don't assume this parental role
unnecessarily. So let's move on to maybe the
next anti pattern which is quite Amin for any administrators
operations people, which is alert fatigue.
So many times there are so many alerts are popping up.
Probably, we don't do actions on most of them.
Can you explain a little bit more about alert?
Fatigue? Solar fatigue is actually a term
borrowed from the medical industry.
It came about from nurses who would not respond to beeping
alarms and hospitals, because the alarms always go off, they
always go off. So, they became desensitized to
it to the point where even when there was Emergency, there was
no way to elevate that emergency beep beyond the cacophony of
sounds that was always going off from these machines.
So we borrow that term alert fatigue and Technology to say
like, what alert do we have that are just constantly firing that
are drowning out more, critical useful actionable alerts.
I think the key word there is actionable, when we design
alarms. We design alarms from the
perspective of Things we think might be bad.
We said, oh well alert on high CPU utilization because that
sounds bad, but it isn't really bad.
We buy these machines to use them.
So this idea that we're worried that a machine is certainly
forty or fifty percent utilized. That's not really a bad thing.
Let's say it is 90% utilize do we care?
What are the other factors CPU utilization on its own is not a
reason for concern, at least not to wake someone up.
So I think we've all gotten those.
Wordsworth like database utilization is high and it's
3:00 a.m. We're running all of our batch
processing. Like that makes sense to me.
But why am I being woken up? And then the other thing to
think about is if you can't design an alert that leads up
engineer to take a next step or action, you need to seriously
question the value of that alert.
So when we have an alert that says replication, lag time is
high. You should be able to say in
that alert with this is actual alert that we have.
The alert will say replication slot, 1 has exceeded
replication. Time chances are, this is
related to the database replication service, being run
by the Bia team. You should investigate that
database and see if that's the cause of the replication lag.
If it is restart the division connector in order to catch it
up. That's a very specific set of
actions where I don't have to do a bunch of things if that's not
the case, if that's not what's going on.
And it's like, well, clearly When we wrote this alert, we had
a very specific set of scenarios that we were worried about and
it's outside the boundary of that.
Maybe I should look deeper into this.
So this idea of alert fatigue is this idea of making sure that
these alerts are actionable and if they're not actionable rid of
it just get rid of it because it's not helpful.
Now, I'm assuming you've been on call before and I'm assuming not
casting any blame on anyone but I would imagine that you've
probably received an alert that you said.
Oh this alert usually clear. Has itself, let me snooze it for
15 minutes. Everyone's done it.
So the question is why not just
increase the threshold of the alert and people like, oh, well
then if there's something wrong, I won't know for an extra 15
minutes. Well, you don't know anyways,
because your first action is always the student for 15
minutes. You don't know?
Anyways, because that's the very first thing you do is like, oh
man, that's stupid memory thing. We always know that when the
code gets into this particular section against memory high and
it clears itself, so it's like Not really doing yourself any
service by having it. So push it 15 minutes when it
alerts you need to react quickly because you're already behind
the eight ball. But guess what?
You're reacting and you know, it's real.
I would rather be a few minutes late to an alert.
But no, it's real then to be constantly alerted and not being
sure if it's accurate or not and having to figure that out and
decide because I know as human beings, we're going to err on
the side of the pattern and just say, I'm going to do this.
If you're at a barbecue, you're having a friend.
You're out with your friends, having a cookout, you're eating
the sun's out. It's a beautiful day, your alarm
goes off, and it's like all except memory alert again.
Never ever. Ever, ever again.
Be like, guys. I gotta go this thing that
alerts every 20 days or whatever is alerting again, and I got to
look at it. No, you're going to snooze it.
You're going to continue chatting with your friends.
And then you crap your pants when you realize it's a real
alert and you've got to do something.
So alert fatigue has really focused on identifying those
patterns and trying to make them better.
Or just simply eliminating them. I think another big thing that
the chapter talks about it as well, is creating metrics that
reflect a business impact going back to the CPU.
Utilization example, if the database is at 90% CPU
utilization, but our transactions per second is
steady and isn't climbing, do I care?
I don't care work it. Yeah, sure. 90% utilization now,
that's not to say that you don't want the metric, right?
You just don't want the alert because the metric is good for
trending capacity planning, all of these.
Great things. I'm just saying I don't need to
know about a capacity planning alert at night at 3:30 in the
morning. That can be an email.
And that's another thing too. Right?
We always default to waking someone up as the default alert.
There can be different types of alert, have a low priority alert
that emails you so you wake up in the morning.
And you say, oh wow. We were at high CPU utilization
last night. Nothing else was impacted, but
that's a good data point to know.
And I'm much more receptive to that data point this morning at
9:00 a.m. Well, I've got my coffee as
opposed to three in the morning when I don't know what I'm
really looking. We're trying to solve so get rid
of alerts that don't mean anything to you tweak your alert
notification settings so that you can do emails instead of
always paging out, try to tie your alerts to some sort of
business impact. So that you know, whether you
really need to wake someone up or not or if they'll or that's
firing is something that you actually care about, because
again, I like my databases busy as long as they are within their
operating thresholds. So you mentioned something that
I pick interest which is it's okay to actually be alerted
late. Inside of always have fin first
minute, right? You got others popping up here
and there because some alerts do actually recover by itself.
Because of this anti patterns for sure, people just put in
alerts, but actually they could recover over time in a short
period of time. But what you're saying here is
that it's okay to actually be alerted late.
As long as you can guarantee the actually it's a real problem and
you are supposed to take an action on it.
So I think it suggests that I think for everyone here who has
been in operation or still working in operations.
You should probably tweak your alerts in order to Behave much
more properly. Yeah, absolutely.
Absolutely, because very suddenly, those extra minutes
actually mean anything, right? And it's one of those
counterfactual is that we talked about in these incident reviews
where it's like, oh man, if we had known about that Alert, five
minutes earlier, we could have prevented the outage probably
not. You have no idea how quickly 5
minutes goes by. You're looking at something,
you're tracking down a red herring.
You're like, oh, yeah. It's probably this thing over
here when it's something completely unrelated.
A lot of times those five minutes are find you as much as
you think. They are.
Of course, your mileage. May.
Very take that with a grain of salt.
If you're in high frequency trading maybe it's a different
ball game, but for the most of us, but I target audience.
It's like, yeah, you'll be all right.
So let's move on to the next anti pattern which is about
wasting a perfectly good incident.
This is interesting because a perfectly good incident.
Maybe he can you explain about this.
Yeah, so actually stems from saying from local politician
here in Chicago that always would say never let a good
crisis go to waste it's the This idea that there is so much to
learn from an incident because when you think about it, so we
have these mental models of our systems, right?
These systems are becoming so complicated.
So complex, so many different pieces.
So everyone in their head has this mental model of how they
think the system works. Then there's the reality of how
it actually works. And those two are seldom in full
alignment because usually something off.
And the Delta, the time that you find out that your model is
different than reality. There's an incident.
Other than that, you can spend the rest of your life and
complete the illusion thinking, you understand how the system
works, but then when there's an incident, suddenly the gap
between your understanding and reality is exposed in its raw
form. So often we sort of just close
it, incident ticket and move on, but it's like hang on.
Let's dig into this incident and try to make our mental models
better and learn from our mistakes.
It goes beyond just the tech side.
It's also the Human Side so simple things that you've
uncover like, okay. All right Henry, I noticed that
this alert fired. You got it in, snoozed it and
then it be alerted, 15 minutes later.
And then you engage what happened there?
Oh, well this system alerts all the time and it typically Auto
recovers. So when I got the alert, I
snoozed it. Thinking it was going to recover
but then when it didn't recover, I realize, okay, something's
really wrong. So, as a manager, be personally,
right? If I'm not part of the on call.
Patient. I may not know that reality that
Dynamic exist. So just by simply asking that
question in the incident review, reveal something to me.
Like, whoa. Okay, we've got alerts that are
so bad people ignore them because they're so common and
I'm sure everyone on your team is going to be backing you up.
Yeah. Yeah.
I know that alert. I hate that alerts.
So now it's like, okay, so clearly I have poor learning
that poor alerting is impacting by on-call team, because every
time you're waking someone up, anytime you paid someone, you're
interrupting their Life, they could be at dinner.
They could be in a movie. They could be taking care of
their sick, mother. You have to think when we page
out. What is this person doing?
In their life that I'm interrupting?
And is this worth it as a manager.
If I'm not part of the on-call rotation, right.
That is information that I can get on the incident review
process. Okay, cool.
Here's another example, a real world example, where two
Engineers were talking about the same system using different
terminology and Didn't realize that they were talking about the
same system because one team use this term a and another team
uses term be. So my Ops guy was thinking that
there's some new system that he doesn't know anything about
that. He's tracking down and he's
pissed because there's no monitoring.
There's no metrics around it. But lo and behold.
Oh, no, we have a terminology difference.
What you're calling sidekick. We're calling consumer Damon and
they're the actual thing. That's a huge disconnect.
But now that he understands we're talking about consumer
David. He instantly has a different
view of the entire scenario because he's like, oh now the
know we're talking about consumer Damon.
I understand that these aren't technical problem.
These for human meat, space problems that are coming out of
the incident review process, but once you start to dig in and
peel back and get into people's head space, it's like, oh, okay.
Alright, this is making sense. So I noticed Henry after that
was all over you restarted. The service.
What made you think to restart the service?
What information LED you to that?
Well, honestly, I was out of ideas and I thought maybe I
noticed the memory utilization was high.
So what made you look at the memory utilization?
Well, I normally don't look at that, but I happened to be
looking at a different screen and saw that it was high.
So, I just said, well, why not restart the service?
Okay, but when you restarted the service, you didn't realize
billing was running and that interrupted the billing process.
So, now, we're not getting bills out on time, which was an
ancillary effect. Oh, I didn't realize the
service, communicated with them. Billing process.
Oh, yeah. It's part of the billing
process, because they share a key or something like that.
Oh, well, it's not like I intended to impact filling.
I just had no idea. All of these sorts of
conversations happen because again, everyone has a slightly
different, mental model of the system.
So, wasting perfectly good incident.
Is this idea that you just say? Oh, yeah, system was low on
memory Henry, restart that service recovered service.
Didn't just recover. You impacted the billing team
who now has to rerun the billing shop you're communicating with.
Engineer didn't realize you guys were using the different
terminology had you known that. He might have made different
choices. If you knew that we were talking
about this particular service. We've discovered that, oh, poor
alerting and because of that people are hesitant to do things
right away. They're waiting until it really
hurts. There's all of this information
that could have been easily dismissed and wrapped up at a
memory utilization with high restarted Service close ticket.
So wasting a perfectly. Good instance, really?
This idea of like there's so much more to be done.
Information in a failure that we can bring out.
If we just really want to put some energy towards thank you,
explain about all these different scenarios based on
anecdotes and all that. I realized this is like what
some people call it post, mortem activity, but one thing that I
do find a challenge sometimes like for those people who are
involved in the crisis of the they solve the problem, of
course, they're like, okay. I don't want to deal with it
anymore. That culture of assessing this
incident assessing. What can we learn from it?
I think it's not there for Those are the people, I would say.
So how do you actually inculcate this culture?
So that it becomes a thing. It becomes a common thing that
people actually want to do it because they feel value out of
it. So the first thing that you can
do as an individual contributor, I'm assuming you have the
willpower. So I'm going to say an
individual contributor in this scenario.
The first thing you need to do is, do it fast like immediately
after the incident within 24 hours.
Why, because things have a sense of permanence in our minds, but
only for a short period of time. Time.
So it's like something permanent, but only, for a short
period of time. That's how it works within that
24 hour period. This is the most important thing
that has ever happened. But after that period, it
becomes just noise in the background while you've got all
of these other demands on you. So if you can get people in the
room as soon as possible to talk about it, the actual incident is
fresher in their mind, so they can recall more Vivid details
are much more energized around it.
You're really manipulating their psyche.
Honestly, by doing it early to say like, hey, let's do it now
while you're super interested but our dealer never let you
lead the dealership. He doesn't want you to go home
and think about it. He wants you to remember what
that SUV felt like moment you're in the store so he can sell it
to you. It's a basic human emotion.
So the biggest piece of advice I can give is like I said schedule
it as soon as possible definitely within 24 hours, but
once you do a few of these, I guarantee you even without any
real training in the process. You're going to discover things
and as you discover them people You will instantly see the value
in it. The hard part, is the follow-up
action items? Because how do you influence
people who schedule and prioritization?
You have no control over to make sure that some of these things
that were brought up, are addressed.
Sometimes, it's worth doing it, just to have the knowledge just
to be able to see like what we know these things now and even
though we're not going to correct it, the next incident,
we're aware of it, but for the things that you need to actually
fix that, where you really need some leadership.
And could be up to like a, these things were things, we discover
in the incident post-mortem process.
We really are going to need help and commitment for your team to
address some of these putting dollar signs next to the actual
potential risk help sometimes. Yeah, we lost seventy five
thousand dollars for a 15-minute outage.
Sometimes that can be persuasive, not all the time
because again, it's not coming out of anyone's paycheck.
So by Design, the only people that really care about that it's
management. But yeah 24 hours proving the
use out of it and Documenting that use reading that
information is part of the sharing portion.
Have it somewhere where people can see and review it and
understand it. The other thing that really
makes it shine as when someone is looking at an incident and
they see this beautiful post-mortem incident review, and
then they go to another incident and it's just like restarted
service. We're good.
It's like, whoa. Whoa, where's this?
That sort of creates this pattern of like, hey Henry, I
noticed the last incident. You ran you didn't do a
post-mortem, like the other pin members of the team you should
be doing. To and it just becomes a
cultural thing. It just becomes this thing where
like everyone rallies around it, but it takes time but you know,
lead by example, do it yourself and when you do it, don't treat
it. Like it's some new fandangled
thing to it as if it's the most logical thing that you should be
doing. Yes, of course.
We're going to do a post-mortem. Why wouldn't we?
We just had an incident to hide the fact that this is the first
post mortem you've ever done before in your life.
Just tell it like it's the most natural Next Step.
Yeah. We're going to post more.
We got understand what happened. Don't you think we need to
understand what happened? Say no to that.
That yeah, I guess we should probably have a better
understanding. Yeah, so that's why we're going
to the post-mortem that's enough.
Come on. So as you mentioned the
beginning, right? Probably all these also is part
of culture. So you can't just switch
everybody in a second like once an incident happened and yeah,
everybody will just do post-mortem for the next few
incidents. So sometimes I think we need the
buy-in and also like I would say, probably policing what you
said is like leading by example, right?
So the people at the top, probably also needs to spend the
time or actually allocate the time to actually do this stuff.
Because it's important and High value for the company, not just
for that particular team. And like, with any cultural
change, you got three categories people, right?
You've got supporters, detractors and fence sitters.
The majority of people are fencers the vast majority of
people and this is true of any context you look at politics.
Most people aren't on the extreme ends of either side.
Most people are in the middle, but the extreme ends of the loud
one. So, those are the ones that we
focus on, it doesn't take a lot of people to change the culture
of an organization. It takes a few.
Supporters or even a few detractors, it works both ways.
So positive influence. It only takes a handful of
people to be able to change to have the fence sitters move
over. Same thing with the - though
with the negative behaviors, only takes a handful of people
to be like wall and also how we're going to do things and
then suddenly your cultures in the tank.
So think about that as you're recruiting people for these
different aspects, right? Who are the people that can
really be boosters for me? And I don't have to convert
everyone. I've got to convert a few really
boisterous. Cheerleaders for this, and if I
convert them, people are going to follow.
And once people start following, you have overwhelmed, the
detractors and they lose. So as I hear your insights about
these and the patterns the audience here, actually can find
more Independence in the book. They are some of the topics that
I think like really relevant, but because of the time I'm sure
we cannot cover all of them. So for people who are
interested, go by the book, or read the book, and you can learn
all the fun styles and Tibetans from Jeffrey.
Yeah, and you should buy the book.
I don't want to go over all 12. So Jeffrey, Before I Let You Go,
normally I have this one last question that I always ask all
the guests which is called the tree technical leadership
wisdom. So this is just for people to
maybe learn from your journey. So what kind of wisdom that you
have in your career that probably you want to share with
everyone. Okay.
The first one is, of course, never let perfect be the enemy
of good or good enough. So there are a solution that is
70% effective is better than A perfect solution that has been
implemented. So, never let the fact that it's
not perfect stop you, because guess what?
It's never perfect. It's all perfect.
In your mind. Keep pushing get it done.
You're going to learn a bunch of stuff.
It's not going to be perfect. Something's going to be screwed
up. Second piece of advice is
belong. Same themes of this whole
perfect. Is the enemy of good discussion
is that there is no point in a project Journey that, you know,
less about the requirements that in the beginning, when you start
that is the most minimal amount of information.
You're actually going to have about your requirements.
So keep that in mind when you're designing a solution for
something because as you move in the project, your requirements
are going to become more and more concrete and understanding
product people and project. People are going to come to you
with all this list of requirements and it's going to
look like they did a bunch of due diligence but know that they
only know so much at this point and cut him some slack as a
result of that. So we have a problem now at
work. We're Ops gets involved too late
in the life cycle of a project but a lot of it is because when
they start they don't know if they're going to need op support
or not, because it's like, well, if it's a feature in the
monolith, we don't need Ops for anything.
We can develop that on our own, they've given us all the
automation tools. We need, we can, you know, do
everything we need to do without the but then if halfway through
the project they pivot and they say, oh this needs to be a
separate micro service will suddenly that's a whole new ball
game for Rob's, but we have to accept the fact that They did
not know that when they started, they made the best choice that
they can make. So, always keep that in mind.
I guess, the third thing is, as an engineer.
You have an implicit bias and just about everything you do and
it's particularly bad in technology.
There is a strong sense of this. I didn't write it.
Therefore. It's crap.
Good engineer knows the difference between a preference
and a problem. It's a pattern that I see all
the time. An engineer comes in.
They get hired and ready to get started.
They look at some code or something in their life.
This is all wrong. Can't do anything.
It's funny because this thing's making this like 140 million a
year. So tell me what is so broken
about it. Is it perfect?
No, of course, it's not perfect. There's a bunch of problems with
it. See my previous two ideas and
suggestions. You have to understand what to
preference and what's a problem. So that you know, where to focus
and put your energy because no one makes a We're out of
rewriting. Something that was already
working and just introducing different problems because you
never actually fix it. Right.
My shifted, you might change the sort of problems, but there's
always some new problem that you're going to be dealing with.
So accept that and know that everything you do is going to be
future. Use thing that they hate, you're
going to make a choice and someone's going to come in 5
years later the who's dumb. Why don't you write it and go?
Why didn't you write insert new language?
That's hip and trendy and I'm going to add a fourth one to
when you make it to try. Choice, document, your
constraints and your context when you make any sort of
technical decision. Why did you make that decision?
What was the reality on the ground?
I remember a story where a guy was telling me about all of this
handcrafted code, that was built at his company and his Chi,
don't understand why it is still in use kubernetes to
orchestrate. All this, it was, like, why did
they write the stuff? He's like, oh, those probably
eight ten years ago. That might be why they didn't
use kubernetes right as I can. Yeah, I thought about that yet.
So he got 10 years of energy built into this thing, just
switching the kubernetes like that isn't an easy thing.
So there's always context around every technical decision that
gets paid. If you can document that it'll
save you some Hassle and future Engineers, some energy around
understanding why particular decisions were made.
Thanks for sharing these visit them.
So I'm laughing, as you said all this because I can see these
patterns over and over again, in any places that I went into.
So this is just by Common, so, Thanks, Jeffrey for your time.
So for people who wants to learn more about you or connect with
you or find the books. Where can they find you?
Sure? Yeah, so I have a website that I
don't really update or maintain. But if you feel nursing you can
check that out at a noble devops.com.
Most likely the best place to find me.
Is that Twitter where I'm at dark and nerdy you can find the
book operations into patterns develop solutions that
Manning.com if you want to order direct from them both physical
and ebook copies, but it's also available on the Amazon book.
Story inaudible I do. Read it though.
Unfortunately, everyone's like, I bought this thing and you were
going to read it. Like I didn't even know they
were turning it into an audio book.
I got the email, the same time you guys did.
So I'm going to make a Lobby for the second edition that I read
it though. So we'll see how that goes.
Yeah. I mean like when you read it,
probably you can impose this fun style.
So for people who are listening so that can also be entertaining
at the same time, which I think I'm doing it as you speak just
now. So thanks again.
Jeffrey for your time. I really learned a lot from this
conversation and I wish you good luck for the things that You do.
All right. Thanks for having me.
I had a really good time. Thank you for listening to this
episode and for staying right till the end.
If you highly enjoyed, please share it with your friends and
colleagues who you think would also benefit from listening to
this episode. And if you're new to the
podcast, make sure to subscribe and leave me your valuable
review and feedback. It really, really helps me a lot
in order to grow these podcasts better.
You can also find the full show notes of this conversation on
the episode page at technology. No, the death website.
Including the full transcript interesting quotes, and links to
the resources and mentions from the conversation.
And lastly make sure to subscribe to the show's mailing
list on technology. No, the deaf to get notified for
any future episodes. Stay tuned for the next
technique Journal episode. And until then.
Goodbye.