Guess engineering is basically this discipline of experimenting
on the system and the system can be anything.
It doesn't have to be massive or Netflix scale in order to
increase your confidence that system will survive difficult
conditions. So the experiments real goal is
to either confirm that your assumptions about a system are
correct or you find a problem or you find place where your
assumptions and the reality don't, Not necessarily add up.
Hey everyone. My name is Henry Surya be Robin.
And you're listening to the tekhelet Juno, the show will be
bringing you the greatest technical leaders practitioners
and thought leaders in the industry to discuss about their
Journey ideas and practices that we all can learn and apply to
build a highly performing technical team and to make an
impact in your personal work. So let's dive into our Journal.
Hello everyone. I'm so happy to be back here
again with another new episode of the package on our podcast.
Thanks for tuning in, and spending your time with me
today, listening to this episode.
If you haven't, please subscribe to Tech.
Did you know on your favorite podcast apps?
And also follow technology on our social media channels on
LinkedIn, Twitter and Instagram. And you can also make some
contribution to the show and support the creation of this
podcast by subscribing, as a patron.
At technology. Know that death /.
Patron and help me towards producing great content every
week for today's episode. I am happy to share my
conversation with Michael. I Polly Kowski.
Mikolai is an engineering lead at Bloomberg.
And the author of chaos engineering side reliability
through controlled disruption. I'm sure that many of you would
have heard about chaos, engineering before popularized
by Netflix along with its tools such as chaos monkey and Simian
Army. I'm Curious about cows
engineering and how to implement and execute it properly, in
order to continuously improve systems.
Reliability in the midst of disruption and disaster in this
episode. Mikoshi 8 in depth about what
Kelsey engineering is. And importantly, what cares?
Engineering is not by clarifying.
Some of the common misconceptions surrounding it
Miko, explain the prerequisites and steps required in order for
us to start doing Chaos engineering, and also mention
some of the chaos engineering tools that we can use at
different layers of our system. The skill set required of a
chaos engineer and how we should explain the rationale and
motivation behind chaos, engineering to get the
management buy-in. And using a fun analogy by
involving hamburger and sharp towards the end Miko.
So shared about Kelsey engineering for people and
interesting exert, taken from his book and his ultimate
Mission over the last few years to make chaos engineering
boring. I hope you will enjoy this
episode and if you like it, consider helping the show by
living it a rating review or comment on your podcast app.
All social media channels, those reviews and comments are one of
the best ways to help me get this podcast to reach more
listeners and hopefully they can also benefit from all the
contents in this podcast. So let's get this episode
started right after our sponsor message.
Are you looking for a new cool swag?
Taglit Journal. Now, offers you some swags that
you can purchase. Online.
These works are printed on demand based on your preference
and will be delivered safely to you all over the world where
shipping is available. Check out all the cool strikes
available by visiting technology, you know that, death
/ shop and don't forget to break yourself.
Once you receive any of those tracks.
Hey everyone, welcome back to another new show of the package.
You know, today I have with me a guy called Michael.
I polakov ski or in short. Let's call him Michael.
I saw me. So, is the author of a recent
book, titled chaos engineering. So today?
I'm sure we're going to be talking a lot about chaos,
engineering how to implement it correctly, and maybe some of the
gotchas or misconceptions about health engineering.
So, because it's good to have you here in the show.
Thanks for being here. Very, glad to be here.
Thanks for inviting me. Yeah.
So before we start, probably introduce yourself to the
audience here, can you tell us a little bit more about your
career? Maybe some highlights or turning
points? Sure, so I don't know how Far
back. You would like me to go.
Yeah, obviously right now, I do a lot of guess engineering.
I run a small SRE. Team that managed communities
and Chaos engineering kind of evolved as one of the very
important tools that we use to basically make our system more
reliable. And it placed nicely into this
entire site, reliability, engineering mindset.
I've been out Bloomberg for a while, now, before that.
I attempted a couple of startups.
So pretty, I think, common take background.
I've been hacking away at coding since I was a kid.
So I think probably a lot of people can relate to that.
Okay, cool. I mean, let's maybe dive deeper
as we talked along about your career.
You have this this book not so long ago, right?
Chaos engineering. I think the first time I heard
about this term is during that time when Netflix popularized,
this idea, chaos engineering, and they've been running in
their production, killing their servers, in order to increase
the reliability, but maybe for all the audience Are can you
maybe explain what exactly is chaos?
Engineering. Sure firing and that's probably
a good idea because there's a lot of misconceptions and the
name itself doesn't help. So I guess engineering is
basically this discipline of experimenting on the system and
the system can be anything. It doesn't have to be massive or
Netflix girl, in order to increase your confidence that
system will survive difficult conditions.
So a unit of course, engineering, the way that I see
it. Is, this chaos.
Experiment. When you basically have a bunch
of assumptions about the system. Obviously we all design the
systems and we want them to behave certain ways.
But only in practice, it turns out that there's a lot of
behavior that we didn't account for or system is doing what we
told it instead of what we intended.
So the experiments real goal is to either confirm that your
assumptions about a system are correct.
Basically your hypothesis works or You find a problem or you
find place where your assumptions and the reality
don't necessarily add up. So this guy's engineering
experiments typically have four steps one as defining
observability because this is like a prerequisite for all of
that to happen. If you can't observe the system.
Any kind of variable reliably than you can't really conduct a
scientific experiment. Then you go for what?
We typically call steady state, which is just a fancy way of Of
saying, this is the normal, kind of behavior.
This is the normal range. So let's say that for
observability. We're looking at variable like
throughput or number of requests per second that some kind of
server can handle the normal range.
The steady-state might be this number of thousands of requests
per second and then we go and we do the fun stuff.
So we try to turn our expectations of the system into
a hypothesis and we say, okay, so we designed the system to be
redundant and means that if we take away one of Servers from
the pool. It should keep working within
this parameters and you go number four, and you implement
that and you verify what happened.
The nice thing about that is that everybody wins?
Because if your hypothesis was wrong, then you discover
something you can fix. And you can save yourself,
trouble, and fix it before you users.
Notice, if your hypothesis was right was, correct.
That means that your system is pretty good.
So you increase your confidence. So, you know, it's the nice part
of doing Because engineering. So this is really it like, it
doesn't need to be more complicated.
And then obviously, it came out of Netflix and it made for some
really good headlines because they were already doing this in
production breaking things. In production is typically one
of the internet meme. So if someone comes out and says
that with a straight face, it's a bit of a controversy.
So that's really what I see. Let's go to engineering.
I think it's simple enough that pretty much everybody going.
At least kick the tires and see what value they Get out of it
from your explanation just now. So there's nothing mentioning
about chaos. So to speak, it's more about
scientific experiment. You know, what?
You're going to test. You have a hypothesis and like
you have the steady steady you want to test about and then you
introduce some kind of tests and experiments.
Not necessarily all that chaotic stuffs, like just killing things
and doing some like one data center takedown or something
like that, but it's not necessary about the chaos
itself. Although a lot of people
actually has this misconception. Okay, let's Just introduced this
scale software and then everything.
Just go haywire. And let's see how the system
goes here. So chaos is a little bit of a
double-edged sword because on one hand, it catches attention
on headlines. On the other hand.
It requires a little bit of explanation.
So at least two things to touch upon here.
One, is that the carriers that we mention here.
It's not about increasing the amount of Chaos in your system.
It's about decreasing the amount of gas in my definition of
Australian to make you picture. Up coat and protective glove
were, and I were and stuff, because if we can't control as
many of these variables as possible.
We can't reliably confirm or deny our hypothesis.
But the other reason for the chaos in the name is that
there's an entire spectrum of things that you can do with goes
engineering, Gas, Monkey, the kind of thing that this entire
discipline started with really was very chaotic in the sense of
randomness of the word. So So, if you don't know much
about your system or you're looking for the emerging
properties Randomness can really give you a lot of value, kind of
out of the box and requires very little setup.
So we take the system, you release the chaos monkey on it.
You let it run and you probably find some things you can
parametrize. You can make sure that it breaks
on your certain percentage or whatnot.
And this already gives you value and I see the sign of a spectrum
as similar to the discipline of fuzzing in testing.
You produce a lot of valid inputs that you would probably
not. Think of if you were writing
unit tests manually, you just run it through whatever you're
testing. And you might discover things
that you didn't think of. If you just cram all this inputs
and like, brute force is approach, right?
And so, if you start with a system, it's a really nice way
to get into the girls engineering, because it doesn't
require much of a setup. You need to have a vague
understanding of the system. You can release it and you can
find thing. And the other thing I mentioned
is the emergent properties. It's also a fancy way of saying
that, but the greatest rate that for those of your audience, who
not heard about that, think about neurons in your brain and
a single neuron doesn't have the property of human conscious or
doesn't have a property of thinking per se, but then when
you put them all together and they interact from within this
interactions, you have the emergent property of Human
conscious. Same thing for another popular
example, of the cells in your heart.
None of them have the property of pumping blood and oxygenating
your body, but put together they create a system that actually
has this property. This examples give you very nice
properties, right? But in any complexion of system,
you're going to have the interactions that you just
didn't predict and sometimes they're going to be pretty bad
for you. So this kind of fuzzing
randomness. Scales engineering side of the
spectrum is great, and it's useful.
And it's part of the discipline over the last few years.
We've been trying to push for the other side of the spectrum
where you go into the system. You already know the system.
Well and you're working in particular properties and you
want to make sure that particular failure scenarios are
covered from the SRE point of view.
If you had an outage that you didn't predict for because the
system worked differently than you expected you would To make
sure that doesn't happen. Again.
One of the best kind of regression tests that you can
come up with is to simulate that failure scenario.
Make sure that the system actually still survives that.
So, the other side of the spectrum of the sophisticated,
very deliberate practice. When there is very, the
randomness is also available to you right now, as a practitioner
of chaos engineering, you can pick wherever its best on that
Spectrum for you. And I think that's It's
definitely good thing. So, as a techie, everyone
understands at least the concept, but when you introduce
this, to the management of the business, for example, so we are
going to introduce chaos monkey or chaos Engineering in our
system. How do you actually explain to
them? What is their motivation
rationale behind this? Because I'm sure any business,
and any Executives, they all want stability.
They don't want some kind of a Randomness or attack to the
system that is working fine. So how do you explain the
rationale and motivation behind this to sell it to them?
Yeah, that's probably like the number one question, and that
goes back to the name, being a little bit misleading, and a
little bit of a double-edged sword.
So it depends who you talk to. And I, in general tends to have
shit. Has two big groups.
Either. It's your colleagues or people
who have had the experience of actually being paged in the
middle of the night. And I think the only real
argument you need to get them on board is to tell them that.
Listen, if we do it right there is a huge potential for you to
be called less at night. So there are like basically the
worst case scenario is that you just don't go some Junior and
you break something, if you do it well and you do it with
common sense and you apply all the same best principles for
deploying the code because the case experiments are code like
any other really you're going to only affect a certain blast
radius anyway, but the worst-case scenario, is that.
Okay? So we did get engineering and we
broke something. So what happens then, one of the
things that It's worth mentioning is that you doing it
on purpose? So if it breaks your typically
in the office, you're typically ready to jump to fix that.
You don't have the complex, which waking up in the middle of
the night. When you're being paged to
figure out what's going on, is nothing Pleasant.
I'm sure a lot of your audience will have had that experience.
You have to wake up. You need to make the coffee,
read through the alerts, that may or may not be easy to
understand deal with people who are panicking because They just
got woken up and they don't know what's going on your contacts,
which, because between all these things, you login, it takes
time. It's typically not a very
pleasant experience. Let's just be honest.
So, if you can minimize the number of times that this
happens, and instead try to do it more purposefully, that's a
big step up. So if you are on rotor for
supporting your system, this is probably.
All you need to know. The other group is people who
Decision-making on this kind of stuff your managers.
And this kind of person for my experience, the best arguments
to start with, its to do some back of the napkin mods.
Now, I can have a phrase for that.
I typically colored the hamburger versus Shark problem.
It's about the perception of risk.
So like I mentioned just a second ago, if you do it, right?
And you are careful about the blast radius.
Jesus. It's pretty a lot like releasing
any other challenge to production, you test things for
the stages and you apply all the same principles you do for all
other bits of code. But if you ask a person
industry, how afraid should they be of sharks?
They have been already primed by Hollywood movies, the Joe's the
Meg, and all of that has to be very afraid of sharks.
But if they actually look at the statistics of how many people
die of shark attacks. It's a very minuscule number,
there is more people dying of Coconuts landing on their heads.
Every year than there is a shark's where us the things that
are really statistically likely to kill them, things like heart,
attacks or heart disease. In general, that take more than
half a million of Americans every single year compared to
probably a single digit number for shark attacks in any given
year. You see that this is actually
statistically much more dangerous D.
So, next time, you see that hamburger, we fold this crease
and, and all of that. Think about it that this.
Is actually more likely to kill you, then the shark.
What was the point. I'm trying to make here.
The but I'm trying to make is that a case engineering when you
first hear about that and about introducing failure on purpose.
It's a lot like end-to-end testing of the unhappy paths the
way I see it. It's like the evolution.
You do the unit test. You just a small little subset.
Then you do maybe component, testing some kind of integration
testing. Then, at some point you get to
end-to-end testing, where you typically test.
Happy path or some popular path and stuff like that.
And then case in June is a lot like end-to-end testing of the
system. When you take it as a home
during unhappy events, when things break, when machines go
down, when Network gets slow, and this kind of thing.
So for someone who is hearing, okay, we're going to add failure
to our assistance and we would like to also eventually do it in
production. It sounds scary, but that's like
the shock. If you think about it.
There are ways to manage the blast radius and they're armed.
Ways to manage that risk and the goal of the entire exercise is
decreased the amount of chaos and decrease the amount of risk
that you have in your system, not increase it.
So just a napkin, a few numbers run through that and you can
typically explain to your engineers that it's really, they
shouldn't pay too much attention to this Curry sending name
because there's a lot of return on investment to harvest.
So, if I hear you correctly or discuss, engineering doesn't
necessarily mean that you have to run this experiment in.
Second, you can also run it in maybe a pre prod, or some kind
of environment where you can actually simulate and do.
All these tests. Is that correct?
Although, if you don't try it in production, you're excluded from
the Coast Community. Okay.
So, you know, this is one of the things because of all the blog
posts. It's shiny doing stuff like this
in production is unorthodox, and it makes for a great
presentation, but that's the Holy Grail, right?
You want to be asked comfortable with the system that you've done
this for so long in the other stages.
That is now part. Out of your routine and you do
it in production, and that's great.
And if you can do that, that's obviously great.
Because if you think about that, you're never fully testing
things until they get into production.
The data patterns will be different usage patterns will be
different. So, by definition, you can never
fully test things on it until the actual proverbial rubber
hits. The proverbial grow, but the
common sense sticking the clothes engineering sticker on
your laptop. Doesn't necessarily give you a
more Absolution to Piercing common sensor behind the scenes,
the less shiny bit is that you are progressing.
That the same way that you progress, all other software.
Rewrite this code. You run it in your test
environments. You might increase it.
You probably going to do the same thing that you do.
When you release any other software and limit the
percentage of traffic that goes through the systems that have
that and then increase it. If anything goes wrong, you roll
back. So the same best Suppose really
apply. Once you get to the holy grail
and the ranking production. That's great.
You can write a blog post and go in the conference and talk about
it as some of us do, but it's not just about that.
And also, it's probably worth mentioning that there are cases
that are use cases where it's probably never going to be.
Okay, if your choices either to potentially introduce this
failure and you're not confident whether it's going to work and
someone might die because of that that's probably not.
Out the right, moral choice to do that or if you have very
heavy contractual obligations. Now from the legal point of
view, you might not be able to do that, but it doesn't mean
that you can't get 80% of the value of harvesting the lower
hanging fruit in the process. So, now, that's a myth.
Thanks for clarifying that. So, obviously, for us people who
are not experienced in it. There are many other more
misconceptions, probably the one that exclude you from the chaos
engineer group, so maybe you can tell us some of The other common
misconceptions that people have sure, sir.
There is an entire Suite of this and I I typically on the regular
basis. I asked my LinkedIn Network to
say what's the biggest blocker and it still seems to be getting
the buy-in in big part because of this misconception.
So the production one is a big one and that's typically
something that people really think the other thing is
Randomness. So a lot of people basically
disregard U.s. Engineering because they feel
like it's a gimmick chaos, monkey, randomly smashing
things, while for bigger systems, like Netflix again,
that example that works nicely because doing something as
simple as taking down. The VA ends is already giving
you value. It might not work for a smaller
system. So explaining what we just
talked about that. It's not just about the
randomness. It's about entire Spectrum from
random to very deliberate. It's also important.
Another thing. I keep hearing is the oh, we
already have chaos enough. We don't need new ha, we've
touched upon this already too. But I really would like to drive
the point home, that, if you still think that case,
engineering is about adding chaos to your systems.
Then you still haven't gotten the memo.
We add the failure so that the amount of incertitude, the
amount of Chaos in your system actually decreases.
So the net net, we do all of that.
A decreased amount of care. So obviously under that
umbrella. A lot of people say that because
they don't feel that confident in their systems, all of our
systems break. And if you are just hearing
about this and you had a massive outage last week, you might feel
okay, fine. I need more maturity or
something like that. I think that's a false premise
to because regardless of where you are on your maturity
Spectrum, you can get some value out of doing that.
Typically discuss experiments unless you go really into the
deep weeds. They are simple to setup.
They don't take that long to implement and especially as the
tooling around that gets better and better by the day.
It's easier and easier to do pretty sophisticated things with
little effort. Worst case scenario, you don't
discover anything and you feel a bit better because you know that
this particular scenario is not going to take your system down
and best case scenario. You can discover something.
So it's really like the small. A batch that can pay off that
are pretty cheap to do. There's also the benefit of the
kind of mindset that goes with that.
If your engineers and your team think about the fact that this
is going to be tested, that this is actually going to be put in
practice and they have to design the systems with this.
In mind. It just goes more naturally to
bake this reliability from day one rather than doing this as it
breaks. So the mindset is I also saw
that if we could probably talk about it would mock.
But we kind of think that who we're not that mature.
We still have outages and stuff like that.
It's really not helping because at any stage in your maturity,
even if you still getting outages from the regular basis,
you can add value out of that. So, no, it's not adding more
chaos. It's definitely not we're not
mature enough. I think one more is about the
scale, people look at Netflix. And they look at Google and they
say okay. Yeah, this are massive companies
and they're bleeding edge and they have all the scale and
that's great. But it's also worth noting that
guy's engineering doesn't require you to have all this
fancy and massive distributed systems.
It can really work. It's more of a methodology that
can really work on any kind of system whether you're working
with a single process Legacy server.
And you would like to make sure that you understand how it
breaks and the kind of things that you expect to happen.
Don't break it, rich rice or whatever, recovery logic.
You can do that. Nothing stopping you.
So really don't be shy, kick the tires and you'll see that with a
little bit of investment. You can get a lot of value out
of that and you don't need to be like, oh no, we're too small for
them. So I think that's like my top
Nate's that. I just keep hearing over and
over. It's probably too late though to
change the way home. So we're probably going to be
hearing them. Thanks for clarifying, all these
misconceptions and myths about Kelsey engineering.
So in the beginning, I think you have mentioned this as well for
people who have heard about this.
So they want to give it a try. Right.
Can you summarize again the four steps that they need in order to
start introducing these scales engineering, sure thing.
So first is the observability and I like the word because it's
very technical. But what it means is being able
to To measure something, reliably, what I mean by
reliably is that in computers, you know, we're kind of spoiled
because we can measure some things reasonably.
Well, if we're Quantum physicists, it will be a bigger
problem because the measurement could affect what we try to
measure. And then we have the uncertainty
principle to Begin work. But what I'm getting at here is
that it's important to be good at observing.
This it can be very simple. This variable can be anything.
It could be whether the server. Up and down, then there's
something can observe our throughput or number of requests
per second or anything, really? And then, once you have that, I
mentioned, the steady state, the normal range of that is how you
verify what's going on. So normally the server is up,
are normally I get this many requests per second.
And then we hypothesize. We say ok, so we are disc right
heavy. Let's see that.
We're sure that we understand what's happening.
Meaning if someone steals a little bit of the disc, so we
might introduce a hypothesis. If the disc becomes 50%, slower.
I still hit, let's say, 50% on my steady stage in terms of
requests per second. And then the last step is to
implement though, take one of the many tools that are
available right now to do that, and make sure that it doesn't
affect your observability. Go run it and see what happens.
More often. Do not, you're going to
discover. Things that you might have not
predicted. Thanks for summarizing that
again. So I'll just to recap first.
You need to have a good Observer bility.
So I think this is a mass, right?
You cannot introduce something chaotic without actually knowing
and observing how the system behaves we got that kind of
defeats the purpose. I guess.
Then the second one is you need to know your steady state.
So what exactly your system at the normal State before you
introduce this chaos and the third you start to hypothesize
and making experiments, I guess like what kind of things that
you want to test against the state?
Ready State. And then the fourth, the last
one will be to implement that. So I hope I summarize it
correctly. You mention as well.
There's this chaos, engineer community and things like that.
Is there a specific skills actually required from a chaos
engineer compared to? I don't know, like normally
engineer or a sorry, or maybe you can clarify on that pot any
particular skills for girls engineer here.
So this is one of the things that I really like about because
engineering is that it really cuts through different stacks
and different too. Colleges and different kinds of
Designing systems. It's really more of a meta skill
the way that I see it. That also means that it's not
necessarily a job description. I mean, some people have this
job description, but it's more of a mindset / skill, that a lot
of different people can have that.
So from what I do daily, you know as a kind of a sorry type
person is that I care a lot about my sister not blowing up
then my system working well. And as the scale Increases and
everything. You realize that the failure is
when rather than if, and you start seeing more and more of
that as it grows. So the primary driver for me is
to have another tool that helps me sleep better at night and
get, paged less, or call it now. So this is something that works
well for the as a radius, but if you're an application team, you
can also leverage the same thing.
You design your system, your application with a very working
on, in a way. It said, it's the most reliable
possible. So you apply the same skills and
there's nothing stopping you from running your own chaos
experiments. The SRE, kind of person might be
running a platform that runs a lot of client echoed that they
know little about potentially and the application side.
You run fewer of the applications, but you have much
more intimate knowledge about them.
So, to give you a more tangible example, let's say that the
chances are that using a database.
One of the things that you write, as you write your test
cases, your unit test and you verify, what happens when the
database becomes unavailable. You verified that right errors
return, for example, so something like that or that
there is a rich way, something like that and that's great.
That's hopefully what everybody is doing.
But now the question is, do you know what happens when the
database is still available? But it becomes a bit slower.
Do you know what happens if it's a all The connections to your
database are now being throttle because there's some busy
neighbor or just networking is overloaded and you get much
slower. Do you understand how the
compounding effects on all of that work?
And do you have things in place to work with that?
Or is your application just going to hang in Forever except
enough connections? Empties up the pool and you get
stuck forever. So at every level of the stock,
whatever you working, you can use the same thing.
NG I really try to drive this point home in my book, that's
much more applied and technical than the other books that were
available at the time. Is that wherever you are on the
stack on the platform, whether you working with the colonel and
you want to verify what happens when Cisco's are blocked or are
delayed or something like that to the platform levels
networking maybe communities level containers level all the
way up to the browser things like Okay.
So JavaScript is everywhere. But do you really test out what
happens when the front end code is having trouble connecting
orbis? Actually getting slow responses.
Do you still display coherent data rather than loading a
little bit here? A little bit there and giving
people a false impression. So it's really applicable to all
the levels of the stack. This is something that's very
exciting for me because you get to look at the And things and
you know, just contains to this and go box.
So when talking about these layers, I'm very interested
because you can do this at any layers, not just platform
levels, which is what we normally heard from Netflix, but
you can also run it at the application Level database,
level even Network level or like what you said doing it on the
browser itself. I haven't heard it to be honest,
but maybe can you give us some examples?
Like the name of the tools and what exactly they are testing
for these layers, maybe some examples would be great for the
Winston. Oh sure thinks.
So let's say that we've all had this moment when someone gave us
some Legacy piece of software that has been compiled 10 years
ago and it runs but the documentation is a little bit
iffy at best. There is more of a tribal
knowledge and you don't necessarily want to go and read
the Fortran code or whatever C code that it's implemented in.
And so if you work in a startup that was started last year,
maybe Maybe you're going to be able to skip that part of your
life, but it's like a rite of passage.
So one thing that you could do about it, let's say that it's
some kind of server and that's an example from my book.
If you want to actually go and play with that.
There's a VM that lets you start that and play.
Is that not a lot of people know that you can use esterase, not
only to trace the system calls, but you can also use it to
implement changes, so you can error system.
Girls. Also, if you have modern in a
version of a stress, you can actually Implement patterns
where you can fail, for example, every other request.
So while the things that you could do at this very very low
level is that the server is doing right to send the request
to you so you can go and you can verify that.
For example, if it can't do it, right?
If it gets an error, there is some kind of ritual Logics.
So if you fail a portion of that, you're going to see what
the return logic kicks in. And you get the right response
or if it breaks completely. So even if you don't know much
and you don't even see the source code of this and all you
have to go by is the tribal notice that?
Oh, yeah, it's supposed to recover, always recovers.
You can actually go and get some mileage out of the most basic
thing that thing does. Because every program your
system is going to have to go and do the right.
So this is something that a lot of people use as trays to see
the Lolo. What's it can do, but not
everybody knows that. It actually lets you implement.
Chaos experiments. Like that stress is also a great
example of the importance of observability because the
penalty of running a program while it's being asterisk is
pretty high. So the measurement actually
effects, the program that you're running, it slows things down
very significantly. So there are other ways that you
can observe this Wings. There are new technologies.
Is that let you do that? Like, EPF the extended Berkeley
packet, filter that are becoming increasingly common and
Powerful, because you can do a lot of similar measurements for
the observability point of view without actually affecting this.
But in a lot of scenarios, it's going to be workable because you
can measure, if all you're interested is success rate.
It might be okay for you to accept the penalty for doing
your test. Something you probably wouldn't
do on a production system because obviously, Let's do
things down and affect the users.
So this is like the lowest level that I could think of and that's
why I went all the way there in the book to show you that
worst-case scenario single process and you can still apply
the same principles and then I mentioned kubernetes when you go
a level up, there is a lot of staff that could bring Auntie
solves for you, but it also introduces its own complexity
and if you don't have a good understanding and just expect
communities To always do the right thing.
You might be in for a surprise. If you are working with
kubernetes as a product, if you're running kubernetes or
someone else. It's also very important to
understand how communities itself fails and what the
fragile points are. And this is something that's
very easy to do. If because engineering and every
one of those experiments is helping, then there are things
like, you can build case engineering into your
application directly. Obviously, in this means that
your Woods is part of the application and it's prone to
introducing new bags, but you can do things like activate the
case, engineering code, only behind some kind of flag and
activate that for some percentage of your traffic.
Make sure that there is no run time penalty for running that.
And the code is not actually hit.
So you can break things for the instances where it said
activated. Although in the browser like I
described, this is something that people react of a smile
initially do. So.
Okay. Well I'm going to Give us this
my browser software, but it doesn't take much to show you
that. Actually, if you just play some
data from the previous request and send data, from the new
request. You might actually trick the
user to do something silly because they have stale data or
inconsistent data. So doing things like this is
also important. Yeah, that's three examples
then. Hopefully give you the entire
spectrum of going all the way up from the bottom.
So I think you mentioned a lot of these examples.
Your book, for those of you who are interested to know more or
even play around with some of these tools.
I think Nico, explain it clearly in the book for this layer, for
this tool. How do you do that?
So make sure to read the book if you are interested for them.
So, I think in the book, that's another thing that you
mentioned, which I find very interesting which is why you
called Carol's engineering. But for people, why do you write
a specific section for that? Actually, it's a little bit
unappreciated, but if you think about your people systems, Also
known as teams. A lot of the same rules that
apply when you're building a reliable computer system.
Also, apply to building reliable human systems.
Probably easiest way to illustrate that.
Is that every team is always going to have some bottlenecks,
and the bottlenecks might be in a form of throughput bottleneck,
or they might be in a form of knowledge.
Bottleneck, if you have only one person who's Capable of
debugging a particular system and that person is on holiday.
You're going to have a problem. So a lot of the job of having a
performant and good team is to continuously trying to find this
bottlenecks and resolve them and a lot of this stuff happens
naturally and if the people on the team are thinking in this
kind of way kind of chaos engineering way, it's going to
be good for your team, long term.
So this is something that the last So in the book is actually
talking about it in detail and it's going to the human aspect
of getting the buy-in. So it's touching upon some of
the bottleneck, some of the misconceptions that we just
described that goes into the curse engineering mindset.
Basically, the way that if you think of your system with the
expectation of failures happening rather than a
possibility of failure is happening.
It's going to help you build more reliable systems from the
ground up. It's kind of like, if you have a
good CI, System that always runs your unit tests and every time
you push a chance Upstream, you're going to automatically
get the feedback from the unit test.
It becomes part of the culture, you get this immediate feedback
quick, turnaround for the feedback.
It works. It doesn't work.
It detected. The problem.
It's similar with the curse engineering for mindset that you
built this in into the automatic thinking about things, and it's
not something that you might address later.
Later, it's something that you expect to happen.
It goes back again to the back of the envelope.
Napkin if you think about it. It's not a question of.
If it's a question of when, and it's fairly easy to calculate
depending on your scale. How often you're going to see
the kind of values. And then the final bit of that
is that was inspired by Dave runs and presentation and he
gave out one of the conference's when he basically described the
teams and I love the description.
I still remember that as a set of By your robots, executing a
distributed algorithm to produce some work and he actually went
much into the details of the games that they came up with to
surface and detect the problems. I basically followed to his
leader to include that into the book and make sure that it's
spread through community. And some examples of that.
I talked about the bottleneck in terms of knowledge.
So, one of the things that you can do is just tell somebody
Buddy, and a particular day that they're not allowed to give
anybody else and help on this particular subject and that will
tell you whether the rest of the team can basically do without
them. And if they can't, you detect to
the bottleneck, he goes into a lot of more advanced games to
all the way up to basically telling people a fake outage is
happening, and seeing what happens.
And what they do to address that this obviously is a little bit
more tricky because you need to get the boy.
In from the higher-ups and people might get confused.
They might not know what's actually happening when they try
to do back. Something.
That's not really happening or you might go all the way in and
actually go and break something on purpose without telling your
team to debug your reliability in terms of responding to that.
So they've really went and created some very interesting
Insight on that in that presentation and I just couldn't
not include that into the book. Look, hopefully, he still enjoys
it there. Thanks for sharing that.
I think it's a very interesting concept teams as a distributed
system, kind of a mindset. So Miko, I know that you are
very active in this community, right?
For me. All I know cool stories about
Kelsey engineering is all about Netflix that chaos monkey Simian
Army and all that. Are there any other cool
examples that you have heard people showcase, maybe in the
conference or things that it's publicly available.
Are there some cool things like that?
So it's interesting that you asking them because That's what
I've been trying to do over. The last few years is actually
to make coats engine. Very boring.
Let me explain why I think boring is good.
What you think about the kind of adoption curve of different new
technologies. It always goes to the same.
Bell-shaped curve initially. It's a novelty.
You have a very small early innovators population.
That's happy to put up with all the shortcomings of that.
Then you have like potential early majority.
They have the late majority and people who drag a little bit.
And so for technology or methodology to reach the white
audience, it has to become mainstream enough.
Basically boring enough that it can be adopted by a lot of
people because not everybody is working is happy to work around
the rough edges. So did the example I really
bring you up is the SpaceX records, not that long ago,
seeing this Rockets go all the way to orbit.
The boosters go all the way to the orbit and then automatically
Land. Look like something from science
fiction. They were the first one to pull
it off and it was amazing. I remember staying up late
because London Times n and watching the nine minutes or
whatever of the flight, and then just Landing like something from
sci-fi, but then over the time as they get better at it.
It stopped being exciting because they stopped blowing up.
They just go up. Go down.
Land on the Drone ship. Of course, I still love you.
It's just becoming so mainstream that I no longer find myself
staying up for that. So now maybe the spaceship is
something that I'm going to want to look at.
But if they start Landing every test and they start doing it
routinely it becomes boring and so boring is good.
The same way that a smart from was something that was
outrageous not that long ago and now everybody has one you can It
on the chair pants outrageously, good and quick.
And you have access to so many different apps.
We have a small super computer in your pocket all time and you
don't even notice that. So I would really like goes
engineering to stop being about the exciting stuff that you can
do and break things in production and not get fired for
that. And instead become this routine
thing that we do, just because it creates a lot of value.
There are benefits to doing that.
Yeah. I'm going to go in the opposite.
The direction of that and say that it's probably more about
making poor, and then you think about it, most of the
low-hanging fruit. It's like cyber security in the
movies. We see the hackers just randomly
punching the keyboard and streams of data going through
and I'm in and probably some nice graphic turned out.
But in reality most of the low-hanging fruit is so boring
because you need to check all the routine things.
I need to stay up to date and you need to pass with the things
that are already known and And you need to make sure that your
S3 bucket is not on public setting.
So that's the boring stuff. Doesn't make it into the movies.
But that's where most of the work is done.
So, yeah, if you want the exciting stuff, there's
definitely things in the internet, but that's not really
where most of the values coming from the most of the values
coming from boring. Thanks for that, valid points.
Hopefully one day. We'll see, all these cars
Engineers, not unicorn. So only the cool companies that
are able to do that, but hopefully, All the engineering
team is able to introduce some kind of experiment chaotic
experiment in order to test the reliability of their system.
So Michael, thanks for spending your time today.
Eventually, we come to the end of this conversation.
But before I let you go, normally I would ask this one
question for all my guests, which is about three technical
leadership wisdom. So, can you share maybe some of
wisdom that you have, maybe from your career, or maybe from your
chaos engineering experiments that you have so that audience
can learn and benefit from you. Wow.
Wisdom, that's a big word. Okay, that's right.
I think one of the things that I learned probably gave me a lot
of mileage. Is that a lot of what we do is
about removing the BS from the equation Engineers apart from
the big Egos and everything. I really very finely tuned to
detect BS. And so, a lot of the tank
leadership and leadership in Tech in general, in my opinion,
is about just making sure. Sure that there is a stable
beers as possible. So if someone asks you a
question, you have few options, you can say.
Yes. Know if you know for sure the
answer you can say, I don't know if you don't know the answer or
you can try to be as your way through it and try to come up
with something on the spot and I got a lot of mileage just by
removing that last option. If you just tell people,
honestly, I know I don't know, yes or no, it builds this.
Ship, that's available to us when they also feel like they
can also say, I don't know. I'm supposed to be an expert in
the field and paid a lot of money for that and Senior and
everything, but there is stuff that I want now and that's
completely fine. You work with a lot of people
who are smarter than you who have more experience than you.
Who are by definition are going to be much better at parts of
your job. If you are humble enough to just
say, I don't know. Well, what do you think it gets
you out of mileage? And I think that really helped
me stay out of trouble and kind of a corollary to that another
piece of an agate. Is that regardless of what you
think about yourself? There's always something that
you can learn from everybody and we work a lot in this industry
and working hard and increasing the diversity of thought coming
from a mindset that okay. If that person disagrees with
me, there is probably something that they think about or they
know about that. I don't know.
So if I just try to convince them to my way of doing things,
I'm not going to come out of that conversation wiser.
But if I at least by default give it a shot.
Maybe they run. Maybe there's something.
I know they do not know, but if I give it a shot, I'm going to
get more value of that. And if you keep getting value
from every conversation, you really going to accumulate that
over time. So yeah, I think that basically
the kind of being humble and knowing that you can learn all
the things from everybody and not be essing is probably what
really helped me get where I am right now, because you asked for
three. I'm going to also try to attempt
the third one. I think you probably hear that a
lot in your sure, but the kind of learning mindset, lifelong
learning is very important. This is something that some
people get built in and they start with that, and they just
super excited about learning by learning.
I mean, like a lot of things, there might be a point where you
do everything about your knees. And there isn't much more to
learn instead of stopping learning.
You should probably start exploring other Niche or maybe
do something completely out of your comfort zone and go learn a
language or to some art or support and see how that affects
your brain. In the recent years.
I've been reading a lot about brain and how it works.
I found myself being able to influence, a lot of weird things
in my life just by picking up new skills that seemed to be
completely unrelated. It's easy this days, you can get
a nap. Up, you go and do a lingo and
you can pick up a language that would normally cost a lot of
money would be impractical, but you can take the few first steps
free or very cheaply. Obviously the pandemic made it a
bit more difficult to pick out a new sports app.
Anyway, the point is that learning things might have
unexpected benefits to you. And this is something that I
really recommend doing. Yeah, I agree with that.
Thanks again for reminding this important learning mindset.
So me go for people. Interested to learn more about
you or maybe a recent book. Where can they find you online?
Maybe sure, sir? Probably the easiest way to
reach out to me is on LinkedIn if you want to interact.
Otherwise, I do have mailing list for the book.
If you want to go to chaos engineering dot news.
I can sign up and get updates. If you have any particular
updates about the book or I mess something up, do reach out.
There is a GitHub repo out of the book where you can download
the VM, and And that's where I can put issues.
I'm pretty sure my some things up.
So looking forward to that. Hopefully, we can curse
conferences when they become in person.
Again, kind of looking forward to them.
Don't like the way I think of it all.
So it's like a chaos experiment in our life where this pandemic
suddenly throws people into different kind of living and
mindset and routines. Hopefully, we will end this
pandemic soon enough so that we all can live through our
previous normal life. So, thanks again.
Go. I hope your cars engineering
mindset and you're being the champion of it have more people
being able to implement that in their team.
They are systems and also the companies so that we can make it
more boring. So to speak.
Like what you said and I wish you good luck for your career as
well. Thank you.
It was really fun being on the focus.
Thank you for listening to this episode and for staying right
till the end. If you highly enjoyed, please
share it with your friends and colleagues who you think would
also benefit from listening to this episode.
And if you're new to the podcast, make sure to subscribe
and leave me your valuable review and feedback.
It really, really helps me a lot in order to grow these podcasts
better. You can also find the full show
notes of this conversation on the episode page at technology.
No, the death website including Doing the full transcript,
interesting quotes, and links to the resources and mentions from
the conversation. And lastly make sure to
subscribe to the show's mailing list on technology.
No, the deaf to get notified for any future episodes.
Stay tuned for the next technique Journal episode.
And until then. Goodbye.