Hey, a quick message. For those of you who are
listening to this episode on Spotify.
I have a small favor to ask Spotify.
Now allows mobile users to rate podcasts.
I would really appreciate it. If you can take a quick, pause
to go to the technique Journal podcast page, and leave your
favorite show. Your best rating on Spotify.
It will help me a lot to get this podcast to reach more
people on the platform. Thanks a lot.
Observability is a technique for ensuring that you can
understand. Novel problems in your system.
That's why it was a necessary addition for me to go from
working on necessary to work in observability.
So, when we think about defining observability, it's basically
this question. Can you understand what's
happening in your system and why without having to push your
code? And to do so, very quickly by
slicing, and dicing existing data that you already have, in
terms of telemetry signals that are coming out of your system.
Hey everyone. My name is Henry Surya with
Robin. And you're listening to the
technology, you know, podcast the show where I'll be bringing
you the greatest technical leaders practitioners and
thought leaders in the industry to discuss about their Journey
ideas and practices that we all can learn and apply to build a
highly performing technical team and to make an impact in your
personal work. So let's dive into our Journal.
Hello to all of you, my friends and listeners.
Welcome to the technology. Now, podcast the show where you
can learn about technical leadership and Excellence from
my conversation, with great thought leaders and very happy
today to present the 88th episode of the podcast.
Thank you for tuning in listening to this episode.
If this is your first time listening to tackle the journal,
subscribe and follow the show on your podcast app and social
media on LinkedIn, Twitter and Instagram, if you are a regular
listener and enjoy listening to the episodes, support me by
subscribing, as a patron, at technology node, Dev slash
Patron. My guests for today's episode is
list. Found Jones list is the
principal developer advocate for a sorry and observability at
honeycomb. And recently.
She just published a book titled, observability,
engineering, which she co-authored with her colleagues,
charity majors and George Miranda.
In this episode. List shit in depth about the
concept of observability and why it is becoming an important
practice in the industry nowadays.
She started by explaining the fundamentals of observability
and how it differs from traditional monitoring and how
observability can help us to run a more reliable and stable
production systems including its relation with the devops and SRE
practices. She explained some interesting
Concepts such as the corn Alice's Loop, cardinality and
dimensionality. And doing debugging from a first
principle in the later. Part of the conversation list,
shared her view of the current state of observability,
including the proliferation of vendor and open source tools and
how we Engineers can improve our systems, observability by doing,
observability driven development, and improving our
practices based on the proposed. Observability maturity model
found in the book. I really enjoyed my conversation
with list diving deep into Tea and understanding the different
nuances of observability Concepts.
If you are interested in this topic, I would also highly
recommend reading further the observability engineering book
in which honeycomb has kindly provided the e-book for free on
their website. Find out the link provided in
the show notes to get your free copy.
And if you also enjoyed this episode and find it useful,
share it with your friends and colleagues.
Who may also benefit from listening to this episode, leave
a rating and review on your podcast.
Up and share your comments or feedback about this episode on
social media. It is my ultimate mission to
make this podcast available to more people.
And I need your help to support me towards fulfilling my
mission. Before we continue to the
conversation. Let's hear some words from our
sponsor. Today's episode is proudly
sponsored by skills matter. The global community and events
platform with more than 100,000 software professionals here
members, can organize their learning experiences around the
Technology topics. They care about most you get
on-demand access to their latest content thought, leadership
insights, as well as the exciting schedule of tech events
running across all time zones. So where the devops our data
science is your bus or you're a fan of functional programming or
all things Cloud, you can make real connections with people who
share your interests head on over to skills method or Cam to
become part of the tech community that matters most to
you. It's free to join.
And you will find it easy to keep up with the latest tech
Trends. Hello everyone, welcome back to
another new episode of the package, you know, podcast
today. I'm very excited to finally meet
someone who I have been following for quite a while.
This funk Jones is here with us. So today, we'll be talking a lot
about SRE and observability, which is the topics and the
trends that are up and coming. Also, these days, this actually
is the principal developer Advocate at honeycomb.
She has been working in this area.
And observability for maybe 16 years by her bio, really is a
pleasure to meet you today and I'm looking forward to learning
everything about a sari and observability today.
I never really get to everything, but we can certainly
do a good chunk of it. Hi.
Thanks for having me on the show.
So list for people who may not know you yet.
Maybe if you can help to introduce yourself, if you
telling us more about your highlights or turning points in
your career. Yeah, so I started working as a
systems engineer in 2004. I've been doing all kinds of
work on reliability and making systems better.
It easier to operate, that includes spending a number of
years working at in studio a number of years working at
Google. That's where I spent most of my
career as a site reliability here at Google and then my
career took a turn towards thinking about.
How do I teach not just my team to work on better systems the
house? All teams across the suffer
Industries, do better, that's wanting to come a developer
Advocate and I switched over to working out how to vote.
So, yeah, I've been following you with all your SRE Contents.
I started probably with the SRE contents, mostly, and then it
took a turn since you join honeycomb.
And now we are talking a lot more about observability and you
have an upcoming book which is titled observability
engineering. So, this is something that you
wrote with Charity majors and one of your colleague in
honeycomb. Yes.
Your friend up here. There's just Mirror and I
observability is definitely one of the hottest topics these days
especially in the technology World.
Maybe you can start by helping us to understand what is
actually observability because this term is commonly used these
days. Let's contextualize in terms of
both as a re and observability. Not all of your listeners will
necessarily have heard of a story either.
So when we start with thinking about what s res do the goal of
a site, reliability, engineer is to try to Systems easier to
operate for the people that are working on member, whether it's
a tourism settle for Ron Paul. Or whether it's the software
engineering team that is kind of building an operating and
running the system. One of those key considerations
in terms of ensuring the system is highly, available is thinking
about what are your service level objectives?
What are your reliability goals for the system?
Mmm, Flipside, how do we think about trying to achieve those
goals? What happens if there is an
incident, does it take 5 minutes to fix?
Or does it take three hours to fix to me?
That's where observability comes in because observability is a
technique for ensuring that you can understand novel problems in
your system. That's why it was an F is
transition for me to go from working on necessary to working
abs or Billy. So, when we think about defining
observability, it's basically this question.
Can you understand what's happening in your system and why
without having to question your code?
And she do so very quickly. Five, slicing and dicing
existing data that you already have in terms of telemetry
signals that are coming out of your system.
So you mentioned a couple of key points.
When you describe observability here, the first without pushing
new code. If I imagine last time, I used
to be a software developer whenever I want to troubleshoot
a debug something. Sometimes I introduce new code.
Yes. Event print after box statement,
all of us are guilty of it. So, tell us more about this
without pushing new code. How can we actually do this?
Yeah, select she way that we think about observability and
kind of that first step of instrumentation is as you're
developing your application. You should add its rotation to
help you understand what's happening inside of your codes
that you don't want to be caught in the bat Lair.
That doesn't mean you have to predict every single failure.
In the spring happen. Advanced be kind of have to
believe yourself, the breadcrumbs at your voided need
to debug in the future. So this starts with, in my view
at least kind of having some I'm form tracing to know when did
each request start and stop in each service.
Where did that request come from?
Which other service called you intern and maybe some other
properties? Like what user is it the more
you enrich your face gives with information that may or may not
seem relevant the time. But if you can freely add that
detail of what feature Factor on that side, which user ID is it,
which grows are they using? Which language are they using?
Then? It means down the road.
You don't. Have to have created by any of
those Dimensions. You can kind of see what you
have any of those dimensions are relevant.
So instrumentation tracing all these are being mentioned these
days. And people also normally
categorize it with these three pillars of observability, called
locks metrics raising. So tell us more are these three
the most important things that we need to implement for
observability. I think that having high quality
signals does matter, you're not going to get great
observability, unless the data is there.
However, you can't just throw a bunch of data into a data.
We can see that you're done. So I think that's kind of why I
pushed back against the three pillars narrative, because one
or two really high quality signal is enough.
You don't necessarily need kind of the Luminous, debug logs, if
you have tracing, that is representative of what you
addressed, but Toronto. Lucky, Metrics can be useful in
high-volume situations, but the problem of metric is that their
pre attitude. You can't re slice the data
what's been pretty accurate data source, right?
So I didn't fix each signal has some benefits and drawbacks that
you need to evaluate and you don't need to collect all three
of them. In fact, there are new and
emerging signals like, continuous profiling.
So they're not really even three pillars were lenses or whatever.
You want to call them. There are many different
Telemetry types that we can think about utilizing.
As we kind of compiled with capability, that's really not a
technical capability. But instead a socio technical
capability. What can your engineers
accomplish with the system? Not what are you measuring on a
system? But can you actually analyze it?
You actually get that result of I can understand, figure out
what's happening. So one thing that is also
interesting for me when I read and study but observability,
right? It mentions that the goal of
observability is actually to provide a level of introspection
or details that helps you. Stand, the internal state of the
system. So the key word here is internal
State. Tell us more about this.
How do you differ this with something that monitors external
State? Yeah.
So I think that when you're measuring external State, you
are measuring things. Like what's be CPU utilization.
What's the memory utilization? Those are things that are
potentially useful but they don't give you an idea of what
was the application doing at the time that this You gonna request
executed. So I think that kind of detail
into what was the code actually doing and not, what are the side
effects of the code. I think that's kind of what
differentiates observing it from the inside out, rather than from
the outside in The other thing that people in the industry, I
used to traditionally, we call it monitoring.
So when we talk about system administrators, last time we
used to talk about monitoring and in the past, I don't know
how many years recently is we start to shift from monitoring
the observability. So is there any significant
difference between 23 observability or is just another
term being coined to Brand? The same thing with the new term
to me? Observability is a superset of
the capabilities that monitoring would provide.
I'd so to leave are not at all synonymous because with
monitoring, you're just trying to figure out when something has
broken but won't necessarily answer why, which is why people
had considered for a long time. You know, I use my monitors to
tell you something has gone wrong and I use my logs to
figure out what happens. Guess what?
In a complex distributed system logging is not going to save you
anymore. It take forever to search
through deep doesn't give you causality of what causes what.
So what we think about the relationship of Monitoring and
observability. I think this is why I mentioned
service. Level objectives, early.
Can me service. Level objectives are kind of
monitoring 2.0 there. The answer to be tell me when
something has gone. Wrong that monitor played before
words, observability kind of maps more closely to me to kind
of replacing London. B does to replacing the metrics
to use for monitoring the past. So what do I mean, - well, what
do you have your service level objective?
You define what a acceptable level of successes?
As for instance, at honeycomb, we aim for India's pipeline, to
be 99.99 percent reliable. That means no more than 1 in
10,000, packets of the coming Telemetry are dropped.
But if we start seeing an elevated error rate, if we have
say, we're allowed to drop 1 million contract for month and
we start dropping 100,000 per hour.
We're going to exhaust that error budget for the month in 10
hours to us. That's an emergency.
I'll go and fix it right away. But if we have like a thousand
bad request that are just being dropped per hour.
Excellent. Think that thousand hours?
That's not an emergency. So kind of it differentiates
between expected levels of errors and an expected levels of
errors and is not susceptible to the same problem that people
have had with monitoring in the past of, oh my God.
It's 2 a.m. 1 out of 1 requests failed.
That's 100 percent error rate. Wake up everyone.
No one likes to paste it to. I am for something that flaps
and goes, right. Way.
So like peanut butter and jelly the go better together and then
together they tend to provide each superset of the kind of
monitoring and logging. Your people used to do the past.
So one thing, when I read your book as well, in preparation of
this conversation is that you mentioned traditional monitoring
is more reactive. You mentioned about getting
alerted, you only see side effects of certain things
happening while observability is more investigative.
So tell us more about this investigative manners or when an
incident happens. What actually does
observability. You.
Yeah, so observability really helps you form and test
hypotheses really quickly because when I am trying to
debug an incentive, my error budget has gone off.
The number one thing nicer asking is what's special about
the request their favorite instead of looking at this wall
of breaths to figure out which lines later at the same time.
What observability gives me is this powder to have caused
Ellen. The power to see on request That
failed to correlate with properties were most associated
with. Then.
We what's maybe it's one customer review.
Its when building, right? Or maybe it's combination of
those two things and the really neat thing is making somebody
going to be. So we've implement this is a
feature called Bubble Up where you can draw a box around the
anomaly and will tell you what's special about that anomaly where
you're telling us what? They're not only is we're not
trying to guess based off of two Sigma or whatever.
Ready work. Just telling you, here's the
difference between your control and experiment.
The cool thing is from there. You can basically go in and
refine where you can Filter only to this population of the vents
and man it repeat the process or to group by this field or to
group by a combination of fields and to kind of confirm that
hypothesis. So instead of saying I'm like,
oh my God, like, how do I set up this?
We re how do I write it in this up to square?
E language? Oh God.
I hope I got it. Right because it's going to take
two minutes, run over all these logs.
Instead, you get that feedback instantly.
So going back to our solos, 99.5% of honeycomb queries,
complete within 10 seconds. There it is.
Not a costume, messing up. You can feel free to just try
and experiment to analyze and slice your data particular rate.
You can resolve within 10 seconds.
If it doesn't work out, try it out of worried.
If it doesn't work out, you now have the MU.
If you need to run your neck square and then ultimately, end
of the day you can visualize this in either can aggregated
metrics. Like what's the distribution of
Lindsay's or you can go and look at the raw data and see it kind
of in tabular log. Click format of where each of
the fields and the relevant queries attached or you.
And look at it as a trees, wonderful.
Okay. Think that kind of allows you to
slice and visualize the data. Any way that you need in order
to understand. Why is the slow worst failure
coming from. So it starts with the forming of
a place this narrowing down the data and then looking at the
data to confirm your hypothesis. Yeah.
So if I understand correctly, the one that you described just
now is actually the what you call Core analysis Loop.
So when an incident happens, this is what the sequence of
events that is going to happen. So you look at a bunch of data
that you load, initially, then you figure out.
Okay. This looks interesting.
Testing maybe you use Bubble Up in honeycomb right?
Where you maybe narrow down the search results.
And then from there you correlate and then you start
again, and then you iterate until you actually find the root
cause do I understand correctly? Yeah, maybe not necessarily the
root cause, but like a proximate set of trigger sent me of hips.
This system over the edge, there is usually no one root cause and
a and then system. The other thing I want to point
out. Is that the thinking about what
a core analysis Loop is, that's work long before I can
honeycomb. So when I Working at Google.
I think to formulate this hypothesis about the for
analysis Loop and make me ask her and then I realized that he
comes for you to be the kind of manifestation of.
How do we bring this technology? Not it's to Google employees,
but to the rest of the world. Nice, nice.
So one thing that you mentioned when we do these cornelis, this
Loop is actually slicing and dicing data, right?
So I think one of the maybe commonly mention about the
requirements of observability is the high cardinality and
dimensionality without this is Very difficult for you to slice
and dice the data because that's just not something interesting
that you can narrow down. So tell us more.
What do you mean by cardinality and also dimensionality and why
it has to be high enough. Yeah.
So earlier I referred to this idea when your instrument in
your code that you should feel free to add as many attributes,
add as many key value pairs as you like to really explain.
As you go along. What's going on your coded?
Kind of leave this bread. Crumbs.
It's almost like adding test. Writing comments, right?
Think it's just a sensible way of leading your future self this
imitation that it's waves. What is this code doing wear
this coat thinking? The reason why people have
hesitated to you that in the past at least with kind of
traditional monitoring systems, is that modern systems bill you
by the number of distinct key value pairs that occur on each
metric that you're reporting. So that's cause people that
either omit, these key value pairs, two logs where they get
lost or to not record them adult, so, When we think of it
out, High dimensionality, what we're encouraging you to do is
to sprinkle these annotations throughout your codes.
And to kind of encode as many keys as you like, but there's a
problem. The problem.
Is that even if you reduce the number of keys in a metrics
based system, it turns out that if you have a lot of distinct
values like user, IE, right? Let's suppose you have millions
of users. It turns out that your metric
system has to create a Time series for each distinct user
and track it forever. If that user is only appeared
once so there is this amortization that an excellent
system expects that your keys and values will get really used
often. And therefore there is a high
upfront cost to having a new key at Value.
Therefore metric system penalize you for having a high
cardinality dimension. So to sum this up,
dimensionality is about the number of distinct keys.
And cardinality is about the number of distinct values perky.
They deal system that supports observability fit allow.
The sand basically and limited cardinality in a very, very high
amount of dimensionality. So, for people who are trying to
understand about this concept, cardinality dimensionality.
Let me just repeat again. What is just said?
So cardinality refers to the number of unique values that
you're storing in your metric system while dimensionality
refers to the number of unique keys that you're sending to your
metric systems. Another thing that I learn about
observability that you mentioned the book.
I mentioned in the beginning that I used to go to the server
Mako. Changes to do debugging.
But in your book, you actually mention, if you use
observability, you're actually doing a debugging from first
principle. Tell us more about this concept
because I still trying to understand this part.
Yes. So we talked earlier about the
core analysis Loop, which kind of sets us while up to think
about this. When we have a really
well-functioning core analysis Loop where you're able to
rapidly form hypotheses and test them, you no longer need to make
these kind of leaps of intuition to the Wonder magically right
hypothesis and instead you can test a lot of different
hypotheses and kind of narrow down your search space and
eliminate red, herrings dead ends.
Whereas in the previous ways of debunking.
It used to be that you would kind of have this one person who
knew the system really well under pad who could immediately
jump to knowing what the right answer was.
Oh, I saw this two months ago. It sticks, right?
That's why I think first principles.
Debugging is much better because it enables anyone on your team
who has learned how to do first principles of honking to step
into any editing, miliar situation and to figure it out.
And if predictable amount of time, sure, your expert, who
have seen. Mrs. Denman, who's been in the
company for 15 years. They might be able to solve the
issue in two minutes, but my goal is to make it so that the
worst case scenario of someone who's inexperienced, but nose
Works whistles, the money they Shield.
Take 10 minutes or 30 minutes, or most of the, but rather than
three hours, five hours, or 24 hours or Worse, what happens at
that person who found the company for 15 years leads to
company over hires that happens these days.
So we talked a lot of honeycomb about this idea of bringing
everyone on your team. So the level of the best of
other. So you don't need these medical
flashes of intuition. And instead that.
Everyone has kind of that base, heat ability.
This is really personal to me because I started the golden
2008 and I Google in 2018, 11 news I need had been on 12
different teams at Google and there's 11.
I was changing teams on average slightly faster than once per
year. I've state two years on one
team, but I'd also say with six months later, another key.
I never got that amount of time on any one team to really be in
the Deep expert on assistant, but they do still have really
great and valued team member because I had gotten really good
at first principles to bite at understanding what?
What software is available in the Google tool stack to
understand any and similar kind of service because Google had
this kind of standardization every service that you will use
the same tracing. The same metric system, the same
logging system and therefore both possible for me to walk
into a completely unfamiliar situation and figure it out in
half an hour or two. That was kind of surprising to
people. Because if you spent like two or
three years on a system, but not 10 years.
It might hate you several hours to figure out what's going on.
You don't quite have that expert level knowledge of the system.
But also you haven't grown that muscle yet.
Develop that muscle of how I walk into any unfamiliar
situation worse. For me.
I've always summer to meeting day.
One on that new team and be 180. I'm Nic team.
I would have to figure things out on the Fly for myself.
That was my survival. So when you mention first
principal pulled those who are not familiar with this term,
what do you mean by first principle?
That's the first thing and tell us more about what are the
skills or maybe techniques that you categorize as first
principle debugging techniques. Yeah.
So in the field of engineering as a whole, when we talk about
going back to First principles, what we mean is throw out
everything, you know, let's start from understanding the
system from the laws of physics or the laws of mathematics
instead of trying to read the manual.
To see what this machine says it does.
The machine is ultimately made up of a bunch of levers and
wires and so forth. So you can kind of Trace
everything back to understand. Okay, what does this lever do
now? What does the manual say this
lever does? Or what does this person?
Remember that? It'll ever?
Does you go and look at the Machinery coltd connected to the
back of this lever, to understand what it does.
I think that's the definition from the first principles to
Mummy is to the wrong, the book and use the laws of the system
wants. Of reality to understand it when
this comes to computer systems, but this fundamentally meant to
me initially my time at Google. And then my time to do
Cunningham is like, I start buying looking at the flow of
execution of one request. Try to understand.
What does spread since a normal request.
Look like, and what does a abnormal reflect where do they
differ? What services are they passing
through? Where are they spending the
kind? This kind of where I find this
idea of exemplars. Very useful, fine.
Meaning example, traces that exemplify both the slow path in
the past, that man just working them up on my skin.
Comparing them to try to understand whereas one passwords
one slow and then we'll bring to mind hypotheses my ideas about
why might these two things be different and then I can start
testing them. So, that's what I mean.
When I talk about first whistles to buddy.
It's not knowing magically what's graphite.
Which gas for to look at instead spending a little at a time,
puzzling out. What's the difference between
these two things based off of what I cannot observe about
them? I'm outside.
Thanks for that explanation. So we've been talking a lot
about the techniques. The internals.
What observability is? Tell us, more white
observability. Now become very hot.
Is it? Because it's just a trend, so
many tools available, or is it actually solving a real problem?
I think the answer and fortunately is yes to both and
it is a non overlapping set. So in terms of the definition of
observability gate, earlier of kind, of understanding complex
systems from Principles and being able to debug in the
puddles. The reason why we need this
today is because we are building some native microservices where
you can no longer use a traditional logging system, that
collapse a lunch from one host where your entire query runs or
where you can do a p.m. Along the entire post to
understand. What are the slow path on
Justice. My post that doesn't work
anymore. If you have microservices a
request that needs to flow through Services maintained by
more than one. A team at that point, you kind
of now have these squishy boundaries where no one person
holds that information about what's going on in their head.
So I think the motivation is that the complexity of systems
has meant that we can no longer the bugs to Stems based off of
the known and knowns the metrics that we thought to create an
advanced or kind of these magical, flashes of insight by
the expert in that, there is no longer an expert on every aspect
of your sister might need advance for not only like one or
two aspects of your system. Not how all the pieces fit
together. That's the motivate.
That's why it's our ability is crucial and needed.
The problem is that observability has you mentioned
three pillars earlier people who are already selling you
solutions to one or two or maybe even all three of these
so-called pillars are trying to persuade you.
That what they do, satisfies this requirement, but being able
to understand your complex systems.
So basically, it's three pillars of the durability, you can have
observability. If you just buy our logging,
Increasing and reference Solutions.
I think that's why he's seen so much marketing buzz and noise
about it. Is that everyone in the industry
historic calling? What they do observability, even
if it isn't this hilarious is like you see all these companies
and you know, make data observability that our light
source code observability and wait.
Does this work mean anything anymore?
So I think basically going to make history, honey, come first
started using the words, our ability in 2017, and it actually
predated Us by a little bit. I think the Twitter observe.
We'll beat increases by one or two years in terms of using the
system's control word, observability to spread this
idea of understanding systems and you'll see this explosion of
people like starting to call it a durability from 2019 onwards,
really that there's kind of this plethora of companies that say,
oh we do observability tool and it's like really, okay.
Well, let's see. Can you actually understand your
systems altogether or is this just a rebranding of your
existing monitoring and logging Solutions?
Yeah, sometimes this is also my confusion.
So, when you see some brands or some products called themselves
observability, most of them are like, maybe white labeling.
So, to speak right away. You can actually see locks
metrics and traces. Sometimes they are not
integrated. In fact, right?
So you'll see three different features and three different
things and you will correlate using a human intuition.
Yeah, and what's worse is they're making you pay because
throughout the day at three to four times, right?
Like they're showing you three different skus.
This should be one set of data so that there's not any Miss
lashes. You can jump.
Leave between 11 and so doesn't cost you an arm and a leg.
Yeah, the cost. But I agree because it can cause
you a lot. When you send a lot better.
I don't know, in your personal experience when working with
honeycomb or when you're meeting developers out there in my
experience, even though we have this microservices cognitive.
There are still people who actually believe it or not.
Do not Implement any Trace. Do not Implement any metrics or
has a very little locks implemented in their system.
That is more. Why is this still happening?
Even though we see bass Trends, but observability and all that.
And the reason is that, it works until it doesn't system.
Complexity is a very sneaky thing.
You think you can understand everything in your head and fate
of your that magical person that's able to debug everything
within 5 minutes until you get the thing that stops you that
takes 3 hours or until when you go on vacation and someone is
calling you because they can't figure out the system that you
built. So I think that's the reason why
people may not necessarily. We invest in observability when
they should be, is because the cost of inadequate,
observability sneaks up on you. It's a form of technical depth
to have a lack of observability. We know how good the industry is
about paying down technical data.
So I think that's what it is. I think the other unfortunately
says resetting is like, it feels really good to be the person who
magically debunked the problem. It feels like job security feels
like accomplishing something. If you're the most senior
engineer I met him, you know, everything about the system and
The one making decisions about the system.
You may not necessarily care quite as much as the engineer,
whose New Year. He man is struggling to
understand how to solve it together.
Now, if people have listened to this episode, they want to start
implementing observability, how they can do it.
Is it like go and buy or subscribe to our solutions that
are available out there install open source, maybe tell us more
how you should start. I think there should be just to
let. I think there's kind of the
foundational understanding of what observability is innocent
and you mentioned our book observability engineering, we
can drop a link in the show notes to a way to download a
free copy of the Missouri, the lady in here.
Here, so I would start by reading that book or at least
reading the first couple chapters just to make sure that
you have that language to explain to your key.
Why do you want them to change their practices?
You know, you figure out. It was the tools in the world,
right? This is what I discovered in
Google. We've had really good tracing at
Google. Since 2000.
Maybe no one used it. No one used it because it was
far to use and no one could see why they needed use it.
There is no easy linkage to people's existing debugging
workloads. So adding a new tool doesn't
necessarily. Things.
I must equal understand the motivation and understand how
they are going to use it. Once you have to be a lot in the
next step is to add the open Telemetry SDK to your
application. So what if open Telemetry open?
Telemetry is a vendor neutral, open source, SDK that allows you
to generate this club of free data.
Whether you metrics formatted log form, editor Trace
formatted. The idea is that it is a common
language for being Be able to produce this data to transmit
around, its you propagate that skeet and contacts between
different writer service. So when you add opportunity to
your application, you're not locking yourself into any
Pickers Lucien, you're making an investment in your future in the
same way that you're adding a new test framework.
Your application is an investment in your future.
You still want me to add some taps or you might need to add
some anyone's tradition, but open plumber.
Tree in general, does a good job of handling that that automatic
Generations faced a Your outpatient be something with
friedrichs that you're using, so you can have a kind of this Rich
data about what requests are going in and out with this brain
work either relationship. Then you'll have to pick a place
to send that data. And there are a white number of
options. You can certainly use open
source Solutions, like, Yeager the trouble with their
difference or Solutions is that. They are great for visualizing
individual traces, but that it's not necessarily going to provide
a comprehensive replacement for cured.
Receive monetary work floats because people often need to
understand what's the average yield by system as a whole.
And then zoom into that trace and gave birth in fulfill these
image of the trace. They don't it's about point.
You might want to look at vendor Solutions and you know,
certainly honeycomb is out there.
I also think very highly of our waitstaff, but basically any
back-end that supports open plum tree is going to be a place that
you can send that big issue and the bonus wasn't alone.
Free is. It's better neutral.
And it supports teeing the data you can They sent a more than
her career at the same time and see which one you like, which I
think is really great. I think competition is better
for the market. It's great for everyone and it
really incentivize his suspenders to do the right thing
by. Therefore.
I think that's why I recommend open Plum Tree because it could
have handles a lot of what would otherwise lead, wrote work
cream, Trace bands, propagating and palmitate around it.
It doesn't lock you in, it gives you that freedom of choice.
So thanks for the tips off, opting for open Telemetry open
Telemetry service like the New standard previously.
It's called open tracing the murderer, open census, and open
tracing. If you had two standards when it
works on at what we actually have managed to deprecate both
of its senses. And I've been tracing, we're not
doing that XKCD comic thing of now, there are 23 standards Alia
Eve, you choose for vendor Solutions.
Normally, they would charge you by the number of data being sent
to the systems. So tell us more bit of practical
choices yet. How do you assess Solutions was
of all? If let's say we want to go With
the vendor says based products. Yeah, I think that you have to
think about the cost of that. Is this how much is it going to
cost you and what are the benefits that you get?
So you may not necessarily want to go with the lowest cost
vendor because lowest-cost vendor might provide with a very
primitive ability to analyze the Bob.
I think a lot of favorites. Her investment from
observability comes from saving your in Heroes time, and
improving your customer outcomes and decreasing customers.
Sure that is far more important than the cost of the Beast
solution for instance. Eat your is free.
But how much time is it going to say?
Do you have the bugging? Really?
So I think there's kind of this Continuum and it's important to
focus on one or your evaluation criteria and to specify a front.
We want to be able to understand issues within half an hour, or
we want to be able to measure or serviceable decades in the first
place even understand where we're going wrong.
So I think that's kind of one dimension and I think that go
along with that. How intuitive is it?
Is everyone on my team adopting it.
So after we Choose all these open Telemetry.
We know which vendor solution. I think at the end of the day.
The developers themselves need to instrument the code.
That's correct. The Ottomans religion and only
get you so far. But other than some patients
never going to capture things through post bodies because
that's how you get X have for fleets right after you have to
pick and choose which attributes you want to send along and the
answer is anything nonsense that basically.
But the Tormentor chemicals, not going to be able to do not
automatically. Yeah.
So although instrumentation can do that automatically.
For up till a certain level, but I think at the end of the day,
developers needs to have a conscious decision to instrument
the code, Click by gravity would laugh?
At someone said, I can automatically create your cast
for you or I can automatically convert code for you.
It's like, no, you can't. Yeah, which brings to the
technique that you mentioned the book.
We have heard about these term, a lot of times driven
development. Right?
So you coined, this observability driven development
and shifting left for observability.
So, tell us more about this technique, this concept, why is
it important? Yeah, I think it's super
important to understand. What is this code going to look
like in production? So the best way is to add that
Telemetry to add those key value pairs into your code.
As you're developing it that way, you can test it inside your
test Suites that way. Inside of your death box.
You can admit that Telemetry data to a tracing people and to
see what was traced. And look like it.
Hey, you might discover, why your tests are failing not by a
kind of running a debugger, but instead by looking at At this
Pharmacy data. So the earlier you adopt
observability injured in all /, lifecycle.
The mortar developers will use it at all stages of development.
Not just at production and we see her you'll catch box.
So that's the argument is that it's really synergistic with
kept certain development. But you're also exercising your
observability code during your tests.
Yeah. So if I can imagine, I'm being a
developer again, so as and when I try to develop a feature, I
have to think about what kind of things that I want to.
Able to observe or maybe look at in my of the Villa T tools so
that when an issue happens, I actually have those data and
then the I implement or instrumented in my code.
I think that's really key. Another thing that you
mentioned, if developers want to have more conscious decision of
adding instrumentation is putting them on call hence the
but their own system. So that's why this is actually
very important. Yeah.
So I think that it is important for developers to have seven
steak in the production writing at their application.
However, that does not Not necessarily have to take the
form of on called, for every engineer, but I think at the
team level is important for kids to build and run your own
topper. So in practice, a lot of teams
struggle with that because you just say, you know, here's the
Peter congratulations, as opposed to, we're going to
support you with high quality observability with that slows of
him. This methodology is to how you
are to service. Now that we've trained, you
pierce that hater. I think that if a lot of
developers understood the payment options, go through
maybe amount of Sound lessons that we formed from years of
being subjected to am radios and kind of almost the masochism
that requires. They might have a little bit
more empathy, but it would take time.
You choose to throw someone who did repairs directly into that
these. We cleaned it up a little bit
before you say hi. Give us a favor by the way.
So, I think kind of that handoff process can be really helpful
for encouraging ownership, but ownership, and responsibility
needs to be accompanied by giving people, the tools that
they need to be able to be successful.
You wouldn't Assign a brand new developer, to design, the
architecture of your future system.
So why are you asking people who have never operated system
before to go swim in the deep end.
I think you kind of have to offer this graceful cat and I
think that observability really is this methodology for kind of
Bridging the Ops and deverill. See are the reason why people
did monitoring before was that monitoring is outside an
observation or outdated measurement because often
operators didn't really necessarily have the ability to
make coaching to system, so that Were limited only to you.
What can my ATM agent matter or what can my monitoring tool
there? So I think that when you have
this shared responsibility of developing the two out of three
and observing and look a planetary at some Libre de Su
World scanner because you're having the instrumentation being
added in this virtuous cycle along with looking at the
results of the its rotation. So you mentioned all these build
and run it it comes back to normally the devops culture and
SRE culture, right? So is this observability also?
Supports the implementation of that particular devops and SRE
culture. Yes, I think so, when you
originally solve some of my videos on S3 and devops.
I highlighted that it is a key responsibility of s res to make
the system debugger to make this system have slos and to monitor.
The disclosed at Google, we certainly didn't use the word
observability until 2019. But what we were doing was
probably closer to observe the latent monitoring husband
Europe, doing this practice. Record Allison and being able to
understand the systems. So for people who have been
implementing this, how do they know they have done it right?
Where they are lacking. I see in your book.
You mentioned this term called observability maturity model.
Is there a way to gauge where people are at this point in time
and what else can they aspire to achieve?
Yeah. I think that there are these
candid key find areas in that Observatory returning model of
where observability isn't isn't helping you.
We talked about observable age of them.
Velopment and kind of getting the code, right the first time
and exercising that it's relation have to really that's
kind of what area of Eternity. The other Airmen material that
we actually didn't touch on yet is TOS delivery for me.
Observability has a you must be this tall to ride that is.
You can skip code every couple of weeks at least because if
not, does it matter that you decrease your time to resolve
issues from three hours 25 minutes.
If you can only ship code every three months or if you can only
shoot view, its rotation, every three months, but no, it doesn't
make sense. Focus on making your glitter.
Jacob Astor. So, kind of, if you are
struggling with your code and be able to ship it, start the
observing your deliberate. I try not your code.
So that's kind of another to maturity area.
Is do we had our skin white or build a sudden late surge.
He can longer to, in our case, 15 minutes, 15 minutes of the
heart up were, oh my God. This is too slow.
So we try to keep our builds under 10 minutes so we can
deploy to production once an hour faster.
So kind of keeping that stuff for delivery cycle Snappy.
We're getting it should be in snappiest kind of another key
area and Then third, we talked a lot about the brakes and
production resilience workflow. But I think also, it's really
important to think about the user analytics to what extent
are people making use of your product.
If no one is using it. Does it really matter these
shifts, that feature? And then finally, I think to tie
it all together. Technical debt and managing that
technical debt is really important and observable a
conservative. Hey, you have the single point
of failure or hey, you have the circular dependency.
So I think that those are some of the key responsibilities that
you have when you're kind of thinking about how to add
Observer. What each shooter suffered
delivery facts? Thanks for explaining about this
maturity model. I think you explained it much
deeper in the book as well. For people who wants to
understand. Where are these areas that you
need to invest time in? So make sure to check the book
will put it in the Show links later.
So let's find myself having a crash course about this
observability. So, thank you so much for
explaining all these Concepts, all these implementation.
Really, thank you for that. But unfortunately, due to time,
we need to wrap up soon. But before I let you go, I
normally ask one last question for every guest that I have in
the show, which is to At three technical leadership, wisdom for
people to learn from your experience or from your journey.
So maybe you can share us. What are your wisdom?
Sure. I'd like to give a brief and
just leave it to one piece of wisdom, which is that it is
much, much more important to make the social piece work in
this happens. It's really important to get
people talking to each other software.
Delivery is much more about people communicating and being
on the same page and kind of having a cheer working model.
It is about what tools you use. So don't think first about
introducing new tools. Think first about align people
on the outcome and making sure everyone is agreed on the
outcome and then you can worry about what tools you're going to
use to implement it. Wow.
That's really wonderful. So social first rather than the
technology aspect and try to align people because yeah, I can
see people implementing different Technologies, even
when you talk about micro service, right?
It's like everyone just wants to have their own services manager
themselves and They don't really care whether it integrates well
and serves the customers. Well, so, thank you so much for
this for your time for people who want to follow you or learn
more about this observability or honeycomb when they can reach
you online. Yeah, I am.
There's the great pretty much everywhere.
So that's on Twitter. That's a good hon.
That's how they look me up. I am will also drop those links
into the show notes. Why is it the gray if I may ask
teenage fangirling about Gandalf from Lord of the Rings?
Alright, so, thank you. Let's hope you have a great day
today. Thank you.
Cheers. Thank you for listening to this
episode and for staying, right until the end if you highly
enjoyed it, I would appreciate if you share it with your
friends and colleagues who you think would also benefit from
listening to this episode. And if you are new to the
podcast, make sure to subscribe and leave me your valuable
review and feedback. It helps me a lot in order to
grow this podcast better. You can also find the full show
notes of this conversation on the episode page at pack Legion
o.f website, including the full transcript.
Interesting quotes and links to the resources mention from the
conversation. And lastly, make sure to
subscribe to the shows mailing list on pack leader dot f to get
notified for any future episodes.
Stay tuned for the next technology.
No episode. And until then.
Goodbye.