High performing team is one the kids to spend almost all of
their time solving interesting problems that move the business
forward, not doing a lot of toil not like working on things that
they have to do in order to get to the things that they want to
do. Not slogging through fixing the
bill. Begin not slogging through
outages, not waiting for test to pass, not waiting for our
deploys to finish fixing something and then realizing it
was a different problem. I think we tend to take for
granted that it just it has to be that way.
If you get your shit in order you you can have a job where you
spend most of your time solving new and interesting problems.
Hey everyone. My name is Henry Surya be Robin.
And you're listening to the tekhelet Juno, the show will be
bringing you the greatest technical leaders practitioners
and thought leaders in the industry to discuss about their
Journey ideas and practices that we all can learn and apply to
build a highly performing technical team and to make an
impact in your personal work. So let's dive into our Journal.
Hello everyone. I'm back here again with another
new episode of the technology, you know podcast.
Thank you so much for tuning in and spending your time with me
today listening to this episode. If you're new to the podcast
know that package, you know, is available for you to subscribe
on major podcast apps such as Spotify and apple podcast Google
podcast, YouTube, and many others also, check out and
follow technology. No social media channels on
LinkedIn, Twitter and Instagram. Everyday, I post quotes and
words of wisdom from the recent podcast episode, and I share
them on those channels to give us some inspiration and
motivation for us to get better each day.
And if you are thinking about making some contribution to the
show and support the creation of this podcast, please consider
joining as a patron by visiting technology.
No dot f /, Patron. I highly appreciate any kind of
support and your contribution would help me towards
sustainably producing this show. Every week for today's episode.
I am extremely excited to share my conversation with Charity
Majors charity. Is the co-founder and CTO of
honeycomb, the observability tools for engineering teams to
debug production systems faster. And smarter charity is a
frequent speaker. And is someone that I admire a
lot, not just for her thought leadership on devops
observability, distributed systems, Etc, but also for her,
Opinions on multiple other topics, such as management,
engineering startup culture, diversity, and many more which
you can also find and follow on her Twitter and personal blog in
this episode charity. And I discussed in depth about
building. High performing teams by having
observability and CI CD. As the critical pillars to
support it. We open up our discussion
discussing about what observability is how it diverse
from me. Monitoring and why the
traditional monitoring tools and pre Aggregates are not
sufficient to help us understand our systems better and to
uncover the unknown unknowns charity shares.
How honeycomb provides unique capabilities that can help
provide observability for distributed systems and how it
works, including an interesting use case that can provide
observability for CI CD pipelines.
Then we moved on to discuss about building, high-performing
engineering teams. Charity, share, it has strong
views on how to build such teams by focusing on continuous
delivery, carrying a lot about the socio technical aspects of a
team and the fifth key metric. As her addition to the widely
known Dora metrics towards the end.
We discuss the engineer manager, pendulum based on her popular
blog post some time back and charity.
Shared what she means by the pendulum and how one should be
conscious about it. She gave her word.
Of advice and explain that going into management, is not a one
way street and that we should also not treat it as a
promotion. This is such an amazing episode
that you don't want to miss and I'm certain that you will also
enjoy this episode if you like it.
Considered helping the show by living it a rating review or
comment on your podcast app or social media channels, those
reviews and comments are one of the best ways to help me get
this podcast to reach more listeners and hopefully they can
also benefit from the contents. In this podcast.
Let's get this episode started right after our short sponsor
message. Are you looking for a new cool
swag tekhelet Journal. Now offers you some swags that
you can purchase online. These wax are printed on demand
based on your preference and will be delivered safely to you
all over the world where shipping is available.
Check out all the cool strikes available by visiting
technology, you know dot, f / shop and don't forget to break
yourself. Once you receive any of those
wrecks. Hey everyone, welcome to another
new episode of the technology. You know today.
I'm very excited to have someone that I really admire from the
devops technology side. Her name is Charity Majors.
I'm sure many of you would have recognized her name today.
We'll be talking a lot about observability, monitoring CI CD
and a few other topics that she is famous for in terms of blog
posts. Charity is a CTO and co-founder
of honeycomb developer tools, that helps to improve the
observability of your application.
She also writes a great Posts at charity WTF and co-host, a
popular podcast, called only cats, which I think is one of
the best observability of monitoring podcast available out
there for charity. Thanks for your time today.
Hope to have a good conversation with you.
Yeah. Thanks so much for having me.
I'm really excited. So charity and maybe in the
first place. Let us hear more from you about
your career Journey, about your highlights and turning points.
Yeah. Well, I'm kind of an accidental
technologist. I went to college for for
classical piano performance when I was 15 and I never had a
computer when I was growing up. I never even had email until I
went to college, but I didn't want to be poor.
I didn't want to be broke and I found myself hanging out in the
computer lab a lot. I never actually had a degree in
computer science. I had majored in classical
studies Latin and Greek and creative writing and philosophy,
but I found that I was spending more and more time working in
the lab than I was doing any of my schoolwork.
I dropped out a couple of X came to San Francisco.
I've been working at startups ever since I've made a niche for
myself, being an early or I like to be the first infrastructure
higher that joins a team of software Engineers when they
think that they've got something that might be real.
And I like to help them help it grow up.
And then I typically start hiring a team, then I manage the
team. Then there comes a point where
I'm like, well, I've got everything I do here and I find
another startup, but I do it again.
So that's what I was doing when I joined parse, which is a
mobile backend for mobile developers, and that was about
six years ago. I was there before beta got
acquired by Facebook. I stayed at Facebook for two and
a half years give you a little bit of context.
Parse was we were doing a lot of like microservices before there
were microservices. We were doing this really hard
code tenancy problems. We had over a million apps
running on purse with one database in one pool of app
workers. And we were going down
constantly. It was really professionally.
Embarrassing to me because every day be like, a new app.
Would hit the iTunes. Top ten parts we go down.
I tried every monitoring tool. Every logging tool, every AP.
Tool tried everything and nothing helped because they were
all dealing with metrics and pre Aggregates pre Aggregates are
great. When you know what questions
you're going to need to ask. Can you capture that data in
that way, but when you have to ask different questions every
day and you have no idea what questions are going to need to
ask, like, all of the existing tools just fell apart.
So, there was one tool that Facebook called scuba that we
started shipping some data sets into.
And the time it took us to figure out which apple is the
root cause of us going down or whatever, just dropped like a
rock. It could be hours or days.
It was like opening. And ended like who knew to like
predictably it was s not even minutes, but every time we would
just like start slicing and dicing we could find the answer,
this made a huge impression on me.
And when I was leaving Facebook, I was like, oh my God, I can't
go back to being without this tooling.
I was supposed to be going to be an engineering manager at soccer
stripe or something. And I was just like this needs
to exist because everyone is starting to have this problem.
Everyone is starting to this problem of high cardinality and
high dimensionality, and there's no tool out there that exists.
So we started building the tools, Would become honeycomb.
We had to start by writing our own database effectively because
there is nothing out there that could do this and that was
difficult, but it wasn't as difficult as trying to figure
out the language because we knew we weren't building a monitoring
tool. We knew that it was something
very fundamentally different that it was, you know, for
helping you develop your application and it was about six
months into having start honeycomb.
When I happened to Google the term observability, which no one
was using at the time when I read the definition which comes
from like mechanical engineering and construction.
Troll Theory and it has to do with how well, can you
understand any state that your system can get into just by
looking at it for me outside and I went, oh my God, but light
bulbs are appeared in my head. This is what we're trying to do.
This is what we're trying to build.
We're trying to build a way to understand your system just by
asking questions from the outside without having to ship
any custom code to understand the problem, which implies that
you could have known it was going to be in advance.
So it's all about the unknown unknowns and it's all about
being able to infinitely slice and dice the Attila.
That's coming out. We started talking about it a
lot and for the first two years or so.
It was like, just nothing. Everybody was like, we've been
told this is impossible. We believe that you're lying,
but blah, and about three years into doing honeycomb.
One day. It was just like everything
changed. I woke up the next morning and
the entire world was like, but of course, we need
observability. We have always wanted
observability. And by the way, we do
observability to number was just like, oh my God.
No, so there's been a little bit of a bandwagon effect.
Now, the frustrating thing is everybody's like, yeah, we do
observe ability to just like, well, no, they're supposed to be
very specific technical definition because if you say
you want to be able to ask any question of your systems, some
things proceed from that some technical prerequisites
perceiving that you have to be able to handle high, cardinality
and high, dimensionality, and you have to gather the Telemetry
in such a way that you have one, arbitrarily wife tortured event
per request per service. And I've written all these blog
posts and stuff. The people who follow it closely
understand. But like the rest of the markers
just like how observability is a new synonym for Telemetry?
Got it, and I'm just like, okay. Well, this is better.
This is progress. You know, yep is coming in
technology industry, any new. Jargon, any new buzzword,
everyone taste used. So if you can help us, like
maybe share from your point of view.
What is observability? And how does it differ from
monitoring? Which, what we are used to?
Yeah, I think it's really important.
That people like understand the way it's different because
monitoring is a very mature. It's an important discipline of
its own and there's a lot of best practices and stuff that go
along with it. I think that the metrics tooling
is great, it fulfills a separate need In the ecosystem from
observability. Monitoring is really about
taking from the perspective of the service and saying is this
service objectively speaking? Healthy is a service healthy.
Do I need to plan for more capacity?
What's my consumption percentage?
It's usually built on a structure of metrics.
There are the generic term for metric, which just Telemetry.
But the specific technical term for metric is just it's a number
you can append tags to. So like it's really cheap.
You can store a lot of metrics very efficiently, you can age
them out, as they get older, you can drop some detail and you can
Your system for a very long time.
The source of Truth for observability is not metrics
because when you capture that one metric that would number.
You've discarded all of the context except for a few tags,
which you bring back. So for example, say you have a
spike of errors in your graph. What's wrong?
Well, what's up durability, you might slice and dice to figure
out. Oh, these errors are for all the
requests are coming from this version of iOS.
From this end point from this language pack from this version
of the firmware from this build ID, just like the long string of
things that like this is what's different about them.
The well, you can't do that with metrics unless you happen to an
advanced capture a metric for this version of iOS this build
ID and the ability is always incrementing and always
changing. So you have to capture those
specific numbers in advance with metrics.
Then with observability. You've captured it in this very
wide event, which captures all that context and it says all of
these things are linked together by this one request from the
users perspective. So you can then ask any
combination of question from that on it.
So monitor is about from the perspective of the service.
Observability is Perspective of the user.
What is the users experience? Like which I think is super
important. So the user for monitoring and
metrics is typically Ops. The user for observability is
typically people who are writing code, people who are shipping
code every day, who want to understand the magical
conjunction between your code, your infrastructure, this point
in time, and that users experience.
So, I think to me, when I hear about it, it seems like it
requires a lot of mindset change.
First of all, like, capturing. All these hide.
Mentionable High cardinality metrics and also like
implementing that. So what do you think someone if
they want to explore this observability further?
What should they do to change their mindset?
Well, I think the shortcut is often.
Just seeing your own system, your own data in and
observability Tool which is why honeycomb has a really generous
free tier because we want people to be able to sign up play
around with it. Even if they don't become our
users. I still want them to like
understand the Mind shift and experience the difference.
So I think that's the shortest quickest route to experience.
It to understanding it. There's a lot of things will
just click if you're used to trying to understand your own
system and you see it in different tool.
You already know the fields your landscape.
And so it's a pretty quick and easy way.
Another way of thinking about this is that this became
necessary when the monolith began to decompose because if
follows failed with the model, if you could always attach a
debugger and step through the code.
Well, now you've blown up the monolith and God knows how many
services and every time it pops the network.
Whoops. You've lost all the context.
You can no longer Trace that code anymore, which is why When
you're Gathering the Telemetry for observability, the way we do
it. Now, you come is we initialize a
honeycomb event as soon as request enters the service and
then we pre populate it with everything.
We know about the environment, the language, the internals go
routines, anything about the context that was passed in
parameters. And then, while it's executing
in that service, you as a developer can go.
Oh this might be useful stuff it in.
We also wrap a lot of common functions database calls.
So any time you do a database called, we wrap it and we record
the time it took and The Rock query.
And the Nice query and results. And every time you do, if you
request, we wrap it and we give you the duration, and the
result, and all that stuff. And then when the request is
ready to exit or error from that service, it gets packaged up
into that one arbitrarily wide track your data blob and shipped
off to a honeycomb. So you can think of this as
like, packaging packaging along all of that context, that it
needs every time a pops, the wire.
Well, now you've got that context because we're passing it
along with request. That's pretty cool.
So, in the first place who is responsible for implementing
this, because in the beginning, you said, Said, it should be
developers. First of all, there will be the
main users of this. But as we all know,
observability monitoring these days are highly associated with
Ops, or the devops site. So, who should implement this
and how should someone actually think about?
Okay? What kind of questions that I
would ask about my system? And where does this?
Observability might be useful for my system down the road.
Absolutely anyone in the chain can take responsibility for
this. I think that most commonly, we
have Ops teams who get it. Set up on behalf of Developers
and start to teach them which is I think a really beautiful
thing. I think that Ops teams are not
becoming any less necessary and increasingly their job is not to
make the code run, its to help developers understand.
This is like the second wave of devops, right?
First wave of devops is like AA. Simple, you must learn to write
codes like, okay. Okay, we get the message and now
it's like, okay, software Engineers, it's your turn.
It's your turn to learn to write operable Services.
It's your turn to learn, to support the services, and to
build systems that are Not too expensive to maintain but we're
here for you. We will help you.
So I think that sometimes it's the developers who take it upon
themselves to bring observability and often.
It's the Ops teams who bring it in, on behalf of their
developers often. It's execs, who are like they've
been paying attention and they've got their ear to the
ground. They know that systems are
becoming impossible to run and scale and maintained without the
sort of tooling because forever the way that we've been doing is
has been by guessing what we see something wrong.
We guess based on past outages and battle scars were like, I
think it's this, right? And then we go and look for
evidence that were right, that works really.
Well. When you have a lot of people
who've been there for a very long time who have a lot of
context in their heads and the systems aren't changing that
much. But when the system starts
breaking in a novel way every single day, it doesn't, do you
much good? So I think that it doesn't
matter who takes it upon themselves to do this as long as
someone does, because it is very important because people burn
out teams. This is an ethical issue.
I believe this is a Humane it. Issue, we've all grown up
working in the tech industry accepting a certain amount of
self-abuse just like masochism, an Ops is just Infamous for
this. Well, throw ourselves on the
grenade, drag ourselves into the meetings.
Just like in the morning, like, I've been up all night.
It's like a badge of honor for us, which it was fun.
I enjoyed being the wizard who points at a graph and goes, so
that's super fun, but it doesn't scale very well.
And our systems increasingly don't reward.
That kind of magical thinking especially these days when
people are using lots and More microservices distributed
systems even platforms like Cloud where you have so many
services first, because whenever you have a platform, you're
inviting your users chaos to live at your house and you're
responsible for it. So can this observation T
concept be useful monitoring business metrics as well.
I think that the separation between business and Technical
metrics is pretty arbitrary and detrimental.
In fact, I think that you want everyone to be reading off the
same answer sheet. You want everyone in the company
to have you seen view of reality.
Reality and to the extent that you can all be looking at the
same numbers. Well, that's just a lot of
argument. You don't have to have.
I believe that tools can create silos because the borders of
your tool is the borders of your knowledge.
I've been in so many meetings, where you spend more time.
Arguing about what was true because of what your tool said
versus the actual problem that you're trying to solve together.
So maybe if you can mention few attributes, I know you have
metrics, you have maybe logging and you maybe have traces likes
pens and things like that. What are the things that are?
Crucial in any distributor systems these days.
I mean metrics logs and traces are just data formats and you
can derive all of them from the arbitrarily wide structured data
blob that I was just describing. You can derive your metrics from
it. You can drive your log lines
from it. You can direct your spans from
it. You can't go in the other
direction. You can't go from a metric to a
white truck. Sure blood.
You can't go from a log line, to a wide structure blog.
So I really think that we need to stop thinking about this in
terms of the archaic tooling that we've inherited and start
thinking about At it, in terms of the use, we get out of the
system. And I feel like the move to
these arbitrarily white shark today to blogs is inevitable,
but it also shouldn't be a big deal for people.
Right? If you squint, they're just
logs. They're just structured that
race is just a wide log line that has a couple of dedicated
expand ID fields, and Trace ID Fields.
So I think it's kind of a red herring.
It pisses me off that so many Legacy tools are just like you
definitely metrics logs and traces, why because they happen
to have metrics products, logging products and tracing
products that they What to sell you and if they're selling you
those three products and you're paying just saved your data,
three different places three different ways and they can't
talk to each other. So I see this as a really
cynical Ploy on behalf of the entrenched Legacy, this Punk's
in the new relics and that data dogs are the world.
They count up all the products. They want to sell you in there.
Like this is how many you need and it's like, no you don't.
What you need is the functionality.
Maybe you can share a little bit more about honeycomb for people
who haven't used it before. I must have haven't used it
personally, but I heard a lot of rave reviews about it.
So maybe we can share a little bit how honeycomb can help and
what are some of those tooling? Like what you said, they sell
you as a suite of products could not achieve compatibility comb.
Well, so, you know, there's a narcissism of small differences.
Of course, we're all trying to solve the same problems here.
So like for example, we usually start all of us, universally.
We usually start our debugging experience by looking at metrics
graph. We see a spike in errors of my
God. There's a problem then because
you can't slice and dice metrics arbitrarily, you typically jump
over to your lung. Tool and you start just trying
to visually correlate. Well, that spiked hair is looks
like this over here. It's not really science.
But you know, you try and find the same thing around the same
time, then maybe you want to trace it or something.
So you pick out a tray side you pick out a request ID and then
you jump over to your tracing tool and you paste it and you
hope that one got captured. And so you're just visually
jumping around and you the human or in the middle of the glue.
Well, you do exactly the same thing and how do you home only?
You don't jump around you just you look at the graph.
You see there's a spike or something.
Well, the way we don't honeycomb is he Just draw a little box
around, this is what I'm interested in and then we can
pre-compute for you for all the dimensions inside the box.
He said, you're interested in and outside the Baseline, and
then we dip them and then we sort them.
So, all the ones that are different rise up to the top,
and then you can click on one and just say, she'll be this
traced to me, example, trace for this.
So you're doing exactly the same thing but you know that you're
dealing with the same data because you're not jumping
around. The cool thing.
Is it? Because traces and moments were
just like two sides of the same coin.
Well, it turns out that Tracy was just visualizing by time.
So you're dealing with it. The same events, exactly the
same data stored, only one time, but you're able to turn it
around, inspect it in all the different ways to try and find
the anomalies. I think, the Bubble Up thing
that we've done is like, it's a game changer because what it did
was it replaced the manual process of what we used to do at
parse was scuba. What we used to do is go.
Okay. There's a spike.
Well, let's see. Is it for all and points?
No, it's just for the endpoints that are like in points.
Well, is it for all the right and points?
No, it's just the ones that are writing to say mongodb.
Well is It to all the mongodb and buts.
Well, no, it's only airing to Feast three primaries.
Weld, what do they have in common?
Or is it to all of them? Well, no, it's just this
application idea. You're following the trail of
breadcrumbs, which is let me point out.
Vastly preferable to guessing and looking for evidence of.
You're right. Right here.
You're actually following the trail of bread crumbs to the
answer every time, but Bubble Up did is it just replace that
whole iterative process because we pre compute them and we show
you them all at once yours like Dropbox group.
There's the answer, which is pretty cool.
This is where we hear a lot these days about AI Ops and like
machine learning and all this bullshit.
I have lots of strong feelings on this topic, but I feel like
I've never want to take the human out of the driver's seat,
but I do want to look for ways for to make machines.
What machines are good at machines are good at crunching,
lots of numbers to let humans do what humans are good at which is
attaching meaning to things. If you see a spike, you don't
actually know if that was good or bad or something.
Expected to see something you wanted to see, you don't know
that until the human is attached me to that.
And then the Computer can crunch a bunch of numbers and back up
the meaning assigned to it. So I don't want any developer to
give up that centrality of be in the driver's seat.
But I do want to look for ways to take out the repetitive.
The loop just like is it this is it that we can collapse a lot of
that work so that you just have to point and click and
understand the way you described it actually sounds to me as if
like, I'm Googling something right?
No problem. I start from google.com such one
thing and it brings me to more and more links until I get Yeah,
so it's about starting up high and methodically working your
way to the answer is a process that we're all familiar with and
our tools just haven't supported it.
We've just had to make these wildly too fancy.
One. Other thing.
I will point out is that something philosophically a
honeycomb that we really believe it is the power of the team
because whenever we're working on these complex security
systems. I'm an expert in my corner of
it, that service that I'm writing right now.
Like the thing that I'm building.
I know everything about it. I know the variable names.
I know rather Gremlins live, but I don't know your Art nearly as
well yet. When I'm debugging, I have to
take the whole system into account.
So we talked a lot about how can we bring everyone up to the
level of the best debugger and every single corner of the
system? Well, we can do that by
piggybacking on the way that I interact with my system as an
expert and way you interact with your system is an expert.
So if I get paged about something, maybe it's looks like
a, my SQL problem. And I don't know anything about
my sequel, but I know that the experts on the team are like,
Ben and Emily. And I feel like we had an outage
like this like last Thanksgiving.
So I just want to go look at like Any of them on colic?
What did they do? What was meaningful enough to
them? That they would run it a bunch
of times. Right?
What did they tag and put into a post-mortem?
What did they add a comment to, or name and like putting their
personal arsenal of useful links?
You should wear grooves in the system as you use it and you
should be able to draw on the collective wisdom of your team
as experts in the system. I trust that so much more than I
would ever trust any a. I so if I understand correctly,
the way it works, is you instrument your coat, your
services to give you this blob of data.
Is there any other mechanism that you use?
And how about the platforms that you mentioned?
So how do you instrument the platforms?
Like the database is the cloud platform that you use?
Well, observability is fundamentally about the first
person perspective. It's about writing a library for
your code. Being the first person actor.
Who's this is what I am doing. This is what I am seeing, but
data is data and it's better to have second best data that no
data at all. When you've got say a database
where you're my instrumenting it and it's not your code.
You do want Some ways to get Telemetry out of it.
The easy answer is, will accept any structured data.
You can throw at us. It's most useful to you as a
debugger. If you've gathered into one or
trailer wide structure. Data Bob per request for
service. But, you know, if you have an
elk stack, you can just put a little stanza in your log.
Stash thing that will just duplicate the output streams and
send us everything. You're sending to elq.
That can be a really nice little bridge for people as they're
getting started because they already knows this data.
They know how to navigate it. You don't have to do anything
change any code that way for things like MySQL.
I'm a databases. Right.
So this is an old school shit. But what we do is we get three
sources of data for my SQL or mongodb or whatever.
We get the slow query log because there's some information
that is only sent to the slow query log.
We also stream over the wire, we do a tcpdump extreme over the
wire and we reconstruct all the transactions and then we ship
each transaction in to Honeycomb as an event.
Because there's some information, you can only get
that way. And thirdly, we also run a Cron
job that connects every couple seconds to the command line
interface and just dumps out all the internal stats.
Ship those in as a beds to those three sources of data.
That's basically the sum total of everything.
You can get out of these databases and it's frustrating
because they clearly weren't designed, in a way to just, like
output all the information that we would like them to have and
welcome place, but you can get a pretty good approximation with
those three sources. So I'm sure you have plenty of
these, right? But any kind of cool use cases
that maybe your customers have shared.
The coolest thing. That was a big surprise to me
was when one of our customers, your to ago made it.
Library called build events for build pipelines and what they
did was basically, they made it so that you can instrument, your
CI, CD pipeline as a trace. Just like your Trace.
So they like overloaded that a little bit if you put the build
of its library in your circle or Travis or Jenkins or whatever,
then it will wrap. So it shows each test when it
fires off and when it finishes you can see how well the
paralyzation is doing, or if it's all sequential or where the
time is going. That's so amazing because
Bringing Down the amount of time.
Takes to build and ship. Your code is so Central to I
think every team really should be paying attention to this but
there's it hasn't been a good way to visualize it and I
thought that was just fucking brilliant.
It's such a good use of visualization by time and that
had never occurred to me. So now we try and get people to
do it all the time because it's such a great gateway, drug to
observability because we're Engineers.
We love to figure things out. We love spotting and
efficiencies and some things seemed really hard and tough,
but that you just see them in the right tools.
Like, oh, duh. I can fix this, but I think
almost no one has this experience with the build events
are like, oh my God, that's where all my time is going and
they go and fix it and it's just you get such a dope, what you
did, such a high off of, it's just, it's so fun.
I never imagined it that way. But yeah, now that you mention
it, I think it's pretty cool. If we can have that data,
especially if you can correlate with the real production system,
but what happened? During build time what happened
during development? Maybe can't ask mrs.
Behave differently during that time.
Yeah, so that's pretty cool. So you mentioned in the
beginning as well that you have to write your own database and
we can share a little bit. Because since you are database
person, why the current database tuning is not good enough for
you. It comes down to the fact that
the trade-offs that we wanted to make our not trade-offs that
anyone else would make for our purposes fast and mostly, right
is better than accurate. So, uh, we're fine with just
throwing away, some very small tail of data in order to get
like sub. S query Layton sees, this is the
other theme of honeycomb. It's fuckin fast.
The 95th percentile for queries is like under a second and this
is really important because when you're debugging, You want to be
in a state of flow, wouldn't be asking this this this this can
imagine with Google took five minutes to return to search.
You be planning in advance. You be like, oh God.
Okay. I'm going to run this query and
then I'm going to go to the bathroom.
That's not good. So we need to really fast.
You're okay. With discarding some data.
We also knew that it was very core that we didn't want there
to be any schemas. We wanted people to be able to
throw in more Dimensions. Any point in time, a maturely
instrumented Services. Honeycomb will have 300 400 500
Dimensions. So the events are that wide.
We don't want the friction of, okay.
Now I'm going to add another one.
We want people to just be able to throw it in, forget about it.
Stop sending it, forget about it.
So no scheming changes. No indexes, because indexes
again, they violate the theory of durability, which is that,
you shouldn't have to predict what's going to be fast.
What you're going to need to ask if you'd all be equally fast to
query it on. So that meant that we needed to
build a columnist or they're not a lot of Open Source freely
available columnar scores. We also knew that we needed the
data to be very distributed. We didn't want to spend all of
our time managing the database. A store like adding databases
for each user, the way it stored is actually it's cool.
So when data comes in through the API, the events get dropped
into Kafka into a topic just for you, your account.
And then the topic is consumed by n numbers of retrievers.
All of our sources are named after dogs, because we're a dog
company. So the retriever is just a pair
of them for each for, each will consume off the queue and then a
thing that we did just unless you're too, we actually made our
database, go serverless. So like At first we were storing
them all on ssds and am. And then we were like would it
be cool if it was just an S3? We thought that the performance
hit would be huge but it wasn't so now like the queries actually
they run as Lambda jobs and the data gets aged out to S3 after a
few hours on disk and eventually I think we'd like to make it so
that you can choose to age it out to your S3, buckets.
So that they're really just hosted on, you know, you can
post it forever for free. We have 60 days of storage for
free, which I think is the best in the industry and if we're
able to do that because we were able to just Outsource a desk.
3r. It's really cheap.
Wow. Sounds really cool.
Thanks for sharing that, of course.
So charity, you mentioned. Also, you're passionate about
building High performing team lately.
I've seen one of your talk about CI CD and also the blog post.
So can you tell us a little bit more about?
What do you think about high-performing team?
What is actually a high performing team versus the low
performing on a high performing? Team is one the kids to spend
almost all of their time solving interesting problems that move
the business forward, not doing a lot of toil not Working on
things that they have to do in order to get to the things that
they want to do. Not slogging through fixing the
bill. Begin, not slogging through
outages, not waiting for test to pass, not waiting for our
deploys to finish fixing something and then realizing it
was a different problem. Just like all of the stuff that
makes this job frustrating sometimes I think we tend to
take for granted that it just it has to be that way.
I've now worked on a few High performing team for it.
Just it doesn't if you get your shit in order, you can have a
job where you spend most of your time solving new and Interesting
problems. And that is just it's a joy,
right? It brings a joy to your work
that is just it's Indescribable. And in order to get there.
I think that focusing on the door metrics is a really good
place to start. How often you deploy.
How long does it take? How often is it fail?
How is it take to restore? Time to recovery and a lot of
improving. Those metrics, involves leveling
up and observability seeing what you're doing, like, putting in
your glasses before you go and you drive down the freeway and a
lot of teams. They're really Flying Blind.
That means that a lot of their work is wasted.
I also feel like It's important to note that high performing
teams. They need to feel emotional
safety. They need to support each other.
They need to be kind and the high performing teams.
You don't get them by just hiring the best people or hiring
the best Engineers. You don't.
In fact, some of the people, I've known, who are the best
individual Engineers are pretty toxic to team's overall you, as
a leader. It's your responsibility not to
coddle the best Engineers. It's a responsibility to build
teams that have a very steady strong level of combined output
because that's what wins the race.
Race, it's not individuals. It's setting your team's up for
success. I just published a slide Deck
with the bunch of steps. The other day.
I do think that the number one, technical point that people
should focus on which, I know that this is something that
nobody would disagree with. Right?
We should have high performing teams.
But another time on moving the business forward.
No fuckers gonna disagree with that sentence.
The hard part is how how, and the place to start for almost
everybody I've ever talked to is, you need to shrink the
amount of time between when someone is written the code.
And when the His life. You should get that interval
down to 15 minutes or less. And if you get that, you get to
spend your time on a lot of problems that are much, much
higher up the value chain, but if you get that wrong, you're
just going to waste all your time.
Tending, to pathologies. It is really an efficient and
effective in my experience. If the number of people it takes
to write and build. The system is say x number with
a 15 minute or less interval. Well, if that interval stretches
two hours, you need twice as many people.
If that interval stretches two days, you need twice as many
people again. And if it's weeks, you'll need
twice as many again, because the cost of context switching and
waiting on each other at the diff, get bigger.
And the reviews take longer and now you need a dedicated SRE
team to like manage all of the deploys in the failures in the
Cheetahs, get patched up. It's just as desk by rho.
So that's the place that I think everyone needs to start is
getting that fully automated and 15 minutes or less and then
everything is going to get so much easier for you or in short.
It's continuous delivery, right? So that's what the team should
Aspire for continuous discrete for real.
So 15 minutes to me, that sounds really short amount of time.
And there's so many things that people wants to do in the
pipeline. Like, for example, automated
test performance test, security test, whatever test.
So is that really important to have 15 minutes as Max limit?
Yes. I feel like I'm being generous
with that. It is important because you
don't want people to have enough time in there to be able to
shift context that moment. When you've written the code,
you want this to be a small dip, right?
You want this to be like a few lines.
How quickly can you deploy? One line of code should be your
goal here, right? You want to be short enough that
you hook it into people's physiological systems, their
muscle memory, you want all of your developers to go and look
at their code after they shipped it.
So, it needs to be short enough that it hooks into your
physical, like expectation. Just like, you don't feel done,
until you've got a new looked at it.
And so, that's an hour or more. I'm sorry.
That's too much time. Someone's gotten switched over
to something else, and that means that they flushed all of
this context of their head and they've gotten distracted and
they're not going to do it regularly.
And if it's an hour or more that I guarantee, you're going to
start batching multiple. Changes up together.
When it's really important that you be shipping, one person's
change, set at a time because that creates ownership.
If you're change going up, you're going to go look at it.
If you know, it's your change and 1 to 20 of your co-workers
going out at some point in the next day.
You're not going to go look at it.
You need to construct that feedback loop and keep it crisp.
And also notice that you are one of the strongest proponent for
testing and production, right? Is that also one of the reason
why? Because you want to achieve this
15 minutes, right? You Don't do a lot of testing
and there it all goes together. I think there's been a major sea
change over the past five years. In terms of startups that have
started, the way people are talking about this stuff.
But used to be that we put all this focus on pre-production
stuff. All of our energy went to like
every developer gets their own staging environment and oh, this
pre-production stuff and everything but we've hit
diminishing returns a long time ago there.
And I feel like over the last five years.
We've started to focus correctly much more on.
Okay, but now, what about production, we Tools for
production. They're like scalpels that are
very sensitive. Very precise.
Developer Cycles are the scarcest resource in our
universe. They're not infinite if we're
wasting them all in pre-production stuff.
Then we're left with nothing for production.
I just think that the importance needs to be inverted there.
We need to be thinking. First about production.
Do we have the tools that we need to ship safely and swiftly
and understand what's going on? And then give 80% to production,
20% to pre-production set of 80% of pre-production, and 20% of
production. But the point is not that
everyone should be testing production.
In the point is that everyone is testing and production whether
you admit it or not and you're better off if you just admit it
and try to do it. Well don't deny deny deny and
then sneakily like SSH into see what's going on.
Now. Everybody has to test the
production you do. I do?
It's not embarrassing. It's a fact of life.
Now, how can we do it better? Yeah.
I mean I must admit. I also testing Productions most
of the times because you can't really predict what actually
happened. Yeah.
You have know what the gold standard is to capture.
And Play like 24 hours worth of traffic.
And I will say the closer you get to laying B on disk the book
paranoid. You should be.
I have written this Suite of software three times for three,
different databases that captures 24 hours of traffic and
replays it. Again and again under different
conditions and stuff. You should know when to do that.
Major database upgrade. You should absolutely fucking do
that. But every day when you're
shipping a few lines at a time, no, of course not.
But you should invest that energy into observability and if
you want to get really precise and really you should be using
feature Flags, so, You can flip that on and off so that you can
decouple your releases from your deploys.
You should invest in observability so that you can
see and canaries. So, if you're really scared
about a change, ship it to one percent of the traffic or ship
it, to one percent of the users or ship it to one node, build
the tools that let you have this very high level of precision
around, who you're shipping it to and then the ability to
compare the traffic stream so that you can see what the
difference is error rates is or what the difference in latency
is or anything like that, build those tools to address your
paranoia. Don't say that you're not
testing and production. So I'm interested to dig deeper
about this 15 minutes. So what actually has to be done
in this 15 minutes apart from building software and also
packaging it and deploy it run tests.
Absolutely run tests. I know that some languages like
was girl. I hardly takes any time at all.
We can rush it to test but this is achievable even for things
like Ruby which is notoriously huge to package, you know, slow
to run. All this stuff.
Our friends at intercom who are the ones who came up with a
slogan that shipping is your company's heartbeat, which I I
really value and respect. They run a ruby monolith and
their entire build and test pipeline runs, and deploys, and
under 10 minutes. It can be done.
Well, this can absolutely be done.
It's not hard. It's just engineering work.
You just need to commit to doing the ongoing work.
Setting yourself a limit set yourself a limit alert.
Would it goes off. Take it, seriously, invest in it
over time and you can hold yourself to a higher standard.
So I'm mostly interested with this phrase.
Shipping is the heartbeat of your company.
Maybe you can tell us a little bit more about that.
This came from intercom and I adopted it cuz I love it so
much. Shipping for any tech company.
Giving value to our users is why we exist shipping.
New changes should be new code where users should be absolutely
as common as ordinary as uneventful, and is regular, it
should be a nothing. Nothing Burger, ask me how many
times a day we ship? I don't fucking know the trip
all the time like a couple times an hour.
I don't know because we're making changes constantly and it
is because those changes are small and uneventful that it's
not a big. Deal, I feel like we need to
invert the way that we think about this is this social change
is a mindset change, but it's not a new thing.
The principles of cic have been around for what 15 years
accelerate was written with all that data years ago.
Like we've known this for a very long time.
We just need to look at it into action.
And the thing is, this is so good for engineers Engineers,
who have worked with true. See, ICD are unwilling to ever
go back because it is like different profession.
It is so much more interesting and exciting and up to the
moment. See your real impact on users
lies like many times a day and that's fundamentally motivating
and inspiring. It's real hard to ever.
Be willing to go back to a situation where releases are so
far away from your lived experience or someone else's
problem. We don't want that and I think
this is related to the socio-technical.
Mm aspect that you mentioned, right?
Like in the first place. You mentioned, you don't need
all the high performer people in your team.
You just need a team that works well together.
Maybe you can also share a little bit.
What we mean by By social technical aspect of a team.
Yeah. So for example, think of it this
way. Imagine if New York Times is a
technical and social organization, imagine that all
of the people in the tech side, got flown to the Bahamas for a
month and they brought in a new set of people who are
everybody's experience talented, good Engineers, Etc.
He was brought instead of Replacements to keep everything
running and then something breaks.
How long do you think it would take them to figure out and fix
even a very small problem? First of all, how do they get
into the system? It's a mind experiment to You
just how much of the system lives in our heads.
These people are not fungible. Right?
Engineers are not fungible. People are very important part
of the technical system and you can't just separate them.
I think it's really important. Especially as we're entering
this age with machine learning and AI.
This AI that just remind ourselves that no people are
really fucking important here. The part of the system that
lives in your head is a part of the system honeycomb.
We interview lots of Engineers and what's really important to
us as their communication. Ation skill.
Now, they need to be able to write code obviously, but the
real interview is not us. Looking at their code.
It says, talking to them about their code.
Just having a conversation about, well, talk to you about
your choices here. Why did you choose this?
What else might you have done or like what trade-offs we did you
have in mind when you are doing this because there are so many
people who are capable of writing the great code but can't
communicate about it. And those aren't the people that
we want for our team because we believe that high performing
teams come from people who are ready willing and able to talk
about what they do. And They've done what they've
done without getting their ego, too wrapped up in it, but just
being like able to have a conversation about it.
Christina and I could have just hired all of our X Google X
Facebook, friends. We knew that we didn't want to
create a cultural bubble. So we made sure to go outside of
our networks, but we were never looking for the best
programmers. We were looking for the best
communicators, the level at which this team performs.
It just, it blows me away. Every single day.
There are order of magnitude better than the elite category
and the door of metrics. And it's not because they're the
flashiest. Coders is because they're dogged
about improving. In their craft at this really
inspiring. So what else do you see from a
candidate? If you interview them apart from
communication skills and some basic programming skills.
Of course, they need to have what other things that you look
for in an engineer. We look for humility.
We look for people who can be gently challenged and don't fall
apart. We look for people who are
constantly, I guess growth mindset.
Is the term of art for it, which feels a little corporate e to
me, but we want people to be able to come curious to a
problem. Not to come feel like they're
going to be. If I'd if it turns out that they
don't know what they're doing. Because it turns out, we don't
know what we're doing, either all day long.
None of us know what we're doing.
Our company values. Are we have values that are like
we hire adults. We don't have your family or
your whole life. But we also do, hire people who
take a lot of pride in their craft.
We don't want to hire people who just want to work.
You just do it to pay the bills either because we take a lot of
pleasure. And what we do, nobody honestly
has more than four hours a day in them of hard, focused,
engineering labor and it's so much more important to me, that
people know how to manage their time to get them.
Themselves those four hours then for them to like sit at their
desk 24 hours a day. We hire adults.
Their values are, everything is an experiment and fast and close
to ride is better than done and perfect, which trees is all the
way back to manifesting in our storage and right?
Like that's our philosophy for storing data, turns out.
It's our philosophy for people as well.
Feedback is a gift. Is another one because I think
that so many pathologies in the modern workplace can be traced
back to the fact that people are scared to give each other
feedback and so it builds up and it becomes a big thing.
And if we can just like deescalate that make it, not a
big thing to be constantly giving and receiving feedback
and to be able to give feedback, knowing that we might be wrong
about it. Just like your something that
I've noticed I thought might help you because we're all on
the same team, right? Like we should be able to help
each other improve which is so easy to say and so hard to do
because we all have our own personal damage.
We all have our own prom has and stuff around the stuff.
So we're all constantly having to learn and unlearn bad
behavioral patterns, but there was a couple of people have said
to me. They started working at
honeycomb to like how you guys talk about your feelings and
awful lot, and I'm just like embarrassed.
It's okay. Yes.
Female founded company. Yes, I guess we do talk about
our feelings about, but you know what, it's better than the
alternative. Thanks for sharing all that.
I think it's a really good Lessons Learned as well for
other teams who would like to build High performing team, you
know, we spend more time at work with each other than we do at
home with our partners and our families.
I think that we shouldn't be embarrassed about talking about
our feelings and being emotional human beings at work.
Because if you're not well, you're just faking it and I
don't believe that lets you perform at a high level over the
long term. I agree with you on that.
So one thing that you mentioned about Dora metrics, I think in
some of the presentations I saw that you actually insert one
moment trick not just a yes, you metrics.
So what is the fifth? Maybe if we can share I think
that every team should be tracking how often someone gets
paged out of hours. I strongly believe it's software
Engineers need to be on call for their own code, but this comes
hand in hand with commitment from management to make Make
sure that does not suck. I'm not just saying software
Engineers. You need to go be masochists to
just like cops have been all this year.
No, the reason to put software Engineers on calls because this
is how we make it better. This is how we close that
feedback loop, how we make it short, how we get things fixed
quickly, so they don't get to infest and like become all
infected and gross and big problems be fixing their small.
So I think that these two have to go hand in hand.
Yes. Developers need to be on call
for their code. And yes manage music.
Make sure it doesn't suck and probably compensate them for
their time to I It's reasonable to expect any engineer who works
on a 24/7 highly available system should expect to get
woken up twice a year or so, once or twice a year for their
work. I think that's reasonable.
I do that's compatible with an adult life.
The exception being, if you have a child that is waking up in the
middle of the night. I would never put one of those
people on call. If you have a baby, you are out
of the uncover rotation until your kid is regularly sleeping
through the night. It's not fair to ask anyone to
wake up for two things. Right?
So with that caveat, these are human systems, right?
So there are exceptions. To everything.
I'm not actually saying every person must be on call at
Facebook. I had a guy on my team who was
totally willing. He wanted to share his part of
the burden, but when he took the pager, he got such anxiety and
it wasn't because he was getting called or page all the time.
He was so afraid. He just had a personality that
was just not compatible with it. We tried for a couple months and
he gave it a good shot. But I was just like I just felt
so bad for him and the whole team did right there just like,
okay. No, Alan doesn't have to be in
called what can Alan do instead and so Alan took over.
Ownership of the build Pipeline and just like with build cop.
So that was like technically more work but it was during
business hours and the rest of us were totally fine to get the
slack. So it's about pulling your own
weight. It's about being willing to
support your own code. It's about management.
Making sure you have the time to fix the things that need to be
fixed because you can't just put a long call and then tell them
to build features all the time. If you're putting them on call
it, you have to give them the time to fix the shit.
So they're not getting woken up all the time and I just feel
like having that metric and holding managers accountable for
that metric. You never promote a manager
who's paging. Alerts are going off at all
hours. That is just a sign of terrible
management. Their team is going to burn out.
It should come down on them. The hardest.
Wow, thanks for sharing that. I think it's really good key
metrics, especially when you have so many distributed systems
in production with problems could happen at any point in
time. And was, if you have a global
system with your users come from anywhere in the world.
I think it's also critical to monitor this like how much you
get paid tonight, right? Because that's what burns your
team's up faster than anything. So one of your most popular blog
post is about engineer and manager pendulum.
So because this podcast is also about technical leadership and
also Engineering Management. Maybe you can share a little
bit. What do you mean by this
pendulum between engineering manager?
Yeah. Well, when I became a manager I
was told it was a one-way trip and I've heard this repeatedly.
People are just like agonizing over whether to become managers
because they see it as a one-way trip.
They're not sure if they can get hired as an engineer afterwards.
They're not sure if they should. You're often told that they
shouldn't and I feel like this has said, the best technical
leaders that I've ever worked with have been people who have
spent time as managers because you learn things about how to
help a team overcome obstacles and work as a unit, that you
can't learn any other way. You learn things about how the
business works. If you can't learn any other
way, but has also, like has been hands on within the past five
years because when you take a step away from the day-to-day of
building shipping code, like you're on a path that diverges,
right? And there comes a point where
you are not an effective leader of people who are writing
shipping code for the sake of your own career.
You need to be cognizant of that and are two different paths.
You can take. If you want to continue managing
people. You can make it, your goal to
work your way up. The ladder.
Now, there are an order of magnitude fewer jobs at each
level, as you go up from manager, to senior manager, to
director or to, you know, see your director at VP or whatever.
There are fewer and fewer jobs. It is more and more competitive.
There's more and more bias involved, I think.
But for people who love it, it could be very rewarding.
But if you want to stay in a line manager, who manages
Engineers who manages the people who are writing shipping code,
You're going to need to make your way back to the Well, from
time to time. You can't go more than five
years without building and shipping code regularly yourself
and maintain that level of tension and empathy, and
immersion in the problem, set. Your engineers won't respect you
as much. You won't be able to mentor and
develop Engineers as well. You won't have opinions that you
can trust about the technology. And there are lots of places out
there who just take this for granted.
They're like manager, shouldn't know anything about technology.
They shouldn't have the opinions.
We rely on our tech leads for that.
And that is a choice. I think it is a choice that is
becoming less and less popular or prevalent because I think the
outcomes are materially worse. I think that it serves everyone.
When manager has opinions of their own that they can trust
about what's going on. Now that said, you do have to
train yourself, not just, you speak up all the time as a
manager. This can be very humbling thing.
Right? Like many of us became managers
because we were Top Dog, right? Like we were the most technical
of the most senior whatever. But once you step out of that
position and you're the manager, it's your job to guide.
People up to reclaim the oxygen that you once, breathe.
It's your job to grow people into that space.
And this can require giving up a lot of ego on our parts, but its
necessary. It's the best way to grow people
up is to have a manager who actually understands what
they're going through and can help guide and nudge and suggest
opportunities for them. So, I feel like this is a change
that needs to be fought on the ground Company.
By company. We need to explain to our
Executives, why it's better for everyone.
We need to build this into our org charts.
We need to stop talking about management as a promotion.
A promotion is just a change of career.
It's a lateral step. What that means is that it's not
a demotion. If it manager wants to go back
to engineering. You don't want to have managers
who are resentful or who miss being engineers and who like try
to reclaim that glory and try to either one, who's always talking
who has all the best dancers, you want your managers to went
to be managing people? You want the people who were
better as Engineers to try it for a while maybe and then be
able to switch back to engineering without any loss of
status without any loss of ego. So it's really We important that
management is not a promotion to change a career.
I think anyone who's trying management out.
You should ask for a two-year commitment.
Otherwise, it's too rough on the team's.
If you're like, shuffling manager around constantly.
You literally have to become a different person to Be an
Effective manager. You have to retrain your own
instincts, your own reactions of the way that you think you're
pacing. You're pacing on it on a daily.
Weekly basis is so much different as a manager.
You have to gain the Instinct and intuition know when you're
doing a good job when you're doing a bad job.
Takes at least two years to build.
And so in the early days as a new manager.
Everybody is all my doing a good job or not.
I do I like this or not. It's like chill out.
You don't know, you have no way of knowing just turn the voices
off. Learn everything you can and get
back to me in two years and then we'll have a conversation about
it. But I feel like the industry
would be so much better as a whole if we demythologize
management. If pretty much anyone who is
interested in trying management, gotta shake at it.
And then if we ask ourselves, honestly, is this good for me?
Do I like it or not? And there should be a path back
and forth. That is well-trodden.
I feel like so much of the pathologies that I see on teams
come from the relationship between management and
Engineers. If we could just make it less of
a thing, this make it. Another thing that you can try
and it's not a big deal. It's not a promotion.
It's not a big ego thing because you're not in charge of people.
It's not like that. It's a support role.
I hadn't really thought of this until now, but I think in the
way that the interval between, when you write the code when you
ship the code is a Building block of a healthy software
digestive cycle. I think that pathway between
senior engineer and manager, being an equal one.
And a well-trodden, one is a fundamental building block of a
really good social side of the socio-technical organization.
I also think like many Engineers, of course, aspire to
try out management. So the swing from the engineer
to manage arrived, and they should because even if
long-term, they want to be a senior technical leader.
There are things that you will learn as manager, that will give
you so, much more empathy will be Q so much more effective.
As a technical leader from doing a two-year stint as a manager, I
think it's a good thing for engineer to try management and
maybe what are some of the tips for like, first, new manager out
there from engineer. They want to try manager, how to
become effective in that new rule.
Yeah. I think that a lot of it comes
down to self-awareness. So number one, I'd say, go to
therapy. This is get a therapist and go
to therapy every week. Learn stuff about power
dynamics, especially if you're a white or Asian, dude, Tech.
I think that there's a lot of stuff that you need to learn to
be sensitive to that. You've perhaps never been
sensitive to before you owe it to your reports to learn about
those things. So, start following feminists
and some people of color and learn about power dynamics learn
how to listen, all of the advice that I would give for new
managers is really about self-awareness and humility and
communication skills, which is another reason that I think it's
a really interesting two or three year journey to go on as
an engineer who's now in your 30s.
Maybe has a family now. I think it's a nice.
Change of pace. I think that it dovetails really
well with just kind of growing up.
And I think one of the reason the pendulum back from manager
to engineer, I think it's about the concern that once they go
back to individual contributor. For example, it's probably
difficult, especially in this region.
At least the way I see it is that you might not be able to
get back that management role, especially when people see the
numbers of experience the bigger team that they managed before.
So, what do you see in terms of this pendulum back?
How could someone assure themselves that Create safe.
I can go back to I see and I can always go back as being manager.
Choose your company's carefully, especially if you'd be worth in
your in your career. You should be interviewing them
just as much as they're interviewing you ask them about
their philosophy of management. Is it a promotion?
How do they think about it as anyone ever gotten back from
being a manager to, I see, would you discourage them?
Talk to them about it. You really want to find a place
that has a culture that matches your own aspirations.
Also, like just Comfort yourself that if it's been two or three
years, you can go back. It's fine.
Especially if you're a dude, right?
Nobody's going to even question you if it's been two or three
years. Yeah, it'll take you a month or
two. You might have to study a little
bit for the coding interview. But, you know, you'll be fine.
You might have to change jobs, if you aren't a company that
values, this, but it is honestly much easier to go back from
being a manager, to be an engineer because this is a
superpower, everyone wants Engineers.
There is so much more demand for engineers and there is
availability. So, if it's only been two or
three years, you're going to be fine, but if you're concerned
about that, then make sure it's all For three years, don't go to
five years or Beyond, or it could actually be very
difficult. So it's been a pleasant
conversation. But before we end our chat here,
normally I would ask all my guests to share their three
technical leadership. Wisdom.
So charity. Do you have some wisdom that you
want to share with all my listeners here?
Yeah. I think number one is your
career is an enormous asset. It is the most valuable thing.
You possess. It is a multimillion-dollar
appreciating asset and you should treat it that way in my
twenties. He's I didn't think of my career
that way I just lurched from one thing to the next, you know seat
of my pants and it turned out okay for me.
I was pretty lucky but I think that a little bit of
introspection a little bit of where do I want to be and
pointing your nose in that direction.
It has a way of making you a tune to opportunities that open
up. If you're like someday, I might
want to be a manager, then you might be more likely to take a
role at a company that has some obvious management roles opening
up. So I just feel like treat your
career, the way that you would want a friend of yours with a
multimillion-dollar appreciating asset to treat their assets.
Think about it for the long term.
Sometimes this means taking a job with a little bit less pay
because you think it makes the story of your career look much
better. I think that being in a position
where you are not spending all of your money.
Every month is a very good tip for everyone.
Have some fuck you money in the bank, don't live paycheck to
paycheck because then your trap, you can't take any jobs that pay
any less than that. And sometimes you're going to
want to look at your career as a long term asset and think about
the story that you wanted to tell.
And it's not something you have to take up a lot of time and
energy, right? Just like a couple hours.
Year, I think will really help. It's really valuable though.
Like, we are so fortunate to live in this time and be doing
this thing because this is the only growth industry of my
lifetime. We live in a different economic
reality than most of the world does.
Most of the United States has been in a depression for 15
years. Not in Tech opportunities,
abound. Yes, Tech has problems with
sexism and racism and yet I often feel like I want to just
like tone down, so the shit that we tell the kids about because
I'm like, if you're telling them, not to come to Tech
because we're two, Here, where do you want them to go?
What is this? Magical mythical industry where
they will be highly paid in valued?
Where they've not going to have to deal with you?
This bullshit. This is called living in the
world, guys. Like it just is this way and so
I think that we owe it to ourselves to get as many women
and underrepresented minorities in detect, it helped them
survive and tough it out because we're so fucking fortunate and
blessed. So that was two or three things.
All in one. I guess try to be financially
constrained. Try to be an ally to In general
other genders and races. The advice that I always give to
young women who ask me for advice is like when you're a
junior engineer, just toughen up.
This is problematic. I absolutely agree.
Tough it up. Try not to take it too.
Personally, put your head down, learn as much as you can just
survive, become a senior engineer and then get sensitive
and then learn to be sensitive on behalf of other people, use
your power for good. Anyone who's a senior
engineering Tech has an absolute responsibility to learn to be
sensitive and to use our power for good on behalf of those who
can't, Wow, thanks for sharing all this.
I think it's a message that we all need to learn especially in
these days where so many things happening, especially in Tech
and around the world. So, thanks for sharing that.
So charity, if people want to connect with you or find you
more, where can they find you online?
I am on Twitter as Nifty Tipsy. I EverQuest enchanters name and
Trudy. WTF is my blog and honeycomb.
Dot IO is where you can sign up for a free honeycomb account and
use the build event stuff to visualize.
Cic pipeline. The way I was talking about.
It's super fun. So one day, I'll make sure that
I'll try that. You expect less than an hour.
It's super fun. You could see all your time is
going. It's great.
Although thanks again for your time, charity, honeycomb, and
everything that you do. Thank you for listening to this
episode and for staying right till the end.
If you highly enjoyed, please share it with your friends and
colleagues who you think would also benefit from listening to
this episode. And if you're new to the
podcast, make sure to subscribe and leave me your valuable
review and feedback. It really, really helps me a lot
in order to grow these podcasts better.
You can also find the full show notes of this conversation on
the episode page at technology. Know the death website,
including the full transcript interesting quotes and links to
the resources and mentions from the conversation.
And lastly, make sure to subscribe to the shows mailing
list on technology on of the deaf to get notified for any
future episodes. Stay tuned for the next
technique Journal episode. And until then.
Goodbye.