Now more than ever, engineering leaders are being asked to be
more transparent with how work is getting done.
Every single thing that an engineering team works on needs
to benefit the business. And your job as an engineering
leader is not to say, well, just trust us, we're in the code all
day. Developer experience and
developer productivity exists in this very strange area where
what is good for your engineers and what is good for your
business are the same thing. Because the bottom line is that
poor developer experience just cripples innovation.
We've really nailed it down to speed, effectiveness, quality,
and business impact as the four pillars to kind of put a story
together about developer productivity and developer
effectiveness. So there's not going to be a
single metric. So for speed, that is diffs per
engineer. These are like PRS or merge
requests or whatever it's called.
That's the first one I always talk about because it is the
most controversial. I think even a few years ago I
would have said like, that's a bad idea, don't do it.
It's not about ping pong and beer, which is often.
I think developer experience has a like quite a marketing
problem. Abby has this thing that just
like makes me laugh every time. He's like, Can you imagine if a
new sales leader, like a new CRO comes in and says, well, we're
going to measure the impact of our sales organization based on
sales Rep experience? Like, there's just no way that
that would ever go over. Hello, welcome back to another
new episode of the Technical Journal podcast today.
I'm very happy to have another DX person in my show.
Laura Taco is here with me today.
So she's the CTO of Get the X or the x.com, right?
So if people who are following into these developer experience,
developer productivity, you might have heard of or seen the
X materials out there be research paper or their own
products, right? So happy to have you again in
the show to talk about developer experience and all the stuff
that you published recently. So welcome to the show, Laura.
Yeah. Thanks so much.
Nice to be here. Right, Laura, I always love to
invite my guest first to maybe share anything from your career,
any turning points that you think we all can learn from
that? Yeah, what a big question to
start with. A good one.
I think there's really 2. I mean, there's lots of points
that have defined my career, but the two that stick out may be
interesting to your audience. 1 was Docker and getting in
extremely early on a new thing. I think as technologists, it's
really hard to know like what's just a new shiny thing that
might be a little bit of a distraction and like what's a
new thing that's like fundamentally going to change
the way that we build, ship and run software.
Even thinking back to like back in 2013, I'm not sure that I
fully understood just how big of a difference Docker would make.
But to me, it just felt different than like a new
JavaScript framework, for example.
Not to put down new JavaScript frameworks, but you know, it.
Just like my intuition, which I think a lot of leaders, we are
taught not to follow our intuition because we shouldn't
act on emotion, we should act on logic.
But truly what your intuition is, is just hundreds of
thousands of data points that you've accumulated over your
career that allow you to make a judgement about something.
So it is, it's not just fully emotion, but working on
developer tools for Docker from very, very early on, getting
started with it before it was really heard of anywhere was
hugely transformative to my career.
So I guess one lesson, intuition isn't just emotion, it is data
points. So sometimes you have an
intuitive feeling about something and you should follow
it. It might pay off in a big way.
The second big turning point was like I left my corporate job,
AVP job that was like had a lot of responsibility to start my
own business and go into coaching and consulting full
time during the pandemic. Like on paper didn't look great.
At the time, I was working with my own executive coach and I
really realized how much of my career I had let someone else's
career framework decide. And that was really the first
time that I said no, if I want something different, you know, I
want more flexibility. I want to be able to go climbing
at 11:00 AM or whatever it is. I need to fundamentally do
something different with my job. And so I took a very big leap to
do that and it paid off with a ton of hard work, a lot of
opportunity and hard work went into creating my own luck to
make that successful. I still do coaching on kind of
on the side. Now I'm back sort of in full,
you know, full product building mode with DX.
But I think the short lesson there is like, don't let someone
else's career ladder define how you're gonna, you know, your
trajectory and your own career growth.
You need to at some point take responsibility for it and be in
the driver's seat. And it's surprising how long we
all go in our careers without actually being in the driver's
seat. Well, thank you so much for
sharing the story. I think those are really great
insights, right? So the first thing is about the
Docker containerization technology, right?
So I think these days almost everything is run based on
container. All right?
So be it, you know, the CICD and all that.
So I think that kind of like revolutionize the whole thing,
how we get things done. And the second thing about, you
know, defining your own career path, I guess.
So thanks for sharing that as well.
So maybe some people also are looking into, you know, more
meaningful career, right? So don't follow other people's
paths or maybe define on your own.
So maybe you mentioned something about intuition, right?
So the engineering leaders have to kind of like listen to your
intuition, you know, gather from past data points probably that
you have gathered in your experience or maybe whatever
learning things that you have done.
So what can you advise us to actually do more in terms of,
you know, following our intuitions?
Because sometimes, you know, technology change so fast, there
are so many things out there, but your intuition is not
something that we always, you know, try to sharpen, right?
So maybe anything any tips from you about this?
Yeah. You know, this is an exercise
and two things can be true because on one hand I say, you
know, you should trust your intuition.
Your intuition is data that you've accumulated throughout
the course of your career that allow you to make a snap
judgement about something or have, you know, a gut feel.
It's not just purely emotional. On the other side, I'm like a
extremely data informed decision maker, but I think you know, if
you get your intuition and actually I think practicing
making data informed decisions helps you refine your intuition
skills. So changing your mind when new
data is available to you, being able to broadly look at lots of
data sets and draw a conclusion from them, but not just draw the
conclusion, but then know what data you're going to look for to
know that you're wrong or what data you're going to look for to
know that you're right. I think those are all skills to
kind of practice that will help you feel more comfortable about
intuition. I think the other part of it is
like, just like in a recipe, time is important.
Like you can't just throw the ingredients in a pan and like,
wait a minute. And they're like, there's a time
element of that. And I think careers are the same
thing. You have to just get the cycles
and get the reps in and let you know experience will teach you a
lot. So time is a factor in career
growth. And I think, you know, over
time, your intuition gets more honed and more honed.
So look out for opportunities where your intuition might be
telling you something to figure out.
Is there a way that I can use data to see if I'm right or
wrong? What data would I look for to do
that? And then when you can get some
more practice in that, it will just become more natural to you.
Yeah, I like that. You mentioned time, right?
So I think one thing that I would add also like your past
mistakes, right? Sometimes in our career cannot
always go up, right? So there'll be ups and downs,
maybe situational, you know, change of culture, leadership in
the company, or maybe you made the wrong decision to take some
career path. But nevertheless, those things
are like data points that you can use to, you know, hone your
intuition. So maybe let's go into our main
topics of discussion today. So the first thing that I'd like
to discuss is something that you actually highlighted before we
had this conversation, right? You mentioned one thing that
piqued your interest lately. It's about engineering leaders
not necessarily being able to become part of the so-called
business leaders, right? So almost in any company,
technology is kind of like enablers, right?
Even though if if it's like technology companies, right,
it's like one of the key pillars of the business.
But many engineering leaders after you mentioned that, yeah,
I kind of like make some kind of like reflection.
I think many engineering leaders are not able to kind of like
switch their technology mindset into more business oriented.
So maybe tell us a little bit more like what made you come up
with this kind of, you know, thoughts and what do you see
from your side there? Yeah, I think right now we're
in, I mean, there's lots of things going on
macroeconomically, right? We've got like the end of the
zero interest rate period. We had this like big hiring boom
and then a bust and like maybe hiring starting to come back.
I think now more than ever, engineering leaders are being
asked to be more transparent with how work is getting done,
how much it's costing. There's just like lots of things
that before engineering was really kind of a black box.
Like we got away with a lot of stuff.
Let's be honest, you know, I think back to the, you know,
like, let's bring this back to Docker.
But you know, I remember the, I mean, the heyday, like the
height of the Docker craze, when it was just like you, if you
knew Docker or knew Kubernetes, like you could just hear pretty
much, you know, no one got fired for learning Kubernetes, right?
You know, that's the saying you could demand a higher salary,
you could do a lot because companies were afraid of losing
your talent and having that business knowledge walk out the
door. And because of that, it, there
was just, I think a lot of, I don't want to say that, you
know, there was dancing around engineering, but maybe it, you
know, we just weren't expected to show the bottom line for
stuff that's happening in engineering org the same way
that marketing, sales, other functions would be expected to.
I think that's really changed. You know, I spent all my time in
developer productivity and engineering metrics.
And so I know that that's the case.
I know that engineers are being asked for metrics in a way that
they just have not been asked to before and they're not prepared
to be able to tell that story with numbers because now we have
to answer those questions. I think there's just a
heightened sense of accountability from engineering.
We're no longer the as charity Majors puts at the artists and
the organization that we can kind of not work without
accountability. But there's just there's more
questions that need to be answered now.
And I think because it hasn't been expected of us, a lot of
engineering leaders just haven't had to develop that skill even
on the broad level. But just think about an
interaction of engineering wants to do this tech tech project
product manager says we need to build this new feature.
And the conversation in a lot of companies is like, well,
engineering says that we need to do tech, that we're going to do
that before we do the feature. I mean, a lot of companies it's
the other way around. But there was never this like,
let's actually build out a business case and see what
customer impact this tech tech project is going to have.
That is something that engineering leaders need to do
now in a lot of companies in a lot of circumstances that we
just haven't had to do before. And it's a really missing skill
set. Yeah, you mentioned something
about, you know, the situation about the economic, right.
So the end of 0 interest rates, I think in the past, I don't
know how many years. You know, they are a lot of tech
startup booms, right? So many engineers being hired
and you know, the engineering teams are keep growing, growing
and growing. Right now, I think it's like a
questionable kind of a decision, right?
If you still want to hire more engineers, right?
And that's why engineering leaders are being asked to
actually kind of like be accountable.
You mentioned, right? So how you optimize your kind of
like resource versus the impact that you bring.
So you mentioned about tech debt, right?
So it's always very interesting, right, whenever engineers talk
about tech debt, right? Because obviously the first
question is like how to quantify that.
And I think you also mentioned things like, for example,
engineering leaders are not able to explain things in the
business sense of business metrics, be it, you know, for
example, number of users, number of growth, right, number of
revenue, dollar that is being brought into the company.
So how do you actually will advise leaders to actually think
about, you know, switching their mindset, but you know, from tech
debt, which is probably more like, I don't know, like
complexity or, you know, maintainability and all that
into something that is more quantifiable.
Yeah, I think there's kind of two, maybe there's like two ways
to answer that. I think for the broadly, I want
to say just as a a general statement that I'm not saying
that individual engineers should be thinking about how much
revenue that their features are going to bring in, right.
That's not appropriate. What I am talking about though,
is engineering leaders who have a product manager counterpart or
like a business stakeholder being able to come with the same
information that you would expect that business stakeholder
to bring about a new feature, come to that conversation with
that same information about a tech debt project.
So I think a, a way to get practice here is like, let's
just take tech debt for example, which is a term that I actually
really don't like because I think it's too broad.
It's not very descriptive but if we could pick AI, don't know.
Why don't you pick a hypothetical tech debt project?
Can you think of 1? The typical one these days,
especially if you're in the big setup monolith versus
microservice kind of thing, you know?
OK, great. One monolith versus
microservice. Like let's break up this
monolith into microservices. Or sometimes what I'm saying now
is like let's combine these microservices into a monolith
again, because it's just, it's just too complex.
But let's think about that like what's the actual benefit of
doing that? So let's just say it's lower
complexity. Whichever way you're going, if
you're breaking up the monolith or putting things back into a
monolith, it's lower complexity. So what would we expect from an
environment where developers have lower complexity code?
We would expect that features can be delivered faster and so
we can pick some measurements to measure that.
We we might expect fewer bugs and higher stability because
it's less complex. We might expect build times to
go down because it's less complex.
We could probably find four or five more examples.
I think for each of those examples then we can actually
tie that to customer impact. So how much does it cost to have
an outage? We can look at contracts.
If you have an SLA of whatever, however many nines you need,
usually if your service goes below that, you need to pay back
some of your subscription fee. Or maybe there's customer churn.
You know, there's lots of different places that you could
look for. What's the actual impact of
incidents and stability problems?
I think the other thing here is opportunity cost.
It's a little bit more difficult to calculate, but let's say, OK,
so every project that used to take two weeks now takes only 1
1/2 weeks. So we're able to accelerate our
delivery by half a week for every project.
Like over time, that's a lot of time saved.
And of course, we can do the back of the napkin calculation
on like salary, you know, just like pure monetary, like how
much salary money did we save? But let's actually talk about
what you could have built instead in that time and what's
the potential economic impact of the stuff that you would build
instead. And so now it becomes a question
of like, well, if we wait, we could do this now.
And that means that features AB and C will be delivered more
quickly, but minus the cost of being able to or the, the
technical debt project, actually the monolith micro service
project, or we could delay it and do it in 1/4 from now, which
means we know we're OK with paying that tax basically.
I mean, we're talking about debt here.
We know we're OK with things moving slower, but the upside of
those features going to market first before we do this tech
tech project outweigh the negative parts of waiting for it
or whatever. So you know, that's just a kind
of simplified example of how we want to start thinking about the
business impact, the customer impact, the user impact of
something that traditionally has been seen as engineering's
problem and engineering's kind of self-directed project.
It's not every single thing that an engineering team works on
needs to benefit the business. And your job as an engineering
leader is not to say, well, just trust us, we're in the code all
day. It is to figure out what is the
way that this actually benefits the business.
And it's just something that we haven't had to do a lot in the
past. So it's a new skill for a lot of
leaders. Yeah.
So you mentioned that I'm also guilty in the past, right.
So whenever we talk about engineering challenges,
engineering project, I think we always kind of like explain it
in the engineering complexity manner and then it stops there,
right? We don't actually translate
further into how it actually impacts the business or how can
it actually bring more capability into the business,
right. So I think culturally in the
industry, I think it's really rare to have an organization
that kind of like be able to balance between these two.
And even you mentioned in the in our pre composition as well that
you have seen two different extremes, one maybe organization
that focus a lot more on just business impact, you know, being
able to churn new features fast, as many features as possible.
And the other one is actually focusing a lot more on
engineering and maybe doing a lot more this, you know,
rewriting micro service and all that.
So maybe tell us about the danger of these two extremes and
how can we actually balance between these two?
Yeah. You know, and I think the career
change that I went through in going to be a coach versus, so
now I get to go extremely broad, not necessarily as deep, but
like very, very broad across a lot of companies, which is a
huge benefit to do like pattern matching.
You know, sometimes I think about myself as like a human
regular expression for engineering problems.
Like I just, I'm just like figuring out which things belong
and which things don't. I've worked with companies who
really have extremely hands off approach to say like, OK, say
every engineer, you just like you're hired to do this job.
Here's the product you work on, do the thing.
And I think what we get in that situation is like a bunch of
people moving in different directions.
I mean, this is an extreme example, but you know, people
are vectors. We have like a speed, but also a
direction. And so people can be moving
really fast, but if they're not aligned and going in the same
place, the benefit just isn't going to be as great as it could
be. So that's on one end of the
extreme. I think the other end is I've
worked with some clients where I mean their engineers need to
know not down to the cent, but like how much impact did that
bring to the the company? You know, I've worked with
engineers who really like that's their gratification centers
light up when they know like, oh great, you know my whatever
optimization I did in, you know, usually it's more like
e-commerce type situations. But like, oh, I optimized this
page load and like now our shopping cart drop off went
from, you know, X percent down to X percent and that accounts
for 1.3 million in revenue and whatever.
And I'm like, it's great that you know that, Like, I love that
you have access to that. But I have also some clients
that take that to the farthest extreme and say if we can't
quantify it and if the number isn't high enough, then we're
not gonna do it. And what that leads to is just
like a slow degradation of their systems over time 'cause you
know, if you cook all day and never do it, a single load of
dishes, it's not going to take long until it's just very
difficult to operate in the kitchen.
And that's sort of what's happened.
So, you know, where in the middle is right for your company
is really an exercise I will leave to the listener.
It's got to be somewhere in between the two of those.
Like how can you save space for self-directed engineering
projects with also, you know, alignment and working toward
common goals for the business? How can you have both?
Because you definitely can have both.
It's just hard to know. Every business context is a
little bit different, but you need to find the place in the
middle that's right for your team and your company.
Yeah, definitely. I think every company has a
different situation or context. So maybe time as well, right?
There's a pressure to increase the business impact and there's
a time probably that is more relaxed so you can do more
self-directed engineering projects or maybe equally
balanced in any product road map.
Typically it's like, I don't know, yearly, quarterly, right?
You kind of like allocate what kind of percentage to work on
business related stuff and also the engineering related stuff.
I think those are definitely good tips, right?
So for people don't go to the extreme, right, I think it's
always challenging if you just follow one approach versus the
others. And I think the other thing that
is quite typical these days is about developer productivity and
developer experience, right? So I remember in the past when I
work in a bigger scale up, right?
So this is always the bigger question to the engineering
leaders, Like tell us about, you know, what kind of metrics do
you want to use for engineering? And I think this is always
difficult, especially if we want to bring up the topic.
We want to improve our developer experience, but building a
business case is always difficult.
So tell us, how can we actually build a business case for
improving developer experience or developer productivity?
Yeah, this is one of my favorite questions to answer because I
think that developer experience and developer productivity
exists in this very strange area where what is good for your
engineers and what is good for your business are the same
thing. And they're so few opportunities
as leaders that we get to like, do the right thing That's, you
know, the right thing that like we as engineers also want to do.
But like, that's also the right thing for the business.
Because the bottom line is that poor developer experience just
cripples innovation. It makes work and value flow
through your company to your users at very slow speed and
reducing friction, not just makes engineers more like
fulfilled and happy and satisfied with their working
environment. It is really good for your
customers and for your business as well.
And so it makes sense to invest in it because if you don't and
your competitors do, there's only one outcome of that
situation, which is that you fall behind.
And so developer experience is this interesting thing that is
sort of like the overlap of the Venn diagram right there.
And I think it's such an enjoyable problem space to work
in. Very interesting that you
mentioned developer experience is something that both business
and engineers, you know, incentivize to make it better,
right? I think this term is kind of
like new in the industry, maybe, I don't know, in the past three,
2-3 years, probably it's getting hotter and hotter, right, in
terms of discussions and research papers and all that.
And more companies are getting into this space as well trying
to understand what is developer experience, right?
Because in so many different culture, different types of
companies, experience and productivity can be quantified
differently. And I know that you have dealt
with this engineering metrics for quite a number of years.
So maybe in your point of view, right.
So what are the good engineering metrics to use these days?
Yeah, well, I can answer that question simply with a new
framework. Actually.
Then Abhinoda and I published along with a collaboration from
quite a few other people. So to give a kind of a richer
answer, I think I would have said it depends.
And my answer would have been it depends for a long time.
I've been working in developer tooling for over a decade, you
know, always trying to like make things more efficient, make them
better, make them faster, make them more stable or whatever.
And like how to quantify this and how to bring that to the
business has always been kind of an open question.
And because every business is so unique, usually the story that
we've needed to tell with metrics to make them matter to
the business has also been slightly different depending on
it. As you said, the last couple
years, like developer productivity and developer
experience are really like having their moment in the
spotlight. There's so many companies that
are practicing improving developer experience, like
really investing in developer efficiency and effectiveness.
So we have just more data to look at more examples.
And from all of those examples, we've really nailed it down to
speed, effectiveness, quality and business impact as the four
pillars to kind of put a story together about developer
productivity and developer effectiveness.
So there's not going to be a single metric.
I think that's an important thing also to call out here.
We've been looking for a single metric for decades now.
Maybe we thought it was lines of code before and now we, I don't
know, maybe we thought it was number of commits or number of
PRS. We're just not going to get to a
single metric because software development is just inherently
very complex. There's a lot of knowledge work,
there's a lot of different processes and social systems
that also feed into it. But I think those four really
cover comprehensively the most important bits.
Yeah, the new framework that you mentioned is called DX Core 4,
right? So it's speed, effectiveness,
yeah, quality and impact. So it's very exciting to always
hear new research, you know, new kind of metrics, you know,
especially, right. So how do you compare this with,
you know, the other metrics that were available before?
So I think many people would have heard about Dora metrics,
right, the four P metrics and also space framework.
And I know DX also published a paper called DFX, you know, a
few years ago. So how would you differ between
this DX Core 4 and the other metrics?
And is it just another metrics that we have to study and
implement our company? Yeah.
So the aim with DX Core 4 is that these, the metrics that we
have in the DX Core 4 framework are like they unify the other
frameworks that come before it. And also the research that went
into DX Core 4, it's all building on top of the research
from Dora, from space, from the Dev X framework, etcetera.
So this is not meant to necessarily.
I think there's all often it's like, which one are you going to
choose one or the other? When in fact, they sort of all
kind of coexist in the same universe and have different
purposes. The DX Core 4 has all of the
four key Dora metrics integrated in it.
It is an implementation of the space framework and it uses the
concepts from Dev X. So the Core 4 is sort of
everything altogether, but simplified enough that it's easy
to get started with. Yeah.
So I think that's a very good thing, right?
So now we can consolidate our effort into this just the X Core
4, right? So maybe if you can explain a
little bit about those 4 dimensions, how did you come up
with just these four? Are there any kind of research
or you know, something that kind of like brought you into this
conclusion? Yeah.
I think fundamentally the backbone of the DX Core 4 is a
concept called oppositional metrics.
And, and you'll see this also in Dora and in space and and you
know, many metrics frameworks. But the idea is that when we
look at many metrics next to each other, we want to have
metrics that are sending signal and work in in conjunction with
each other, that if one goes up, we want to see the other one
like also go up. And if the other one goes down,
we know it's a, a signal that something is wrong.
And so, for example, we could take speed and quality because
that's an often the classic trade off is like, are we going
to go faster or do we want to make it good and, and quality.
And usually it's seen as a trade off.
And what core 4 and a lot of these others metrics frameworks
suggest is like, it should not be a trade off.
You should be able to move faster while increasing quality,
and so we want to see those things move together.
The same for effectiveness, which is like the the measure of
developer experience. How effective can developers be
in the system in which they develop software?
If we see speed go up but the developer experience degrade, we
know something is wrong. Or if we see a really great
developer experience, but quality goes down, also not
good. So we want to have these sort of
levers and checks and balances built into the framework.
And so speed, effectiveness, quality and business impact
really cover the biggest corners of software development and keep
those things in check. For the individual metrics, what
we're doing is suggesting a key metric and then several
secondary metrics if you want to go a bit further into your
measurement. So for speed, that is diffs per
engineer, these are like PRS or merge requests or whatever it's
called. This is a metric and it's, you
know, it's the first one, it's the first one at the table.
It's the first one I always talk about because it is the most
controversial. Definitely.
I think even a few years ago I would have said like, that's a
bad idea, don't do it. But we have seen lots of
companies use this metric to provide helpful but limited
insight into the health of their software development cycle.
One really important thing about this metric of diffs per
engineer is that we're not trying to see, oh, how many PRS
did Henry close and then how many did Laura close?
We're not looking at that. We just want to look at a
developer in your organization, you know, are they closing on
average 3 PRS a week or 10? You know, what is that?
What does that look like? Because that can provide us
information about how easy it is for developers to develop code
just like fundamentally what we're doing here, right?
So we wanna keep track. Like are developers able to
actually produce code that has a lot to do with effectiveness,
which is the measure of developer experience in the
system. So companies that have a great
developer experience have developers that you know, just
don't lose time to things that are wasteful, are friction, you
know, a lot of drag. They are satisfied with their
tools, their processes, etcetera.
They, you know, they can make room for innovation, those
things. So we want to measure that and
make sure that that's again one of the four pillars and that
it's keeping balance with the rest.
Quality is change failure rate. That's a classic good old Dora
metric. If you recognize that change
failure rate. And then impact is the kind of
the business number here, which is percentage of time spent on
new capabilities. Thanks for sharing this overview
about those 4 dimensions, right? I think definitely the first
thing that always kind of like controversial is counting the
number of something, right? So this time in the DX Core 4,
it suggested that we should count the number of diffs per
engineer, although in the aggregate not like individuals,
right? So this this diff could be the
PR or merge request that we are accustomed to these days.
So tell us why this is something that is more prominent versus
the Dora metrics which previously advocated something
like number of deployments or maybe the lead time, right time
to bring value to production. So why diffs become more
prominent rather than those? Yeah.
So Dora metrics are designed to measure software delivery
capabilities. It's really focusing on like
once the code is written, how can we get that code out to our
customers? So you'll see things like lead
time, deployment frequency change, failure rate, time to
recover from a failed deployment, which used to be
called MTTR. These are really focusing on
that delivery side of things. The diffs per engineer focuses a
little bit more on the code authoring side of things like
the, you know, everything that happens before the Dora metrics
kick in more or less. So the activity measurements.
So you mentioned the space framework before.
For those listeners who haven't had a chance to work with the
space framework or read about it, the space framework suggests
that there's five main dimensions to developer
productivity. We've got satisfaction and
well-being, performance, activity, communication and
collaboration, and efficiency and flow.
Those activity metrics are usually what we think about when
we think of developer productivity, like commits,
lines of code. It's not a great metric.
Other things like that number of PRS, for example, those all fit
into the activity side of things.
We're not saying like we should ignore those activity metrics.
That's not what space came out to say.
They're saying activity metrics exist.
They provide limited insight into how well the system is
performing. Also, you need to make sure that
there's all these other things accounted for as well.
And so that's what we've tried to do with the core 4 is to not
ignore the activity metrics. It's still an important part of
system performance, but give it more context so that it's
actually meaningful and gives us information, which is why Disper
Engineer not in an individual level, but not as the only
number. We never want to look at that
number on its own. We always want to have it
accompanied by effectiveness, quality and impact.
Yeah. Also not to mention that you not
just have the key metrics, right, the one metric for each
dimension, right? You actually also have secondary
metrics, which I think in this speed dimension, right, you also
have likely time and also a number of deployments as well.
So I think again, like coming back to what you said, there's
no one single metric. So it's always kind of like
combination of a couple of them. And the other thing that I find
also interesting is about impact, right?
You're not actually measuring in terms of dollar revenue or
number of user or number of whatever, right?
But actually it's kind of like the number of new capabilities
and kind of like new innovations probably.
So why is that more important rather than, you know, the BAU
stuff, right, bringing more revenue and profit to the
company? Yeah, part of this is like every
business is different and I guess like when we get into it
depends territory like we are not able to provide outstanding
benchmarks about like what is a great business performance for
your particular company in terms of like monthly active users or
daily active users. That's not really something.
But what we can do is look at in general how, what's the ratio of
like what's revenue per engineer, for example, for every
dollar of revenue that we get in how much are we spending on
developer salaries. That's something that we can
kind of look at standardized and see and compare, you know,
companies or like R&D as percentage of revenue.
We can look at different ratios and see, OK, how do we like how
does this look across the industry, but then more
importantly is like, how do we look compared to ourselves and
is this number going higher or lower?
We're not suggesting that you shouldn't measure daily active
users or monthly active users, like please do that if that's
the if that's the metric that you're looking for.
But when it comes to kind of broadly speaking about business
impact and how organizations are approaching this across the
industry, these are a bit more easy, like they're a bit easier
and more straightforward to standardize on.
You mentioned about the IT depends answer, right?
I think this is quite typical in any kind of discussions about
developer experience. I even have a friend telling me
that, hey, in your past episodes, why is it always kind
of like it depends, right? So I think it's a more
opinionated answer is something that many of us would kind of
like appreciate or kind of like want to follow, right?
But in this case, the X Core 4, I think it's kind of like a new
new, kind of like a new paradigms.
How can we actually start measuring this, maybe in our
engineering team or engineering organization?
Is there some kind of different mechanism that you would
suggest? Yeah.
So in the DX Core 4, we'll go over the key metrics and
secondary metrics, but we also have a third row, I guess or
column depending on how you're looking at in that table, which
is about data collection. So I'll just like I'll get right
to my sort of opinionated answer.
Everything in here can be collected via a survey.
So surveys are super powerful. They're not just about gathering
people's opinions. You can gather quantitative data
via a survey. It's self reported data, which
quite honestly is high enough fidelity to make the decisions
that you need to make with the data.
So whether your lead time is 46 minutes or if it's between 30
minutes and an hour usually doesn't matter when it comes to
what do we need to focus on and what do we want to improve.
So you can gather all of these metrics with a survey and self
reported data. You might have to ask some
different people for different surveys, especially for
different survey answers, especially when you get into the
business impact stuff because individual engineers probably
don't have information about R&D as percentage of revenue, but
they can tell you how much time they spent on maintenance work
versus new features last week, you know, So my opinion, the
answer is start with a survey. You can get all of this via a
survey and then figure out what would be useful if the
collection was automated via, you know, scraping data from an
API or integrating with some tool and then go build that.
But don't delay getting a baseline because a baseline is
going to give you direction into what you should start improving
and then help you measure your progress.
And so delaying a baseline to build something that you're not
even, you don't even know if it's going to be useful because
you haven't gone through the process of getting the
information, making a decision, changing something, measuring
again to me is a a bit of a waste of time.
So I would start first with the baseline, make your changes,
measure them, and then figure out what you want to invest in
when it comes to automated collection.
Yeah, getting the baseline, I agree.
It's I think it's very important as well because in the past I've
used the X as well, right. So we were actually like having
a lot of challenges, right. We know we have problems like
anecdotally, we can hear from engineers, OK, we had built
issue, built time issue, you know, quality issue, but it's
all about opinions, right? Rather than kind of like, you
know, quantifiable, right? So I think get getting the
baseline across different teams, across the whole engineering.
In fact, I think it's something that is super critical,
especially if you're a leader, right?
Because without that baseline, I don't think you would know what
kind of things you should improve.
And plus there are so many different dimensions and
problems, right? I think it's also difficult to
kind of like improve everything, right?
You're gonna, you know, put focus on some certain areas
rather than, you know, throwing all your effort into all
different dimensions, right? I think, I think the baseline is
definitely super critical. And I think another thing that
is one of the dimension that I would like to ask you is about
effectiveness, right? So I think in your DX Core 4,
right, you mentioned about Developer Experience Index or
DXI. Maybe this is something new for
some of us who haven't heard about it before.
Tell us what is this DXI? Yeah.
So Dxi is a developer experience index.
It is a predictive benchmark of developer experience and all
that it is, is just a mean average of 25 developer
experienced drivers that you can find online.
It's like the DX 25, we can find it on DXS site.
What we found over time, now that we have a lot of companies
using DXA, lot more data is that companies that score higher in
these Dev X drivers have less time wasted.
And so that again, that time waste is not just in terms of
like salary and, you know, frustration, but really the main
benefit there is like, what would you have built instead?
What could you have built instead?
Because even if like so for example, the average developer
wastes 20% of their time each week on inefficient processes,
meetings that are not useful to them, waiting around, waiting
for a feedback, you, you name it.
I mean, if you've worked as a developer, it probably probably
doesn't take you long to list out all the ways that your time
gets wasted in a week. Or when you're just kind of
spent, you're, you're just waiting for stuff.
Even taking that down to 19%, I mean, this is like a minuscule,
but like, let's give developers back just like a little bit of
time every day or like don't take them out of their flow a
little bit more. It does not take long for that
to add up to some. A huge amount of time of money,
but also opportunity like what else could you build instead?
And so that's really what the idea of this developer
experience index like it's not about ping pong and beer, which
is often I think developer experience has a like quite a
marketing problem. I don't know if there's a, we
should name it something better, but you know, when Abby has this
thing that just like makes me laugh every time he's like, Can
you imagine if a new salesperson or a new sales leader, like a
new CRO comes in and says, well, we're going to measure the
impact of our sales organization based on sales Rep experience.
Like no way. Like you would not like there's
just no way that that would ever go over.
And so I sometimes I feel like, you know, when developer
leaders, CTOSVPS directors and we talk about developer
experience, the perception is a little bit like, oh, these whiny
engineers want to play ping pong and drink beer after 3:30 PM
every day. And like, that's not what this
is. This is not a ping pong and beer
problem. This is like a crippling, you
know, this, this stops innovation and it's going to
cripple your company's ability to innovate if you don't fix it.
And so putting it front and center in the Core 4 along with
the other things sort of emphasizes like this isn't just
a ping pong and beer, it's truly like a fundamental business
problem that needs to be solved. But the DX I I guess to go back
to your original question is just a mean average of these
developer experienced drivers that are correlated with lower
time loss, better performance and engineering generally.
I think it's very funny when you mentioned that, right?
Yeah, I think it speaks true to the situation, right.
So why there is a developer experience but not, you know
other departments experience, I don't know like sales
experience, operation experience and things like that.
So I think it's very important that we kind of, I've put
perspective on this, right? I think productivity is kind of
like general term, right? To kind of like measure or
improve. So another thing that I would
like to ask you, since you have published this paper, right?
Has it been implemented in maybe some of your customers or maybe
other organizations be big or small?
Is there any kind of impact? So once somebody has, you know,
used this framework, new framework, right?
Is there any impact into how they improve their developer
experience? Yeah, there's quite like
hundreds of companies using the DX Core 4 right now.
And I think like when you start using it, you shouldn't expect
that, OK, now all of a sudden developer experience is going to
improve. What it does is it shows you or
or gives you some models to figure out where do I want to
focus my efforts and where should I prioritize our
resources in order to fix problems, to see the impact that
we want to see. The other thing that DX Core 4
really does for an individual engineering leader is it just
gives you better set of vocabulary words, common
language to take the problem of developer experience and the
problem of developer productivity, quantify it, but
also tell a better story so that you can walk into your exec team
meeting or whatever. Meaning that you have to go to
and draw a line in between developer experience and
innovation capability, for example, in that impact
category. I think before with other
metrics like Dora. Dora is really great, but it
only measures software delivery capabilities.
It's not tying it all together and really bringing it back to
the business, which is what we've designed Core 4 to do so
that an individual engineering leader, whether you're a manager
or AVP or CTO, you can have better conversations about why
productivity problems or, you know, lack of investment in
things like tech debt or whatever it might be, why it's
bad for the business. And then specifically, when you
change something, what are the results that you should expect
to see and then hold yourself accountable to that prediction?
Yeah. So when you mention about, you
know, not expecting sudden, you know, improvement, right.
So I remember in the past when I, you know, used to do this as
well, right? So after, I don't know, like two
quarters, we only saw like a mini improvements in some of the
numbers, right? So I think some of these
problems are very complex, especially the social dynamic
aspect. And plus, you know, people
change, you know, direction or maybe the mission of the
organizations also change, right?
And I think don't forget about that aspect as well.
It's not like something you just quantify and you know, you put a
lot more effort and it will be improved after a while.
So I think the other aspect is how do you actually motivate
engineering leaders to actually try bringing business case for
improving developer experience? And do you suggest also doing it
quarterly or is it something like a yearly thing or is there
any kind of a, a best practice that you would suggest us to do?
Yeah, I think always all the time, I think, you know, I don't
have a great answer to the cadence part of it, but let's
take a maybe a, a bigger picture answer, which is like, who are
your enemies and who are your allies?
So I know we're going to talk about like my tech leadership
wisdom at the end of this episode.
But like, you know, adjacent to that answer is like there's
always going to be someone who benefits from an inefficient
system. Doesn't matter how bad something
is, there's always going to be a group or an individual that's
going to benefit from it. And so you really need to figure
out like, why is the systems like, why do we tolerate
developer experience as it is? Or like, why do we tolerate poor
productivity or like bad tooling?
Like who is it benefiting? Is it benefiting us because we
don't like change and it's just easier to not change than to go
through the discomfort of having to change, You know, why is
that? But try to figure out like, what
are the, the organizational factors that are keeping the
status quo in place? So that's like your enemies kind
of category. There's also your allies and
those are like your Co conspirators, people who are
going to, you know, just as fired up about developer
productivity and about developer experiences.
You are that might be another engineering manager or leader.
It's really great if it can be like a product manager or
someone from a cross functional function or a senior individual
contributor, like staff level, principal level, like those are
really great allies to have. Because then instead of it being
you going to your VP or your CTO and saying, hey, we want to do
X, it's like a bunch of you. And there is strength in
numbers. There is strength in numbers.
I would also look for the person at your organization who's going
to sponsor this. So like an exec who is going to
pave a path for you. You know, you might need more
budget, you might need resources or headcount.
You're also going to need sort of like a mandate to be able to
get everyone on board with these types of projects being on the
road map, so to speak. And so who's gonna sponsor you?
And then concretely, what are, you know, if you do establish
this sort of like working group, like what are you really
responsible for? What are your KPIs, so to speak?
And that's really where the core 4 comes in, which is like, how
do you measure the impact of what you're gonna do?
So that's where I would encourage if, if an engineering
leader is like interested in pursuing this is like first
figure out like your, you know, like your tactics on the ground,
like who are you working with? What are the factors working
against you? What are those objections?
Who is an executive sponsor that you can, you know, bring this up
to and try to kind of make a plan and then pitch it that way
and then take it from there? Very interesting advice you have
there, you know, figure out your allies and kind of like your
detractors, right? So the one who benefited from
the inefficiencies or maybe status quo, right?
Sometimes we don't like changes, especially if it's too drastic,
you know, coming back to the example of monoliths micro
service or the other way around, right?
It's kind of like a big change, right?
It's something that not everyone is incentivized to make it
successful or make it happen, right?
So I think figuring out allies and work together in order to
come up with business case definitely is a very, very
crucial. Another important thing that
typically is always discussed when we talk about developer
experience. Lately, it's about introduction
of internal developer platform or even AI these days.
So maybe in your view, do you have any take of these two,
right, IDP and AI? How do you see them as a core
components these days to improve developer experience?
Or is it something just nice to have for some organizations?
Yeah, I think developers see it as a core component.
And So what I'm interestingly seeing, I've had quite a few
conversations about this in the last few months, is like, how do
I as an engineering leader, make a business case for investing in
an IDP or investing in Copilot, for example?
Especially with AI, some of the promises are very lofty.
Let's just keep it that. Like, let's just say that
diplomatically. I mean, you know, I don't know
who really believes that, but like, AI in general, you know,
is just sold as like, oh, it's going to make everyone twice as
productive and you can reduce half your staff or, you know,
whatever, whatever the promise is.
That's just not really what's happening.
It does reduce cognitive load and make developers work faster,
help them work faster. But you know, it's still, it's a
tool that needs to be operated by a skilled individual and
using an irresponsibly can lead to more problems when it comes
to maintainability and creating more problems for yourself.
So figuring out what purpose these tools should serve and
then figuring out a way to measure that impact and tell a
story with numbers and make a business case for it again is a
very important thing. So, you know, one thing we
haven't really touched on is like attrition or being able to
attract talent. A lot of developers, if you ask
them like if they were interviewing for two similar
companies and one of them had a copilot license for them and the
other one didn't, most developers are going to go work
for the company that has a copilot license for them.
And So what does that actually cost you?
And you can figure out the numbers like you might have to
ask some other people, but like how long does it take the
average from job requisition to be posted online for that role
to be filled? Then how long does higher like
training take? Like on boarding, you can figure
out how much expensive that is. And then how expensive is it
when someone quits and goes works for someone else because
they're not happy with developer experience.
Like we can put a number on attrition pretty easily.
It usually it costs like 75 thousand U.S. dollars, I would
say at a minimum to replace an engineer like that's mid level
or above. And that's not nothing.
You know that that's pretty significant when it comes to
like what are you going to pay for copilot licenses?
But these are some of the ways that I might put together a, a
narrative around something like Copilot or, or an IDP.
It's not just about like what's the tangible benefit in terms of
productivity gains, but also let's look at space.
What about satisfaction well-being?
What about communication and collaboration?
What are those other things that are positively impacted by these
tools being in place? Yeah, thanks for highlighting
that. We have to understand the
purpose before we implement those kind of thing, right?
Because many of us, we read from the research out there, people
raving about, you know, IDP or AI, right?
And we just follow, right? Implement it without actually
understanding the the impact or the purpose or the even the the
things that actually sometimes like in the past episode, I have
this Andrew Boyaji, right? So he highlighted the state of
developer experience report. So he highlighted that leaders
are just following, you know, maybe the trends, right?
Implement AI and thought by doing that they actually can
improve developers, but developers thought differently.
They just find like maybe a few percentage improvement, but it
doesn't really actually necessarily improve the whole
developer experience. So understanding the true
purpose behind implementing those I think definitely is a
key. So we discussed a lot of things,
you know, maybe is there anything that unintuitively you
figure it out but many people are not aware of in terms of
developer experience, is there something that you want to
highlight? So for us who may not have heard
before in the industry or in the research these days.
I think that maybe the most surprising thing is that your
teams just, they know where the problems are.
They work in these tools all the time.
And I don't care what metrics framework it is like if it's the
TX Core 4 or Dora or any combination of metrics aligned
to the space categories. Do not expect that capturing
workflow data and looking at GitHub data, JIRA, GitLab,
whatever, is going to point you to some like unknown about
developer productivity that like suddenly now that I have all
this data about commits and story points and number of PRS
and how work is flowing through the system, I'm going to be able
to find this magic bottleneck that I'll be able to fix and
like unlock productivity for my team.
I have never seen that happen. I have worked with literally
literally thousands of engineering leaders at this
point. I have never come across a
single case where engineering team, let's just say, has
discovered something new based on workflow data or quite
honestly even like survey data. The thing is like your team is
in these, they're in the tools all day long.
They know where the problems are.
But surveys can be so helpful to just tell you how big the
problem is. Like we know the problem is
there and then we can put some kind of formality around
measuring how big the problem is and if we're fixing it or not.
So I think this myth of like I'm going to implement, I'm going to
get some developer productivity dashboards and like suddenly I'm
going to be able to forecast and see ahead when my team has
productivity problems. Or I'm going to kind of find
this like secret bottleneck, like it just doesn't exist,
doesn't exist. Your team knows where the
problems are. Just believe them because they
are professional adults who do not have incentive to lie to you
unless you give them incentive to lie to you.
So gaming the system isn't actually as as big of a problem
as we think it is as well. That's another surprising thing.
Wow, I think thanks for highlighting that, right.
So I, I, I left when you explained all that because,
yeah, I think many people try to automate, you know, you know,
getting the workflow data, getting metrics out
out-of-the-box, right? Only to find that if you ask
maybe the engineers or the teams, right, you could get the
signals master. Other than spending all the
effort to actually come up with the dashboard, maybe.
And in the end you kind of like, no, that's the the like the
problem exists from some people who may have shared it with you.
So you don't need to actually spend a lot of effort.
So Laura, it's been a pleasant conversation.
So I think we all learn a lot about improving developer
experience. So as we reach the end of our
conversation, I have one last question for you.
I call this the three technical leadership wisdom.
So you can think of it just like advice that you want to give to
us. Maybe if there anything that you
want to share with us today? Yeah, I have so many things.
I'll try to keep it to three. On the topic of metrics, I think
the saying that like all systems are wrong and some are useful is
really important. And so I would encourage anyone
listening to have a healthy dose of skepticism for any type of
data that you're looking at, making sure you understand where
it's coming from, understand the purpose of it, all the things
around it. Because measurements are going
to be wrong. Even a broken clock is right
twice a day. So just don't let data push you
to incorrect conclusions. I think that's a very dangerous
area in this developer productivity metric space.
Yeah, I think, again, as I said before, things work poorly
because it sometimes benefits us to have them keeping, like to
stay working poorly. Maybe that's ourselves and we're
the enemy there because we are resistant to change.
And maybe there's another factor in there.
So I would figure out what those things are.
You know, you'd really try to debug the system and maybe to
follow on this like, you know, metrics thing.
A lot of us that are, you know, were previously individual
contributor engineers are going to try to pattern match or bring
a lot of the skills that we had as an engineer, for example,
debugging skills and try to debug our teams and
organizations in the same way that we debugged like our
Kubernetes clusters or whatever. The thing is like, your nodes
can't talk to you and they don't have feelings, and talking to
people because your teams are made of people isn't being
illogical. It's actually the correct way to
approach measuring a system that's made-up of people.
If we fail to recognize that people make up teams and we try
to treat them like a Kubernetes cluster, we're actually missing
sort of like half of the whole system.
And I would just say that's sort of categorically incorrect to
approach it that way. So make sure that you're not
forgetting that software teams and software development is a
socio technical phenomenon. So we need to have the social
aspect and the technical aspect together.
Both of them are equally as important.
So don't forget that humans exist.
Developers are adults. They're professionals.
They don't have reason to lie to you.
Treat them as adults. Believe them when they tell you
something is causing them friction in the deploy or the
development process. Wow, that's really beautiful.
I don't know why when you explain that, right, I always
come back to this phrase, you know, pet versus capital thing,
right? So I think for engineers, we try
to make everything kind of like uniform with big container
Kubernetes cluster, right? And we treat everything like
capital, right? But actually all individuals are
different, right? We have our own needs, our own
challenges, right? And don't forget about the
people aspect. So I think that's really
beautiful. Yeah.
So, Laura, if people want to follow you or you know, maybe
discuss with you further about some of these topics, is there a
place where they can find you online?
Yeah, go to laurataco.com. That is where you'll find more
about me. My blog.
I do have a course on developer productivity metrics.
I'm going to be running it more frequently, so you can check
that out as well. And you can find me on LinkedIn.
Just search for Laura Taco. I think I'm the only one.
Right. I'll put that all in the show
notes. Thank you so much for your time,
Laura. I think we all can learn a lot
about developer experience from you today.
So thank you so much for sharing.
Wonderful. Thanks for the lovely
conversation.