Technical debt is not really a technical problem.
It sounds as if it is, because it's got the word technical in
there and it's got this debt. But really what it is, is it's a
problem about trade-offs. Today's guest is Andrew Richard
Brown, the author of Taming Your Dragon: Addressing Your
Technical Debt. He has spent over two decades
uncovering the hidden human factors behind technical
problems that challenge how we think about technical debt,
risk, and project estimation. I always find it quite
fascinating, because we all hate technical debt, but every time,
either consciously or unconsciously, we are also
creating technical debt ourselves.
The way we make any decisions is we use our emotions.
We may think we use logic, but actually we use our emotions.
If you look at the technical debt, potentially you're gonna
have to repay some debt in the future.
Something that is in the future, it's gonna appeal to your logic.
That's principally why the technical debt does badly.
From all these layers, technical aspect is just a small part of
the technical debt. There are so many other things
like organizational system, the bias thinking, the emotion.
How can an organization start managing their technical debt
better? What you really need to be doing
is avoiding creating the technical debt that you can
avoid creating. You can't do development really,
really fast. And then when you become
completely tired, now we will slow down to a sustainable rate.
It's a bit too late. You need to make an arguments
that are appealing to people's emotions, not to their logic.
You can sort of paint a picture. You can tell a story.
If you're coming at it from the viewpoint of we need to manage
this technical debt or burn it down, you are almost starting
too late. Once you do that, it's almost as
if you've given the business permission to create this
technical debt, because you can just pay it down later.
Hello, guys. Welcome back to another new
episode of the Tech Lead Journal podcast.
Today, I'm very excited to have Dr. Andrew Richard Brown, or
I'll call him Andrew Brown in this episode.
So Andrew is the author of a book titled Taming Your Dragon.
Um, so it's a book about tackling or managing technical
debt, which I'm sure all the engineers here, uh, know about
technical debt. Uh, we hate technical debt, but
also at the same time, sometimes we create technical debt
ourselves. So today we are going to learn
what is actually technical debt and how can we manage it much,
much better from Dr. Andrew. So Andrew, uh, looking forward
for this conversation. Thank you so much for this
opportunity. Thank you for inviting me on.
Right. Andrew, in the very, uh,
beginning, I'd always love to invite my guest to share a
little bit more about yourself, right?
Maybe you can use like career turning points that you can
share so that we can learn from you.
Yeah, sure. So, um, yeah, so I think that it
was perhaps about three career turning points for myself
really. So the first one was I did a PhD
in semiconductors. And so I had a plan.
And my plan was to have a career in semiconductors because I
could see that as a bright future.
And there was gonna be a big future in semiconductors there.
But when I graduated, the semiconductor industry was in
recession and it remained in recession for about two years or
so. And so I think then that, that
basically, that taught me a, an important lesson right at the
start of my career, which was that actually, although you may
have flat plans, you are gonna have to be flexible about those
plans. And so I looked around and
eventually I got another job at a, in a different industry,
which was in scientific instruments.
And I traveled around the world actually as an engineer
installing these machines. So they were pretty big
machines. They were called mass
spectrometers. They were about the size of a
car, weighed a couple of tons, and they cost about half a
million pounds, and they took about three months to install.
So you're spending a lot of time in foreign countries there.
So I saw a lot of different countries, a lot of different
cultures, and that sort of woke you up as well too.
Things could be different, equally valid there.
And actually that's where I met my wife or my late wife.
Um, I was working in Japan and I then met her whilst I was
working out there. And again, that came to another
big career decision and I decided to move to Japan so that
we could be together. And so I moved to Japan.
We had a nice time and then eventually we came back and we
lived and settled down in England there.
The other big change or the big career turning point was that
when I started, I was in a lot of hardware there.
And I resisted. I could see that software was
increasing, but I resisted going into it.
And part of the reason was that I could see that I knew some
friends and they'd been in software development.
But the technology had moved on. The languages they were using
had gone out of fashion and they were out of a job and they found
it very difficult. And so I was very reluctant to
get into an industry where they just threw people out of work
like that. But actually, eventually, I went
in there and I'm glad I did because although, and I have
been thrown out of work a couple of times.
Actually, so long as you're keeping your skills up to date,
and that's a really important thing in software.
If you keep your skills up to date so you've got something
marketable, you may lose one job but there will be another job
there somewhere that you can get into.
And so I would say that that was an important lesson there really
of making sure that you always stay up to date and current or
you are learning something new. Always, always be learning
something new. Yeah.
Oh, thank you for sharing your story.
I think it's very insightful and very timely as well in this kind
of a time, right, where people, maybe some of us are feeling a
little bit concerned about our jobs, our career.
Maybe because of the AI disruption, maybe because of the
economical situations and so many changes, uh, happening
globally. So I think, uh, I really love
the first point as well where you should be flexible.
I think many people career, right, doesn't go as planned in
the very beginning, right? We always love to plan, oh, I
wanna do this, I wanna stay on this career track, to climb the
ladder or something like that. But most of the people that I've
talked to actually said that their career plan actually are
different than what they used to think about in the very
beginning. Before we continue, I want to
tell you about our sponsor Jellyfish, who helps engineering
teams measure the real impact of AI adoption.
Let's hear what they have to say. 90% of engineering teams
use AI, but few can measure its actual impact.
It doesn't have to be a guessing game.
Jellyfish gives you a comprehensive view across your
AI development activity. Know where AI hurts, where it
helps, and how to guide your team with confidence.
Learn more at jellyfish.co/ai. And now let's get back to our
episode. So I think maybe, uh, let's
touch on a little bit on this. So for people who are maybe in
lots of concerns these days or whatever situation that they are
in. So maybe looking back, what kind
of, um, confidence, that made you, you know, brave enough to
take those kind of, uh, risky moves, maybe back then, right?
Because you just said your friends, you know, got out of a
job because programming language changed and things like that.
So maybe people can get some inspiration from your journey as
well. Yeah.
So I would, I would say that in terms of, uh, making risky
moves, yeah, it's something that actually people talk about
afterwards, but at the time, they're not actually as risky or
you make it as- as un-risky, at least risky as possible.
So for example, if AI's coming along, it would be madness to
leave your job and start learning AI or something like
that. You should be continuing in what
you are, seeing what is happening.
And most of us can read fairly clearly that big things are
happening in AI. And you can actually then start
to learn about AI and get some skills there that are marketable
in the environment and in the industry.
'Cause one of the, you know, terrific things about the
internet is that so much knowledge, so much information,
firstly, is available, and secondly, so much of it's
available for free. You don't have to be paying lots
and lots of money for these things.
You can do it, and there's people out there that are happy
to take your money off you, but there's enormous amounts that is
free. And a lot of it's from very,
very good institutions like the MIT, Massachusetts Institute of
Technology, is giving away loads and loads of its online courses
for free there. So look for education that's
free and get yourself skilled up before you need to be skilled
up. Yeah, I think it's always, uh,
very useful to always keep yourself kind of like hungry for
knowledge, right? Keep learning as and when new
technology coming in. Uh, definitely we have fears.
I also have fear about, you know, how AI is gonna disrupt my
role. But I guess, um, depending on
your passion, depending on your interests as well, you can
always keep that hunger to always learn, upskill yourself
and make yourself marketable, that which is something that is
very important as well. So, Andrew, today we are going
to talk about your book, Taming Your Dragon: Addressing
Technical Debt. So in the first place, maybe
let's go to the definition of technical debt, because many
people in the engineering, software engineering industry
would have used technical debt as a term, maybe quite loosely.
Um, so maybe in the first place, what do you mean by technical
debt? Okay, so I think that I would be
fairly conventional with what I mean by technical debt.
I would take technical debt to be the additional work which you
will or someone will need to do in the future as a result of
choosing to a solution which is expedient in the short term, but
is going to cause you longer term issues there.
So, I would say it's no more than that really.
Wow. This is, uh, quite a very
interesting definition because, uh, yeah, you can see that
there's kind of like a trade off here, right?
So long-term kind of problems, but a very maybe short, fast
delivery, right? Uh, it's kind of like taking
shortcut in a sense. So I think I like that
definition, right? So, because many people
associate technical debt in software engineering, especially
with financial debt. So, you know, thinking about,
okay, we take some debt here maybe for our loans, mortgage,
whatever that is, right, and we'll kind of like repay them
back sometime in the future. So I know in your book you
mentioned about this kind of analogy maybe is broken in some
ways. So maybe tell us why financial
debt may be not a good association with technical debt.
Yeah, so certainly. I think that, well, the first
thing to say about the analogy to financial debt is that it's
given us tremendous benefits there.
And because first and foremost, everybody understands financial
debt. So everybody has taken out a
loan of or owed money in some way, and so they understand
that. They understand the benefits and
they understand the consequences.
So they understand that perhaps if they take on some debt, they
can do things that they couldn't otherwise do.
And that's the benefits of it. So, you know, you, if you, um,
you want a job in the next city, but you need a car to get there,
then you take out a car loan. And so that's a good debt to
take out. And you also understand that
you're going to have to continue paying that after you've
received the benefits. It's very useful in that way
because it gets people to understand it very, very well.
So it allows people very quickly to understand the essence of it.
The problem with the analogy of technical debt is that it isn't
an exact analogy. And so technical debt is, it
would be like you took out a car loan in order to get this job,
but you don't know what your interest rate is, and you don't
know when you're gonna have to repay it.
And maybe you're never gonna have to repay it.
Maybe somebody else is gonna have to repay it, you know, in
which case we'll take out some more.
And you don't know exactly all these details of it and you
don't know what the interest rate is.
And you don't know the repayments or anything like
that. And also, perhaps the most scary
thing is that at the same time you're taking out this technical
debt, perhaps someone is handing out lots and lots of credit
cards to that account there. And all these other people,
including your daughter, spend thrift daughter, she can then go
out and rack up lots and lots of debt on this debt to get
benefits for herself, which you are gonna have to pay out.
So that's perhaps a closer analogy to what the technical
debt really is. It's a very scary, very strange
kind of financial debt there. Right.
So I love that you mentioned that we don't know the interest
rate that is gonna be accumulated if you don't pay,
repay the debt, so to speak, right?
And you kinda also don't know when you should repay the debt
fully, right? So I think that's a very good,
uh, reminder, right? Because when we talk about
financial debt, usually it comes up with all this commitments,
you know, contract, whatever that is, right, upfront.
But when we incur technical debt, probably we don't know how
to answer some of those. So you mentioned about, for
example, people take technical debt for shortcut.
Um, some of these are really not just technical kind of decision,
right? But in your book you mentioned
it could be organizational issue or some kind of human bias.
So tell us these two elements. Why is it also important to
actually put in place when you want to understand technical
debt? So, I think the first thing to
understand about technical debt is that it's not really a
technical problem. It sounds as if it is, because
it's got the word technical in there and it's got this debt.
But really what it is is it's a problem about trade-offs.
So nobody takes on technical debt just for the fun of it.
We take on technical debt for a reason.
And usually, we take on the technical debt because we want
to be able to do something that we couldn't otherwise do if we
didn't take on that debt. So it's usually, it's going to
be something like we want additional features there.
So for example, the product manager might, they know that
they that this technical debt is gonna be bad for the product
long term, but they need to squeeze this extra feature in.
Or perhaps it is for a project manager, it is, they know that
piling up technical debt is going to be a problem, but they
need to get this project in and they are being measured by that.
So it's technical debt is just a consequence of a trade off that
people have made. So they've made it there.
That then leads to an interesting point because
practically every company you go into, they will say, we've got
too much technical debt. So then you've really gotta ask,
well, what does that mean, you know?
It must mean that you've made trade-offs that now you are
regretting, in which case then you have to ask, well, what has
caused you to make trade-offs that you now regret there?
And then that gets into the technical debt, the trade-off
there. So really, if you want to
understand why you've got too much technical debt, you've
really gotta understand how your mind makes trade-offs and how
you trade one thing for another and why it is that technical
debt comes off so badly in that trade off, really.
Yeah. So I always find it quite
fascinating because I personally also have this kind of thinking,
right? So we all hate technical debt.
Every time we go to organizations, seeing any kind
of code base, we can point out really, really, fast that, okay,
this is not good. This design probably doesn't
scale well. It's a technical debt here and
there. But every time we create, for
example, new code, develop something new, conscious, either
consciously or unconsciously, we are also creating technical debt
ourselves. I think also maybe because this
is the nature of software development, where you can't
always, you know, create a perfect design since the very
beginning, right? So maybe tell us this kind of,
uh, I would say is a paradox, right?
Sometimes you hate technical debt, but you're always
incurring it no matter what. What do you think will be a good
mindset for software engineers out there to have a healthier
relationship with technical debt?
I think that really, to have a healthier relationship with
technical debt, what they need to be doing is the engineers
need to be moving beyond thinking of technical debt as
this technical problem there. It's actually because it's a
trade-off problem, almost certainly, they are only gonna
be partially involved in making that trade-off.
Often, it's gonna be the stakeholders.
And so the stakeholders, they're going to want this project in or
they're gonna want these features.
And they are prepared to put up with, either put up with this
technical debt or they are unaware of the long-term
consequences of it. So it's really, it's having that
conversation and uncovering and surfacing these contradictions
there. And basically between the
technical debt, balancing technical debt and balancing
whatever those business needs are, because if you built
software so there was no technical debt, it would be
delivered so late that it had no business purpose there.
So there has to be some kind of trade-off, and it's probably,
it's gonna have to be made by the business.
But the business aren't sufficiently well informed about
the technical consequences of this that they can make a fully
informed decision. So it's having those
conversations there with the business around it so they can
do a better balancing act there. Right.
I think it's a very good reminder because, uh, sometimes
technical people, software engineers, we love to make
decisions solely, purely from technical point of view, right?
But I think you just put up a good reminder, uh, because you
need to communicate. You need to un-surface, uncover
this kind of a potential issues, if let's say you take a certain
decisions. I would say it's more like an
informed trade off rather than, you know, purely just technical
basis, uh, kind of like a trade-off.
And sometimes also when we develop, we know that it's going
to go into certain direction, uh, and we just assume, many
assumption in our head that, okay, this will go this way.
But actually, if you talk about it, maybe with your team, maybe
with the stakeholders, I mean it could turn out to different
direction. Maybe different trade-off will
be taken altogether, right? So maybe speaking of which, um,
for putting a healthier relationship, I know that you
also come up with this technical debt onion model, which is kind
of like framework or maybe perspectives how we should, I
don't know, like uncover different layers of technical
debt. So tell me, uh, well first of
all, the idea of this onion model.
So the idea about the onion was that, well the first part was I
had this realization that technical debt is not a
technical problem. It's this trade-off problem.
And then understanding that. But then I gradually, I realized
that even if you address the trade-off problem there, so in
other words, if you understood how we make trade offs there.
And the way that we make trade-offs is, or in fact the
way we make any decisions, is we use our emotions.
We may think we use logic but actually we use our emotions.
And what we use in our logic is it's just post-decision
rationalization. We use the logic to rationalize
the decision that our emotions made.
In which case then you've gotta understand, well, why in that
emotional decision does technical debt fared so badly?
I look at that in the book then and talk about that.
But then what I realized was that even if you address those
problems there and you know that this technical debt is bad, and
you know and you understand that emotionally, what happens is
that actually you are not making your decision as an isolated
individual. You are making your decision as
a role in an organization. And in that role, you've got
some responsibilities, some demands that are made upon you.
So if as the project manager you are making that trade-off
decision about technical debt, you may know that it's really
bad for the company to take this on.
But also you are being measured by getting this project in
you've gotta getting it in on time.
And that's what you're being measured by, not by the
technical debt. So you are acting in your own
best interests, and you are acting actually as the company
is pointing you with its incentives and its rewards and
its punishments there. So you are, you are doing what
is appropriate for that role there.
So you need to understand how decisions in systems are made.
So you really need to understand systems thinking as well for two
reasons. The first is that these
decisions in a system will be made differently from
individuals. But secondly, you need to also
understand how things work when you have them in a system there
and how, for example, by piling up technical debt, you can get a
runaway effect there of suddenly too much technical debt causes
the development to really slow down from which you cannot
recover. So you really need to understand
these systems as well. And then what I realized from
there was actually, if you're talking about individuals making
decisions within a system, essentially that's their
economics. That's what economics is.
And so then I looked at, well, what can economists teach us?
And economists have been looking at problems similar to this for
hundreds of years. So they've got a whole series of
problems that they have highlighted and addressed.
And the really good thing about looking at it from the economist
point of view is that, firstly, that they, they've studied it
already, so you don't need to reinvent the wheel.
But secondly, they are talking the language of business very
often. So very often we, we find it
difficult to engage with the business.
And we may talk about technical debt and we may talk about these
risks and this and that. But all the business people here
is ya-ya-ya-ya.. And it doesn't really go in.
Whereas if you start talking about externalities or the
principal-agent problem there, it's likely that it's something
that they met at business school.
And so straightaway, you are talking their language.
So that's an advantage of this, looking at the economics layer.
And then underneath that I put a final layer, which was around
wicked problems. And so really, technical debt,
and in fact all, every, practically everything that we
do in software development is a wicked problem.
And so I talk about wicked problems and tame problems.
And so tame problem is it's a problem like chess or a game of
some sort that has some rules and there is a single solution
or there's a solution which everyone will agree is better
than another solution there. Whereas in a wicked problem,
it's got several characteristics.
Uh, one of which is that different people may have
different opinions about what is the best solution there.
So for example, on that of the road I live on, perhaps there's
a bit of noise. And my best solution is that
there's no noise at night. Whereas for the young people
that are coming back from the pub, there's a better solution
which is that there's a bit of noise as they come back from the
pub. And... but it's all quiet by
three o'clock. And so you can see that's one
thing that you could have there is that there's different people
may have different opinions about what the solution is.
But another thing as well, which is very highly relevant for
software development, more so than any, most other types of
engineering problems, is that a wicked problem, very often, you
will not know the, um, you will not know the solution until you
find the solution. So you won't understand the
problem until you develop the solution.
And very often for us, we don't know what the software is going
to be because it's part of the exploration.
We think we are building one thing and it ends up we tell
people we are building something else, and in the end it ends up
as something entirely different. Whereas if you looked at say,
civil engineering, say constructing a bridge, you're
pretty sure what you're gonna end up with before you even
start. But that's not true in software
development. So very often we, we don't
understand the problem until we got to a solution.
So that's sort of a part of this wicked, wickedness of the world
that we live in there. Wow, thanks for kind of like
walkthrough of all the layers. Just to recap, right, so you
have kind of like five layers. It's kind of like an onion
model, right? Layers by layers.
The inside, maybe the core, maybe it's technical layer.
And then you have the trade-off layer.
You have the systems layer, kind of like the systems thinking
aspect that you mentioned. Then you have the economics of
game theory layer, and the last one is the wicked problems
layer. So maybe let's try to peel it
slightly, um, smaller layer, right?
So maybe I won't go deep into technical layer because I think
for software engineers, it's kind of like assumed we know
what kind of technical problems. Maybe the design, scale, you
know, lack of tests and all that.
We'll just skip it all together. But let's go to the trade off
layer, because I think this is quite important as an
individual, right? Whenever we make decisions, we
sometimes make a lot of trade-off, either knowingly or
unknowingly. So what are some of the
interesting aspects of the trade off layer that you want to
remind us or you wanna kind of like advise us today?
So essentially, yeah. So the trade-off layer, when
you're making trade-offs, it's important to fully understand
how you make trade-offs, which I said is we use our emotions.
And although we might talk about logic and rationality, most of
the time the rationality comes in afterwards, and we just use
it to justify the decision that we've made.
And so you've really gotta understand what it is, how you
make decisions using emotions. Essentially you use something
called the effect heuristic, which is, it's like you're a
sort of, it's like your- an emotional sort of gauge there.
And what that does is it guides you in the decisions that you
make. And there's several important
things to realize. One of which is that this
process is entirely subconscious.
So you don't understand why you make these, um, decisions.
Although you will be able to justify it, you'll be able to
give a good reason, but that reason will be, uh, somewhat
spurious. And so they've done quite a lot
of experiments on people making decisions or having preferences.
So one of which was, um, they took some, I would call them
kanji characters, but that's because I'm coming from a
Japanese background there. But it would be the Chinese
characters, and they would flash this, uh, they'd have this
Chinese character up on the screen.
This is to Western people who don't, who cannot read, uh, the
Chinese characters. And they would ask them to make
a preference. Which one do you prefer?
Now what they didn't know was that although they were looking
at a screen there, it was a special screen and it had a very
fast refresh rate. And so what they were doing was
they were putting up onto the screen, every so often they were
putting up a picture of a happy face or an angry face.
And what people did was they had a preference for this character,
and they would give a reason why.
Because it was a, you know, it had smooth curves or it did this
or did that. But what it actually was, was
that behind their flickering too fast for them to see, there was
a happy face. They didn't like the other one
because it had an angry face, but they never realized this
because it was so fast. So our emotions or our decision
making, it happens so fast that it's outside of our
consciousness, which makes it very difficult then to control
really. But we do need to be able to
control it and we can control it.
And an example here would be, so there's a, there's an analogy
which is quite close to technical debt.
So technical debt, it's you're trading off between a feature
and taking the debt on. So the characteristics of the
te- the feature are, it's gonna be a media.
You're gonna get the feature as soon as it's coded, it's gonna
be concrete. You get the feature or you
don't. So it's gonna be those things
there. Immediate, it's gonna be, uh,
concrete there. Whereas if you look at the
technical debt it's gonna, potentially you're gonna have to
repay some debt in the future. But you don't know when, you
don't know how much, you don't even know what it's gonna be in
there. It's very indefinite.
It's very abstract there. And so you can see that
something that's concrete and immediate, it's gonna appeal to
your emotions. Something that is in the future
and indefinite and in abstract, it's gonna appeal to your logic,
but not your emotions. And so that's principally why
the technical debt does badly. And a useful analogy is to
smoking. So in smoking, most of our
governments, they realized several, many decades ago that
smoking is bad for the populations.
They wanted to stop it. And so one of the things they
did was they decided to put, um, health warnings on the cigarette
packets. And the, um, the tobacco
manufacturers, they fiercely resisted this, and they fought
it through the courts, but eventually they had to put the
warnings on there. But they didn't need to worry
because what happened was, if you look and you read that
warning, it appeals to your logic.
It's talking about a potential future hazard to your health,
which may or may not happen in the future.
And that appeals to your logic, whereas if you looked at the
cigarette packets, you had packets of pictures of sexy
cowboys and things like that. So as a teenager, that was
appealing to your emotions. And so they put these warnings
on that and it had very little effect.
Whereas if you look now at a, in certainly in our country, I'm
not sure about yours, you cannot get packets of cigarette like
that. They're all plain branded, so
the manufacturers cannot affect your emotions.
What they've done in our country is they've put horrific
photographs of damaged body parts, damaged by smoking, which
of course, appeals very strongly to your emotions there.
And so now it's been switched over that the health
authorities, they are controlling their message
through the emotions. And it's not the only reason,
but it's part of the reason that smoking rates have gone down in
many countries there. So that's like a clue of what
you need to be doing with technical debt.
If you've got, or you want to change the level of that
technical debt, you need to make an arguments that are appealing
to people's emotions, not to their logic.
I talk a bit about this in the book where I say there's several
ways you can do it. So one is you can sort of paint
a picture. You can tell a story and you
could put together a scenario of you wanted to do some
development, but you're prevented from doing so by this
technical debt there. So that would be one thing
there. There's other things as well you
can do to affect the way that you make decisions.
So one is something called a Ulysses contract.
And a Ulysses contract, that's where you commit to something
that you cannot later reverse that commitment.
They're used quite a lot in, um, they're like living wills.
So if you come to the end of your life and you are losing
your faculties, you can put in this Ulysses contract so that at
a certain stage you won't be able to reverse those decisions
there, and somebody else will be making those decisions.
So you can put in a Ulysses contract for, say, technical
debt where a project agrees to take on this technical debt, but
if they do that, then they have to apportion part of the
project, uh, funding to addressing that debt at a later
stage. So there's a few things that you
can do like that, yeah. Yeah, so I think it's very, um.
Interesting the way you explain about these kind of a trade-off
analysis, right? Because sometimes we think it's
very kind of like simplistic, uh, mindset, right?
Like we think, oh, we take this debt simply because we want to
go faster. But sometimes it's more beyond
that, right? So you mentioned about emotions,
right? It could be maybe also that day
when you make the decision, you kind of like not in a good mood
and sometimes you take a shortcut simply because, I
dunno, like you just wanna finish things faster.
Could also be the trade-off that we mentioned in the very
beginning, right, with business, right?
Business always wants, you know, things to get delivered faster,
many features. Project deadline that gets set
without, you know, being asked from the engineers, right?
So those kind of things definitely make trade-offs, um,
kind of like more complex than what we think about when we take
technical debt. And another thing you also
mentioned about kind of a few solutions, right, that we can
do. Maybe painting the pictures, uh,
trying to kind of like put the kind of like the story.
If let's say we take too much technical debt, what would
happen? Maybe in the business terms
could be, yeah, we couldn't scale beyond x number of users,
uh, or we couldn't process transactions, uh, you know,
beyond whatever number of RPS and things like that.
And Ulysses contract I think is very interesting.
I'm not sure like how many people have that in their
day-to-day team, uh, kind of like contract, right?
So maybe in your life exp, yeah, in your life experience, have
you seen this Ulysses contract kind of like implemented?
I've advocated for it, but at the organizations where I've
advocated and put it forwards, it hasn't been taken on there.
But yeah, so I think, yeah, I think like most of the time when
I've been advocating, I've probably been a bit too late.
We've already starting to hit crisis point, at which point it
really doesn't matter what you suggest, it is just gonna have
to try and get out the door and worry about everything later,
yeah. So maybe one implementation of
Ulysses contract that I can think some teams actually
implement. So maybe it's, for example,
three sprints of, you know, business technical features
more, and then you spend one sprint just to focus on
technical improvements, technical debt.
That could be a simple contract that any teams, I believe could
actually implement in their day to day.
Yes, that-that, yep. That's a way of looking at a
Ulysses contract. But in some ways you'd need to
be a little, you'd also could be a bit more granular in that if
you get two solutions, right, we can... and I can think here of
when I used to work at HMV. There, we had a particular
problem in that we wanted to display a product in two places
at the same time, which is understandable.
And basically in order to do that, to do it properly was a
big, big rewrite of the database.
And we couldn't possibly do it before Christmas.
And Christmas was really important for HMV, because about
a quarter of all of their sales are just at Christmas time.
So you are never allowed to do anything that messes up
Christmas for HMV. And so what we worked out was
there was a quick and dirty solution, which was we could
create a duplicate of the database so you could put this
product in two places at once. But then you had a lot of
maintenance. So anything that needed to be
updated or you had any feeds in, all had to be duplicated.
The company with the business agreed yes, we'll do the quick
and dirty. We'd get it in before Christmas,
and the first project that we will do after Christmas is we
will undo this and we'll do it properly.
I'd like to say that's what happened, but it isn't.
The first project is when we came to it actually, we realized
that so many code fixes had gone in during the Christmas period
there, that all of a sudden it was much, much more difficult to
actually go back and redo it properly.
So they decided not to do it and they were doing any future
project and they kept on putting it back until eventually I left
the company and it had all got it had all gone on.
Yeah. Right.
Yeah, I was about to say sometimes, yeah, we got this, I
dunno, like promise, right, in the very beginning.
Yeah. We will let you do, you know,
like repay your technical debt and all that, but sometimes, uh,
because of whatever business situations, we may never get it
fully repaid after all, right ? Thanks for sharing that personal
story. So let's move on to the next
layer, which is, uh, you call this systems layer.
So you kind of like associate this with systems thinking a
little bit. So what interesting thing that
you wanna share about systems thinking or maybe systems layer
that could actually affect technical debt?
I would say. So, yeah, I mean, so in terms of
systems, in software development and IT, we're actually starting
from an advantage, because we already have a fairly good
understanding of systems. Because practically every
project that we work on is within a system.
And if we are putting in a big project, say like an ERP system
or something that's really big, almost certainly there'll be one
or several people looking at the problem and looking at it from a
systems point of view. So we've got that understanding
there of systems. So we have an intuitive or it's
more than an intuitive. We have a specific, um, and a
concrete understanding of systems.
But really that's IT systems and that's like physical systems
there. As well as the physical system,
which is important, there's also the social system that we are
working within, and that's where these decisions are made.
And so what you'll see, an important thing about social
systems or living systems, not just social systems, is that
different parts of the subsystem, they can develop
their own sub-goals and they can pursue their own sub goals,
which may be in conflict with the goals of the whole
organization there. And that's something you don't
normally get that in physical systems because the designers
will have... you won't get in a well-designed physical system.
And mostly you won't get it because it's been thought about.
Whereas when we're putting in or we're working in software
development, the testers, they can have their goals.
The software developments, they may have their goals, and of
course, business will have very different goals there.
And they can be clashing there. And so this can lead to
unexpected developments there. And so I mean the, one of the
most famous ones is you've probably heard of Prohibition in
the United States. The banning of alcohol is about,
about a hundred back, it was about a hundred years ago.
Beginning of the 19, of the 20th century, a lot of countries had
a lot of problems around increasing rates of alcohol
consumption. There's various reasons for
that. People were getting a lot more
money. They were working in factories,
they were getting better paid and they were going out and
drinking. That's what happened.
But this was causing lots and lots of social problems.
And so different governments tried to deal with it in
different ways. And the United States, they
decided they were going to use prohibition, which was just ban
all alcohol. But what they hadn't realized
was that, or they hadn't considered the system inside
which the alcohol is sold and consumed there.
And so they banned alcohol. And so all the bars closed down.
All those breweries and distilleries, they all closed
down. Everything.
Everything like that stopped. But you still had people that
they used to be drinking and they want to continue drinking.
And so anybody who's done economics, economics 101 is
gonna tell you if you have a demand for something and you
suddenly cut off the supply, there's gonna be a substitute
supply from somewhere. And this substitute supply came
from criminal organizations. It actually, it came from
criminals who actually then became organized because they
weren't particularly well organized before that.
So then it, this led to lots and lots of problems in Prohibition.
So you had a rising crime, a lot of violence.
You had enormous wealth being generated for these criminals
who then, they would then use it to bribe the police, bribe the
courts, whole series of things like this.
And they built up these very, very strong organizations there.
And so one of the longest lasting effect of Prohibition in
the United States was something that was entirely unexpected or
unanticipated and highly undesirable, which was the
growth of organized crime in the United States.
In our country, we tackled it in a different way and we didn't
get those gangs building up because of that.
Wow. Quite fascinating story, right?
So I think many of us could relate with this kind of like
social system aspect, uh, especially if we work in a
bigger organization, right? So things could be also like
political reasons, hierarchies, right?
Or silo teams that work, you know, quite independently
without talking to each other. It could be so many other things
like incentive system, reward system, punishment system,
right? So those kind of things, maybe
from one small perspective point of view, it seems could solve a
particular problem, right? But unconsciously, you can
create unprecedented effect in some other parts of the system,
right? So I think this is where systems
thinking kind of a mindset is very, very important, especially
for us who are working in software engineering.
So never kind of like focus or invest too much on one
particular solution, but thinking holistically how it
could affect the other parts of the systems as well, I think, is
very important. So yeah, let's move on, yeah, to
the next layer, like economics game theory.
You mentioned about using this model so that we can communicate
with the business much better. Um, maybe some of examples of
economics or system, uh, or game theory layer here that you could
advocate for us to use next time we talk to our stakeholders?
I would say in, right, in terms of that layer there, there's
lots of very fascinating things in there, and it's very tempting
and very easy to become completely embroiled in it and
look at some fascinating aspect of it.
But I'd say perhaps the, two, if there were, if I was to choose
two things, it would be, firstly, you don't need to
communicate this to the bosses, but...
Actually, it wouldn't hurt to communicate it, but it's the
principal-agent problem, which is that it's a problem where
basically the principle so that the boss or whoever has the
money, you are paying or rewarding somebody else to do
something of your bidding there. And that's essentially when
we're working in the company, we are the agents and then the
organization itself is the principal.
And so if you look at the principal-agent problem, what
you will have is that, they may have different goals and they
may have different motivations there.
And so what you should always be aware of and thinking about is
how do their goals line up? You're going to be most
successful if the principal's and the agent's goals line up or
coincide there. If, and you should also
understand that you'll be unsuccessful to the extent that
those goals don't line up there. And it's of course, it's not
just the employees, but you've got an added complication in
most organizations in that they will often, they will bring an
external help to manage large projects.
And what you have to realize is that perhaps the goals of those
outside organizations, the consultancies, may not be very
well aligned with the organization that they're
working for. And certainly, I've been on a
lot of projects - I imagine you have as well - and you will have
seen some of the antics and the activities that some of these
consultants will get up to which, and not necessarily
helpful for the client there. And here I'm thinking of things
like perhaps rivalry between organizations there.
Another one might be that actually one of the things that
perhaps some of the less ethical consultancies might do is as
soon as they go in, is to try and remove all of the capability
of the organization so that the organization is now dependent
upon the consultancy. So the whole series of things
like that. So there's this principal-agent
problem there, which I think people should be aware of there.
That's gonna play out in the technical debt there of that the
company may want lower, low levels of technical debt.
But the principal, if the principal is the, if the project
manager who is going to be perhaps moving to a different
organization at the end of the project and their goal is to get
the project in or peer in, and it doesn't matter to them how
much technical debt there is, you've got that principal-agent
problem there. So I'd say there's the principal
agent problem that is there. And a second thing that I think
that I would also highlight to the business, because they won't
be so well aware of this, is the problem of externalities.
So an externality is, basically it's where one party can impose
a cost or a benefit. But usually it's a cost.
You don't worry about benefits. They can impose a cost on
another party and the other party has no say whatever in
whether they accept that cost or not.
And the most commonly cited example is pollution.
If you drive your car into the city and you are there perhaps
to pick up your wife and she's out shopping, and so you sat on
the side of the road there waiting for it to come out the
shop there, And you had your engine idling.
Well, you are creating all this externalities in terms of the
pollution that you're creating. But you're seen in your car and
you are not suffering that externality.
It's others that are suffering it there.
And that's very much what happens on projects there, where
projects may be making decisions around how much technical debt
they are piling up. But they're not necessarily
gonna be the ones that pay off that debt.
Perhaps it's gonna be the support team.
I mean, this was brought home to me once when I was at, um, or
not once, many, many times, irritatingly many times when I
worked at HMV. We had a project manager there.
And whenever he caused a problem that affected a problem, that
affected another project or support, he would just turn
around and someone said, oh, right, you, what you've done now
has done this. And there he would turn around
and he would say, that's not my problem.
And say, he was just saying effectively, he was saying, I am
putting an externality on you. And the structure of our
organization is, I don't care whether I do that or not.
And the company doesn't care and I'm getting away with it.
And technical debt is very much, very often it's an externality.
'Cause the people that are making the decisions about it
very often are not the ones that are gonna end up paying that
back at the end there. So those are the two things that
I think, that I would take from economics to technical debt.
Principal-agent and externalities.
Wow, I think now that you are kind of like explaining and
giving examples, right? I'm sure some of us kind of like
laughing, you know, hearing those examples because we could
actually realize it's happening to all of us, uh, in our career,
right? So I think definitely this kind
of principal-agent problem, right?
Externalities where someone makes decision without actually
taking the responsibilities later on is kind of like quite
common, especially in software projects where you kind of like
engage consultants, third parties, or project basis rather
than product basis, right? So I think, uh, we can see it in
real life. And let's go to the next one,
which is the wicked problem. You kind of like, um, touch on a
little bit earlier, right? In that wicked problem is
something that maybe you don't even know in the very beginning,
what could the solution look like?
You can always explore, experiment until the solution
actually comes out. This is so much evident in
software, right? Because we'll iterate, right?
So we'll come up with a very simplistic design that could
maybe solve the problem, the features.
Show it to people, people will ask for more, and then we
continue evolve it until it gets to a different kind of solution
altogether. So why do you think wicked
problem is kind like the outer layer, which I assume is gonna
be the most complex thing to consider.
Um, so what kind of insightful thing can you share with us
today about wicked problem? The wicked problem, I would say
that the... There's multiple parts to the
wicked problem as you are saying there.
So one characteristic of a wicked problem is that you don't
know what the problem is until you actually have a solution.
Other things are that there's no right or wrong, there's just
better or worse. But perhaps the really critical
thing for us is that different parties will have different
beliefs about what the problem is and what the better solution
is. So we may think something from a
technical side, uh, there that, well, this is the best solution.
But from a business point of view, they may be thinking
something very different there. And it's how you resolve those
things or attempt to resolve them or accommodate them there.
That's something which I think most of us will find very
difficult, particularly within software development, because we
tend to be very technical people and we tend not to think of
things in terms of political, political sort of landscape and
how we need to accommodate different things there.
And really the people that perhaps are best equipped to
deal and understand those, that political landscape and so on,
they're all sitting in the business there.
It's almost as if we need to get them or someone from that side
to come over and advocate for ourselves there.
Otherwise the risk is that we end up in just like a standoff
there, of the technical people saying this technical debt is
bad and we have too much of it and we need to get rid of it.
And the business saying, yeah, but we, we are getting do- the
market is being dominated by somebody else, we need to get a
bigger market share which means we need more features or we need
more this or more that. And once we've got a bigger
market share, we can sort out all these other things later.
It's almost as if you're having this argument but you are
arguing in two different languages there.
And I think that's tremendously difficult and wasteful thing to
do there. And I think that it's not just
for technical debt. I think there's a whole series
of soft things around here or we have a whole lot of problems in
software development, which are wicked problems there.
And I think we should un, we should perhaps try and
understand better about wicked problems and what we can indeed
do about them there. I think, uh, after you explain
it, I'm sure, um, you know, people can relate, right, what
do you mean by wicked problem in software engineering and why you
put it as the outer layer simply because, yeah, it's very
difficult. Um, sometimes you could wear
multiple hats. Only then you could see where
things are kind of like not aligning, you know, things are
always miscommunicated, misunderstood.
And sometimes, yeah, like you mentioned, people don't have the
same definition of problem, doesn't have the same definition
of solution, right? Or maybe we don't know the
solution in the very beginning, uh, only until, you know, we
reach a certain point. So definitely software
engineering, I find it's a lot of wicked problem kind of a
thing. And we can see from all these
layers that you just explained, right?
Technical aspect is just a small part of the technical debt.
And there are so many other things like organizational
system, the bias thinking, the emotion, social structure and
things like that. How would you advise software
engineering team or organization to start managing their
technical debt better or at least introduce technical debt
in a more healthy manner rather than, you know, like trying to
burn as many technical debt as possible and then try to repay
in the end which will not happen.
And in the end, you know, it cause a lot of problems, or a
lot of damage in some of the stories you share in the book.
So maybe how can organizations start managing their technical
debt better? I'd say almost certainly if you
are managing technical, if you're coming at it from the
viewpoint of we've got this managed, this technical debt,
and we need to manage it or burn it down, you are almost starting
too late, really. It's what you really need to be
doing is avoiding creating the technical debt that you can
avoid creating there. And so I think that really there
is, it's almost like, uh, an analogy would be doing a
development at a sustainable pace there.
You can't do development really, really fast.
And then when you become completely tired and knackered,
now we will slow down to a sustainable rate.
It's a bit too late, you know. So it's getting people into the
mindset that they are making these trade-offs and these
trade-offs will be there for an awful long time.
And getting them to make better trade-offs there.
Now there's always going to be technical debt, and they need
to, you need to get this tech- you need to take on this
technical debt. If you didn't, you couldn't get
the products out there in time, you'd be too late in the
marketplace. But you don't need to take on
all the technical debt that you're taking on.
And some of it would be unnecessary there.
And in particular, I would say it's things such as when you're
trading off technical debt for additional features there, you
really need someone to push back on do we really need these
features? Because it's very easy for the
business and the marketing to say, this is absolutely
essential. This feature, we must have it
in. And no one can prove otherwise.
And so it's just because it's so easy, it's so tempting to do
that again and again and again, until you end up with this very
bloated product with lots and lots of debt in it.
So it's kind of starting at the beginning there.
And one of the things that I talk about in my book, in the
last third of the book there, is really how you can address the
technical debt. But it's addressing the
technical debt from the mindset of avoiding creating it in the
start there rather than getting rid and burning down this
technical debt. In fact, that's a very bad thing
to do in many ways because once you do that, it's almost as if
you've given the business permission to create this
technical debt because you can just pay it down later.
Try and get them into this mindset of avoiding this
technical debt and the costs of this technical debt there and
why and how they can avoid it or minimize the amount of technical
debt there. Yeah.
Wow, I think it's a very good, uh, reminder, right?
So, uh, whenever we talk about managing technical debt,
probably it's a bit too late, right?
So I think avoiding it, prevention is always the best
cure, right, rather than taking pills, medicine or adding
people, buying software, whatever that is, right, to
actually kind of like solve the technical debt problem.
And I think you mentioned a very good keyword that I think we all
need to always reflect, right? Sustainable pace over a long
term, right? So the moment you feel like,
kind of like the velocity slowing down, the team cannot
deliver things as smooth as before, right?
Maybe that's, uh, kind of like also a signal for us to kind of
like think about how we can tackle or manage our technical
debt better and communicate that to the business stakeholders,
right? And in your book, you also
mentioned a few common anti-patterns.
Uh, I think people who would love to kind of like advocate or
champion, you know, managing technical debt better in an
organization you should check out Andrew's book.
So it will be amiss if I didn't, you know, mention or discuss
about AI with technical debt, right?
I think with the advent of AI, we can certainly see there are
many aspects of technical debt that could be, I dunno, kind of
like remodeled, right? So the first aspect is like with
a lot of AI churning out code, I think many people say it could
create a lot of technical debt, right?
Or maybe also, and some other people say, oh, we have a very
high technical debt system. Now with AI, we can use it to
actually solve technical debt. Maybe simply because, I don't
know, at least documenting the feature because the people have
left or things like that. Or creating tests automatically.
So what do you view, uh, about AI?
Um, is it something that is going to increase technical debt
or is it gonna be reducing technical debt?
I think that the most prominent there and that the most salient,
visible part is the big increase which is gonna come from
non-technical people doing vibe code development there and
creating enormous amounts of technical debt there.
So I can see that being a huge, a huge thing there.
But I could also see that it may be that certain types of
technical debt may become less important, particularly if you
are able to create this coding and this solution so much more
easily. It may well be that the debt is
less important perhaps. But I would say certainly it's
the AI and the vibe coding does give you a very strong feeling
that this is something that could create enormous amounts of
technical debt. Particularly since I imagine
vibe coding being written, and if something is useful and it's
taking off, it's probably not gonna get rewritten.
It's let's build on top of that there.
In which case, yeah, you're building on something that's,
uh, probably got a lot of debt or a lot of limitations in it,
yeah. Yeah, so there's this danger,
obviously about vibe coding or you just blindly accept whatever
AI suggests, right? And I think I wanna touch on
also like the emotion aspect, right?
Whenever we use AI, sometimes we feel very confident and it's
very easy, right? We could just churn out code
after code after code, building features on top of features.
Sometimes there's, I think there's a dopamine hit as well
whenever you build such thing rapidly and you could see
results. You almost see the results, you
kind of like polish it more and more.
Uh, but unconsciously you introduce a lot more technical
debt. Do you have like real case
experience or examples where people use AI effectively?
Like cautiously also not to take on more debt, but also kind of
like build a system that can also last like much, much
longer. Well, okay.
So I've spoken to people that have been using AI, and what
I've... Of course, what they're telling
you may not be exactly the same as what they're doing, of
course. But what they were saying was
that they were using it quite a lot for investigation and
research there. And if they wanted to find out
something, they were perhaps posing a question to an AI or an
LLM model there. They were getting the answer
back, they were looking at it, they were then refining their
question, firing it back again, and going through a cycle then
multiple times. But then often at the end they
were manually coding. They were doing it there.
So they were doing it for this research there.
That, at least that's, or that's what they were telling me.
Of course, what people tell you isn't always the same there.
And perhaps there'd be a reluctance to say, yeah, I just
let AI do all of the stuff for me there.
I, guess we will see. The results will come through I
think. And we'll be seeing those surely
I think. Right.
So I'm pretty sure, uh, one day we can, because now everything
is moving, everything is changing.
Uh, and AI technology itself is being kind of like reinvented,
you know. Every week, every month, you
know, maybe there's a new model coming, new techniques coming
out. Agentic thing now is also kind
of like the crazy thing. And I'm sure AI will pop up in
your layers. I would assume it would be in
the trade-off layer and sometimes also in the wicked
problem, right? So I think, uh, definitely it's
gonna be fascinating this, uh, AI trend going in the software
development world. So, Andrew, uh, as we reach the
end of our conversation, thank you so much for outlining about
this technical debt. I find it very fascinating for
us, educating also, uh, for us, like how we can make technical
debt trade-off better. Before I let you go, I have one
last question. Uh, it's like a tradition in my
podcast, which is to ask this question called the three
technical leadership wisdom. So if you can think of it just
like three advice you want to give to the listeners, uh, maybe
if you can share your version today, that would be great.
Yeah, sure. So, well, I would say, um, that
basically read more deeply, okay.
If you think about where you're spending your time, quite a lot
of your time, it might be you are just reading something on,
uh, LinkedIn or perhaps watching something on YouTube.
There are some really wonderful things that have been written
out there, and here I'm thinking of, published books, okay?
Something that is published and something that is a book,
someone has spent a lot of time thinking about that, doing that.
They will have put in hundreds or thousands of hours to produce
that. So there's hundreds or thousands
of hours that have gone into that thought, and you can get
that thought for an investment of just, uh, you know, 10-20
hours or something reading through it there.
That's not always gonna be true though.
If it's, say, a blog article there, it won't have been
through all of that process there.
But if it's been published and it's been published by a
publisher that has got... They need to make money.
And in order to make money, they need to do something that's good
enough quality that people will read it there.
And there's a lot of stuff out there.
And here I was thinking about, um, you know, uh, one of your,
more recent guests was, uh, Karl Wiegers with his Programming
Pearls. And he's like, what a wonderful
book there, you know, with all of these lots of good ideas
there. And, you know, similar, there's
one with, um, uh, it's Lessons Learned in Software Testing.
Lots and lots of really great ideas in there.
I think developers, we probably read perhaps two books a year or
something, two technical books a year, if that.
We could- we really should be reading an awful lot more.
That'd be the first thing, reading in our own subjects.
But then I would also say you should also, you should read
more widely. And here I'm taking a sort of a
hint from something called red teams here, which is where you
should read outside of your experience there.
So, I mean, I do a lot of, a lot of work and a lot of research
into the human factors. But the human factors that had
not, you know, I had nothing at all to do with it until I
started thinking about why have we got all these problems in
software development that have been around for decades that we
haven't. And so I started looking around
for where else have they had similar things.
So, you know, I've looked at that and I've looked at systems,
and systems thinking and dynamics.
So you should broaden out your, um, your ideas and read more
widely. And particularly, you should
also read things that you don't necessarily agree with.
You should expose yourself to different points of view.
Now this means though, then this comes to the third bit here.
You need to set some time, you need to allocate time to do
this. So put a slice into your, uh,
calendar there. Maybe it's gonna be a morning or
something every fortnight, just for reading and nothing else.
Wow. I think, uh, that's a very,
again, very good timely reminder for some of us who, you know.
Because these days, social media or maybe short blog posts are so
much easy to access and sometimes we get notified as
well. I find sometimes like all these
creates fatigue as well, right? So you have so many content
thrown out to you. So we kind of like spend maybe
short time reading it. We forget a lot of things.
I think you, your advice is very good, timely reminder, right?
Read more deeply, read more widely as well, and set the time
to actually do those two things right?
I think for software engineers, yeah, I also think myself don't
read enough, especially those technical books that could be
very thick, uh, and sometimes very dry.
Rather than entertainment. Yes.
Yeah. Yeah.
That is true, yeah. But I think it's very important,
especially if we wanna keep ourself up to date.
Re-skilling, again, like coming back to the first thing that we
discussed, uh, in our conversation.
So, Andrew, if people love this conversation, they wanna reach
out to you, ask you more questions, maybe about technical
debt or social behavioral, uh, aspects, which I know that
you're passionate about as well, is there a place where they can
find you online? There's LinkedIn.
You can probably find me there, Andrew Brown at LinkedIn.
Or I'm on YouTube as well. Or you can fire me an email
which is. We can perhaps put it in your
session there. It's Brown, Brown Sensei is the
email to get me at brownsensei@hotmail.
I took brownsensei when I was working in Japan actually at the
time. Yeah.
And yeah, I'd love to hear from people.
Right. Very nice, uh, nickname,
Brown-sensei. So I think, uh, I'm sure like we
all learn a lot of things from you today from the Sensei, so
thanks for sharing your knowledge and your experience.
So yeah, thank you again Dr. Andrew.
Thank you. Thank you very much, Henry.
Thank you. It's been a pleasure.
Thank you.