Spend some time looking at the system in which he work,
understand how the width is working.
Understand how flow is for your organization and then you can
work to optimize that you can find.
You can look at where those hierarchies are the
informational hierarchies, it might not be on the orchestra,
it might be in the value stream maps and work to change that.
Hey everyone. My name is Henry Surya, we
Robin. And you're listening to the
Tecla Journal, podcast the show, where I'll be bringing you the
greatest technical leaders practitioners and thought
leaders in the industry to discuss about their Journey
ideas and practices that we all can learn and apply to build a
highly performing technical team and to make an impact in your
personal work. So let's dive into our Journal.
Hello again, to all of you my listeners.
Yes, you are listening to The Tackle, a journal podcast, the
podcast where you can learn about technical leadership and
Excellence from my conversations with great thought leaders in
the tech industry. If you haven't, please follow
the show on your podcast app and social media on LinkedIn.
Twitter and Instagram. And to appreciate and support my
work. Subscribe as a patron at
technology. No, dot f / Patron or buy me a
coffee at technology. Not deaf / tip.
My guest for today's episode is James Lewis.
James is a director at thoughtworks and a Of
microservice architecture. In this episode, we went back
Memory Lane to the time when James first coined and
popularized, the micro service architecture.
James described, his definition of a micro service and it's
important characteristics. He also shared the reason
microservice Evolution, including the swing between
micro service and monolith in the second half of our
conversation. James shared, his insights from
complexity science related to different scaling patterns,
particularly he explained how different Archetypes can affect
an organization's growth rate and towards the end James gave
some tips on how organization can detect signs of suboptimal
growth. And what we can do to maintain
organizational agility. This is such an insightful
discussion with James and I find it a really interesting story,
how he first got the idea of the micro service architecture.
I hope you enjoyed listening to this episode and learning a lot
from it. And if you do, please share this
with your colleagues, your friends, and your communities,
and also leave a five-star. And review on a podcast and
Spotify. Let's go to my conversation with
James after hearing a few words from our sponsors.
Are you looking for a new cool swag?
Tackle, it Journal. Now offers you some swags that
you can purchase online. These wax are printed on demand
based on your preference and will be delivered safely to you
all over the world. We're shipping is available.
Check out all the cool tracks available by visiting
technology, you know, dot, f / shop and don't forget to break
yourself. Once you receive any of those
tracks. Hi everyone.
Welcome back to another new episode of the technology on our
podcast today, we have another thought workers in the show.
So his name is James, DeWees a software, architect and
director. Thoughtworks is based in UK.
If some of you don't know about James Lewis, he is actually the
person who coined the term microservices with Martin Fowler
back then in 2014 and I think some of you would have known by
now that since then it has been a craze and hype and so many You
transfer following microservices and today we will touch on a
little bit about the history of microservices and some Evolution
since then and we'll talk about some other things as well.
So James thank you for the opportunity to have you on the
show. Looking forward for this
conversation, hiring your thanks very much for asking me to be on
the show. It's a real privilege and
pleasure So yeah, thank you. So James in the beginning, I
always love to ask my guess to actually share any courage, you
need highlights or turning points that you have in your
career to be share here for the listeners to learn from Yeah,
that's great question. Actually, what wants to create
hearts? Are there's been so many.
I've been, I think genuinely so lucky over the 26, or so years.
I've been doing this or adjacent roles in technology.
I think I use the words luck deliberately, right, because my
career has spanned really work. If you think about the last 26
years of what's been happening in technology, expand the birth
of the internet and the World Wide Web.
I was one of the first generation, the UK to grow up
privileged enough to have access to computer in school.
We do. This BBC micro thing that every
school was sent to computer by the government or by the BBC SE.
So my career really spans from that when I was sort of 89 years
old through Java through the birth of the World Wide Web
through Apache, through sun, Microsystems, all these
different milestones. And I've been very privileged
along the way to work with some bread people, we go way back, I
started off doing physics to be honest, I wasn't diligent enough
to be a good theoretical physicist.
I didn't do enough practice and I'm definitely Not accurate or
precise enough to be experimental physicist.
So I said fell into systems Administration and programming
after that, which I've really loved ever since.
And I start off as a sysadmin that I was a Oracle database
administrator for a while. Well, that pl/sql goodness back
in the day and then I studied with programming on.
Would you believe TCL to control language from things like chill?
Control language for vignette story server, which was a sort
of very early content management system for the web.
Java are going to Java, I guess back into of 2001 ish.
I joined for weeks about 2005 as a Dev to don't know what that
meant at the time and I still do it.
But that was my title death to. And I've been a thought, which
ever since, which is it's meant. I've been incredibly privileged
to spend time with some amazing people, you know, forwards,
London 2005. Until today has been tremendous
amount of talents for which in general forward, slow ball has
this huge amount of talent, but for which London, but I joined
there were these People who went on to really change our industry
day Farley and Jazz humble where they're being they went on to
write continuous delivery. You had in that price and Steve
Freeman who with a crisis of J, more from went on to write
growing object-oriented software.
Guided by tests down North, was there who went on to create
Behavior, driven development at area fifty Daniel tools and
obviously very evil, very influential and wise figuring
our community and, you know, Cindy Mitchell, who is the MD at
the time but was former son? I think provides the swing
libraries for Java. So these amazing insects amazing
people. I should mention Eric doing a
boot as well because he was an old friend before I joined the
likes of reunited with him and thought ways but this is a huge
talent that really helped me to embrace the fun, right?
The fun of working with technology, you know there's the
old adage, if you enjoy, what you do, don't work a day in your
life and I think I've been very privileged and lucky to have
that come true for me and I enjoy, what I do enjoy working
with the teams of people at thoughtworks, I enjoy meeting
new clients and learning new. Domains and still do.
I still find it fascinating? So, yeah, I guess that's a brief
precis of my early career. So you've been at dogs for I
think about 17, 18 years by now. So adapting.
That's pretty long, super long. And I think you naming some of
the people just now. Right.
I think some of these are thought leaders in the industry.
They also change some of the trends in the industry as well.
So I'm sure. Today, we'll be talking a lot
about some of the trends that you also yourself a drive in the
industry. So let's go back to the time
then. C14 right?
You are the first to coin the term microservice with Martin
Fowler. I don't know who came with the
term but I think if we can go back and reflect the story, how
did you actually come up with this idea?
It's a really good question. And actually you need to go
fears before in fact. So one thing about thought works
is there are so many amazing people working that the weights
doing amazing things, right? So I mentioned a couple of them
days Farley mentions Jazz humble.
So continuous delivery eventually.
Daniel to host North his thinking wearing the A really
informs me. I worked on a team, building our
services, our retail bank, with him restful services.
That really influenced my thinking Jim Weber idea,
Robinson, who were Christian practice, along with other
co-authors service that really tied together.
For me also Ian's observation that the be of the web, not
behind the web, this idea that. Why is it in big organizations?
We spent so much time, creating our own infrastructure is our
own protocols. We try and reinvent the wheel
the whole time and actually there's this massively
successful distributed system called the World Wide Web.
Why shouldn't we just Use the Technologies from the world wide
web and your breasts Walker can choose level 3 Richardson
maturity model level 3 architecture.
So I was lucky enough to be in that on the place and the other
thing I was lucky enough to experience a really formative
experience for me as he was in 2007.
I think I went to my first cross disciplinary or cross technology
conference it was called Yahoo in all who's Daniel tears nor
they'd really pressed me to go to this thing.
Blew me away, right blew my mind, people talking about Cheri
different things, different types of integration different
programming languages coming, Of where Java came from virtual
machines in frustrations just this massive, interesting
information so that when it's my home, and I think that started
me on this sort of Journey to try and synthesize a bunch of
the things that I was seeing that were successful and to come
up with some way of doing the all at once.
I guess in the same way that XP extreme programming came about,
right? It's a bunch of practices or
bunch of things that people were doing that, we're really
successful independently. So let's put them all together
to do the ball. That's going well, I think my
brain. Works that I see curtains and
I'm able to phone connections between things.
I think there's maybe a legacy of the physics earlier my life.
And so gradually, these sort of things I was seeing in the
waiting outside of thought, work started to fit into place, then
there was one event. Actually in 2011 was lucky
enough to be invited to something called the software
architecture Workshop, this was in just north of Venice.
Actually, I can't remember the exact name and there were about
30 or so people. There had been invited to all
across Europe and the US Africa. Actually, it's an unconference.
So we were talking over 3, Days form your own agenda and one of
the themes that came out over that three days.
Was, why are things so hard to change?
Why is it that these big products?
There's some product people that we've got this big product.
How do we break it up? How do we change it more
effectively? How do we improve maintenance of
it? Do you build a new one or do we
strangle it all these different questions I can?
Honestly say, I had a sort of never, it'll never happen to me.
Again, you'll happened once at least I literally had like a
ding in my mind. You like, literally, almost Like
a chime, right? I know.
It's like a cliche to say, but genuinely what?
My thought. Maybe, we're solving the wrong
problem. Maybe if we didn't have, the
problem of things that were too big, maybe, if we had lots of
smaller things that might make things easier to deal with.
So I came downstairs, very excited in the morning to Jimmy
Nielsen, who was organizing the event.
The Great's domain-driven design expert in voice person.
I said, Jimmy Jimmy, I've got this idea, his wife and kids
with our he often travels with his wife and children and he was
looking at me quizzically and I said Jimmy I've got this idea.
Like, what happens if we just have everything as an act of the
route and they talk to each other across the network and he
looked at me as like, yeah? Okay.
That's interesting. James, but you're crazy, you
know, he probably remembers it differently and yeah, and then
we sort of went our separate ways and I went back to a client
projects in the UK and we started putting some of this
stuff in the practice. So rather than doing threading
within the monolith or having a structured model, if we decided
to scale via processes. So every time we wanted to have
a new capability or subdomain if you like we've created new
service, And along with that, there came a bunch of other
practices. We have one repository per
service and the reason we have one repository service is
because I'm stupid, right? And generally, that was the
reason. It's like if you put more into
the same repository, then I won't be able to avoid finding
the abstractions and extracting stuff into shared libraries.
And we don't want to do that, right?
We want to have across the wire a pi to 3 Pi communication.
We started putting all these things into practice.
We had something that looked very much like ansible free
ansible in order to manage all this stuff on.
Aw. Yes.
And it seemed to work pretty well.
And I did a talk at the time called Java, the Unix way.
And I first did that I think in Congress that weekend ever walks
in Poland, I also then did it at short notice in Q Khan, I think
it was coupon 2012 in San Francisco, Jose humble.
That asked me to go and do a training course.
I did a training course on micro Services which was a little bit
scary because I had the guy who sat next to Roy Fielding Adobe
on my training course, I was trying to teach him about
resting, like for this is a bit awkward, and then I did this
talk and then it sort of went a bit quiet.
Martin Fowler, you know, Martin took a friend are, he'd been
trying to persuade me to write it up?
Should write it out. We should write to that.
We should write it up and we funded a internal Summits.
Always microservices some food people in from around the world
to talk about our experiences and eventually he decided that
the only way to get me to write it up, was to Fly Me To Boston,
and to go for a sleepover at his house.
So that's what we did. I flew to Boston, I spent a week
living with Martin and his amazing wife, Cindy drinking,
good craft beer. And writing about microservices,
I should also point out that the history here.
Yeah, and this is where the whole thought, waves, diaspora
and the sort of Corrections to the rest of the industry coming.
There were other people at the same time, talking about exactly
the same thing. So whilst I would argue that
Martin and myself, we wrote down what became the description of
the characteristics Fred. George at the time, was also
talking about micro services in a different style, but he was
also talking about these really small things that he was told
about functional microservices. If you like and Adrian cockroft
at Netflix, he was also on a That clicks to what he was
going, fine, grain service-oriented architecture.
But then, you know, Adrienne for a George's.
Explore wigs Adrienne has been a longtime Ally and friend of
thought whips and marries all the people I know.
So it's no surprise really that these ideas all came together,
so you could probably say that the three of us Fred Adrian.
The myself with the people who sort of put it all together in
one place and then I were to that.
So yeah, that's, I guess, a very long or very short version of
the story depending on cover. All right, I think it sounds
really interesting. II didn't know some of these
historical moments, right? Like when you shared the
lightbulb moment that you got and then you shared this idea
and then it became like a thing that we found out from Martins
block, right? So, I think it's still there.
Beyond 2014, both of you publish this on his blog.
And I think if we have to go back to the definition, right?
If you would Define microservices.
Now, I know that people take microservices into different
types of understanding different types of characteristics, but
maybe from you, originally, how would you define microservices?
This now I probably and we've talked about going back and
writing division to a writing updated as addition if you.
Like think it holds up pretty well.
I mean, there's always going to be I think Martin calls its
semantic diffusion right where the original meaning of a term
is lost over time. As many people start to add
their own meanings to it but I think it actually if you look at
those characteristics they stand up pretty well and might make
more of a focus. She on the business side of
things, rather than the technical side of things, I
think the technical side of things is accelerated off in all
sorts of directions. Remembered all came after this,
right? And kubernetes, these are
technologies that have been invented or popularized because
we doctors being the other line Tech has been around for ages to
help obviously a lot with building distributed systems.
It makes it a lot easier. I think we're, I'm Consulting,
the things I see, least the pride are the things around
products versus projects and the organization around business
capabilities and really embedding domain-driven design
Into the Heart of what you do and I think that's because
that's actually harder in some ways to solve than the
technology problems of breaking things up and promote
communication. And why do I say that will?
Because it involves humans. Right.
So, whenever we're in a position where we can pull humans, and
teams and people and structures, and this may be related to some
of the stuff I'll talk about later, but the time scales to
change and for experimentation, on much longer, it's much harder
to change your developer ization of your organizational structure
frequently. Like we can do when we refactor
code bases or where we actually redesign.
A Nori architecture, because it takes a long time to see if
something's working. I think the other thing is with
the characteristics, I'm pretty early with emphasize Conway's
little more in a sense because I think that again, is something
that's a real driver for a long. The microservices, I'll all the
problems I see with implementations, we often get
asked a question. Like, you know, we've got these
two thousand Services, I how am I supposed to understand all of
those to which my answer is? We you know, right?
Yes, that's the whole point of breaking things up into smaller
units. Old Rising around the
capabilities, rather domains and then split the team's up like
that. Conway's Law happened in there.
So that all you need to know is about the things that you know
about and the things may be adjacent to.
You don't need to know about all 2000 or 4000 of these things
that I think that's definitely something that I would spend it.
Go back and focus a little bit on some Newman is going to
lovely phrase. He talks about, you know, when
you've got a big problem, how do you solve a really big complex
problem? Break it up into a lot of
smaller, but still complex problems and solve the small
problem. That's really what We were
trying that's one of the hearts of microservices.
The other thing I probably would have emphasized the bit more and
this is may be due to my journey as a software professional since
is the concept of flow. So I probably would have put
more effort into describing. If are these optimize your
ability to get things done and to scale teams around these
things because that fundamentally is such a huge
issue in our industry. Now how do we go faster?
How do we get more stuff done here?
Pretty developer experience. How do we get value out the door
improve cost of delay, all these things and I think microservices
at the heart or a way of doing that, but only if they're done
properly, are quoting properly by paying attention to how you
split up your teams and that you don't get this, massive
distributed model, if that everyone has to understand
everything and everything has to be deployed along once, but that
just might be more able obsession with flow and flow
efficiency that I've developed over the last few years.
I'm not sure. Yeah.
So I think what you said is really insightful, because many
people actually took to the extreme on the technology side,
right? So things like for example,
lines of code like how big is a micro service right?
Or what technologies that we should use maybe like kubernetes
contain, and things like that? But I think you remind us the
important things which is already defined by both of you
in the article itself 9 characteristics, right?
And things that stand true to the test of time.
I believe things like for example, componentization fire
Services, organizing around business.
These products are projects and things like that.
So I think that's a really insightful.
And I like the way you also mention about Conway's law, I
think 11 also mention that it's like a law of gravity.
You can't escape from that and you just need to adapt to
Conway's law, right? The other thing about
microservices lately is that people have been swinging,
right? So some started with like
service-oriented architecture, then they went into
microservices, monolith became quite a trend as well because
people implemented microservices in the wrong way.
And our people started to think about right size service
microservice. So, what do you think about all
these different swings right now?
What could go wrong here or what is the underlying Trend behind
all these? That's a good question.
I mean, going back and I'm fortunate enough to have been
involved with industry from family or time.
You mentioned service-oriented architecture.
But what was the driver for SOA to start with, right?
So this is pre ATI economy. This is in many ways.
This is pre Thomas. Write the ideas behind
service-oriented architectures. It was about how to maximize the
investment in our To like to use date, right?
If you think about how organizations grew in the 80s
and 90s, what you had internally, you'd end up with
lots of these different platforms that were sometimes
doing the same thing. Sometimes naughty is our big Erp
systems that were like sucks. All your data is really hard to
get stuff out of, maybe one department.
But one thing, another department.
But another thing that they both did the same thing to end up
with duplication of Investments and things and service oriented
architecture. Really, was this idea of how can
we create a set of common building blocks that we can?
Using our organization, right? So this is going back to
capabilities as well as business capabilities.
Were first talked about in the late 50s.
So, this is nothing new, right? But when we talked about
capabilities, they were talked about, not in the sense of
Purity, is in the software. I adjust the services.
They were talked about in the sense of people processes and
tools that make up an important part of what your business does
and serious oriented architecture was about then
almost like, creating these units that will allow these
things to talk to one another without a duplication.
So on so reducing cost by doing X, Y Zed, and I think as things
change those, then you move into the internet age, into the world
of of where everything is digital.
Doesn't matter, if your bricks and mortar retail, you're also a
digital retailer. In a sense, at least you have to
offer your catalogs digitally Etc.
Then the meaning of a service-oriented architecture
started to change and we started thinking less about internal
integration between systems and how to make that more stable,
which is really where my head was back in the day into this
idea of K. How do we offer apis?
Externally, how do we build software?
That's providing these apis that either our channels can use or
that we can offer as a service, Google Maps as an example.
So maybe there's been this long term Trend towards more
service-oriented architecture everywhere.
If you like backing 2014 15, in my training courses, I'd be
asked how big should one be, you know, do you think we'll end up
with thousands of these small things that are 50 lines of
code, each which some proponents of them at the time were saying,
you know, there was one classically This is my chance to
chat. It comes with a minute, but his
rule was you can write your service in any language.
You like, as long as you can rewrite it in half a day in
another language, which means things are really small, you
know. So there was that movement.
Then there's also the idea of modular monoliths do we have to
start with microservices? This seems like a really complex
way to build simple systems and it is really complex ways to
build simple systems. I don't think anyone's ever said
that you shouldn't build structured.
Bolus, I think the problem came that over time.
I'm we saw this really hard once you go past a certain scale to
keep the model with modular. If you've got something that's
relatively simple and small, then that's fine.
But at a certain point with a certain amount of tune on the
team with a certain number of people working on the platform
excetera, you tend to reach a point where things start to
become Tangled. Yeah, you end up having to put a
lot of energy into managing Tech debt or a lot of energy into
avoiding entropy if you like in the morning.
So I think naturally So the strong one way, I think.
Then they're actually we're swinging somewhat in the other
direction but I also think there's something in the middle,
right? I suspect we still haven't
settled on that yet. Which probably looks a bit more
like fine grains, service-oriented architectures,
which you guys agency because at certain points are certain
scales, you can't solve the problems without using this sort
of distributed system. There are certain things you
have to use these sort of patterns for.
And obviously, as an industry we are amazing.
Well bit like, uber Asia know this.
Snake that eats its own tail. We keep on going back, round and
round and round and do the same things.
I had that there's a funny conversation with someone
recently about what should we use, Simple stack to build like
an internal simple internal ever.
That's never going to be exposed to the outside world is going to
be a really simple line of business, kind of thing, not
many users, and of course, the immediate reaction from many
people were as well. You first of all, you start with
react. So then you need maybe graph to
hourly this massive, massive stack of stuff, right?
Just to build something that you could do really simply via so.
Side rendering and I think that's the safely microservices
will hit everything with the microservices tournament.
The getting that you don't always have to solve every
problem that way, I still think they're very useful tools
though. Although I do often when I
introduce myself as the sort of grumpy old full of
microservices, I do say and I'm sorry about that, you know?
Right. So I think the one of the
classic things also for people who are building internal tools,
like what you said just now, right?
So there are only a few developers but they create more
than the number of basis that they could handle as a 1%.
So for example, one person could handle three to five
microservices on his own. I think that's also probably not
the right thing to do. So I think, let's see how it
goes in the industry. These days I want to continue
our conversation to the other thing that you mentioned, just
now conversation, right? We are always trying to optimize
ourselves, how can we deliver things faster and Lady?
You'll have also been giving talks in various conferences
about this topic which I had the opportunity to attend in y'all
Sydney back then in December, which I found really insightful.
I want to go and maybe cover some of the things that you
mentioned as well. So one of the most important
thing that you mentioned in the talk is about most companies
right as they get bigger, actually, it becomes very
difficult for them to scale even bigger while in the top.
You also mentioned that AWS, interestingly didn't have that
characteristics as they get bigger actually, they become
much, much bigger looking back from my experience as well.
Right. I can see many companies as they
tend to become bigger as more number of people join the
company's things start to get slow.
Our. So maybe in the first place.
Let's go there. Then try to analyze why things
become slow as most organizations Ruby?
Good. Sure you're absolutely.
Let's do that. My observation of Max is and I
should point out that again, this is matter research, this is
me summarizing. The whole water of work.
Really insightful stuff that's been going on in the world of
biology and economics in the world, things like City
Planning, or all these different areas that generally come under
the term complexity science. So it's been a huge amount of
work going on. Is Singapore's a big sound
reflects the sciences that is an Institute in Singapore.
For be the home of complexity science has been the Santa Fe
Institute in New Mexico and there's just been some super
interesting things coming out of there.
Well, what I find fascinating is they put together a
cross-disciplinary group of scientists of academics, right?
People who wouldn't normally talk to one another talk to each
other, and they get them together and they bought a
conversation if you like. So you end up with a columnist
and biologists and physicists and chemists and geographers,
and all these dead. Different groups of people
coming together to talk about problems in a holistic way or
think interesting things that they're seeing in a holistic
way. One example would be economics.
So Santa Fe Institute is a driver of this idea of
complexity economics or non equilibrium economics where I
want to murder this definition but the idea is that traditional
economics is based on four Librium States frankly because
the math is easy. The math easy.
If you're talking about systems at equilibrium, the observation
from a bunch of people coming together in Santa Fe was hang
on. It most things answer, accurate
rewrite, most things are not every dream states like the
world is in a non equilibrium state.
So why is it we're trying to approximate what's really going
on and economics with these massive oversimplification.
And the answer is because the economists are economists, then
our mathematical physicist, right?
So, the mass, that is easier for the mathematical physicist, is
not so easy if you don't have that.
So, that's one example of the sort of cross-disciplinary stuff
has been happening and they've been doing some amazing things.
Usually, agent-based models they can now do things like say,
Simulated economy is not growing economies in the large and they
can produce by putting in the rules that the difference
countries in the world, apply to the capitalist systems of their
end. So the different Market controls
just by using these simple agent-based models exchanging
value. And these Market controls, they
can produce the gini coefficients for Nations around
the world so you can say left before in the Netherlands we've
got this type of Market control. All right, let's run the model
with these Market controls own turns out.
We can get the gini coefficient so that's the ratio between the
richest in society in the poorest in society where the
wealth is it pops out for the Netherlands?
If you put them off in controls, the UK has its unique
coefficient pops. Acts, they do some fascinating
interesting stuff across the board.
I think what really interested me was how it then related to
have some of that research into scaling laws into how things get
bigger relates to what we do as technologists, or what we do in
the area, or the organizations, in which we find ourselves.
And there's Seems to be, I think, in, at all, that talk,
about Geoffrey West hooks to paraphrase.
Every time you see straight lines on a graph, in lots of
different places, you all these straight lines, basically,
replicating across different domains different parts of the
world, different types of complex system.
Then you should find something interesting in it.
I you should be looking like maybe there's something
underlying maybe there's something that's like a truth.
We can go after and that we can use scientific methods to try
and determine and that's what they've kind of been doing in.
Santa Fe around scaling Jeffrey Webb.
The professor West, I should say, I don't know them, they're
the gentleman. He's been instrumental as well
as some others. And what they've sort of found
essentially is with different types of network.
And I'm using the words, the network in the kind of
mathematical sense, if you like so, whether you've got and like,
hierarchical like graph, like directed graph, like networks
versus things like social networks, different types of
networks, coming explain the different straight lines on
graphs, that they've been finding and all sorts of weird.
Weird and wonderful ways with wonderful places.
So, in example, the straight line on the graph that is the
scaling law for metabolic rate in mammals, you can draw a
straight line. There's now is getting bigger.
There is the straight line that can be drawn about how many
calories, essentially, they need to intake to take in to survive
and there's another straight line for how the infrastructure
in the city's expands as a city gets bigger, can you water pipes
and much more water pipes? Do you need?
And it's a very similar straight line with a very, very similar
experience and it's the same A companies as well.
And Company, growth revenue is the same and all of these
different things appear to be related to a particular type of
network, a hierarchical network. So water pipes and cities are
hierarchies in. Mammals it our circulatory
system is a hierarchy essentially for heart, down to
the quarries, in organizations, we have hierarchies of
information flow, right? So, where do ideas come from?
Well, is direction from from whether you orders in some
senses come from, and they tend to be hierarchies as well,
because as companies grow It tends to become easier to create
a hierarchy which information trickles down through in the
same way that water flows through pipes in the city than
it is to do it any other way. But the interesting thing about
these hierarchical networks is that they all scale using what's
called a sub linear scaling. Also this scale with its UB
exponentially. What that means is as you double
the size of a city, you don't have to double the number of
water pipes. Instead, you get less, you have
to do less than that, and the same as you double.
Size of a metal. You don't have to double the
amount of calories intake in its scales, less than that.
And it's the same for all those ations for physical
infrastructure. As you double the size of an
office, the network cables are double, right?
I mean, it's the infrastructure in the office, doesn't double.
This is what we call or columnist, call economies of
scale. So, as you get bigger, you pay
less for things essentially, and all of these complex systems
exhibit these behaviors. But it does also implies the
other things employers. The fact that things slow down
right things, move more slowly when you have deeper
hierarchies. So humans experience, life more
slowly than a marriage does. And it's down to a mathematical
relationship related to the death of our civil HP system
compared to the depth of the circulatory system of most all
seems to be explainable that way.
Anyway, I should say science, we might find something else out.
And similarly, with cities have cities, get bigger.
The higher Arc is give us anything.
And then the issue thing is with companies, I what happens with
clump, I think we experienced this in our everyday lives here.
If you work for a big copy as you get bigger, things do get
slower, we do experienced this personally day today, you do
individually day today but also you can see it in the things
like the results that public companies Post in terms of
Revenue and so on. So as a company doubles in Size,
Doesn't double the amount of Revenue.
Interestingly, the other thing is like, mammals companies die
all the time and this is another fascinating similarity between
the two which It is also expandable via these lobbyists,
your own complexity science, complex adaptive systems, so has
mammals get older as we get older we're taking the same
number of calories but we're having to direct more and more
of those categories to self-repair to repairing
ourselves rather than you know vigorously youth without having
to do that. And so therefore we slow down
the age, things like brain down and eventually we passed away.
Seems like companies do the same thing with something interesting
facts in the book scale. Think it's the half-life of a
company on the stage. Is 10 years, so every 10 years,
50 percent of the companies and the exchanges will change.
So as I say, there's all these interesting things that seem to
be related to hierarchy things. Slowing down, things taking
longer as things get older. And so on, which I thought was
really interesting. Right, because in my experience,
when you plot out, things like development, life cycle is like,
you look at the big old organization.
You look at the software development lifecycle you drop
that as a value stream map, you'll end up with these huge
long processes, right? Massive, where you've got like
it takes a year from an idea to get even to the project
management function to be able to see scheduled to have some
work done, right? Because bouncing around between
different review, board bouncing between different architecture
groups and the technical business analysis.
This is the business analysis at La one.
Never know what? Any of these things me then you
hit development and then maybe some development gets double.
It takes however long, it takes and then it pops out into some
really long involved process of testing, you know, Testing and
user acceptance. Testing regression testing
performance testing there until you eventually production and
you see this in every old, big traditional company that I visit
conversely, if you look at younger companies, often they
don't usually they have much shorter time to market, right?
They're much more able to be much more Innovative because the
time it takes for idea to go from in someone's head that
information passing through a series of process steps to make
it may. Tends to be much shorter and
slowly over time. There's this thing that
companies are add all these processes, these constraints,
they add hierarchy to manage all this constraints and all these
processes and things start slowing down.
There's also another interesting idea that larger organizations
do the same thing. Bigger older organizations
rather do the same thing that happens with mammals where they
have to spend more of their revenue on keeping the company
running. In the same way that, as we get
older, we spend more of the color if it intake on Ariel
cells companies, they spend more money on just being the company.
Whereas at the start, they were spending money on Innovation and
products, and people and all this cool stuff and that's why
they were successful. Eventually, they end up in this
position where they're spending. So much just keeping themselves
going. They stop thinking about
Innovation or innovation becomes hard R and D becomes hard.
I've got this great idea but I don't know where to put it
because there's no one interested in me.
Because planning, everything you've got on, will be rude to
say marketing, I don't know. But yeah, that's the own
observations but the The issue thing is you coming back to what
I was saying earlier, the two types of network it appears,
there's another type of network which in the natural world is
associated with a different type of scaling or something called
super linear scaling. So, super linear scaling is when
you're in advance of exponential growth.
So as you double the amount of one thing, as you double the
number of people, you get more than double the revenue.
For example, and in the cities, cities are another example of
social networks, massive, social networks exist, should people
communicating as you double the size of a city, you would get
more than double The amount of socio-economic things out of it.
So you get more than double The Innovation.
You get more than double wage growth, other effects negative
effects to get more than double Prime pollution disease, these
things, right? And all these super linear
scaling super linear factors are related to these social
networks. So people talking with people, I
like to sort of explain it in the sense that if you live in a
village of 150 people, right? And I'm a software developer.
I live in a village of 150. People, how many other software
developers live in that Village for me to commit to, on a
professional level? Maybe one, who knows maybe more
than one these days, but probably not that many.
Whereas, if you go to, Yo, Sydney, there's a thousand
people just at the, I've Sydney talking about Innovation,
talking about bootstrapping each other's ideas.
Like the original conference. I went to, and we talked about
right at the start of this, that boot strapped on my ride is so
you get a bigger city. As you get bigger places where
more people live people tend to Together we'd like addressed.
All right, so you get this bootstrapping effect where
people with like interests get together.
That's true for not just software and cool stuff with
Innovation software. You get more than doubled, the
number of lawyers in bigger cities, right?
You get more than double the number of accountants, etc, etc.
And it related to these social violence, these social network
graphs for me, that's a really interesting observation that we
know mathematically there's this relationship.
If you structure yourself, if information is allowed to flow
in a particular way, then you get Get like the whole is
greater than the sum of the parts.
If you like, you get more out of than you think you should.
And if you structure yourself in a different way, conversely, you
get less act than you think you should information.
You know, if the information flows hierarchical E versus via
social network mathematically, there's a choice between the
two. So the question is, how to take
advantage of this, how to use this knowledge, to improve our
ability to get stuff done, because that's really what we
want to do. Not necessarily make more money,
maybe it's save lives, maybe it's work.
Hospitals and organizing teams such that they're communicating
super effectively which they do incidentally.
Soon in the UK are constantly for the rest of the world.
Paul, maybe it's schools. How do we organize our schools
such that? We've got the right information
at the right time and stuff doesn't have to trickle down
slowly from above or if it's a big organization, how do we
organize ourselves? Such that people have got
everything they need within a short number of hops from them.
And this is why, you know, we took and go back to
microservices microservices at all.
About product teams, which have everything there.
You've got everything you need on your team to solve the
problem, right? So, we've all of your products
towards have been about a price, which is it's the same idea and
it's related to Commerce law as well.
How do we avoid having to go outside our team?
Ask other people to do stuff for us because that's from the
hierarchical way of doing things and the observation are making
the talk. Of course, which was related to
me originally by someone at AWS wasn't that scary WS structures
themselves. These Any small independent
teams relying on Self Service infrastructure.
That's another key thing that I should talk about in terms of
microservices more there. So decoupled from one another
that they're able to add new things on the side super fast.
So their ability to spin up your product teams is super quick.
Interestingly, we also sort of point out, don't know if you've
ever done much work with AWS but, you know, their apis you
could be using one set of apis. You look at different set of
products and they bear. No relation to one another
right, like the user experience that developer experience it
Across the different products is very different, but that's
deliberate, it's not deliberate per se, as if we want things to
be different, but it's deliberate in the sense is we're
not going to pay the coordination cost to make our
apis consistent because the coordination costs is going to
slow us down. So with allow the teams to be
independent and that means we have to put up with the fact
that the developer experience isn't going to be potentially
the best, which of course has interesting implications for
things like micro front ends or how do you build Those apps that
are able to use multiple teams to deliver parts of an aircraft
parts of a website parts of the page or whatever it is retain
that experience because most organizations that they want
their brand to be relatable across the difference.
You don't want one the recommendations bit to look,
very different from the main product bit, but it means you
have to pay a coordination cost to do that.
I think that's the direct result of some of this complexity song.
Wow, so many interesting stuffs that you brought up here, right?
So things from the type of networks that you have in the
organization, whether it's hierarchical social, Social so
amount of flow how much information trickle down easily.
So one thing that I want to bring up first before we go into
recommendations for all organizations or software teams
here many times actually, I also read that software teams is also
the analogies like it's also a complex adaptive system, right?
So for many people who may not be familiar with this term, they
always think like okay it's just another department within an
organization. You can just produce whatever
code that you can do and then just deployed but I think we all
know as a software Engineers that Not that easy, right?
So there are a lot of things that actually going on and some
people actually call it complex adaptive system, maybe if you
can touch on a little bit and explain why we should also think
that software engineering team or product engineering team is
also a complex adaptive system. Here are good question, what is
the definition of complex adaptive system, right?
I mean again from the center face-to-face, Ethel dollar
definition which is a complex adaptive systems, display four
characteristics. The first one is that inherently
complex so that the whole is greater than In the sum of the
parts, that's the first thing. So for example, I normally
explain that by saying I am a complex adaptive system.
James, if you put James into a blender and came up with like,
James soup, that's not very nice idea.
And then you pulled James soup into a James shaped jar, would
it be James? And the answer is no right now.
This is the interesting thing, this is why the history is
fascinating. It comes back to an old
physicist called Murray gell-mann.
He was fascinated by the idea of pairing.
You get such From such Simplicity, he wrote a book
called The quark and the Jaguar, right?
How do you get something as complex as a Jackie or from
something as simple as a subatomic particle pulled
acquire right? What makes a j hero and what
makes a human? What makes a team or
organization or city? So this is idea of complexity,
right? The whole is greater than the
sum of the parts. There's also this idea of
emergent Behavior, so when we say measuring Behavior, would
mean that you can't reliably predict the outputs of a complex
adaptive system from its inputs. Every time it's not a functional
problem where you can Say, given this stimuli this thing will
always happen. Everything coming back to the
thing about teams which teams exhibit that behavior all the
time. All right?
You could never run the same situation, the same maybe even
use a story or Sprint twice, you couldn't do it, you'd get
different outcomes if you buy it a second time, the third
characteristic, is this idea that they're made up of self
similar parts. So in humans, the self
similarity is the cells. We are self-similar our cells
have self similar to one. Another actually in mammals, and
this is where the scaling laws were.
Come from all cells are self-similar to you and Marcy
sell or to an elephant cell, will to a blue whale cell and on
teams to self similarity, is the people on the teams are
self-similar, we were not within limits and then this idea of
self-organization. So there's no Central thing
telling a complex adaptive system, how to organize itself.
Mammals, in our cases, DNA, right?
There's nothing, when we're in the room that says, okay.
Now, build a heart. It just evolved over time, and
that happens According to some weird magical, biological
process. Yes, and there's obviously
similar to the teams are as well.
And so that's why I talk about teams being complex, adaptive
systems example, I used again, the Sprint thing.
If you just change one member of a team, you've got a different
outcome. If one person is sick at a
different time, you get a different outcome.
If people pay or don't pay you get a different outcome, it's
almost like every single line of codes could for, your could not
be written depending on so many different factors.
And that's why we talk about teams has been complex, adaptive
systems, and of course, organizations are made up of
these components. Made up of these things.
So that organization is also a complex adaptive system.
Hope that someone else's look wet definitely and it's not just
within internal, the team's themselves.
I think the external factors are so, they are so many
complexities. What people call the vuca world,
right? The disruptions the new
technologies that come as well. So I think all these play A Part
as well, to make it much more complex.
So, coming back to the topic of how can organization avoid this
sub linear optimization growth. So, would you be able to suggest
some things like How can we actually first identify?
If we are sharing a sign of Aging.
So things get slow down. Things are not flowing much much
faster as when we are small and how she'll organization start to
think about the more optimal way to actually have this super
linear growth that you mentioned for really great question.
And what we normally do is I told you that by saying hey I'm
just here to provide the information you can do with it
what you want. But actually that's not very
helpful at all, is it? I mean I do have in that talk
some concrete advice, right? Which is for example, you can
use Use the metrics from the door reports of the accelerate,
Burke the full key metrics. You should become very popular
rightly. So as indicators leading
indicators, actually of organizational health.
So you have the four key metrics need time to Value.
They put a weird definition of lead time is from commits
through to production. Whereas real lead time is
actually from idea through Soup To Nuts, while don't be used to
lead, time to find mean, time to recovery change the failure
rates and number of deploys per unit.
Time those things I think are really interesting to track now
Now, a lot of people then fall into the Trap of saying great,
you've got tools that will tell us this and then they spent
eight months trying to automate measurable, this stuff.
There's no need, right? We don't need Precision, we
don't need the Precision you get from knowing on average we have
exactly 30 2.5 hours between each commit.
That's not what we're after here.
What we're after is saying, yeah, look, accuracy finger in
the air, your tea. Hello, why average does it take
to get stuff into production? That's good enough.
And as long as you track that somewhere and as long as,
Reflect on it. Maybe at the end of iteration or
whatever it is. But you doing quarterly training
or when you're okay, ours are being refused.
And you look at our we better or worse.
That's what we're really looking at.
You know, are we trending in the right direction?
Save me, time to recovery same for the others.
We can use those Trends to work out which direction we're
heading. How are you?
Use the analogy from biology or Medicine of going to doctors?
Start general practitioners in the UK, they'll take your blood
pressure, your temperature, your heart rate, that kind of stuff.
They'll know if There's something a bit dodgy going on.
Right? You got super high blood
pressure maybe there's an intervention that needs to be
Sage doesn't tell me exactly what the problem is but it might
tell you there is an issue. Now there are a couple of
caveats to using the full key matter.
It dry before, key metrics are only going to be improvable to
the limit of the system, you find yourself it.
So, if you're in a deeply hierarchical organization with
these massive, long value streams to get stuff done.
You can optimize for key metrics, but you're only going
to be able to optimize to the limits imposed by System that's
long. Keeping I think that's kind of
interesting is that would be one thing.
I think the other thing is this idea of signals when we have
some of this information, by how we structure our self
fundamentally impacts performers to such a degree, which we kind
of no, no based on a bunch of data that's out there in the
public to me. How do you take advantage and
how do we listen to signals from our own organization.
So that's one set of signals. I've just given you which is a
full key metrics but the other thing is taking time setting,
The organization up so that people in the organization can
take the time to listen for the signals.
Now I often these days, use the phrase that people are so busy.
Doing the work. They don't have time to think
about how the work Works. They don't have time to think
about the system and optimize the system.
These people are just head down, just doing stuff, doing stuff,
doing stuff all the time. And often that's actually why
they hire people. I thought Works to come in right
because we come up with tool box with a bunch of techniques.
We can say hey we'll build you this value stream map.
Now we can optimize it, I probably shouldn't say this.
Secret, there's no reason that you for example Henry films do
the same thing with the same tools, it's just that often we
don't have time to do that ourselves, you know, because
we're so busy doing other things.
So that will be the other piece of advice I would give is spend
some time looking at the system, in which you were understand how
the work is working. Understand how flow is for your
organization and then you can work to optimize that you can
find. You can look at where those
hierarchies are the informational hierarchies.
It might not be on the orchestra, it might be in the
valley. Streamers and work to change
that. And one thing I think I
mentioned it briefly, I think one thing that is often
overlooked is self-service and why self-service is so powerful
because as I mentioned the problem with hierarchies is you
tricking information one way and often by have to ask someone to
get something done and waiting on someone else doing some work,
right? That's kind of like blocking my
flow broking my circulatory system in this Century.
So how do we make those systems such that their cell service?
So I don't ask someone to do Something for me, instead, they
provide that in a self-service way and that's what's so
powerful that AWS is that they recognize that by providing the
undifferentiated, heavy lifting was the original term.
They use all the stuff that the original AWS platform did when
they provide that self-service, they solve a couple of problems.
The first is you have to wait on anyone to get stuff done.
All right. So that's a flow Improvement
problem and the other thing is you solve the problem of scarce
resource because it's queuing Theory, right?
If you've got a request coming in, To a service into a queue
rather and you've got one thing able to take that request off
that you and processes. The only way it by adding
another message in at the same rate, if you've got multiple
producers feeding messages into a queue, then you have to have
multiple consumers to process, the messages at the same range.
Keep the cue from backing up. And it's the same with what we
do is exactly the same as what we do when we are asking people
to spin up the ends for us when we are asking another team to
make a change for us. The only way for them to scale
is for that. Team were asking to add more
people, as they get more requests, they have to add more
people, it's unsustainable. As we asked, cops to do more for
us, they have to add more people and that's unsustainable.
So, you invert the relationship and say, make these things
self-service make providing infrastructure.
Self-service make, providing changes self-service, things
like develop a platform engineering teams, providing
self-service access to tooling to release pipelines to
databases on, demand, all that stuff, that's all about
improving flow, but you don't get to see that.
Really unless you are able to visualize your system, unless
you're able to step back, take the time and understand where
the blockage is, are where the cholesterol is building up in
your arteries if you like. So, I think that's a very good
reminder yet. Like you mentioned, right, when
we are working in the organization in the team, we
tend to just get things done, right?
Okay, we have new features, we have new ideas, we have new,
roadmaps, whatever that is, and we just get things done, and we
just live with the same habit. Maybe let's put it this way.
The same habit that we used to do.
Right. Maybe it proves successful in
the beginning and we just continuously do that without
actually monitoring and review the system, right?
How the system looks like after some time that we go through it.
So I think that's a good reminder to always look at the
systems. Think about how we do the work,
right? And I think the concept of
platform teams, the self-service thing, it goes back also to the
team topologies, right? I think this work by Emanuel
Pious and metal skeleton is really called it in everywhere.
I think I would say, you know, the concept of for the
Streamline team and the platform team enabling team and things
like that. So I think that's really good
concept as well for people to look at how you can actually
optimize your organization and you kept mentioning about flow,
right? Optimizing flow.
I think this is also, another thing that is really critical
for organization to figure out how the flow in your team
actually moves. Is it fast?
Or is it slow? And we can use the key metrics
from Dora as one indicator. It's not the true answer of
everything and we don't have to be accurate so thanks for
sharing that. So James unfortunately, due to
time we have to wrap up pretty soon.
I have one last question though which I would like to ask you
which I call this a question 3. I think the leadership is Adam
so you can think of it like an advice that you want to import
for us here to learn from your experience and your expertise.
So can you share maybe two or three technical leadership
wisdom? Yeah, this is a big word, right?
And you may expect me to come up with some deeply technical from
a set of answers here. I'm going to go completely the
other way, okay? These are three things that I've
learned from people who work within four weeks.
And in particular, I want to call out and with what Daniel to
his North year because he's one of the people that I've really
learned a lot for is a very, very wise person and The first
thing is that empathy is a superpower, especially as you
get more. Experienced being empathetic is
absolutely key to getting things done, I think from a personal
perspective so much so that you know, I'll go back to 2008 2009
with another team with them. It was when he wrote the article
that became the bdd article. It so just invented whatever it
was he originated bdd Behavior driven development and in that
it's all about being empathetic is all about understanding.
Not just what your team needs, what you need, it's about.
Understanding what your other stakeholders need that aren't
just the business stakeholders that are, your stakeholders, in,
security, and operations, you've Downstream or Upstream, folks.
Within the large Apollo, the suffered, a long process, but
empathy of this is all your empathy with the people you're
working with. You don't know what's going on
in people's lives often, and be their pathetic as a long way to
creating a really fun environment.
And that comes to my second piece of advice or maybe rather
it's a statement and I would say that seriousness does not equal
professionalism the best projects I've ever.
I worked on the most successful products have ever built the
best teams I've worked in. They understood that seriousness
was not professionalism. Having fun makes a better team
having a good time. Being empathetic with your
colleagues, sharing good experiences.
That makes the team pure as much as anything else.
I think we sometimes make the mistake that if we seem to be
having a good time doing something.
Then surely we're not doing it right, you know.
But actually often I think it's completely the inverse so that
would be my second one serious does not equal.
And then the final one, which is interesting, only, then this
like awesome down doors. It comes from a conversation
with you like gold, brass the great Management Consultant, who
wrote the goal and Beyond the goal, and a number of other
books. And he was asked about it.
Why organizations often fail at adopting new things, new ideas,
new Innovations. And he says that in his
experience, it's because there are four questions they need to
ask, right? And they forget to ask when.
So the four questions are, if you've got a new thing, whatever
that Innovation is microservices Docker, kubernetes continuous
delivery. You should ask yourself, okay,
one of the cool things that this thing gives.
What can this unlock for my organization?
If we adopt this Innovation is technology and the second
question you should ask is OK, what current its limitations in
the company? Will this new innovation help us
overcome? And then he says, you should
then look and say, okay. What rules do we have to manage
our existing limitations? And he says, when people adopt
new technologies often what they do is they stop there.
So as an example, continuous delivery would be a good example
of this. When you adopting continuous
delivery, that's going to give you the ability to speed your
time to Market to the Ultimate Security audit, all these really
cool things built by quality etcetera, etc, etc.
And what limitations do we currently have are we've got
this massive manual testing process that we could, probably
use continuous delivery would help shorten the amount of time
it takes to do that, okay? And then they are all.
What rules do we currently have? Well, you know the rules are
that you wanted to get into you, 80 years have to pass this
barrier in order to integration. Pass this thing about it.
And so you adopt continuous delivery and the development
teams, go super super fast, but you forget to ask the fourth
question. What new rules do?
We need and the new rules are the important bits because if we
try and marriage new stuff with the old rules, you don't get
anywhere. So, in the continuous delivery
case you can say, well, we'll manage our new features delivery
process, but using the same set of manual gates to get through
to different environments, it doesn't matter if you've got a
computer doing it, if it still takes you six weeks to get
through, if you do Anything. So what are the new rules look
for the new rules that you need and implement the new rules?
That would be my third one. I've ever listened to this.
Four key questions that you mentioned right whenever we want
to adopt new things. So thanks for sharing that and I
left the second one. Seriousness does not equal
professionalism. So I think yeah, in some
organizations they tend to want to be very strict and follow the
process. You always hit the target,
right? But that doesn't always equate
to professionalism. But you mentioned right?
Or like we need to create fun. Fun.
And I think when we work in a fun manner, I think people tend
to try for, I tend to produce the best results.
So James, I think it's really a great composition.
I learned a lot. So if people want to continue
the conversation and maybe connect with you, is there a
place they can find you online. They're arming, the best place
is probably very email. If you want to chat with me very
long dreams, don't leave yourself a full which to come.
I'm also on Twitter at Boise be. Oh, I see why I don't like data
that. Use it that much.
So those are probably the three places of the best.
Thank you for this opportunity. James, I hope people also
learned a lot from this conversation.
So thank you again. We thank you so much variation,
actually, pleasure talking with you and hopefully I'll get to
see you in Australia again before too long.
Yeah. Looking forward for that.
Thank you for listening to this episode and for staying, right
until the end if you highly enjoyed it.
I would appreciate if you share it with your friends and
colleagues who you think would also benefit from listening to
this episode. And if you are new to the
podcast, make sure to subscribe and leave me your valuable
review and feedback. It helps me a lot.
In order to grow this podcast better.
You can also find the full show notes of this conversation on
the episode page at technology Arnold death website, including
the full transcript, In quotes and links to the resources
mention from the conversation. And lastly, make sure to
subscribe to the show's mailing list on pack leader know that
deaf to get notified for any future episodes.
Stay tuned for the next technology, no episode.
And until then goodbye.