I saw a lot of people doing was just assuming that micro
services were mini web servers. And sure a mini web server with
a small API and sort of database is a micro service, but it's
horrible architecture. So I think the way to do micro
services, the Tao of micro services, is not to be obsessed
with the network and Kubernetes and all that other stuff.
To think of them first as software components and all the
other stuff is, is just a consequence to get Legos, to get
components. You can't have complex
interfaces. I was very inspired by Unix
pipes. So if you think about it, that's
one of the most successful composer models ever built in
the history of computing. And that's because the interface
is really simple. It's just a stream of bytes from
one process to another. Just because the network is
unreliable doesn't mean a monolith is unreliable either.
It's a fallacy to assume that you can build an error free
system. The seven network fallacies are
100% true, but you don't deal with them by trying to make the
network infallible. You deal with them by accepting
that system overall, even if it's a monolith, has a baseline
error rate. And that's a business
requirements issue. Hey guys, welcome back to
another new episode of the Techni General Podcast.
Today I have with me and author an interesting book, I must say,
at the Tower of Micro Services. His name is Richard.
Roger Richard, welcome to the show.
Henry, thanks for having me on. Yeah, let's start arguing about
micro services I'm looking forward to right, because, you
know, everybody agrees on what micro services are and how to do
them. There's no controversy at all.
I'm sure we we'll get to a lot of different perspectives about
microservices and monolith and things like that.
But maybe let's start the show by asking about yourself, as if
you can introduce, maybe telling us any highlights or turning
points in your career that we all can learn from you.
Yeah, I graduated in 97 and I studied mathematics and
philosophy and by random luck the mathematics course at AC
plus plus module. So I learned to code on VI,
original VI on VT100, green screen terminals in the attic of
the maths department. So of course I ended up in going
into web development, right? It was just the pre.com boom
era. The website that I'm most proud
of, the fastest one I ever built was in C pure C.
It was a real estate website. I didn't know about malloc so
everything was statically allocated which is why it was
really fast, but literally every operation was a buffer overflow
waiting to happen. But it remains the fastest
website I've ever built. I spent the 1st 10 years of my
career as a back office Java developer.
I pretty much went to no conferences, did almost no
community stuff. I did read a lot of stuff, but I
never really kind of engaged outside of what I was, what I
was working on till I discovered Nojs and the Nojs community and
kind of got out of my shell. I got into startups and the
first successful startup that I was involved in that did exit
after I left, unfortunately built the core tech as the CTO.
We built something like a Roku for mobile apps, but if you use
JavaScript on the back end, use the Rhino engine for JavaScript
because we wanted it to be easy and quick to, you know, kind of
like almost like no code, nearly quick to build a back end for
mobile apps and developed a love for JavaScript, which is a weird
language, but I don't know, once you kind of get over all the
strange stuff, yeah, I don't know, it just kind of clicked in
my brain. You know, I like the object
prototype approach rather than the sort of hard classes.
When I was doing Java, I read the Gang of Four design patterns
book. I could tell you off by heart
all the patterns, right? Visitor everything.
And then when I discovered JavaScript, I realized you
didn't have to actually use any of the patterns.
You can just go pretty much functional.
So myself and three other guys set up a consultancy based
around Node JS and you know, sometimes you can get lucky.
People talk about product market fit and if you have to ask the
question, do you have product market fit, you don't.
No, we were a product company. We tried, but we were just a
consultancy. But Node was taking oath around
that era since of 2012 and we were pretty much the biggest
node consultancy in Europe. We started a few conferences, I
started speaking at various conferences, that type of stuff
and nothing took off. And within the space of two or
three years, we had a big business, which is kind of
crazy. So that was super fun, super
stressful. And then we discovered that
JavaScript is not a great language for writing large
enterprise systems, but our clients wanted us because it was
the hype of the thing. So that's how we ended up using
micro services or the idea of micro services, but also ended
up with a kind of a weird take, which we can get to in a little
bit. But basically the limitations, I
mean, it's weird, isn't it, right.
I love the language JavaScript, but the limitations of
JavaScript. And that's, I mean, I coded
TypeScript now, but you got to remember JavaScript back then,
ten years ago was pretty bad, right?
It was kind of a sucky language. Once you've got any sort of code
volume, you can't even refactor it.
I mean, there's no type safety. Hopefully you wrote good unit
tests, but they're probably tied to the implementation, so
refactoring is almost impossible.
So out of the vadnais of JavaScript, we ended up using
micro services to compensate. It's funny how things like that
go with softer, isn't it? You have this idea that there's
some sort of perfection or ideal way to build systems, but it's
very, as the physicists say, it's very path dependent.
So where do I get to? I, I, I have to get to today
because you asked about the career history.
So yeah, yeah, consulting is great.
But if anybody's working in consultancy, they know it's
really hard work, tons of travel.
And I'd still wanted to do a proper SAS, a proper traditional
VC funded SAS. So because I've been doing a lot
of conference speaking, I thought I'd spotted an
opportunity in the event tech market, specifically a SAS
solution to help people run booths and do sales league
capture and recruiting. And because I mean, I was
speaking at like a couple of conferences a month, but
organizing it all using a spreadsheet, which kind of
sucked. So I sold out of the consultancy
and sold out to the other partners and set up a SAS
business, did the Angel round, took in a good sized round,
built the product, everything was fantastic until March 2020.
So we lost all our customers to go in the space of a month and
we had to pivot. Eventually, though, it took us
about a year to figure out what to pivot into, but we realized
that a key part of the market that we've been looking at was
developer relations. And luckily, developer relations
still has to happen even if there are no conferences.
So yeah, we pivoted into that as a market, which is where we are
now. However, it's never a simple
straight line because of COVID, we never did a Series A.
And you know, how do you survive?
So guess what? Consulting.
So I tried, I tried to get away from consulting, ended up doing
it again. Luckily we're kind of refocusing
on the product now, but I ended up doing consulting again for a
few years, which, yeah, really put the ideas around micro
services to the test again. Because I guess the interesting
thing was that although we didn't make the entire Eventech
platform open source, about 80% of it was open source plug in
based. We'll get into why that approach
was taken, but you know, that means that if you've built
something that's based on a micro services architecture with
open source components for one vertical event technology, it
was a good litmus test for the adaptability of micro services.
Can they be used in other consulting projects, in other
domains? And I think we found that the
answer was yes, but for a very particular reason.
Now, I'll just tease it. We'll get into that later.
And that brings us up to today. So yeah, the one regret I have
and if anyone here is if any of your listeners are a junior
engineer, I would say that a couple of years in a much larger
big tech company is probably worth doing at some point in
your career. I nearly nearly worked for SAP
when I lived in Germany, but decided to chose a start up
instead. But I probably should have taken
the job in SAP for a couple of years.
I mean, I, I've ended up having clients that are big companies,
so I suppose I ended up understanding in that way.
But even if you do intend to do a startup, I think it's
important to work at a really big company to understand how
they work so that you could sell to them.
So that's I guess one, right? But apart from that, I've had
lots of fun. I'm still coding, coding for 27
years. Not going to stop.
Wow. Do not intend to stop.
Oh, that's the other thing. Yeah, you've got 20 good years
from about 18 to 38. After that you've got to
compensate with good design, because the old nighters don't
work anymore and you can't code yourself out of complexity just
with brute force because yeah, you lose IQ points, the brain
starts degrading, unfortunately. So yeah, enjoy.
Enjoy it while you can. Thank you for sharing your
story, I think it's really interesting.
Sounds like a roller coaster as well, right?
So you've been into so many different journeys.
So definitely there are a lot of learnings.
But today we'll focus more on the micro services, right?
In which you have written this book called The Tao of Micro
Micro Services. But first of all, I'm quite
interested, like why do you choose this title?
You know the Tao. Is there any specific meaning
behind it? Yeah, yeah, it's interesting.
When you write stuff, sometimes you you can sort of forget to
fully use a metaphor. And I'm currently doing the 2nd
edition of the book, so hopefully I'll get to actually
use the metaphor fully this time around.
What it refers to is the fact that what I saw a lot of people
doing was just assuming that micro services were mini web
servers. And yes, OK, micro services is a
very broad term, like everything in tech.
And sure, a mini web server with a small API and some database is
a micro service, but it's horrible architecture.
It's you know, all those articles that you read
criticizing microservices for people saying, oh, we're going
back to a monolith because we had a really bad experience.
They are all legitimate criticisms.
I came at microservices from a totally different place.
I felt there was a different way how it's kind of the way I felt
there was a different more philosophical way to approach
it. Let's talk about that because I
think that's where a lot of the potential has been missed.
It doesn't mean it it can't still be taken advantage of
because the way we develop software is continuously
evolving. So I worked as a researcher for
a while. I had my very first startup
failed. It's very important to have a
failed first startup. I was completely broke.
So I ended up working as a researcher at a local university
and ended up working in the telecom space on IP telephony,
which involves something called the session initiation protocol
SIP, which is kind of like HTTP except state based and has loads
more verbs. And if you imagine 2 phone, one
phone ringing another, then it's you're sending like ringing
state messages that I don't know.
Henry, have you ever played around with any IP?
So yeah, it's horribly complex. No, I wouldn't recommend it.
In the startup that failed, I'd really gotten into, if you go
back to the game, four design patterns, really got into the
command pattern where you kind of encapsulate commands in the
system. You can try to express all your
business logic as commands right now.
I mean, this is a very, very early version of something like
CQRS, event based approaches, event sourcing perhaps, but very
locked into the code, very hard coded.
My experience working with IV telephony led me to think about
the power of pattern matching. So the problem with receiving
different stateful messages for IV telephony is that you have to
decide what to do with them. So you're trying to write a
little state machine, but first you have to match what's going
on. And the application we were
building, you have to provide customization.
So we just ended up writing a really, really simple pattern
matching algorithm. And so I mean, if you've written
something, if you've used Haskell or something like that,
you understand this idea of pattern matching.
In fact, I think it's being introduced in a whole bunch of
languages now. I think there's even something
for JavaScript. It's literally the most brain
dead simple approach thing you can think of where you're just
look at a bag of properties and you're just matching against
some of them. I mean, you can get more complex
regular expressions or matching interior properties or whatever,
but it doesn't really matter. All that matters is that you
have this bag of properties, which might be adjacent document
or HTTP request, and you're just picking a few that you care
about and then using that to do more business logic.
So when we got around to building Node JS applications, I
kind of put these two ideas together and said, how do we
turn the command pattern into a component model that uses
pattern matching? So here's the thing about
consulting, right? And have you ever worked worked
as a consultant? Have you ever built websites for
people? Yeah.
OK, so here's the problem. It's the same job every time,
really. OK.
Because what we do is consultancy is we turn Excel
into administration interfaces, right?
And there's always user management and permissions and
projects and groups and there's always data ingestion, there's
always report generation. Maybe some of them have a
mapping application or something like that.
But business applications are always the same.
So if you're building a consultancy, you want to be able
to reuse stuff because that makes you more efficient, better
job. And Node was very open source,
so a lot of the MPM registry was growing exponentially.
I know that has its own problems, right?
The problem is a lot of those components were technical
libraries. So stuff for like sending e-mail
or reading an Excel file or database drivers or Orms, right?
That type of thing. Yeah, they're components, but
the problem is every one of them has like an API that you have to
learn. And they're infrastructure.
They're not business logic. So let's say you're building a
referral system. Your users can invite each other
into the application. The logic is pretty much the
same every time. So how do you build a referral
component that you can reuse? That was the problem that we
were facing. So we built the system where it
was event based. The only interface between
components was a Jason document, and the routing was based on
pattern matching. Your system consists of.
This is before micro services, right?
It's a single process application, just a single node
process running on a bare metal server.
This is the way we built things back in the day.
I built an entire newspaper system using this description
bottle, Apple Pay, everything. But the whole architecture of
the system consisted of very isolated components.
The only interface input and output was effectively Jason
documents. The routing to decide where a
message went was based on pattern matching.
And that meant that you could define a referral service, let's
say by a set of messages, so an invitation that's happened, or
generate an invitation code, or give this user a reward based on
the fact that the invitation code has been redeemed by
defining Jason documents. And you could have schemas if
you wanted, right? But essentially you didn't have
to. So when a message comes into the
system, you might have some tags that say this is a referral
message and it goes to the referral component.
So that was fun until about 2014 until like I said, things get
too big, right? It's very hard to manage a
system like that. It's hard to scale it.
I mean, you can for sure, you can deploy multiple monoliths
and put them behind load balancers and all that sort of
stuff. But then we started noticing
things like serverless and Docker and micro services idea
was kind of in the air and it was just an obvious next step to
say, OK, well, why don't we take the referral component and turn
it into a micro service and then we can deploy it independently
and we could scale it independently and get all those
benefits. But the important thing is it's
not a mini web server. It's not a mini API endpoint.
It only knows how to receive and send Jason documents.
That's it. So that's how we ended up with
that. That's how we ended up in a
place where we were using micro services.
But with a component oriented approach.
So I think the way to do micro services, the Tau of micro
services is not to be obsessed with the network and Kubernetes
and all that other stuff. To think of them first as
software components and all and all the other stuff is is just a
consequence. Right.
I think that this is a quite unique perspective because I
think from all the journeys of micro service that I have
followed, right, I think it has gone from ups and downs, for
example, from monolith to micro service and then it goes back
into monolith. These days some people call it
modular monolith or whatever that is, right?
But I think that there are still places where micro services
could actually work. And sometimes some people
advocate things like domain driven design, bounded context
to actually create a proper boundary.
But the one that you just mentioned is actually to think
of micro services in terms of three different things that I
could gather from the book, right?
The 1st is about designing the message first.
So things like the Jason document you mentioned.
Second is the pattern matching, which is kind of like, you know,
associate the kind of shape of the message and then you route
it differently to different services.
And the last thing is about additivity.
You know, like think of it like you can compose, for example, if
you want to work on some workflows, right?
You compose that kind of workflow using different
components or micro services. So maybe, yeah, this is I think
the true maybe the tower of the way of you defining micro
services. So tell us about why these three
are so important and why do you advocate this approach instead
of, you know, some of the more popular way of designing micro
service? Yeah, it comes back to this idea
of components. First, you actually mentioned
the most important one, which is composition.
If you're designing object oriented system these days, you
should prefer composition over inheritance, right?
So inheritance people realize that inherent, you know, basic
inheritance is probably a bad idea.
You end up tying yourself in knots, whereas composing these
together is the way to go. So I mean, what's a component
system? It's Lego, right?
It's meant to be like you're 10 years old and you're building a
spaceship with Lego and the things just click together.
So how do you get to that point? How does that work?
Because everybody who's worked on large enterprise systems know
that they're not Lego. The refactoring is horrible.
I mean, everybody's had a iteration where you have to stop
the world and refactor. So the composition element is
really, really important. And I think the pattern matching
gives that to you in with a really nice mental model.
So first of all, I should say separately, the implementation
details here don't matter. We built an open source
framework for doing this. But it's actually pretty small.
You could do this in any language.
You could literally write one from scratch for a Greenfield
project or whatever. You don't particularly need to
use a framework. It's it's more of a philosophy
than than a framework. Pattern matching gives you
composition because it allows you to easily go from the
general to the specific. So one of my go to examples is
let's say user management. When you start building the
system you might already have, you might already decide there's
one type of user in the system and there's a little bit of
logic to deal with that. So I don't know.
Assigning a user to a project is 1 Jason document with a few tags
that you could have and match up.
But that only gets you through maybe two or three iterations
because of course, systems don't have one type of user.
They have admin users and they have project managers and they
have group permissions and all sorts of crazy stuff.
So if you have an assigned user to project message, that is
general, very simple, right? It's just adding an entry to a
database table and setting up very basic permissions.
That's your first implementation, but you might
have additional business logic that has to happen.
If it's an admin user, there's a special value that has to go in
or something into the database or they have to be put into a
central registry. There's some sort of extra thing
that has to happen. So in most systems, what you'd
end up doing is adding a whole bunch of if statements in that
logic, say, oh, if it's an admin user, do this extra stuff, but
that's not composable. What you want to do is you want
to leave the old code alone. Pretty much that just handles
normal users. Run that code and then also run
extra code that handles the admin case.
So what you can do is either before or after.
And again, this is pretty simple to implement because all you're
doing is you're saying, take a look at Jason documents and if a
few of the properties match, do something.
So I can say, OK, my properties match for adding a user to a
project, add the user to the project.
Oh, there's also A tag that says this user is admin, so I'm also
going to trigger the admin user message.
You mentioned additivity, right? So that's additive.
The old code stays the same and then here's the new code and you
click it into the system like a Lego brick.
And you can do that for all sorts of things.
It's particularly cool for cross cutting concerns.
So one of my other favorite examples there is, let's say you
want to have like creation time and an updated time in your
database rows across the board for all of your data entities.
Well, if you encode the data entity operations, you know,
saving and loading as messages, then the pattern matching can
intercept those messages and add in creation time and save time,
auditing permissions checks, whatever, all sorts of fun stuff
without that being visible in the business logic code.
The point is, you want to keep your business logic as business
logic, right? Business logic shouldn't be
setting creation times or worrying about permissions or
any of that sort of stuff. So if you have these additive
components, you can see how it's pretty easy to start saying,
well, maybe they actually live on separate parts of the network
and maybe the messages can flow over the network to make these
things happen because it's all just pattern matched routing.
The business object code has no knowledge of whether something
is on the network or not. So you can run the whole thing
as a monolith. And this is how we develop,
correct? We run the whole thing as a
monolith locally. I mean, local development with
real microservices. And I have done this and I have,
you know, killed my machines like 30 Docker services and
crazy stuff like that. These are not lessons that
weren't learned without a lot of blood.
But you can run locally as a modular monolith because you've
got your modules. And then our favorite deployment
these days is split up the messages by domain and deploy
them as Lambdas. And that's really cool because
you don't have to worry about running things in separate
service locally. You can use Terraforms,
something like that to automate the deployment of the lambdas.
So you you don't you're not worrying about working with
lambdas and deploying stuff to debug and all that Lambda
development is kind of sucky. And then the other big thing
that the component based approach gives you is really
simple mocking. So anybody who's ever tried to
build unit tests and build mocks is some, well, I don't know,
have you have you heard of the banana monkey jungle problem?
No, I haven't. So this is a, this is AI tried
to imagine it in a very posh English accent, because I
learned about this from a friend of mine who used to be a fusion
scientist. He worked on the joint European
tourist. So he's very clever chap.
So you must be right. So the problem with a lot of,
and I mean, this is a big problem in Java.
The problem with writing unit tests in a lot of systems is
that you have all this dependency and you have to do
dependency injection, all that some crazy stuff.
So you need a banana to unit test the banana, right?
But unfortunately to get the banana you need a monkey.
And if you need a monkey, you have to get the whole jungle,
right? So you basically have to
bootstrap your entire system to do a small unit test.
Now, if the only way that your system is exposed is via Jason
documents, then mocking them is really simple.
You just hard code the response. You don't need all the rest of
the infrastructure. You don't need to have mocking
utilities that will build complex.
AP is, and I think this is, it's a point I make in the book,
actually, I just remembered. To get Legos to get components,
you can't have complex interfaces.
You think about modern languages, they try to present
the class as a component, but classes have methods, they have
interfaces, they have static methods, they have member
variables, they have annotations.
Are they singletons? Are they?
And it only gets worse, right? Because in order for the
language to have power, the class construct has to have lots
of different ways to interface with it.
So now you have to learn a new API each time, and someone has
to model that whole idea of user types in a system that would
have to be modelled. So now you've got to interface
with it somehow, and it's very hard to compose together.
I was very inspired by Unix pipes.
So if you think about it, that's one of the most successful
composer models ever built in the history of computing.
And that's because the interface is really simple.
It's just a stream of bytes from one process to another.
So that has its own problems. It's a little bit too simple.
I don't recommend building. I mean, people have, right?
But I don't recommend building enterprise systems with streams
of bytes. So you need to just up the
complexity a tiny little bit more, right?
So Jason Documents, I used to be really big into XML, by the way.
I've written XML parse results of fun stuff.
It's cool, but it's so painful, which is why I like Jason so
much, right? Because it does make life a
little bit easier, even though I could really do with having
comments. But anyway, if you up the
complexity just a tiny bit and you allow the messages that go
between components to be Jason documents, and then you, you
know, apply that leniency principle where Postal's law, I
think it's called, where you're leaning too much, you accept and
strict in what you omit. It's just powerful enough to
build enterprise systems. When you use pattern machines,
it covers about 95% of use cases.
For the 5% it doesn't. You always have to be able to
expose the actual underlying API, right?
Always have a get out of jail free card, but 95% is pretty
good. I've kind of covered a lot of
ground, but I'm trying to get the philosophy Yeah, in one
body. So I have some follow up
questions actually. So when we think about these
messages, right, you keep mentioning about Jason documents
and things like that. Some people also these days have
even driven architecture, right, even like modelling software as
events. Is this message you are talking
about the same as kind of like an event?
From my understanding, it seems not.
I mean like it's not fully 100%. And then how does this pattern
matching work against the message?
Because when you talk about composability, I think it's kind
of like easy to, how should I say easy to think about when you
talk about function calls or maybe like Unix pipe.
But when you have different micro services and you have
different network calls and databases, how does this
composability work? And not to mention the
transaction as well. So maybe if you can maybe
elaborate a little bit more on these two parts.
Yeah, so my concept of messages is a bit more general and I'll
actually refer to the book because I did spend a bit of
time in the book creating an architectural structure for
thinking about it. So event sourcing is very much
unidirectional flow and that's a great idea.
And it's a core benefit is that writes go in One Direction and
reads come from a different place.
It's a kind of a one way system. The real world is a little bit
messier, especially since you might be dealing with legacy
systems and all that sort of stuff.
And a great many AP is are designed to be request response.
So I identify kind of two axes that messages could operate on.
So this is one of them is whether they're synchronous or
asynchronous. So synchronous messages expect a
response. You send a message and you you
expect a response, you're waiting for a response.
So if I save something to the database, I expect a response
confirming that it's saved. It gives me a new ID, something
like that. Asynchronous messages are ones
where you just emit them and you don't know, right?
Somebody might act on them, you just don't know.
And then the other one is whether they're consumed or not.
So you can think of a classic message queue where you're
sitting there as a listener and the message comes from the queue
and you take it and process it and then the message is deleted.
Amazon simple queue service, something like that, right?
Or you can think of something like Kafka where there might be
multiple readers. So one of the people that I
found really inspiring in the early days, microservices, and
if you go onto YouTube and look for his talks, they're
absolutely fantastic, is a guy called Fred George who built a
lot of the early micro services systems.
One of the examples he gives is insurance quotes.
So you're trying to do dynamic insurance quoting.
So somebody types in their details and they give me quotes
and the system could process for about 30 seconds.
So the way he implemented that system was that messages would
go out to lots of different micro services that could handle
different types of quotes and then they would all provide
quotes and then there would be some business logic to rank them
or something like that. And that's an example of a
system where the messages are not, they're not consumed by
single service, they're broadcast.
So when you're building a system, you can think of it in
terms of whether the messages request response or whether it's
more like event sourcing. So whatever way you get messages
from micro service A to micro service B, whatever way the
routing works, whatever the pattern matching works, give
yourself that freedom because you will need it.
But I mean, sometimes the criticism that I get and
something that sometimes I've heard people struggle with is
that they think I'm hand waving when I'm saying our message
routing is do whatever you like. So let me give you a very
concrete example. So recently we built an AI chat
bot for podcasts, right? It lets you kind of interrogate
guests. It's kind of fun, just as a kind
of fun side project. And that runs in Amazon, uses
Simple Q service. And if you think about it,
there's all sorts of stuff. You have to download the RSS
feed and then you have to download the audio.
And then if you have transcripts, and then you have
to chunk the transcript and then you have to put it in the vector
database and do all the fun stuff, talk to the models to get
the vectors. So it's very much batch
processing flow. It's very much messages go from
A to B through different lambdas.
And if you built lambdas and use SQS, the way that works is you
have named topics and one Lambda will submit a message to a named
topic and then another Lambda is listening to the named topic.
So that doesn't fit my model because if you think about it,
those named topics are hard coded.
But I won't have a situation where I'm free and easy and I
I'm not stuck with topics. I just want to send my message
and it magically gets to the right place.
So in my my modular monolith that I'm building locally when
I'm developing the system, I just say that my, let's say my
audio downloader microservice will listen for messages that
say here's the RSS. And then it gets the RSS and it
can look for the audio file download.
Which means that my component that downloads the RSS can emit
that message. Everything happens in one
process. I have a tiny little algorithm
that looks at the keys of a Jason object in memory and then
makes a method call to the right place.
So somehow I have to translate that to Lambda SQS architecture.
Initially when we started doing it, stuff like that was moved
into configuration. So you would say, OK, these
patterns of messages, yeah, it's like when you have a hammer,
everything is a male, right? These patterns of messages all
mapped to this topic queue. So I have an extra bit of code
that or an extra plug in, let's say that I deploy, when I'm
deploying to Amazon, that intercepts the messages.
This is a common pattern, right? Intercepting.
It's just pattern matching like anything else, except this time
it doesn't do any business logic.
This time it does transport. It'll transport the messages for
you. And for a long time that
configuration was done like any configuration system, right?
It was in the code you configure that and there are option
configurations for these plug insurance.
But one of the reasons that I decided to do a second edition
of the book is model driven architecture.
So this is giving anybody who's my age nightmares at this stage.
So about 20 years ago, it was really, there was this a lot of
hype around UML diagrams and you would literally architect the
system as a blueprint and then all of the junior coders would
just type it out basically, right?
Total failure because that's how my software actually works.
But I think there was a kernel of an idea there that was really
good, which is that. And sorry, then you can combine
that with like domain driven design and things like that to
say there has to be a higher level abstract description of
the system that's above the code in some way.
And I mean, we use that all the time these days, right?
So YAML definitions for serverless deployments,
Kubernetes infrastructure as code, right?
But you can have architecture as code as well, not diagrams,
right? Not visual programming, because
that never works. But you can essentially use all
of the very robust configuration tools that we have these days.
There's a lot of them, right? So take the ideas in Terraform
or Palumi or things like that and create a type safe model of
the system where you are listing your database tables and you're
listing your services and you're listing your messages.
And then you can do things like say, oh, here's the Lambda
deployment context. And in that context, these
messages go to this topic in SQS and Bats.
Now, deployment configuration completely separate from
business logic. So I never have to, when I'm
writing my podcast subscription code, my audio download code, I
never have to care. I just don't care how they talk
to each other. I just know I can take for
granted that the message will get delivered.
And then you add in lots of the modern observability, like open
telemetry for actually debugging that.
And yeah, it's kind of fun. We've a lot of fun these days.
Right, I can also see a few alternatives to pattern
matching. I mean tell me if this is not
correct, right? So you can use something like a
server path routing, right? So where you have HTTP/A goes to
a service slash B to B service. Or you can also take the message
bus approach right where you have some business rules so to
speak, right in the message bus to actually route to different
topics. Or maybe in like Rabbit MQ you
have different channel and exchanges and things like that.
Is that also the same kind of thing when you mention about
pattern matching? Yeah, I have a very big hammer.
They're all nails. So the issue is the power that
you get from keeping everything to pattern matching is that
things like unit testing and mocking get much easier.
Or especially if you have junior developers on the team, the only
API that they interact with is essentially you can send
messages or receive them. Now we do add a little ORM
layer, right? Not Object Relational Mapper.
That actually becomes messages. Anyway, we'll talk about that in
a minute because that is quite powerful.
But when the junior developer is writing business logic code, all
they should ever be doing is sending and receiving messages,
loading and saving stuff to the database.
That's it. So if those messages are coming
in from Rabbit and Q or they're coming in from HTTP request or
whatever, the routing should translate that first.
So for example, on the HTTP side of things, what we've written
for ourselves is little gateway plug insurance that will
translate. So locally, I'm old school, we
still use the Node library Express for local development.
So we have a little gateway that can translate Express inbound
Http://requests into a Jason document.
Yeah, there are, you know, a subset of requests where you
need to be able to access the headers and all that sort of
stuff to do both. But again, you can use the
pattern routing because that's a cross cutting concern.
You can put it elsewhere and you can just restrict business logic
to saying here's a Jason document.
It happened to come in this case from HTTP.
But you don't know, maybe it's a unit test, right?
So you know, the way unit testing APIs is really painful
because you have to simulate AP Http://requests.
You know, there's a few people that do it really well, like
Fastify. Yeah, I'm obviously using no JS
examples here. Fastify is really good system
for simulating inbound HTV requests.
But the issue there is to use that you have to commit to
Fastify. The problem is if you're
deploying to Lambdas, well there you're not really using Fastify.
I mean, you could, but generally you just want to use the events
that Amazon sends. So now you have a whole
different system. So you have another little plug
in that can translate Amazon events.
We recently had a scenario where we built a system on AWS, spent
six months building the system, enterprise system to be told one
month before deployment that corporate policy was that
everything had to run on Azure. So you can imagine that in most
scenarios that that's a total disaster.
If you have used any Amazon AP is directly you're in trouble.
But because we've modeled the whole system as messages, all we
had to write and we already had some because we obviously built
other Azure systems. All we have to do was write
things like a little gateway plug in for Azure Functions
which have a slightly different inbound message for which by the
way is adjacent document anyway, right?
There's another good analogy for this approach which is AWS
Eventbridge. So if you've used Eventbridge at
all, it's actually very similar or Erlang OTP, similar style.
These are all either influences or same basic idea.
So yeah, it is a very, very big hammer and I doubt advocating
for a very specific approach to building large scale enterprise
systems. But at the same time, it's all
driven by this idea that you really want a composable
component model and the only way to get there is to have a
simplified API. So you always want to hide
things like Rabbit in Q or the fact that it's actually be
requested at the end of the day. One final thing I'll say on that
is I'm very much aware of the seven fallacies.
Network architecture. And I know very well that what
I'm advocating for is to pretend the network doesn't exist, but
that's a different discussion. Yeah, I think the one that you
mentioned like it's called transport independence in your
book, right. So yeah, even though you use web
server, HDP web server or you use message queue or you use
some other mechanism, I think it doesn't matter as long as you.
So I'll translate that into messages and you use some kind
of pattern matching to be able to route to different services.
And I think this key about composability is something that
maybe not many developers naturally think that way,
because if you think about adding user types, most of the
people will just refactor the code and tweak the portion and
do if else and that kind of thing.
But I think doing pattern matching maybe is like the
functional paradigm as well, right?
The functional programming paradigm. 100% I mean, I stole
the idea from Erlang totally, yeah.
So I think these are definitely the heuristics that you advocate
whenever you design micro services.
And obviously in your book you also covered about model driven,
you know, like modeling code. Yeah, yeah.
Which, yeah, some people like it, some don't, but we probably
won't. Diagrams.
That's that's the secret. It's all code.
Yeah.
So maybe as part of this everlasting debate about micro
service versus monolith, what's your take about monolith, by the
way? So do you actually ban monolith
at all in your software design, or do you actually find that
monoliths still have a case? No.
So, you know, because we use a modular monolith effectively for
local developments, it's a wonderful get out of jail card
because that project that we're selling that where we had to
move to Azure, we converted everything from Lambdas to Azure
functions, set everything up. And in the last week, I turned
out that the admin who could give us appropriate permissions
was on holiday. So there was no way for us to
get enough permissions to make the system work and go live by
the deadline. So our solution ended up being
to run AVM and run the system as a monolith in production.
And now there were some performance issues, right?
Because this is where micro services are great for scaling
because parts of the system can scale independently.
So we did have to deal with some performance issues, but we had
to get out of jail card and that we could, It was nothing to do
with the software architecture, but the dynamics of the
politics. The project meant that the only
way we could go live was to deploy monolith.
So we still had that option. And I would not have had that
option five years ago. I mean, this is when people say
micro services are crazy and monoliths are easy.
They're right. And when people say, you know,
if you're building an NVP, maybe you should just build a monolith
to begin with. And I actually agree, right?
Just deploy it on a bare metal hexer server or something for
$10 a month. But wouldn't it be nice if it
was a component based model that you could turn into lambdas or a
big Kubernetes deployments without having to rewrite the
whole thing? It doesn't have to be my
specific approach to pattern matching.
It doesn't have to be node JS. You just have to start from a
place of saying that my system should be composed of
components, and all the components should have the same,
just like Legos, they should have the same interface.
And that's really where all the power comes from.
Yeah, it's one of those silly debates that isn't real, but it
comes from places like, I mean, I'll tell you, we were brought
into a large enterprise client. This would be maybe 2016.
Micro services are getting really popular and they built a
whole load of mini web servers and they were running about 60
micro services. And the problem is that even
though they've given all the developers really powerful
laptops, as a developer you couldn't run the system locally.
So every developer was also given a large Amazon instance to
run the system so they could develop against it.
And I mean, of course, if you come into a set up like that
where you have a team of 50 developers that can't even run
the system locally, of course you're going to cope with micro
services are utter nonsense. Why on earth would you do this
to yourself? The only people that I've ever
seen that have actually managed to make the mini web service
thing work is Netflix. I mean, they had the resources
and the brain power to make it work.
If you look at their open source and all their tooling and all
that sort of stuff, wow. I mean, it's amazing.
I mean, yeah, there's a fantastic way to control
everything, but it kind of tells the story, doesn't it in, in
that they needed all that brainpower and resources to make
it work, you know, And I, as an advocate of micro services, you
know, surely the easy Rd. is to use Netflix stuff.
But I don't have a team of 20 Stanford graduates to make it
all work. Not to mention running all those
Netflix tools right? I think that might be quite a
set of people. Right.
Exactly. I mean, I'm sure there are other
people who've who've done a really good job of it.
Netflix open source, I kind of tell, yeah, it's what's the
Spider Man purse. With great power comes great
responsibility. So I guess services are really
cool and you can do really cool stuff.
But as the Australians say, they can be a massive foot gun as
well, right? So, so maybe let's go into what
you mentioned just now about the network policy and also the
data. I think these two typically are
the the two biggest things, right?
So network is you can't assume network is always reliable and
fast. And second thing is about data.
When you have your data passing across multiple services,
there's no atomicity anymore, right?
Everything is like an eventual consistency and how do you
reconcile if there are failures in one part of the service into
the other? Yes, yeah.
OK, so here's the thing. Just because the network is
unreliable doesn't mean a monolith is unreliable either.
You know, when I used to deploy monoliths in C and Java, stuff
like that, VMS would run out of disk space because the logs have
used up everything, right? Or memory leaks, right?
I mean, when I used to run Java servers, but we used to just
restart them every 24 hours because of memory leaks.
But the problem is if you just randomly restart a server at
4:00 AM in the morning, well, that's not 4:00 AM for
everybody. You could be in the middle of
transaction and they can just die.
So if you walk into a supermarket, do you think that
the prices are 100% accurate across every single label of
every single product? No, there's actually in the USI
think the law is the federal regulation is that they allow 2%
error rate. So the price that you see on the
shelf is incorrect up to 2% of the time.
I think, and again, this is inspired by Erlang, you have to
design your systems to accept and a baseline error rate of
some kind because weird stuff is always going to happen.
Yes, the network is going to make that a bit higher.
Of course the trade off is always the trade off is that you
get scale. So you have to look at your
business case and think about what the mitigations are going
to be. If you're building a, a consumer
app, you know that's to do with user generated content.
I mean, how many times have you used Reddit and it doesn't load?
It's like we try load again, right?
So I mean, who cares? There was a network failure,
just reload. I've built banking apps and I
can tell you that they're very careful with their mainframe
COBOL systems that do the actual transactions.
But I can also tell you that there are backroom staff that
reconcile problems. I mean, have has your bank been
100% correct over your entire lifetime?
You hear these stories of people who are like magically have
$10,000 appear in their account, this type of stuff.
So it's a fallacy or telcos and they're five nines up time.
I mean, have you ever had a cell phone call fail?
It's a fallacy to assume that you can build an error free
system. So then you have to look at the
business case around what the mitigation is going to be and is
that is it serious enough that you have to have a backroom
staff overnight like banks fixing things?
Do you solve it economically by compensating your users in some
way? Are you Google where your profit
margins are so great that you know, what if your search
results aren't that good anymore, it doesn't really
matter, right? So yes, the never, I mean that,
you know, the seven network fallacies are 100% true, but you
don't deal with them by trying to make the network infallible.
You deal with them by accepting that system overall, even if
it's a monolith, has a baseline error rate.
And that's a business requirements issue.
And sometimes it's just, I mean, for a lot of systems, it just
doesn't matter. It's just hit reload in the
browser and keep going. Especially a lot of back office
enterprise systems, that's just the way they work.
It's a little bit different for some types of consumer apps
where a baseline error rate might mean that you get hundreds
of support requests a day, right?
So that's going to suck. So here's, I mean, here's the
interesting thing that the modular component based approach
can do for you. If you're finding that this a
micro service A and micro service B that are really
codependent and you're getting an error rate due to network
issues that's too high. Well, combine them into one
deployment artifact, right? Turn those network polls into
back into function polls. Now you're, they're still
modular because the the interface is still just the
messages, but you may find that you're beautifully designed,
partitioned, are domain driven architecture has 10% of cases
that need to be optimized. And when you have to optimize
code, the code just looks bad, right?
So you stick them together and deploy them as one artifact.
So that's the networking side of things.
The data side of things is interesting.
So first of all, it's kind of similar in that if you actually
look at the details of transactions in traditional
database systems, if you actually look at what they
guarantees they offer, they're not actually that good.
If you look at the specific details, like if you look at
Oracle, right, there's actually, I think 4, there's four
different transaction levels. You might see old data and you
might end up having to use them because you need the throughput
of your system or it might not matter.
So designing a system where you need to do distributed
transactions, yeah, that kind of sucks.
It's a void if possible. Unfortunately, it may not be
possible, especially if it's a large enterprise system,
multiple teams. But hey, you know what, even in
those scenarios, there's usually like a lot of different models.
So you still have the same problem, right?
So then you're in enterprise service buses and reversing
things and all that sort of stuff.
You just use SAPSAP Solved this in the 1980s.
No seriously. Like if you look at R3, if you
look at the SAP system, it's like the perfect micro service
deployment system. They have solved all this stuff.
It's just you need like $2000 a day consultants to actually make
it work. So especially if you're not
building mega scale systems, try very hard to keep transactions
contained inside one service, one unit of deployment.
And if you've got this modular architecture, then again, you
can stick things together. The other thing which I advocate
in the book is don't just go to transactions automatically.
There's loads of other approaches you can use.
I mean, even sort of brute force things like having a batch
process that will correct data that's gone slightly wrong.
Reservations. So reservations are really cool.
That's kind of a standard algorithm to prevent like double
spending. I mean, I won't go into the
specifics of it, but you're basically creating database
entries that basically block other things from happening.
But it doesn't need, it just needs atomic rights.
You don't need transactions. I'm a little bit naughty when it
comes to the database. I have to say I don't mind micro
services sharing databases or tables.
I don't know where that idea came from, this kind of
obsession with purity of the database per micro service.
I guess if I may provide my understanding, right, it's
because of the leaky internal data model that if you use two
different services sharing the same database, right, the other
service could just say, oh, I see this table, I see this data,
I'll just query it and, you know, think of it as if like,
it's my data, right? But actually, yeah, yeah, yeah,
yes. Yes, yeah.
So the data becomes a secondary interface which is now needs to
be managed. I mean, I don't see how
monoliths are immune to that problem either, right?
OK, so that kind of leads me on to the way that we use this idea
to represent data. So I don't know if you were
aware of another argument going on in parallel between people
who dislike ORMS and people who speak ORMS.
You should have ORMS, and I've used both.
Obviously my original C code was directly talking to my SQL and
I've used Hibernate, sharded Hibernate for Java, which is
wow, wow, how are we going? OK, so here's the thing.
I think one of the big learnings from the no SQL movement was
that joins are overused. And almost always 90% of cases
you really don't need to join tables.
Now that does introduce certain inefficiencies for sure.
But the nice thing about that is it keeps your business logic
relatively simple because you have very strong ideas of
specific entities, like there's a user and there's a shopping
cart and there's a line item. And if you denormalize, again,
you have to add in maybe batch processes to correct things and
reservations to keep everything consistent.
But mostly if you denormalize and you don't use joins, mostly
you're OK for most enterprise. And certainly if you're doing a
consumer application for scale, you're going to need to
denormalize anyway, leave the joins to the reporting engines
and things like that, right? So I mean we preferentially use
things like Dynamo DB these days.
You can't really to join some of that, right?
Yeah. So one of the underlying things
I find that where people are not enthusiastic about microservices
is because they also tend to have to give up the idea that
you can join tables and write custom SQL inside your
application. Microservice component based
approaches tend to push you in that direction.
That's a big thing to give up. But I think it also goes back to
the idea of being inspired by Unix pipes, because simpler that
you can make the shared interfaces, the less trouble
they give you. So even if you have shared
tables, shared schemas, if people aren't doing joins mostly
OK. And then if you only expose the
data via messages anyway, if the schema drift or something like
that, then you can add interception messages that will
temporarily adjust until you can figured things out, right.
So in the book, I give some examples of migration strategies
where you deploy multiple services or translator services
to handle those types of scenarios.
You know, they pay us good money because this stuff ain't easy.
So I don't think there's any easy answers to that.
I, I just think it's not as it's all about trade-offs, right.
So my personal trade off is I don't really use joins apart
from reporting. I'm still here.
I live, I'm alive. I think the point where you make
you use less joints. So I think in your book you
mentioned about data duplication, right?
So maybe think of like instead of normalizing and doing joints,
right, you can also duplicate data.
Yes, it may still a little bit, but also like thinking in terms
of eventual consistency, you won't get like a strong
consistency anyway, right? So I think that's really, really
interesting view as well. So I think we can't cover any.
Everything interesting in your book?
You can tell I could keep going another three hours.
So definitely for people who are interested in this new way of
thinking, you know, the tower of micro services, I do recommend
checking out Richard's book. I think it really provides you
with a unique different views of how to tackle micro service
problem. So Richard, as part of my last
question, I would love to ask you this question I called the
three technical leadership wisdom.
So if you can think of it just like an advice that you want to
leave to ask the listeners here to learn from you, what would
that be? You have to let the developers
that work for you make their own mistakes and you have to create
a culture of making mistakes blame free.
So that means sometimes as a senior or the architect allowing
code in the system that is maybe not the quality that you would
write yourself. And I think a great deal of
stress, especially for new team leaders and new architects,
comes from this tension where it's literally impossible to get
everybody to write code to the I mean, maybe you think you're
good, I don't know, right? But it's whatever your
perception is. You're never going to get other
developers to write code the way you want.
And I mean, that's why components are so important,
because if if something is written one way inside a
component is less, the blast radius is smaller.
People talk a lot about empathy and they use the word empathy a
lot, But I think in technical leadership, a lot of that
empathy is showing leadership by accepting a diversity of
technical perspectives and allowing people to maybe make a
mess. I'd much rather if the developer
came to me and you hear stories of people who've like destroyed,
deleted the production database, you've told like, I would hire
that person without thinking. I wouldn't even look, I wouldn't
even think about it. You know, I don't even want to
see your code, right? You deleted production database.
I'm going to give you a job. That experience is something
that you can't Rep. I mean, I've deleted live
websites. You can't replicate that level
of experience. But for an organization to
appreciate it, you've got to have empathy, right, Instead of
looking for that kind of blame culture.
So yeah, go easy on the PRS. That would sum it up.
Doesn't really matter. Anything else or that's all.
I think we can also wrap up with that.
I think that's the most important thing, right?
So thank you so much for the wisdom.
I think it's really powerful and beautiful, right?
I like the empathy side. Obviously, as many architect or
tech lead, sometimes we are very strongly opinionated about
certain stuff, be it architecture, design or even
coding style, right? So sometimes it all doesn't
really matter as long as you know the right outcome that you
want to achieve. So Richard, if people want to
follow you or they want to chat with you about all these
different perspectives about microservices, is there a place
where they can find you online? I am RJRODGER, still on X
Twitter. I'm started using Mastodon,
things like that. Same username.
But yeah, Twitter's still where it's at by the looks of it, for
the moment for tech. And yeah, the microservice is
British places like that. Right.
Henry, thank you so much. It's been super fun.
Yep, thank you so much for this time as well.
So I hope people enjoy our discussion about Microsoft today
all. Righty.
Thank you. Take care.
Bye bye.