Hey, a quick message for those of you who are listening to this
episode on Spotify. I have a small favor to ask.
Spotify now allows mobile users to rate podcasts.
I would really appreciate it if you can take a quick pause to go
to the Technically Journal podcast page and leave your
favorite show your best rating on Spotify.
It will help me a lot to get this podcast to reach more
people on the platform. Thanks a lot.
So let's start. To your book.
So you have had a lot of experience in the industry.
You have written thesis work in large financial company.
Why specifically? Writing an API design book
because API has been around for many years, especially when you
said about SOA, right SOA. And there are so many books
already published as well. So is there anything different
that you try to convey or try to teach us from your book?
Yeah, definitely. Thanks.
So the story behind the book is a little bit longer.
I need to go back some years. All I have to my money was the
first author and the team lead of this pattern approach.
I knew him for quite some years. I said OK I want to have this
idea. We are building all these
services we have and then back in the day microservices were
really the hype rebuilding all these and FI need to teach those
two. And I have corporations.
We only need to build API. So I want something to teach.
I at the time was new at a very large scale project where they
are connecting land, registries, banks, notaries and some other
parties across whole Switzerland.
So it's a system where you have like 1000 different
organizations connected and automating all these business
processes. As you can imagine there are
many services, many APIs involved for doing that.
A bank places a request, the platform generates some
documents, send them back to the bank so that they can digitally
sign this. It goes forward to the notary
and then to the lender Just ran whatever you can do there.
There were some people very new to service design and we had
many discussions about things which I would have thought are
basic, but obviously they weren't.
So my motivation was more for my daily work life to just say,
here's a pointer, please read this there and then we can have
this discussion again. Yeah.
So that was essentially how it started.
And then other people joined like Merkel and Sizar and Uber,
I can't recall now exactly in which order, but quickly became
this team of five people. Uber and Sizar both were very
experienced as patent authors. Already Olaf and I and Mirko as
well had some practical, more practical experience, although
Olaf and was a professor and Merkel actually now as well.
I'm still in industry. I think that's the motivation.
And then the question is, how do you teach these things?
So I mean, design, architecture, always about decisions and
decisions have these nasty aspects of that.
You get something, but you lose something.
And if you go through the right door, you can't go through the
left door. That sounds easy, but people
always assume that there's this perfect solution.
You just do something and then it's perfect.
And you do this every time because it works all the time.
The silver bullet thing. And we all know it doesn't work.
I think even if you're fresh from university, at least after
three years, it doesn't work that way.
And patterns make that explicit. I think that's so great because
firstly you have a name and I think we'll talk about names as
well. But you have the section bound
forces. So what are the influences on
your design? And some things are better if
you use a certain pattern and some things are worse, so you
pay for your decision. And patterns make this explicit,
and that's why it's so great as both a teaching tool and also
reference tool. So even nowadays if I'm in
projects, it's just nice to go through the pattern list and
check whether there's something I should have considered or not.
Because it's always busy when you're doing things, it's nice
to have checklist. I always love reading pattern
books because it is practical. That's the first thing, right?
And also it kind of like lists down the decisions or the
trade-offs some people say on why you choose a certain
pattern. So for example if you opt for
scalability, then maybe this pattern might be better for
example microservices, right? But obviously we know that
microservices is not silver bullet.
In some cases it might not work as well.
So Fred Brooks have this write up last time, no silver bullet.
So I guess it has been mentioned many times in any kind of
architecture kind of discussion I had in this Technician
episode. Your subtitle of the book also
mentions specifically about Loosely coupled.
Message Exchanges. So tell us more why you?
Choose specifically loosely. Coupled message exchanges as
your subtitle. Well, actually the subtitle was
one of the last things we decided.
And if we look at the book, what's special about this?
It closes the gap between high level books like Enterprise
Integration Patterns, which are not high level or very abstract,
but they're certainly on a high level of abstraction.
And then you have these books about technology.
So how do I use REST or GRPC or graph, QR or whatever, and what
verbs to I use and which circumstances, how do I build
good UIS and all these things. And in between there is
something. So you have a conceptual API
design, something where you decide, OK, I need these kinds
of operations, I need to have these quality attributes and
because of this I perhaps to pagination.
I chunk things and not transfer a whole message and that's
independent of the integration technology of the protocols you
use. Back in my first project was a
banking system or a banking platform and they used CORBA all
the way. It was all CORBA already there.
There were many patterns which we still use today and more REST
http://apis because these are really long living and even now
whatever comes up next, you will have these problems and you will
have these patterns hopefully addressing those problems.
So it's not tied to any specific technology.
First of all, so you can make your API design on a conceptual
level using these patterns and then shoots your technology.
So that's one aspect. The patterns themselves are not
coupled to any technology themselves, but obviously it's
probably more about the coupling between API clients and API
providers. I think a good API and that's
why nowadays finally we all agree that API first design is
the way to go is you don't want to expose your internal data
models or your internal logic too much on your APIs.
And the more your clients are not under your control, the less
you want to do that. I think most patents here will
allow you to deal with this problem.
So how do I build APIs which well if I do them by the pattern
book so to say, although it's not your goal shouldn't be to
use as many patterns, but if you just follow these forces and
things then you will hopefully have a design which allows you
to have these loosely coupled exchanges which are not
dependent on your internal models.
So you mentioned something. API first design.
I think this is mentioned many times.
In fact, I think I have one episode James Higginbottom also
talking about that and in the book you also.
Mentioned a couple more. Good practices for API design.
So I think these days almost all back end developers will create
API especially REST API or maybe GRPC or Graph QL.
Are there any other things that you would advocate for all these
engineers? What kind of good practices they
should do or they should consider when building API?
Well, first of all you should think about how to express that
if you start with any software development, it's just not only
APIs, but it's in general. You need to understand your
domain thing, that's important. The problem with APIs is let's
go back to this project. You have a team of people who if
you want to build a good solution or a very good
solution, you need to understand what banks are doing, what
notaries are doing, what land registries are doing.
And the interesting part is in the last couple of 100 years, I
mean this domain is very old. Houses and government management
of houses have a long time. So land registries are around
like forever and banks are very old concept and no trees I think
as well. And over these couple of 100
years, these three parties still haven't learned what they are
doing. So that was very interesting.
A bank doesn't know data and model of land register, which is
very interesting and surprising. So if you want to build APIs for
domain or for clients where you don't know exactly their
requirements, it's the first thing where you need to reach
out. To clarify this and it's easier
said than done. There are a couple of
approaches. So most of all you're doing
workshops, building your first API version, release it, try to
evolve it, but then you need to deal with life cycle issues.
So one part of this loosely coupled is also you need to
decouple the life cycle of your client and your provider,
because you won't deploy new versions simultaneously, so you
will have points in time, many points in time where they're
different versions. So client normally supports an
older version and the provider rolls out a new API version
which should be backward compatible so client doesn't
break. So that's kind of the ideal
scenario. Well, many people say OK, just
build backwards compatible APIs, yeah, but it's not always
possible. Sometimes you have breaking
changes. If you say OK, I don't know my
clients well enough, I want to iterate and I want to improve
things. It's more likely that you will
have breaking changes. For instance, banks usually had
one to many relationships where the Land Registry internally
manages many to many relationships.
These kinds of things if you know them later on.
So you have your API defined with one to many and now you
need to support many to many. It's likely going to be a
breaking change. Also in your environment there
might be breaking changes. So in Europe we have the Euro as
a currency, which before were multiple national currencies and
if you had online banking bank then to money transfer, there's
no way of doing that now as it was a breaking change that
there's a euro and not all these local currencies.
The same goes for account numbers only have been local
national account numbers and now we have the Ibens International
Banking account number. So there will be changes in your
environment which will break where you can't do anything.
And there will be changes where you learn your domain better and
where you can say, OK, in theory I should have known this
already, but you didn't. The more you iterate to get to
your requirements, the more you need to think about your life
cycle. Some patterns we have there is,
for example we have an experimental preview pattern.
So you say, OK, let's say we roll this out and currently I
don't give you any guarantees, you can check this out, you can
give us feedback, but it will break without us telling you so
That's not nice for the client, but the client also has some
benefits because he gets a preview, so a better version
where they can use the API, preview it, use it, give
feedback, and perhaps hopefully getting a better API down the
road due to this feedback, even if it has breaking changes.
And obviously some organizations or some developers will say, OK,
no, under these circumstances, having only an experimental
preview, I don't integrate, I don't use this API, which is
fair because it's known. What's unfair is if you say, OK,
here's an API and then you just say don't support it anymore, I
break it in some way or another. So being clear about your
evolution process, I think that's important.
So have your requirements, know how good you know them and then
plan ahead because in the beginning you can.
So if you just had guarantee like we support this for three
years and then you just try to alter this later on, people just
come to your office and hit you. Probably These are decisions
which you need to do as early as possible, and then obviously you
need to flesh out all the domain knowledge in your API payloads.
So yeah, it's always wise to start with the domain
understanding first, and in fact things like DDD or Domain Driven
Design is also something that advocates you to actually
understand the problem. 1st. Rather than coming up with just
the technology or the solutions, people these days whenever they
build API, the first question that they have in mind is
definitely which technology like HTTP, Ras, GRPC, Graph, QL and
things like. That, but actually.
Specifically in your book you also covered the other important
parts, which is the payload itself.
You mentioned the messages. Maybe for us engineers here, can
you give an overview what aspects that we should consider
from the messages point of view, the payload point of view, in
order for us to design our API better?
Well yeah, I think the most important part is how many
requests do you have and how large your message payload is.
So if we look at that, obviously you could make an API operation
which might be an HTTP resource where you say get me all
customers and then you get back all customers and everyone's
happy. You can deal with the data
whenever you like. This approach obviously has some
drawbacks, So much load on the API provider because it needs to
fetch all these customers. You have much load on the
network because you need to transmit all these customers.
So we don't do this design. What we usually do is get a
certain customer or search for customers, but it's that
everything we need to consider, No, usually not.
So if we look at the search, even a search might return too
many records. That depends a little bit on
your domain and how many objects you have, what domain objects
you have. So if we have a service which
would return list of currencies, we know how many there are and
it's quite manageable. If we are fetching customers
from any large organization at the Microsoft on the whole
Salesforce database, whatever you like, that's too much.
So we need to have some mechanisms if we are running
into this problem to limit our response sizes.
So you can either say OK I only returned the 1st 200 hits I have
might be a good strategy. We could have pagination.
So I give you the first hundred and if it isn't sufficient you
can query the next hundred. We could also say, OK, perhaps
you need not all customers, but query all sessions in the
conference. And that's a very huge
conference, like 600 sessions. To each session you have some
information like the title. OK, just a string who's giving
the presentation? That's just a string, the image
of the presenter. So that's obviously some large
payload part. When I put this in my message,
it depends on the use case, but most likely not.
I would just put a link to this image and whenever the client
probably pops into the detail view then it can fetch this data
as required. So we have this decision on
whether we need to include data or whether we reference data.
Depending on your use case, you often have things which you
don't need. Coming back to the Swiss land
registries, you have hundreds of years of historical data and for
most use cases you don't need historical data.
So it makes sense in one way or another.
And there are multiple options. Address what data the client
needs and you could do many operations.
You could do some parameters which there's a pattern called
wish list. So I say I want this historic
data or not. So it's just a boolean flag and
if you set it you just get back 20 times the data what have If
you have no historical data and if it gets more complex then you
might have not only wish list but a wish template where you
can define in your whole data structure and all the nested
things what data you want to have or not.
Which then closely or easily gets you to graph QL.
And all-purpose is just to specify a request with exactly
the data you need. And so there are many things you
can do to adjust or trade off the number of requests and
request size on the response size especially.
And I think that's the point where we need to make more
conscious decisions. On average, I think most people
just dump out everything which they have and then at a certain
point they just see OK, well it doesn't scale, the database load
gets to have me and things are getting so slow.
I hope you enjoyed this short clip from Tech Lead Journal
Favorite Playlist. If you find this episode useful,
please help share it with your friends and colleagues who you
think would also benefit from listening to this episode.
And if you want to listen more from this conversation, please
go back and listen to the original full conversation with
my guest. Stay tuned for the next Tech
Lead Journal episode, and until then, goodbye.