Software engineering involves a lot of decisions and that
decision has some trade-offs. So we have bars and cards and so
on. It's not like one decision is
always better than the other. Sometimes you are not aware
about trade off at the decision time, but later after a year or
two, this is costly. So maybe this cost could be
validated. Hey everyone.
My name is Henry Surya Barragan. And you're listening to the
tekhelet Juno, the show will be bringing you the greatest
technical leaders practitioners and thought leaders in the
industry to discuss about their Journey ideas and practices that
we all can learn and apply to build a highly performing
technical team and to make an impact in your personal work.
So let's dive into our Journal. Hello, everyone.
Welcome. Back to another episode of the
technology. Not podcast with me, Ojos Henry
Surya with Robin. Thank you for tuning in and
spending your time with me today, listening to this
episode. If you're new to this podcast,
please follow technology, you know on your podcast app and
social media on LinkedIn. Twitter.
And Instagram also consider supporting the show by
subscribing as a patron at technology.
No, dot f /, Patron, and support me to continue producing, great
content every week. Speaking about software
mistakes. I'm pretty sure that many of us
have made such mistakes in our software projects in whatever
shapes or forms, or how about having to decide on important
trade-offs to incorporate in your software design.
And if you look back to those experienced, some of you who got
them, right? Will be patting yourself on the
back for being able to choose the right solutions for your
software problems. However, I also believe that a
lot of us could point back in time and say, yeah, I think I
got that one. Wrong.
And what a big mistake that was personally.
I've made a number of those costly mistakes and how I wish I
could use some guidance, that can teach me how to avoid
typical and common software mistakes and trade-offs.
Fortunately, Our Guest for today's episode has written a
book just on that particular topic.
Thomas leg is the author of software mistakes and
trade-offs, how to make good programming decisions.
In this episode, Thomas shared what?
Let him to write this book and also shared one of his past
software mistakes taken from his career experience.
He also gave advice on how we as software developers should
approach the potential software mistakes.
We could make and explain some trade-offs that we typically
face. When making software,
engineering, design decisions these days such as code,
duplication forces flexibility, premature optimization versus
optimizing hot path data locality and memory and finally
different. Semantics and there are
trade-offs in building distributed systems.
There is so much to learn from Thomas about software,
trade-offs, in this episode. And I hope you learn a lot from
this episode as well. And if you do help the show, by
giving it a rating and review on your podcast app or share some
comments on the social media channels.
It may sound trivial, but those reviews and comments are one of
the best ways to get this podcast to reach more listeners,
and hopefully, they can also benefit from All the contents in
this podcast. So let's start our episode
right. After our short sponsor message.
Are you looking for a new cool swag?
Tackle it Journal. Now offers you some swags that
you can purchase online. These wax are printed on demand
based on your preference and will be delivered safely to you
all over the world where shipping is available.
Check out all the cool strikes available by visiting
technology, you know, dot, f / shop and don't forget to break
yourself. Once you receive any of those X.
Hello everyone, welcome back to another episode of the package,
you know, podcast today I have with me someone named Thomas
lelik. He's actually the author of
soon-to-be published book by many called software mistakes
and trade-offs. So when we talk about writing
software, most of the times actually, yes, we do care about
correctness. We do care about performance and
things like that. But sometimes we also need to be
aware of what kind of possible mistakes or trade-offs that we
need to think about when. When we write software so that
it performs based on the context that we want to build it from
and also the business requirements.
And also in terms of maybe functional correctness and also
performance. So today, I think Thomas will
discuss a lot about this and hopefully, we can learn from him
about all the mistakes that maybe have seen throughout his
career or we some of the Epic stories Thomas.
Currently works at data stacks for company that built databases
such as the Sandra and also a few other things.
Yeah, Thomas really happy to have you in the show and looking
forward for This conversation. All right, welcome.
Oh, so Thomas maybe in the beginning for you to introduce
yourself, maybe telling us more about your career Journey or
maybe any highlights or turning points?
Yeah. Sure.
So I've started in the ships that company.
It was like on the Scandinavian Market at all Scandinavian
people were reading for content stores, like newspaper and also
websites providing news, but it was only to millions of people
and users. So that scale was not so big.
So, Really? I promise to Allegro the send
eCommerce website here in Poland.
The biggest one, we have 20 million of active users.
And there was a lot of interesting problems that are
big data problems. And we were collecting users,
traffic and saving dots for the future analytics, much learning.
And so on the data was like eight years or so, so there was
a petabytes of data. So was involved in streaming
Solutions when using Kafka also microservices architecture that
will Amount of services a picture with like hundreds of
Miles. Davis is living there was 700 of
them and each of like a service was deployed to multiple nodes
that was auto-scaling. Also rolling deployment, of
those batteries that are well known in this ecosystem.
And also be greatest solution, space Millennials park because
it's Advanced and has some advantages over how to bunt and
those old Solutions after almost four years.
I've proposed to data stocks because here we have a lot
bigger scale as you mentioned. We are providing ecosystem
around Cassandra and some tools around it.
Like for example, I know Stargate API that provides a
variety of apis to our customers that could speak to Cassandra
but not deep directing VAR C ql but also for example grdc
graphql and so on. Also, I was involved in a cop
car collector. So it was based on connect three
more cred by confluent. So we provided a way to insert
data to Cassandra and Link instruction that price from
Kafka platform. Was quick out there for sure as
well. So we needed to take part was
consideration. Seriously.
I was involved in some performance test things as well
and so on, and main product was also double driver.
So the main entry point for some data, from jvm ecosystem to the
jvm world also design trade-offs, interesting
decisions that I was involved as a part of the team.
So that's a summary of my journey besides that.
I am also a technical trainer here in Poland.
So that's also an interesting. Audience because I can teach
folks from different companies, but also I can learn from them
from their questions by doing. So I'm better at the world.
I am teaching them or something Kafka or caching Solutions
performance. That's and so on.
Thanks for sharing your journey. It seems like throughout your
career. You have to work a lot with kind
of big data problem, distributed systems microservices and also
like optimizing to the low-level you mention about building like
connectors drivers and things like that.
So obviously you have seen Of different types of software.
Most they are for high performance characteristics.
But in the first place, why are you interested in writing a book
about software mistakes and trade-offs?
This is quite interesting because I don't see much of such
book except yet some mental patterns.
Maybe there is, but in the first place software mistakes and
trade-offs. Why do you want to publish this
kind of book? Yeah.
So from the beginning of my career, I've started this thing
that software engineering involves a lot of decisions, and
the decision has some trade-offs.
So we have pros and cons and so on.
It's not like, one decision is always better than the other.
But of course, you can have more than two options as
possibilities. When I was involved in this
variety of systems, in some business, strictly e-commerce
systems, be data, streaming processing us on the team
tackle. A lot of those decisions and
trade-offs are different levels. So the solutions are the
low-level, like, code patterns. That's, every supplier engine is
to make them almost every day, but also more architecture.
Just that you can like discuss with your team which screened or
h 2 weeks or even more high-level positions that will
influence your evolution of your software.
Its maintenance flexibility also performance during those
breaking points in my career. When we choose one solution over
another I've started to writing a personal list of those
decisions and how they end up. Like, from the time perspective.
For example, how this decision impacted our software after one
year after, Three years and so on because all psychic pleasure
to remove those systems, not only during the development, but
also made the nurse and when the system was deployed to
production and I so actual usage of it.
After that. I've created this list of
decisions. It was like, 15, hold them.
That were really important. And after some time I saw that,
those words are quite generic, so I decided to share it with
for the engineers Architects and so on.
So they will not make Same mistakes and they will be aware
of those trade-offs because sometimes you are not aware
about trade off at the decision time, but later after year or
two, this is costly. So maybe discussed could be a
lady that after we will read one of my chapters of my book.
So it's interesting that you started this from like curating
your personal list of decisions and trade-offs in your career.
Maybe if you can share maybe one or two, the costliest mistakes
that you have. Senior career based on this
list. Welcome.
So first one and I'm touching on that in chapter 10 of my book.
So we're building a streaming solution and we needed some kind
of scheduling Library scheduling framework that will work
efficiently. So we need to count like ten
thousands of requests per second.
At least our system has distributed was based on Hazel
cast work. Then we picked one solution one
Library without realizing all its trade-offs.
We hold properly analyzing its source calls because on the one
hand, and this library was documenting that it should work
properly in distributed system, but in practice, in reality, it
turns out that this problematic, if you have multiple nodes,
there were some problems of synchronization, the domain was
totally different and the library was not well prepared
for that. So, this was caused to be saved
because we needed to debug those problems.
And then after some time, turns out that this is a lot easier to
just implement the me. Even product.
That provides those functionalities for arms,
instead of using food party software, that we didn't fully
understand and we didn't have enough influence on fixing that.
So I think that was one of the mistakes that could be judged.
That would be generalized to a fact that we should know the
trading model of a libraries that we are importing.
So for example, even if you have some long block right now, there
is a chance to influence microservices that works using
non-blocking Solutions can gladly Node.js and so on.
Sometimes if you are using libraries that are not well
suited for that reading model. You may block you on, mine dress
processing. It made turns out that it Blocks
Your Mind flow and impact your performance.
So that's should be found at the design stage.
And also, we find a way around if you have quite simple flow,
that works in synchronous way using an awesome College
library. Also, you should be aware about
problems that may arise because you need to be our about
ordering about multitrack. Being feisty.
When you are introducing most challenging to your code.
It means that you need to be aware of that, and it means that
we will have additional problems or maintenance costs.
Maybe sometimes there may be hidden, like, for example, in
chapter 5 of this example of java streams API and using
parallel stream of Destruction. So it hides the internal for
between pool that is doing processing in the multitrader
dry. So, at the first glance at the
code, complexity level, there is no change at all.
You are just changing strings. Method, invocation, two,
parallel streams, but underneath you on creating goal of Triads
and your compass multi-threaded. Now attaching, I'm not into
chapters of my book basically, so, as a software developer, I
know that a lot of times as a developer's, we tend to work
based on just business requirements and you know, just
ensuring that the functional correctness, the input produces
the correct output. So in your opinion, right, what
should be the attitude of developers or software engineers
in terms of looking at? Bible Software mistakes and
trade-offs. Is it something that they have
to be aware since the beginning before they even write the
software or is it like during the implementation, or like, how
should you advice for software developers to relook at all
these potential software mistakes and trade-offs.
Yes. I think there are two different
approaches to that. So, first one is when you are
peaking at some framework library, that will shape your
code base and influence your code base alone.
So, for example, in the VM Ward, if you are picking spring
framework and well if you guys are called base and will be hard
to change it to something else in the future.
Also, for example, if you are picking a queue solution and you
are picking Kafka, we need to be aware about the stray dogs.
It is favoring availability over consistency, depending on the
settings of acknowledgements with a distributed system that
you are integrating with, the, you need to ask the same
questions like how to influence, consistency, availability.
And so on. So for those situations, I think
it's better to fry those up front.
You can see possible to know them upfront by doing good
research and prototyping. On the other hand.
There are those low-level decisions, like refactoring of
the cloud and picking a specific pattern or design in this
situation. Maybe it's part of those to
experiment with the code and comparing like to approach
sighs. So maybe it's not so cozy to it.
For example, Implement a solution using inheritance, but
also implant. Another solution is
incomprehensible. This is going to be one imploded
by two engineers and then you can compare your Solutions at
pick, the better one. So basically in the second
scenario, the cost of experimentation is slower.
So maybe you can postpone the decision about which one would
you choose and in the first one is the cost is higher because
once you have this in your applications alucard's to
migrate some of its better to read about this educated about
these and all those trade-offs up front.
So maybe let's go into some of the software.
Where mistakes and trade-offs that you covered in the book,
let's start with the first one which is code duplication versus
flexibility. Maybe tell us more about these
kind of trade-offs. Like what do you mean by
duplication forces flexibility. So I'm considering this problem
to wise. First one is at the
architectural level. I would focus on that firstly in
the microservices ward or multiple Services.
Each service is responsible for some specific Business Online.
Ideally and contains some Self-contained functionalities.
So if you have for example to microservices, it is often the
case that they may share a similar code in some way like
for example autorización set up being the token and submitting
it or maybe some kind of validation or things like that.
So if you have this duplication between Services the first idea,
maybe to just remember that will unify it using.
For example, library and use it in both Michael statuses, but it
also has a trade-offs. So for example, if you have the
common Code. You have the tight coupling
between two microservices and this Library.
So it means that it may impact the evolution of your code base.
Also, the fact that this code was duplicated at the beginning,
doesn't mean that it was the same functionality.
Maybe they needed to evolve in a different way in that situation,
really hard to evolve it because you will have some obstruction
that is not fitting properly. All these cases, you will need
to add additional content to the library development and so on
and suddenly, I've applied coupling and also flexibility of
your software is lower. So you are not able to deliver
your software as quick as possible or the solution would
be to just extract this new functionality to order
microservice. But also it has other problems
like additional Network request response time, additional
latency deployment, but we need to create for this micro service
and so on. So at this architectural level,
sometimes sound code duplication.
It's not blocker. For example, you have two
components that Are doing a similar job and you slowly start
seeing some obstruction in those two components and you are, for
example, decide to use inheritance to reduce.
The duplication of components solemnly are dependent on this.
New component, got an inheritance and the same
situation can happen here as it meant result.
That's obstruction, doesn't fit the evolution of those two
components. Maybe you decided to extract it
too soon. And I'll be stages, may be
really hard to get it back to the previous solution and of all
those components. The palanquin, similarly, you
will introduce tight coupling and impact, the speed of
delivery, flexibility of your code.
I think this is quite a common case, right?
Whereas the software Engineers. We are also kind of like hot.
So you have to remove duplication.
Don't repeat yourself dry principle.
Also, like, when you see things over the internet, you have so
many open source project their basic creating libraries
Frameworks and reusable components and things like that.
So we are kind of like hot in a way that okay.
Location might not be a good idea and also reuse abilities or
important. So in the first place when we
are tackling a particular problem, what is your advice for
software engineer to actually be look at this perspective.
Should you start with just a simple implementation first and
over the time, you'll see some kind of abstraction that could
be built around that or you should start seeing the
obstruction in the beginning based on the particular context
that you know, like, what's your approach here?
That I think the same solution is the second one.
I don't. Why not starting from the
abstraction but why some time until this will settle out.
So if you have a couple of components and they were leaving
for some time, so we are quite stable state.
So it's not a frequent ablution place.
If you see some extraction, then it's a good time to extract it
at the beginning. You may be doing that too soon
and it may be hard to unwind that, but there's also this
argument that if you don't start earlier, where you You can reuse
some of these stuffs. It can also be costly in the
future where you have to tweet your existing software refactor,
it in order to make it more reasonable generic, and things
like that. So any fog around this, there is
no simple answers here. Also, if you will go to the
extreme, your code will not be good.
But yeah, I think it also comes to a fact that you need to have
a good test coverage of your components.
So this is not an excuse to not create good quality code, also
need to have one that stood component if Our what the status
is, of course, easier to refactor, but there are some
functionalities that you can be sure that they will be reused.
For example, some simple string manipulation, like what you have
in standard as the case, some connection to Excel on systems
and so on Coop, so let's move on to the next trade-offs, which is
what you call premature optimization versus optimizing
hot path. May be the first term.
Almost all Engineers will be familiar.
With premature optimization is the root of all evil.
They all say, The second term here.
Optimizing Hot Pot. What do you mean by that?
So in chapter 5, I will propose in your cream work.
If you have code, that you don't want to prevent to optimize how
to make it an early optimization or just in time optimization how
to have enough data to optimize it.
So we need to have two things at all.
Lakewood decision, which code values to optimize.
So the first one is number of requests per s.
Also expected to both on your path so assume that we have two
endpoints. Let's say 1 and point is only
one request per second. And second one is only done
because per second. So having that data, you might
be tempted to just go and optimize.
The centrepointe block is executed 10 times per second,
but it is not enough because we need to have latency data.
So once we have data about latency, for example, you can
get average latency for the first opponent s 1.
Then you can multiply a number of requests and average latency
and have a distribution of overall Legend.
She overdosed to and phones having the data.
You can detect the power of that is executed.
Most open. I'm calling it hot bath.
It means that it is executed almost for every user requests
in a system. So that was working with, it was
very often, the case that they were following this Peridot
principal distribution. So for example, like 80% of
value was delivered by 20% of the code or it was ten ninety.
Thirty seventy and so on. And it means basically, that's
it. Focus.
Your optimization efforts on the smaller part of the code you
will gain a lot of benefits. So I'm proposing, there is how
to detect this hot tough. So the smaller part of code that
optimizing will result in future benefits and how to make this
decision. So, we are basic, you need to
have this requests per second smash of latency.
When we have the data you can detect hot puff and dig deeper
in your code. So you can go to the specific
parts of this hot bath and measure each of those.
Those parts and then you can focus on optimization efforts on
specific thing. It is basically a way to limit
the scope of your work because in production systems, it is not
possible to optimize every possible path.
If you detect the whole class, we will can make a rational
decision. It is a place where you can do
some additional efforts and they will readily be worthy.
After optimizing. The whole path is going to
result that you will reduce latency so much that it is not
Responsible for the most of the overall utilization on your
system, but it may also turns out that still optimizing this
hot path will give you better benefits than optimizing other
groups in your course evenly, for example this first endpoint
that was one request per second average latency is for example,
500 milliseconds and the hot pipe is even 50 or 10.
If you multiply that with this number of requests, we have the
full context and make turns out that series 1.
Optimizing. Maybe a little bit of recap.
So yeah, you should know the true put of your system requests
per second and also the latency of each of that request.
And then you multiply it, get a sense of distribution.
Right? What's your P95, maybe your
average your maximum and things like that, maybe a little bit of
tips here, because sometimes Engineers when they write
software, I don't think they have this in mind in the first
place, what kind of tools or what kind of techniques that
they should introduce to their software in order to get all
these numbers easily. If you have system have defined
as alive, so if there is a, some kind of a nice light should stay
at or system should be able to accept some requests per second
within specified like like and see if you don't have this data,
then you'll need to measure it somehow to get latency.
You should write benchmarks that will validate your application.
So you can, for example, use got linked to WR K tool.
There's a lot of stomach acid also has its own testing, three
morgue. You can, of course, use it to
extract the latency, but there are a number of requests per
second in it to help from us alike or from some product
information. Maybe what kind of traffic are
you expected, Albert, traffic will grow over time.
So then you can adopt your test and measure again.
So sometimes also I think these days people, classify these
things as observability, right? So having good monitoring or
Matrix, as part of your software, maybe instrument your
code base to that you can get this data easily.
Maybe even a profiler. Some tools these days.
They can embed instrumentation code inside your code, so that
you can get this providing statistics easily.
So, thanks for highlighting this importance, about optimizing,
Hot Pot. Let's move on to the next one,
which is what you mention about data, locality and memory.
Maybe can share a little bit more about this.
That's also in the Big Data Systems, but also all the data
intensive systems. There is an important
optimization that you should make.
And all those recent remarks are doing those.
Optimizations. But also the impact the way how
the system's Big Data remorse on spark.
How do a song are built and some complexities in them.
If someone is new to a big data, ecosystem may think that those
complexities are unnecessary or easily hard to enter point is
high. But they do quality means that
you are moving your computations to data.
And not they got two computations.
What does it mean? So, in Luxor example, if you
have your data saved, almost People notes like a dog knows,
five sister knows even database nodes.
If you have some big data processing or processing
ecology, like for example find the average of some value and
your processing is on different nodes.
Then you are not fetching the data from the nodes or perform
operations on the computation node, but you are serializing
the computations and sending them to the data nodes as
sending computations is fast, because they will not take a lot
of data this way. Basically sterilizing your Java
Scala or other language program into binary format, sending it
over the wire to the data nodes on the diagonals.
Also, some kind of an Executor has to be running to the
sterilize, the computation and apply it to the data that is on
the local node. It can be done in a distributed
way. So multiple data nodes.
Each of those is processing part of the data and then there's
another place of coalescing, the result is sometimes display.
Involves sending data because for example, it depends on the
partition or partitioning scheme.
Sometimes you need to send some data, but the amount of data we
usually will be substantially reduced.
If you do it from the beginning, send the data at the beginning.
That's basically as the simplest description of data locality,
but implementation of it is provided within those
distributed systems, and I'm focusing on Apache spark because
it's a recent technology that is available planet.
It's over the old how to based Technologies, mainly because it
does a lot of processing in around not on the disk.
The good example of data locality could be using the join
operation. So if you are joining data sets
and we have one big data set and second maybe smaller ones, for
example, you have a list of accounts and all purchases made
by those accounts. So there will be a lot of
purchases and number buttons, will be smaller, leveraging
data. Locality also would mean that
you are making a sign. Visions which data is sent to
other nodes. In that case.
We would send a smaller data set to the bigger data, set to
reduce this network traffic impact within scope Network
shuffling and that's one of the biggest value.
We need to focus one writing the data processing to optimize -
processing, but with Calculus to generalize to just normal to
occasions, right? If you need to fetch some data,
you need to answer the question. Maybe it's better to tell the
system to compute that and then we only Results.
Also, the second aspect here is partitioning that is important
and picking the proper partitioning scheme for your
processing. So we need to optimize your
partitions to your patterns. For example, if you need to
analyze data pronounce, then it will be nice if one data node
will keep the data for the whole month.
If that's feasible. Then you will send a computation
to this data node and competitions will be local to
the whole long and they could be executed in a local way.
On the other hand, if you have data Partition by, for example,
account ID in that case, and you need to extract data for the
whole month. It would mean that you will need
to scan the data from each data node, and then it involves
Network partitions. So yeah, I'm showing couple of
those examples in this chapter and how to basically help to
feed this partitioning depending on the processing that you plan
to execute. So, thanks for sharing all these
details about how big data distributed system works because
I think like for small kind. Data, normally people, I'm not
really aware of such trade-offs. But yeah, it makes a lot of
difference. If you are processing a lot of
data and where you store the data, especially if your data
cluster is large. So the thing that I pick up from
your explanation, just now, instead of sending the data to
the computation. Why not doing it the other way
around? So you sending the computation,
which is your coat, and you can call it the functional code,
serialize it to Binary and then sent to the data, do some local
processing that and just send the result back or maybe you do
some kind of joy in and Reduce operation right.
In terms of Hadoop Paradigm mapreduce, you basically send
all this map operations, two different notes and then you
reduce them and aggregate them into one possible result.
So look at this data locality of.
Could you also actually apply this to like micro service?
Is that kind of architecture pattern?
Is that something around that or is this just applicable for Big
Data? Yeah.
I think it's going to be also, of course applied to
microservices Microsoft is often leads to such a lot of data from
other services to Eight and and result for customer, or maybe
for another microservice. If you start noticing that a
Microsoft business to such a lot of data from multiple services
and speech services returning, thousands of records.
And then you are just iterating over those records.
Filtering some subset of those circles or even coalescing into
the one result. Maybe that's a situation when
you put send to the second microservice.
Just Computing this body. When the twin this one value.
In that case, you will Reduce the traffic, the computations
will be done on this second microservice.
And only the result will be returned and that case you will
save Network bandwidth. And also processing, CPU time,
and so on. So, yeah, maybe a little bit on
this, right? May be some kind of a Viewpoint
because last time in traditional way, we used to have this
database stored procedure to process database queries, and
sure that you can actually compute on the database node,
but over the time people move on from that strategy then actually
put the application. Coat.
So to speak, like a back-end API, right?
Where you just query the data and you compute on the
application back. And so what's your view on this?
I think the last time we used to do it near to the data, which is
the database itself, maybe throughout the time, because of
scalability, because of may be other considerations.
People are moving away, that kind of computation away from
where the data is residing. Do you have any thoughts around
that? Good stored procedures, hit a
couple of problems, may be testing the stability over time
and also, it could be very vendor-specific. 60, you are
coupling to a specific vendor and other problems so that
implementation of it was not ideal.
But the under Direction, had some classes, those trade-offs.
And so you are keeping the computations on the known as you
mentioned. There is a trend to move,
computations back to the data and all the data promoters are
doing that. But they are doing that on a
little bit higher level to provide some obstruction to note
tied to a specific vendor. For example, when you spark, it
doesn't matter what format They tell you have it can be a brawl
Arquette or I system just plain text files and so on one
Cassandra and database collector that crisis abstraction and all
the logic is in actual programming language.
So it's easier to test easier to maintain.
There are some forms of these approaches.
Of course. Yeah, looking at the distributed
data framework. I must say that.
It's also exciting. So I've used like Apache, beam
data flow as well. Like spark.
I think there are lots of Innovations around That I'm
really glad that some people actually writing this kind of
software, so that as an end user, so to speak software
Engineers. They just need to understand the
Paradigm and write the code appropriately.
Let's move on to the next trade-offs, which is what you
call delivery semantics in distributed systems.
So what's the importance of delivery?
Semantics? Or maybe in the beginning
explain tool some of us, what is delivery semantics?
So system that you are interacting with gives you on
the data. It will be delivered to us, at
least, once it could be at least.
Once at most once or exactly, once basically, if you are
interacting with network call or any distributed system, then
those delivers a man things are important for you.
Even if you have two simplest microservices and one is sending
a request to the second one, there could be a network
partition. For example, when the response
get back to the first microservice, in that case, the
second microscope is accept the data process.
It properly may be saved to the database, but the first service
doesn't know what. Happen.
And in that case, you need to make a decision from the
perspective of the first service.
What to do, we can retry the request.
In that case. You will have a consistent view
of your system if the request succeeded.
But it means that you have submitted more than one request.
So, you may have to duplicate and this will be at least once
you will deliver the message, but you could deliver it more
than once, it gives a good quantities on both systems, but
it also has some problems because the second system leads
to be Prepare for that. So, it means that the could be
duplicates. You can design a system to the
item potent. So it doesn't matter.
If you are replaying in my search.
It will result in the same outcome.
Like, for example, deleting the body for a specific primary key,
maybe item portal, because it doesn't matter.
If you lose it once or 10 times, it will be just deleted.
But there are some operations that are harder to design them
to be. Identical told like updating,
for example, some counter because it will be
Inconsistencies there. So we have at least once in that
situation, if the customer system is not aware of this,
those applications may result in the data.
I'd inconsistent State. And on the other hand on the
producer side. We might decide that in such a
situation. You are not be trying, but it
also has a problem because the fact that response didn't count
to the first service. It can also mean that it wasn't
processed at all. It was a failure in that
situation. The first service will not We
try and it led to a sound that the second one didn't get the
value at all. So it was lost at most once.
So it means that it could be delivered 20 times or one time.
It will be no duplicates but possibility, 40 and delivery, of
course as everyone can be aware that can have problems because
you lost son. Had a gun in transmission.
There are some cases of the disease.
Also, good enough, like, for example, collecting celebrities
that are collected every millisecond or something like
this and if you will Lost some love them.
It may be not problematic. Lastly.
You can have effective exactly once because when the network is
involved, you will need to make decision about retrying at some
point. It'll be done even on the
protocol level ACP layer and so on so ineffective exactly.
Once basically you have this duplication that can use hidden
somewhere or some kind of a transaction built into your
system. It's hard to implement because
some problems complexities also big performance overhead view.
We need to build it and be aware of that.
Also, what's important, if you have to smuggle service
architectures, when there is multiple, Microsoft is
connecting with each other, even if there is effective ones
between two of them, the next one, if does not provide
effective, exactly. Once they will be the same
problems. So your whole pipeline needs to
work in this way. So maybe just to recap again.
There are three delivery semantics.
So like at least once at most once and exactly once.
So from my experience in my career, Most of the systems
especially the distributed systems, they will opt for the
at least once delivery, semantics because maybe it seems
like it's a much cheaper to accomplish.
And also it's less complex in terms of how to guarantee that.
So in your point of view, is it fair to say that for most of the
use case, we should design our system using this at least once
delivery? Semantics.
Yes. Yes.
The other effects correct. We can remove the need for the
duplication by designing our system to be item button with
require. So I'm thinking for example, if
because events based system which requires some thinking but
it's feasible. So, for example, you can send
the whole state instead of following updates.
Like if you have the whole state, you don't need to do this
counter updates as I mentioned, so there will be no possibility
of making the other that inconsistent.
So if you are designing your system to behind important, at
least once will work in a very good way.
Also, you can build up the duplication mechanics of.
So, for example, each event, each message can have some
unique Eid. It's going to be uuid and at the
consumer side, you can keep just some kind of a table persistent
map or you are mapping the ID to the fact if it was processed or
not, but also it has its own problems because you might
operate with distributed database that also lay involves
Network partition. That's why this update of the
fact that even most processed or not needs to be done also in an
academic way. But yeah, I have this in my
chopper, as well. In that case.
You'll need to apply this modification.
On the database itself, not supposed to say the state and to
do some intermediate operations, but it needs to be one Atomic
operation. For example, comparing, some
kind of a compare and set on the database and it's nosql
databases offers that as well. So you can Implement applicants.
Another thing that is, commonly related to this delivery,
semantics is when you choose your messaging cure, I think
maybe these days, a lot of messaging queue up for, at least
once as well, but there are some products that offer different
delivery semantics Maybe. From your point of view, any
kind of thought, what kind of messaging you should people are
four or maybe some kind of technologies that people should
be aware of, with all these different delivery.
Semantics in this talk to, and I'm referring to Kafka, because
I have to because experience in this technology of coral is nice
in that way that it can allow you to configure your producer
and consumer Behavior to work in either of those ways.
You can have at least once at most once depending on your
needs. So then you can tune your
settings properly. Come.
Offsets in the proper way and you can design your hands to and
pipeline to work in either of those situations.
Because she has mentioned, you can have these cases that are
good enough with utmost words, but also some use cases when you
have and you need at least once. So using Kafka, you can have
both of those. It's only the matter of
configuration and configuring Consular producers properly.
I have this in chapter 11 of my book.
Thanks for sharing that. So for those of you who are
interested in, In looking into more details about delivery,
semantics make sure you check the chapter 11 in particular
about messaging, qubit Kafka and all the configurations that you
can do. So, Thomas, I think it's been a
pleasure learning from you. But all these different
trade-offs. I'm sure as a software engineer.
I hope we can all upscale ourselves to be aware of what
kind of mistakes that could possibly happen and what kind of
trade-offs that we should think in the beginning before we
Implement something so that we don't make costly decision.
That is quite difficult to retrofit or quite impossible.
To actually fix sometimes because when you have the data
actually involve, right? Sometimes of fixing data is not
so easy. So Thomas Before I Let You Go,
normally, I have this one last question that I always ask for
all my guests, which is to share your tree.
Technical leadership is done for all of us to learn maybe from
your journey, or from your career.
So, can you share your three technical leadership?
Wisdom, but also the first of all bleeding, but example, if
you want your team to enforce some specific standards, we
should give an example. For example, if you wants to
health good test coverage, you should also do it.
If you want to have good description of your work also
should do it. And all I will follow is the
good direction, second one. So it's always ask the question,
why? So if you are given to do
something Implement a specific feature that will result in
complexity. You should always be aware what.
Value it will bring if it gives you an eval you or not.
What's the ants to Aunt Flo of the city?
Turn how it will impact your customers.
And so on. So being aware of this, not only
about the technical things, ask why we do even need to
implement, but because every code is maintenance overhead and
cost. So, if you are able to solve the
problem without code at all, then it will be ideal.
And the put one is to create a blameless culture.
So that there is no blame of a single person if you are doing
so blender with some error. Or failure on production.
There are high chances that there was a problem of a process
and not a single person. So it means that you should time
to process fix the process to not do the same mistake in the
future and learn from their mistakes.
So create the some meeting discussing.
How does it happen? And try to incorporate that in
that new process that will not have it.
Thanks for sharing all this wisdom.
I find it interesting. Especially the second one always
asks, why especially if it involves Some kind of a complex
or a big kind of effort before you actually go down the rabbit
hole and you realize, oh actually it's not so important
all. That's another alternative way
where you actually don't need to write so much code.
So, thanks again. Thomas for people who wants to
reach out to you or learn more about the book itself.
When is it going to be published?
Where can they find you online? So on LinkedIn, also Twitter, my
Twitter handle is stomach. I'll 007, and it's also for get
help. I will respond any questions.
Feel free to ask. Ask them regarding bold.
You can also join demanding slack Channel this week.
We have book of the week and our book is book of the week.
It means that everyone can ask any question regarding teamwork,
and I'm answering them. That's cool.
So, thanks again.
Thomas. I wish you good luck with the
publication of the book. So looking forward for that.
Thank you for listening to this episode and for staying right
till the end. If you're highly enjoyed, please
share it with your friends and colleagues who you think would
also benefit from listening to this episode.
And if you're new to the podcast, make sure to subscribe
and leave me your valuable review and feedback.
It really, really helps me a lot in order to grow these podcasts
better. You can also find the full show
notes of this conversation on the episode page at technically
journal, the death. Site, including the full
transcript, interesting quotes, and links to the resources and
mentions from the conversation. And lastly, make sure to
subscribe to the show's mailing list on technology.
No, the deaf to get notified for any future episodes.
Stay tuned for the next technique Journal episode.
And until then. Goodbye.