One thing I've realized in the past, probably 15 years, is this
whole notion of transaction and consistency and acid compliance
and all that stuff is many a times, a composed, as in all the
business wants is don't lose my transaction.
They don't really care if it's like acid compliant, or this all
that, right? And we as technologists should
not make choices. It's the business that you make
that decision. Hey everyone.
My name is Henry Surya Vaughn. And you're listening to the
technology, you know, podcast the show where I'll be bringing
you the greatest technical leaders practitioners and
thought leaders in the industry to discuss about their Journey
ideas and practices that we all can learn and apply to build a
highly performing technical team and to make an impact in your
personal work. So let's dive into our Journal.
Hey again everyone welcome to the technology.
Now podcast the podcast where you can learn about technical
leadership and Excellence from my conversations, with great
thought, leaders in the tech industry.
If this is your first time listening to this show,
subscribe and follow the show on your podcast app and social
media on LinkedIn, Twitter and Instagram.
And for those of you longtime listeners who want to appreciate
and support my work, subscribe as a patron at technology.
No dot f /. On or buy me a coffee at
technology node. Dev slash tip.
My guest for today's episode is promotes Adel.
GA promote is a director thoughtworks and the quarter of
several data and architecture books such as software
architecture, the hard parts and the jolt award-winning
refactoring databases. In this episode, we discussed
data Essentials. In software architecture,
promote started by explaining why dealing with data is hard in
software architecture, and some data related concerns, we should
Think about when making architecture decisions, he then
shared the thought process of how we can choose the right
database for our purpose and shared insights on data
modeling, differences, between SQL and nosql.
Promote also touch on the important considerations in
managing transactions, and the trade-offs between acid and
eventual consistency towards the end, promote shared practical
advice on the step by step, how we can split a monolithic
database through database refactoring.
This is such a Great discussion with promote to discuss, all
things about data and databases and I hope you enjoy it as much
as I do. And if you do, I would
appreciate it. If you can help, share this
episode with your colleagues friends communities.
So that more people would also be able to learn from this
episode. Please also leave a five star
rating and review on a podcast and Spotify which I will highly
appreciate. Let's go to my conversation with
promote after a few words from our sponsors.
Are you looking for a new cool swag?
Tackle it You know now offers you some swags that you can
purchase online, these wax are printed on demand based on your
preference and will be delivered safely to you all over the world
where shipping is available. Check out all the cool tracks
available by visiting technology, no, dot f / shop,
and don't forget to break yourself.
Once you receive any of those tracks.
Hey everyone. Welcome back to another new
episode of the technology on our podcast today.
I have with me consultant at Bollocks, he has been there for
25 years, that's pretty long. And in my view, his name is
promotes Adele, gay is actually someone who is very experienced
in dealing with data databases the intersection between data
and applications. He's also the well-known author
of this book called refactoring databases, even though it has
been written long time ago, I think it's still applicable and
relevant these days whenever you want to deal with rdbms and how
to make changes and evolution towards your schema.
So the cost of nosql distill recipes for continuous database,
integration software architecture, the hard parts and
the recent book evolutionary architecture, which is the
second edition. So promote, welcome to the show.
Really looking forward for this conversation to talk a lot about
database and data. Yeah, thank you.
And we data in databases have been my passion for almost 30
years. Now I'm happy to talk about
nice. So, promote in the beginning, I
always love to ask my guess to share a little bit more about
yourself. Our first, maybe you can share
some career Journey or any highlights turning points that
you feel are worth to share with the audience here.
My journey started back when I was in 7th grade computers, or
spoon, had these two computers for Des Moines school, and you
can go play with them. And you sounds, I need a little
bit of basic games and computers all through high school.
I got into computer engineering. We call it a finish that and
during that time that is amazing.
Professor cordac examines each worker and he really inspired
how to think about operating systems or distributor.
So, I did a project on this team.
What is the most efficient way to distribute work on cluster of
computers? It's really interesting.
And then after that job hunting stuff and ultimately ended up in
quotes around 99, your four years in the middle, different
places. And we were doing this Amazing
Project called a class where we had an existing database that
was being worked on big upfront design kind of database that was
already designed 600 plus tables.
And then this whole notion of Isaiah 60 Blake was Catching
Fire. And Martin Fowler, Edward
curling up giving coaches and coaching sessions per period of
time where so mind, changing that you would think, why was I
doing up for Designing? The first place ever?
Like why did I even do that? Right?
So we had this first iteration planning and we said, oh, we'll
do this much first, happy whatever.
And I went back to my test and build it on the people's that's
posting a day and that's a scary thought for A person, but I did
and every iteration as planned, new stuff, you do new
functionality, you back things. You've got things to the table
and stuff are more stable, snow bunch of that constant change
and there are about 30 or so developers in that team plus
plus terms plus b, a is plus a bunch of other people and I was
the only divorce. So anything that came to the you
do this, I literally walked into their space discuss with them
the chain. With them.
And then I came up with this scheme of pregnancy, Henri meets
a change. How do the rest of the 30 people
that teach because by that time I had come up with this concept
of every question, working on the code base should have their
own schema they shouldn't be like shit excuse, right?
Because early in the beginning and I noticed that if they're
sharing schemas and I make a change for Henry John here, mix
affected because the see my history not quite okay.
So I said, okay, let's play everyone.
Schemas I came up with like a bunch of scripts.
They can create their own schemas, they can create their
own, like popular, that schema, and all kinds of, but a bad
script, I wouldn't call them devops, but a bunch of bats
grips on Windows machines that you could run like, create my
schema do that and it would create a scheme of Iran.
And once they happen and then we came up with this concept of all
I want to modify or splits, I would speed with them and come
up with a neat and then at some point of time, it so happened
that this is all going good but now we are trying to figure out
okay, how do I give away all the power and simplicity and work
was last particular, slide? What are the discs between the
requirement? So that's where the whole
concept of version controlled migrations in Nevada.
So we didn't really have a number The thing is all these
things. I just keep track of each one
teach to tease myself and I kept applying them and I knew what to
use like an attractive tracking metadata, in those schemas in
what's amazed at what level bunch of that kind of stuff and
then ultimately, we figured out progression of changes.
And so when you are either in of the New Path somewhere, like you
encounter a bunch of problems as we go, right?
And then Just trying to like, find out solutions to those
problems. One problem was, how do I know
what TJ was with this? Big was big?
So then we started publishing, what was the last chain that was
available when this bill was. And we took that put it as a tie
in the Village, self-published Package.
So who are unpacked the build new?
What change was needed for that to work?
So, once you have that small stuff like that, if you really
think out there was all of them with inconsequential, Initial
selection of them was really a say-so, not some of those 10
stars and my opinion at some want.
Look at the whole scheme, I had set up.
And so, this is a really interesting from was like, you
should speak about this talking about this, let other people
know about it. And so he took me to Madison
Java user group and they were like 600 people in the big
article in there. Everyone's had come mrs.
Martin, not me. But Marty put me in front of
Music. Some people for the first time
and I was shaking, you know. Up.
But I did give that talk and I sense like that was a good way
to share knowledge, like you learn something better to share
instead of this, keeping it yourself.
So I sat in sharing and then from there on, like lots of
other cops, I think speak on said, XD day is different
places. And I refined based on
questions, like, in your project, there's a certain
context. And once you've solved, all that
context, there are no more new problems to solve.
So when we go out, People conference stocks generally
speak to people. You hear different contexts,
right? And then you want to learn how
to solve those make branching and merging.
We will do Braxton. Because at that time we always
did trunk waist up, but some people did.
And like, how do you manage team says when you grunting really
interesting stuff like that. So ultimately, I think around
2004 or something additional, Martin came up with this idea of
whatever I was doing at similar stuff, was also being done by
Scott Ambler. Basically put both together and
say quality the refactoring database is evolutionary
database design book. So we then took all the changes
that I was doing on projects, like move columns, column table,
to table and codified them and came up with the structure that
just like how Martin's refactoring book.
Say if you tell Annie - black method, if someone says that,
you know exactly what so our language or naming. 1/2, a bunch
of things that would happen, so something similar to that.
So, by the time IntelliJ had shortcut keys to expect method
and a bunch of that kind of stuff, we are inspired by that.
So we came up with that name came out over 70 or so patterns
that we put in the book and we talked a lot about the book and
later on, I got about 13 years integration.
And so other 2008, I'm see this whole nosql movement was like
really finding out and lots of stuff.
Like we have look paper, come about Lots of people are doing
different things with a mongodb. I think point five point eight
come out so I did a project with mongodb.
We also need a project agree on and then I started talking
thinking about the value designed objects in object
oriented languages and how you store them in relational
databases, a translation that needs to happen.
It's not the same with no escalator, right?
Like object-oriented stuff. How does it translate to
document? How does it?
A spare, key value stores. How does it pass the columns or
how does it translate to graph stores?
So that gave me an idea of we should probably talk about how
this modeling works and why you choose one over the other.
I like to talk about this concept of like, an automatic
car like gears, like you put in dries and you just try it.
You don't worry about what's underneath, right?
Worry about first gear, second gear, this and that like,
automobile reference here, that's what relational databases
where they picked. See or anything else, they
picked a bunch of that kind of stuff for you and you don't have
to worry about anything else. So by default, you get those
things and developers don't have to worry about other choices
because there was no choice. But once you go to another
escalator basis, there's a lot of choice.
So you have to decide what is the choice that you want?
Because you can just go blindly saying, I don't care what choice
once you go into that the availability of choices creates
lots of options. And at the same time also, So
creates lots of pitfalls with what if you make the wrong
choice. So that's why we thought about
writing that book, The nosql distilled Utah.
This is a broad survey of what types of nosql databases are
there, how we pick them? What is consistency?
The Gap here on distribution, mapreduce bunch of, that kind of
stuff, all together. And how we should dispel people
from like, oh no skill schema-less.
So, I don't have to worry about schema.
That's not true. The schema is there.
A code it's not in the database but it is still there in your
courts. We have to crack of it and stuff
and how you shouldn't have to be like these.
It's equal. Or is it nosql?
It's not that. It's just like with in the same
Enterprise or within the same system you could be having more
than one. And that's where the polyglot
word came about is you could have more than one type of
storage and we give like couple of examples and so that was
pretty good. And then later on, I wrote
something with Neil around software architecture, the hard
part. It's about how do you pick?
Because I personally think many times the data people are
excluded from architecture discussion and it may be by
choice or it may be like a like once we decide the architecture
we can just store data somewhere and I think that's a misnomer or
you should be including data Architects, people who be with
data into the architectural decisions because long-term
where data is stored, how it's stored, affects a lot of
different things. Ultimately all business really
wants to He's not your code but bait, right?
Because business runs our data data is, what is stored persists
applications? Come in go.
So how to like, make sure your data is available, find about
interoperable. Like this are principles, how
can they be applied? And all that kind of stuff.
So, inclusion of the data protects in this whole,
Enterprise architecture, architectural generating, should
be encouraged and even data architect should be saying, I
want to be Be there in that discussion.
Yeah, so that's my journey. I've been here.
I've got books for almost 25 years and still enjoying it.
And right now, I'm doing a bunch of stuff around data mesh type
of architectures and implementations of that idea.
Well, thanks for sharing. Your story is really
interesting. How you got this, all started
right surgery from refactoring, databases from project
experience, having to share it in front of the audience, for
that large amount of audience. I think that's pretty scary to
me, as well in the beginning, I believe.
And I think since then, yeah, you Dealt with a lot of our data
challenges and wrote A Few books, along the way.
So I had Neil a couple of episodes before talking about
software architecture, the hard part.
And I think with having you here in this episode as well, I want
to continue our conversation about the architecture but
dealing with data I think, as we all know, dealing with data is
very difficult. In fact, many people think it's
the most difficult thing because your applications come and go
like what you said but the data always stays there.
So the first thing is probably if you can share with the
audience here, how much has that changed these days with all the
advancement of technology? Different types of databases is
data still, the hard part of the architecture.
The Ada is still hard because bunch of data is locked inside
commercial systems that are bought off the shelf, so bunch
of stuff, just sits there. And then how do we get it out?
So it's useful. Other parts of the
organization's is very important.
Many, a times data is also stuck behind some kind of I would say
walls because oh it's available in that day only.
You can object to it or it's available there but you don't
have the right Keys. You don't have the right way to
get it out and stuff. So right more than the desire, I
think the paths to get to the data physical and people are
working on that. And I think that's why betameche
is become so popular is because it gives you like this domain
oriented weight that getting at data and you can deal with data
and piecemeal stuff, instead of oh, I need to build this whole
data warehouse before I get access to it.
Now, people are saying okay. Can we like talk about it in a
smaller chunks of pieces and time to Market has reduced
because of that, right? And that I think is making life
easier. There's also this notion
nowadays all Cloud providers give you some type of storage
mechanisms and those are also like all the way from like five
storage block storage, all the data analytics based storage and
stuff. I mean, there's a bunch of
choice there. But as people are using Cloud
Technologies, interoperable of that data is become a little bit
easier. And having access to that but we
are still dealing with the problem of people decide to put
stuff all over the place and then knowing what did I store?
Where was it stored? What's the definition of that?
What's the meaning of that column, or table or whatever is
still harder, right? So we need to be a little bit
more disciplined and structured about how do I store stuff.
I mean, the basics of computer science is meaning things like
the name, your favor, right? Calibrate, all kinds of stuff.
Like, even if you have a Json to do, you mean They sauntered
attributes, right? So once of that kind of stuff,
still needs to be followed religiously so that someone who
has not worked with, you can understand what that structure
is. Because again, we have so remote
and select and or consists of in someone waiting, looking at the
Json that you publish. And they may not even know
either that you exist, right? So how do they get what we
desire means without spending? A lot of Cycles, easy challenge,
right? And specifically you also invite
that like just now when you were sharing, The beginning, right?
You invited people to include the data architects in the
beginning whenever people are talking or deciding about the
architecture so that they're involved, maybe specifically in
your experience, what would be some of the areas of concerns
for a bunch of Architects and data Architects to discuss about
whenever they want to make decisions about data-related?
Yeah. So I think the first thing
probably higher level thing that you should be thinking about is,
what are the forums where beta people?
Should we invite and once I've had clients that we go at, I
tell them. Anyway away, this architect
involved, you should have a data.
Having said that the areas of concerns is generally you want
to give people enough freedom to low-weight do things and that
kind of stuff. But at the same time it should
be such a blank canvas that they do whatever they want.
So a good example of this is a CR, a specific Cloud usage and
of company. And someone says, hey, that
other Cloud gives me that feature and this will Use that.
So you should be having some guardrails saying, okay, you
could use certain things Within These parameters and we won't
ask anything. But if you have to go outside of
this, then you have to come and justify why we are not saying
no, but at least come and justify why.
So at some places, we have come up with a flow chart for picking
the right database for its right.
So you go through this tree of information, decision-making,
tree and then you arrive at something.
And if you want to use that file, You make me want ask
anything, you just start using it, but if you arrived at that
decision and you say, oh, that other thing over there is better
than this, for my use case, then you should have a conversation
someone before you pick that up. So architecture, governance
matters a lot in this kind of scenarios because for solution
sake purposes, that may be great, but for million and sake,
operational sick that other thing over there is a hindrance
from the operational side. So you have to have very good
reasons why that is this. So that's one the other it is
sometimes And to pick Technologies or Skies
architecture sites based on our own biases of understand.
I, if I am a data type of person, my bias is to make sure
all data is correct. I have taken care of very low
doses, right? Load versus really low and all
that kind of stuff. And then do the right kind of
design. But somebody else coming from
the other side may not be thinking about those things they
may be thinking about what is a rarity, they may be thinking
about coupling versus orchestration versus
choreography. He and a bunch of others, all
valid concerns. But when you don't pair that
with the data side of the shop, then you end up like, hey, this
is all on software set is All By Design but I am taking all of
that and storing it as blob, somewhere that nobody else can
read and then you have visited with Corpus of modularity
because now the data said you're not model so stuff like that
matters. So that's why I say, like any
kind of new stuff comes up. They should one other place
where I think this painting really helps is And product
managers are business, people are coming up with new ideas,
and they are saying, let's at least for by out and see, okay?
How much this will cost is this your physical and stuff?
And they generally tend to talk to The Architects and say, oh,
is this a workable? Next with the data, it should
also be because they can tell you, where can I get the data
from? Where is the data available?
Do we have this kind of data available and stuff like that.
Then also give you a way to make the feasibility question, answer
in a complete sense. So the other place is also to
talk about especially on the analytic side like are reporting
side of things like that. How is data being given to you?
What is the lineage of that data?
How what kind of Transformations are like?
What are the standards being followed?
What kind of quality rules you are in application development?
I think that living standards quality standards bunch of those
kinds of stuff is way of data people can really help because
if app is gone through development or Going to software
has gone from development for ten iterations.
And then you tell, oh, this quality rule.
This standard has to be applied, nobody's going to listen to you,
because there's no time now because the D, or the product
manager of the product owner is going to say, I don't have time.
I need to take this to Market, and now, you're carrying that
TechNet forever instead. If there was only your you have
talked to them, you would have gotten those unique standards,
you would have put all those things in place.
So you would not have that. So stuff like that, but
developer type level those things matter a lot over a
period of time. The other thing also is probably
coaching and mentoring in there should be a constant flow of
ideas from software side to the hate, aside from the data side
to a software service, and those could be like, what can I use
for certain things right? There may be some techniques, I
can use on the data side that can scale this much easier
versus use killing the app side like crazy, right?
They made some things, I In say he'll if a package this data up
and give it to you, then you can distribute it much easier than
me, coming up on the data set its side.
So those give-and-take, I think as a team you can come up with a
better solution. Thank you for sharing all these
areas of concerns. And the I have to say for me
personally as well dealing with data related Tech.
That is not fun first, it's like a resume, it's risky.
Also and second thing is how we deal with the existing data,
right? Sometimes we have to patch them,
migrate them and do all these stuffs in order to comply with
the new decisions, right? So I think it's always not fun
and I think that you said if you can deal with it earlier that is
always the best preferred options you mentioned earlier as
well about picking the right. It database types, right.
And these days, I think I remember when I read your book,
nosql distilled long time ago, there was probably only four
types of databases like the document, the graph, the key
value store. But these days, there are so
many. In fact, there are plenty of new
database companies being formed with their different type of
characteristics. So maybe if you can guide us
also for people here who are new with many different types of
databases. Well what will be some of the
attributes or Three sticks that we should think about whenever
we want to pick the right database for our workload.
Of our use case, maybe some guidance here would be great.
Sure. I think there are three or four
things at a very high level that you need to think about.
And then you can go down as you go down, you can have more
choices to worry about the what is productivity like right now,
human time costs, more than anything else.
So if you are working longer hours that cost is much more
than the cost of any other technology.
Ey. So, if something makes you more
productive like, for example, if you're writing JavaScript and
node.js and stuff, I have Json, I just need to store the Json
and retrieve the Json. Go document.
I spy database has been a simple choice, so I think productivity
is one thing that drives a lot of these decisions.
Like, oh, this thing makes me productive.
I don't need to worry about other stuff and then the other
decisions, they'll optimize later, but that's the primary
decision. The second is sometimes use
cases which need not of thought to be picking the right
database. What I mean by that is let's say
there is some use case where you have like oh I need to store
like trillions of rows and that won't fit in a single machine.
So I need to find something that can distribute itself, easily,
shower date, or whatever. So let me find the right type of
or they may be some instances where you say, oh I have this
external. The fire that I always have and
I can get data based on that, always, alright?
Lee and based on that then you pick the style for that type of
thing. So like what are my read
patterns and what are my weight times?
You need to analyze that a little bit and figure out what
is the style of database that works for that?
And the reason I'm saying is twenty years ago, we used to
think a lot about this space. How much disk space we have
using, and the cost of the disk space and all that stuff.
Today, the amount of time you spend in thinking about that
will pay for this, right? So, the cost of the disk space
is no longer a question. So it's more about how much can
I write on my skin? I read, how fast can I read how
fast you are? Need to read and a bunch of
that, kind of questions come into play and then you pick the
right kind of database for that. Like, some databases you write
only once but we'd be in sometimes.
So then maybe you need something that is read performant and I
may want to write the same data 34 times in.
Different styles like different. Aggregations Sky, that's fine
because I'm reading so many times and I want to make Vida
fishing. It's our other places.
The right efficiency in a matter of a lot, because data is coming
so fast at you and you want to make sure you can write to write
efficiency. So then you pick something that
caters to, right? And the third style is, what is
it that you want to get out of that database, right?
Like for analytical proposals are you doing lots of rows of
executions that kind of stuff. So let me pick something that
helps that. Or are you doing like graph
traversal or graph analytics or relational analytics or that
kind of stuff, right? So let me pick something that
works for that. And nowadays, there are many
databases that coming up, but if you really look deeper
classically, there are like five types.
It's at a very broad level like relational databases key value
stores documents and databases white column stores and crafts
stores. So most of them kind of fit in
this style. Like even a block storage is
nothing but a key value writing a Blog.
The file in is a key. Everything inside is a whale,
right? If you think about it, that way,
then there are five big pipes and then you can decide which
type of database that you want to look at and then you can
within that type. Then there may be multiple
Choices available and then you pick based on that, right?
So if you take an example of how I need a graph database because
I'm doing graph traversal fine. Now you have options of year for
Jay, Neptune, DeGraff tiger grass and a bunch of other
choices. Now, how do you pick that?
Choice is a question that always comes out, right?
And then do that. Generally what I tend to do is
set up our scoring Matrix of what is it that I would pay
right in this graph? The sense, I can pick new Forge
a paragraph B graph and Neptune put them on like a header type
of stuff. And I have metrics on which I
want to measure because every company can have different
measurements, right? Some people may say familiar, it
is very important, and I have people who know neo4j, so no
discussion for me. I just some people will say, oh,
I don't want to leave the ews landscape that we are in, so
I'll just pick Neptune in. You're done.
Some people may say open source matters, Me and distributed
databases matter to me and that kind of stuff by nodes are gonna
be really large. So I need distributed graph
database. So people will say, okay will be
graph so that I think is what you need to set up for your own
company or your own situation, and come up with.
What are you going to measure the products on, right?
So set up a rubric for yourself and that score, these products,
very easy. In fact, most of them you can
try, you can run up, you'll see, you can do some research.
And then set up that rubric and score yourself.
And the answer will just hopping speech and then you click one
based on. Thank you for the guideline.
I think is really interesting the way you first mention about
the ease of use or the productivity, right?
So based on the developers or the language stack, that you are
using one question related to that, in my experience working
in multiple projects and different types of products.
One thing that always happens is that I've heard multiple times,
they develop choose a certain Database.
It works well in the beginning but after a while, it didn't
perform or it doesn't skill and a lot of learnings as well.
Also point out that eventually actually many people referred
back to rdbms, you know, things starting to become like a big
quickest again, right? With rdbms may be either MySQL
and postgres or even Cloud databases.
These days also have rdbms compliant interfaces, although
maybe the way they store or query is different.
So I'm interested to hear your thoughts about this coming.
Back to sequel. Coming back to rdbms.
Is it still the preferred default choice?
That people should choose. And when you talk about
readwrite patterns as well, maybe if you can give rough
ideas when there's rdbms actually does not scale that
people need to start thinking about other types of database.
So many a times when people pick up new types of databases for
themselves, they have picked a new database but they're
designed thinking still is in the relational world.
So I have seen a lot of document database, This is where the
collections have been designed as if they have tables and then
the complaint comes in, like, I can join this Corrections.
A document database is not supposed to join collections.
You have to have like a aggregate version of the
collections, right? That's where this domain driven
design, way of thinking really helps.
So, in the nosql distill book, we talk about what is the
boundary of you're a greedy. So good example of this would be
lets say e-commerce system. You have a customer that has
multiple orders and each order as ordered by items as payment
information shipping information, probably packaging
information on it. Like is it a gift?
Not a gift that can stop right in a relational way of thinking.
You have a customer table, we have addressed legally, have
ordered it when you have order item table, probably have a
payment information and bunch of these states.
Right? And when you go to document
databases, you can say oh I'll take that idea and just have
multiple elections of these things.
I have a customer collection, I have a dress collection and that
NASA, because when you put this back together, you have split
the aggregate into a really small parts.
And now we have to put it all back together and the database
was not break for putting all those things.
Like, so where do you pick your aggregate boundary?
One example would be. I have a customer agree, gay
inside the customer we get. I have all the customer
information and I have a collection or a list of orders
inside it, and then eat, Order in turn the Json object.
If we think about it will have the order Handler will have the
order items. We have to paint everything all
inside. So when you go to the database
and you pull the customer out, everything comes with it, so
that's one and somebody may say, oh what is a customer has
thousands of orders. Then my objects will become too
big, like, you're transporting all of that over the wire,
sending it all back. Okay, now it's time to think
about how the split disagree it's to be.
So then you can say, oh, Want to have a customer Aggregate and I
have what I have orders. Agree.
So you have one object for customers that has all the
customer information, probably their preferred shipping address
before, packaging whatever and then you have one object for
each order and the order has a reference back to the custom.
So when you go back to when you want to show, all the orders
about the customer, you go to the customer, pick up the
customer and then go to the orders list and say, give me all
the orders that have this customer.
So now you have split the eye. Aggregate but not too much just
a little bit so that it can manage this one.
So you have to think about these kinds of stuff because when you
go back to the standard relational way of thinking and
you speak too much, of course databases, So, I think the first
thing generally, I recommend when people pick a new database
style, is do pet project. Like, don't do your actual
project look at project play with it, try to understand,
okay, how do I need to think about it?
How do I need to worry about it is need to get training, get
training. If you need to like talk to
someone who's experienced in that database, talk to them,
understand this design patterns that are different.
So again the classic example would be like graph traversal,
for example, some Probably say, oh, graph traversal.
I can do this in databases, like, relational databases.
I just put pointers and stuff and after two or three levels of
whiter, you'll figure out relational databases to work and
then they go to graph databases. And then in graph databases,
they start treating each node as if it is a table, it is not.
It is a single instance of row that is connected to something
else. Alright?
So we have to think about these things a little bit differently
and Then you realize that the gains of using that technology.
So I think that is what I would recommend people.
Like figure out how this tool Works.
Get some training topic, someone who's experienced play with it a
little bit, solve a simple problem or something and
understand how this thing works. One other thing I have also done
is if you know that in this space of this database type, I
need to pick one technology that was talking about the rubric
before, right? So performance can also be a
rubric on that like, does it Handle of billion notes, for
example, or does it handle a billion documents?
So put a billion documents nowadays, you can create fake
data, put it in the database and see how it works.
Setting up that experiment, we take a little bit of time but it
will save you time in the future.
So create a framework or create like a harness, which puts in
all this data in there and then you run your query load.
And this is where knowing you're pretty pattern.
Makes a lot of sense. Like if you don't know how many
rights and how many reads you have it, Cannot set up a
performance harness. So you basically once you've set
up like a billion documents for example, and then you know, can
going to do like 60% of my load is read forty percent is right.
Okay, so what does that mean in one hours of production time,
I'm going to do this many nights.
I'm going to do this video reads, right?
Okay. Set that up and run it and see
what it does and then you will figure out.
Okay, should I change my design pattern?
Should I change my product, should I change my Bye.
Should I change my technology? Whatever it is.
It tells you where you need to focus, I think you're sharing
about. The aggregate boundary is gold.
I feel this will save a lot of developers time and effort as
well. I think it's a very good advice
that we have to look maybe from the use case the business use
case product requirements. What kind of aggregate boundary
that we are dealing with right and then not to use the old
paradigms, the rdbms, these data modeling, apply to different
types of databases which is Is probably a majority of
performance issues come from there, right?
So the impedance mismatch between the data model and how
you query them. So, thanks for sharing that.
The other thing that I want to ask is about whenever we have
all these data types. Now, people tend to think about
touring special type of data into different databases or what
we call also polyglot persistence, right?
Especially now with the microservices, maybe one service
could have multiple databases but it also comes with a hard
challenge, right? It's about the transaction.
I It's always very important to think about transaction, and
especially with polyglot, persistence multiple data types,
right? How do you have this in a
distributed manner? So maybe from your view, how
should we think about managing all these transactions may be
multiple databases if they are involved, right?
Is there any special thing that we have to think about?
Yeah. So I think that is always such a
use case, specific discussion that needs to be had with the
business like one thing I've realized in the past probably 15
years. Is this whole notion of
transaction and consistency and acid compliance and all that
stuff is many a times, a composed, as in all the business
wants is don't lose my transaction.
They don't really care if it's like acid compliant or this or
that, right? So many times I always tend to
go back to this discussion of how much money do you want to
put? Like, money is a proxy for time
here. Basically, how much money do you
want to invest to make this as a top client or This 100%
consistent or whatever, and the business should be driving that,
right? They should be saying it's okay
if it's famously. Okay, then I have a different
solution for that worship. Someone say, oh, I need to make
sure all stores are consistent all the time and that cost a
lot. And is a totally different
solution. And we as technologists, cannot
make that decision or should not make that decision.
It's the business that should make that decision because they
are the one who are deciding, how is it that they want their
data. How is it that they want their
systems and how much do they want to invest in that desire
that they have? So we should expose this to
their business. That's the first thing I want to
say. The second thing is let's say
someone says, I need a highly consistent database or highly
consistent data then, okay. What are the patterns in which
we can implement the consistence that comes into play?
And then within a micro service, if you have, let's see more than
one database. Then how do I make sure they
consistency? Of the rights that are going on.
And in most statistical, test systems, even if you're writing
to the same database, you will have consistency issues because
let's say you are writing to Cassandra which is a distributed
database and the data mean reach one node.
And you may say oh it's done but that nor sales that we have lost
in it. So you still have to think about
you and it's the same database you have to think about what are
the consistency requirements that I consider.
So every right could have different consistency
requirements. Well, I'm writing just some kind
of audit log that will be a little bit different consistency
requirements versus I'm writing an order.
So every ripe you can decide what is your consistency level
and many of these databases especially if you're writing to
single database, provide you different types of consistency
guarantees, right? So I think they are called like
forum is one when it says majority of the nodes got it.
And then you can say, oh, just neck as long as one node, got
it. Find out as I go, there's more
than two notes have it? I'm fine Arabic in mongodb.
I think for example you can just say, oh as long as the not
receive they can find. I don't even care if it does it
come to the desktop. So a bunch of these options are
available and it'll pick which option you want based on each
different type of right even within the same application.
So I think making these kinds of decisions inside act like
probably a tech lead level or at architecture level.
You can say oh this type of Rights will always Go with this
type of consistency, this type of Rights will go with this type
of consistency and probably encoded and don't make the
developer think for every right. Just have some kind of a
convention about it and you go. The other part is when you're
writing to two different databases, like one probably is
like, let's say you write a relational database and the
other is going to a graph database or something.
Then now, you are in this. Like, if I want to maintain
consistency like, a hundred percent consistency, what are
the patterns that are available and implementing those patterns
that you can use? Keep our transaction coordinator
or something like a two-phase. Commit is a really hard problem
and you really need to think if I want it.
And what is the types of challenges are willing to take,
on Nick? So, Many Items, what happens in
a polyglot, like the relational and cross.
As an example, you are writing to relation now for operational
purposes and you're writing to graph or graph traversal
analytical, probably recommendation engine, that kind
of purpose. And Those things can be
eventually consistent and relates now can be like 100%
classes. So you can even make a decision
between that like, what is use case in which I am writing to
multiple layers, right? So someone writes takes an
order, your forces, the order and the data.
Eventually within the next five, ten seconds goes to the graph
database and some other person is going on web page.
And you want to show them a recommendation based on the
grass, how much wrong is it for the recommendation to not based
on this last outer material? That's a trade-off, right?
And you say no no it has to be in a hundred percent, all the
orders that are committed here. Recommendation, has to be based
on that, then you click pick a transaction coordinator, you put
all this, the architecture complexity increases because of
right and that's a trade-off. You have to take on like oh I
cannot even be five seconds behind.
So I'm increasing my architecture complexity.
That's a bit of you mean and is it okay you should ask the
business, is it okay? And then if they say yes, that's
how I want it. Then yes, then you increase the
architecture of complexity to meet that business, need the
introduced a transaction coordinator with the night
happens. Here you go there, right?
That side. And then of course, those come
with its own trade offs, introducing transaction
coordinator zookeeper, but then you made a informed choice about
using. Thank you for sharing this
guidelines, right? I think it's really key for
people to hear, right? So always involve business,
whenever you make this type of decision, maybe sometimes
business actually don't require. Real-time 100% consistent within
milliseconds, right? Sometimes it could be delayed,
they accept delay. And knowing this is actually key
whenever you create your solution, right?
Because then you can have different types of options.
So I have one interesting thought, right?
Whenever you said in the beginning, that it stuck in post
and how much money or effort you associate with choosing this,
right? But actually many types of this
decision, like you said, be stackin post is developers who
make the decision and actually from their arguments.
Choosing like an acid compliant database is actually much much
faster and saves a lot of effort because you are when you deal
with eventual consistency, that means you need to write a lot
more code. You need to do a lot more
testing so maybe a little bit of advice for developers here,
which I'm sure that we have plenty of a developer listeners,
how we should think about the effort here because storing into
an acid compliant is pretty well known.
These days, there are Frameworks very easy to do.
First was having to do it in an eventual, consistency Manner and
the effort that requires Them to do that.
So maybe an advice here for developers as well.
Yeah, certainly I made again, I like to think about everything
as a trade-off, right? So you when you say as it
complied it makes my life easy great, you traded off, Skilling
for that, right? So when skill comes in, then you
start having this replicated data bases.
Relational databases setting up replication and making sure
replication works properly. All of, that is extract stuff.
You took on because you traded off.
Scaling for asset, no plans. So when that happens,
unknowingly you have three did something for something else and
I am saying make that visible for yourself, like, okay, I pick
tacit compliance. What did I give up?
And once you start thinking about those terms, it makes a
lot more sense. And someone will say, oh my
database doesn't have to go beyond 50 gigabytes fight like
that trade-off is perfect for you, but if someone saying oh my
business is used across the world, I will have billion
millions and billions of Actions Rose reads, writes, whatever.
At that time, we cannot just blindly say, oh, I pick asset
our minds because that comes with some very tough choices and
making those visible to yourself.
That's why I was saying the clearing, is someone else in
this time, makes sense, because you have your own biases and
everyone has given, I have so, I generally say, oh, let's sit and
work together because they will expose my vices, which is good.
Well, I think that's a very interesting insights, right?
So yeah, sometimes we develop, as we tend to have our own bias
and we tend to think of the problem, may be from limited
perspective, right? So having other people, giving
the same thought process may be from different perspectives,
maybe design, review, process or even pairing whenever you come
up with this decision at the is always important.
And yeah, like you said, in the end, there is always a
trade-off, like Neil also said that, right?
So architecture is all about trade-off.
So knowing that, when we pick something basic, We also may be
compensated for other aspects, which we may not realize, but
sometimes it happens later on, right, which is normally the
performance scalability issues. So, another thing that is
commonly being discussed in this data world is we all have
microservices, right? But most of the time, start up
start with the monolith and then hence we need to break our
databases, you know, doing more like if illusionary, schema
changes. So, since you wrote this book
refactoring databases long time ago, and I think it's very
applicable. I will be split monolith into
microservice. There will be some of your
advice here for people who are going to Embark or are embarking
on this journey to split their so-called giant monolithic
databases into more multiple types.
So is there any key advice that you want to give here?
Yeah definitely. I mean there are a lot of
stepwise things you should probably do I think in the
software architecture of the hard part to do, we put up like
a five-step process for this. The first step is to identify
what are those boundaries? He's being the links, right?
Like, if you are going to take a monolithic application and split
it. Let's see. 3 for example, say
then there are probably going to be three domains of three
services and then you should decide within your papers which
service owns Which tables. So let's assume for Simplicity
sake. They're operating table.
So each service is going to get six tables or maybe some service
gets for some service ks8, whatever.
What are those tables? They belong to each one of
those. Right.
So create a boundary, kind of a situation and then once you
have, that cannot be find, then there are Educators here because
one service may be right into the table and the other service
me reading from it. So you need to decide who is the
owner of this table so that you can clearly see.
So, first step you can do that. And the second step is in the
same database start, moving these Stables owned by service,
a into service, a is scheme like many tables.
Any database products, especially relational database
products. The give you the facility to
move between schemas like you can see, outer table moves,
schemas you can just move it to a different schema and it just
gets allocated to a different scheme.
So you are logically segregating not physical situation.
Yeah. And the monolith is still
talking to all of them, probably you will have to do some kind of
a synonym or something so that the underlying movement is
transparent to the application, right?
So once you have On that you're physically separated the tables.
But the application is still talking to one database, schemas
wire synonyms or something like that.
So this is where now, you start to work on the application side,
say, okay. Now all of these tables, this
part of the code is talking to that.
So maybe we should create a separate connection pool for
that or maybe create a separate thing for that, that is talking
directly to that once and you do that on the application side and
your deploy. Make sure everything's working
and throughout this journey, our goal is always to iteratively
keep work, right? It shouldn't be that you do all
these first five steps and deploy once, it's every step you
deploy, every step you deploy. The because something was wrong
rollback is he's so Second Step. You do though, it's tough and
third step. What you do is now that you have
three schemas in your sample application three schemas in the
same database that are physically the same place, but
logically separate, right? And the application is
independently talking to them. Some is a is talking to schema a
service, be stopping to schema be service, he's talking to
schema see for all practical purposes.
They are independent. The only thing that are
dependent is on one machine because they are all running on
lunch. Okay, so now this is where you
start bringing in replication, give install two other or maybe
three other databases of the same type and replicate schema.
A to machine a replicate schema, beat machine, the machine And
you just set up. Replication, don't do anything
else and then whatever is happening on here with rights,
get replicated to other side. And one fine day, you can just
switch connections to those others.
So now service is talking to machine, hey service, be
stopping to, she be and service, he's talking to machine, see and
you can get rid of this database break, the replication get rid
of this database. Now, those become the main
databases and now they are all split into Socrates.
So it's a five-step process. Her some complications in this
of course especially like I said, one service is writing to
one table and other services reading from that vehicle, that
service now needs to figure out, where do I get the data from?
Because it's in a different database.
So you have to create probably some API changes to read, don't
make the service do a cross database connection because that
defeats the purpose of microphones.
So you have to expose an API to do that and a bunch of that kind
of dependencies will come in, but it at least gives you You
starting path think about this. Like, how do I split water
plants and how do I get to the end goal?
And many a times, what we do is you don't split everything at
one girl, you just take one piece of functionality, take it
out and then do whatever you need to be done.
Just take that one thing out. So if you think about this
monolith and percent is gone out or different ways, 90% is still
emotionally and then you think about what's the next thing I
can work on again, business value plays a big deal here
because Within Only the reason to split generally is because of
Architectural Components, like, scalability auditability
deployability, bunch of those kinds of stuff is being hindered
because of a male. That's why you want to split and
scaling is the only problem you're trying to solve.
You take one or two of these out.
The rest of the application is fine because it doesn't have
that much load. The to that you took out has a
lot of loader and you can scale them out and whatever you
probably don't need to split the rest of the 81st.
So you have to think about it this way, what is it that I want
to split first, split it out and then do I still need to split
more? Okay.
Then what is it that I need to speak?
Next does the remaining needs to be split if the answer is?
Yes, keep going down that path. The answer is no just stop stop
right there because you don't need to split it.
So I generally tend to think about it, like, did I achieve
what I wanted to achieve the eligibility, requirement
irritability, whatever it is, that you are trying to fix.
Once that speaks you can stop splitting I think in the
application software worked, it is also known as the Strangler
pattern, right? Yes, exactly.
I think that was written maybe around 2008 or something.
Yeah. And I think your advice here,
don't do it in one go because sometimes we want to do it in
one go because we feel it's risky.
And we don't want to deal with this kind of work because it
tends to be quite long right. Whenever we want to evolve our
database, schema. Right.
So I think don't do it in one go.
Do it iteratively right? Know what's the target as well,
like what you said? So maybe we don't need to split
everything in two different databases.
Some could still exist in the model.
It as long as you have the proper boundaries and you define
the ownership really well, while some may be different quality
attributes, require you to actually split them and hence
yeah, you can do this migration. And I think all your books here
have the recipes. I would say, the software
architecture, the hard part, I think like you mentioned, there
are five steps you can think of how you deal with this splitting
of databases and whenever we deal with eventual consistency
and The transaction is also covered there.
And eventually also, if you deal with rdbms and you need to
evolve your schema, we can always go back to your classic
book, refactoring, databases, where you can have splitting
column, renaming columns and things like that.
There are 70 patterns that you mentioned in the beginning.
So I think for people here who are dealing with all this data
Journey, please do check out promotes resources books, right?
And also lately about data measure, I also had an episode
with drama. I think it's also becoming
trendy especially me. Missing the operational and
analytical concerns, and also how to get data out of it
easily, right? So, I think all these definitely
are some of the trends in the data of space.
So promote, thank you so much for this conversation.
I really love and learn a lot about how to deal with data
better. We are reaching the end of our
conversation, but I have one last question that I normally
ask for all my guests, which I call the three technical
leadership. Wisdom, you can think of it,
like, an advice for people here to learn and listen from you.
Right? What would be your tree?
Technically, the It was them. The first one I would say is
keep learning new things, whatever it is like in your own
work area, or it would be outside of work or whatever it
is. Keep learning new things.
The second is, make sure you're mentoring.
Someone always could be other developers, it would be the data
people, it could be someone outside.
Make sure you're always mentoring others and I have
found that mentoring others, gives me New Perspectives on
things because I may be telling them things, but I'm learning a
lot. Why?
I think I find that really interesting.
I do enjoy mentoring. The third is always, I have said
this problem with three times already, this, whatever
decisions you're making involve the business because it has cost
implications. And as technology people, I
don't think we can decide on cost.
It is businesses decision to take and give them the choices.
Because sometimes they don't know the choices and they just
assume like using big words like we want asset versus you in
terkoz's, they have no idea what you've done.
So give them the choice in the show.
Oh them like a rubric or show them, comparative analysis, show
them. If I choose this, these are the
implications. If I choose that Nations and
here's the cost for a versus cost for p.
Not pick one. Right?
Because that gives them, lots of information on which they can
make a decision and it will make you a better Technologies,
because you are thinking as a neutral person, giving them both
choices, right? And that forces you to do deep
analysis and research, when you're putting those two
choices. If one choice is lots of pros
and the other choice is not sub cons.
Then they know that you basically want this and not bad
and that makes you focus more on.
And how do I analyze this? How do I make sure there are
Pros proper, prozac, both sides, proper content website so that
people can make. And I think it makes you a
better Technologies when you make someone else take the
decision and you are giving all the information.
Wow, I really love that better Technologies is not the one who
make the decision but actually they're giving the people the
chance to make the decision knowing the well-informed
information given by us from Technologies things rights.
Very, very lovely and also what you mentioned sometimes.
Yeah. Business didn't know what
options exist. They're all know is just like
jargons here joginder and yet they will just leave it to
developers to decide. So I think as developers we have
to give them the information in the may be easily
understandable. And yeah, decisions should be
taken collectively. I would say, not just my
business also, but also with the Technologies.
What are the implications of the decision?
I think that's where we need to be making decisions, probably
just toss a coin and make decisions but what the
implications of the decision Downstream our rate of X is the
key part. Yeah.
Correct. So implications is always
something that we regret whenever we reach that stage.
So I think you're the implication should be well,
thought out and discussed in the beginning.
So promote, if people would love to continue the conversation
here, would like to ask you a reach out to you asking about
data related stuffs, is there a place where they can reach out
and contact you online? Yeah I do have a Twitter handle.
Its promotes analogous at promotes evaluate.
I do also. In a website this is not a.com
so you can reach out to me from there and kind of like very easy
to spot me on Twitter and Linkedin my website.
So any of those forms are fine and I generally tend to reply
because I am super interesting data and its effects on people.
All right, thank you so much for your time.
I really love our conversation. And again, I hope people here
are much well-educated about dealing with data and doesn't
think it's a hard problem anymore.
So, thanks again. Promote.
Thank you, and we thank you for this time and thank you very
much. Thank you for listening to this
episode and for staying, right until the end if you highly
enjoyed it. I would appreciate if you share
it with your friends and colleagues who you think would
also benefit from listening to this episode.
And if you are new to the podcast, make sure to subscribe
and leave me your valuable review and feedback.
It helps me a lot in order to grow this podcast better.
You can also find the full show notes of this conversation on
the episode page, at Tech Legion o.f website, including the full
transcript interesting. Quartz and links to the
resources mention from the conversation.
And lastly, make sure to subscribe to the shows mailing
list on pack leader. No dot f to get notified for any
future episodes. Stay tuned for the next
technology. No episode.
And until then goodbye.