So it's no longer the case that you just need one technology
that we do all of it. In fact, in the cloud.
You will have completely dedicated Technologies, which
are meant to do. Exactly.
That one thing. I would say.
It's actually par for the course to re-engineer on, okay, that
happens all the time. This often does need to change,
right? I think that's some, in my view
inevitable, but I think we design principles, they hold the
test of time. For instance, if your bathing
business is business decisions, right?
On top of it, you have to Absolutely, give the guarantee
that the data is correct by the data is wrong.
Then you making wrong decisions in potentially affect people's
lives in some form. Hello, and welcome to data.
Shatter the podcast on all things data.
This podcast is a series of conversations with experts and
Industry leaders in data. And each week.
We aim to unpack a different compartment of the data
suitcase. I am your host got the chassis
that I'm a blogger newspaper. Columnist book author and a
former data adds a jury consultant at currently head,
analytics, and business intelligence for delivery.
One of India's largest logistics.
Companies. You can follow me on Twitter at
Karthik s that is ka r. Th IKS and read my blog at no
Intruder.com., That is n. 0e n t, hu b, a.com all opinions
expressed in this podcast, belong to be and iPod touch base
and it did not reflect the views of any organizations.
We might be Associated. Nothing discussing this podcast
to be taken as financially for me like this.
I will put delivery one. India's largest logistics
companies at the time of recording, we deliver around 1
billion packages each day and each package gets scanned about
20 times, at least, as it makes its way from origin to
destination. So you can imagine the amount of
data that we are dealing with. It's firmly in sort of Big Data
territory and as head of analytics, I have the job of
making sense of all this data and the first step in making
sense of data is to organize it effectively.
Today's guest is rangarajan, vasudevan founder, and CEO of
the data team. The anger was my classmate at
IIT Madras and then went on to do a master's in computer
science at the University of Michigan, Ann Arbor.
He was a founding engineer at Aster Data Systems, which was
acquired by teradata, in 2011, in 2015, vanga starting the data
team, which is building, AI solutions for customer.
Intelligence is also a guest professor at IIT Madras where he
teaches a course of Big Data, you can follow him on Twitter at
ranga vasudeva. That is our Ang a Bas ude VA n.
What is Big Data? Well, I think we.
The original connotation used to be about very large scale data
occurring at selling very high velocities.
And there was also this added connotation that this was data
that was traditionally beyond the way the classical database
is used to store data in an Excel sheets, in a spreadsheet,
row column that kind of format, but that that was when the
concept originated by back more than a decade ago decade and a
half ago. Yep, nowadays, if you really
think about it, most the most of the conversation is around,
definitely the volume, and velocity, that's definitely
there. And then, there is this, this
added implication that there are so many types of data that's
been generated in the, in the digital world.
Yeah, some of which is not very conducive for analysis by
regular methods. Right of of what Use as a
favorite visualization of the bi to e.
So it's a broad term that encompasses all of this.
Yeah. For the most part.
I think the volume and velocity has stuck on as Key Properties.
I would say thank you. Okay.
So let's just get to the problem statement.
Right? I mean if you think about the
problem statement, is that like you but nowadays it's you have a
lot of data flowing around. It's easy to collect the data.
It's easy to store the data because of the so called Big
Data and things like that, right?
And cloud and All that. So how are you supposed to sort
of organ? Now?
The reason we collected data and store, it is with the hope that
someday we'll be able to make good use of it.
That someday someday could be tomorrow, but like that, we
could do some analysis, make some use of it there, different
kinds of analysis that you can do on the data and so on so
hard, even with the cloud and big data and so on.
How do you organize the data? However, what are the different
sort of mechanisms in which data is organized?
And like how has that evolved over the last 10?
15 years ago? Yeah, I guess.
That's the one who doesn't things there.
Firstly, I think one of the key motivations right for why why
big data on makes sense the way it does right now is that
storage is always become cheaper and cheaper right as a years of
program. So the topic of whether to store
the data has become almost like a no-brainer for most companies
and even individuals, right? You get a 1 terabyte hard drive
for what three hundred bucks. That's really really cheap.
Yeah. So then the question is Is it
enough that you're able to store the data?
Obviously not right? But yeah, in at least a decade
ago, it used to be the case that people were throwing away data.
And that was because the traditional methods of storing
data which was databases and things that are or even storage
and networks and network attached storage is and things
that all of them were fairly expensive when it comes to the
form factor versus the amount of money that you're actually
paying for it. Yeah.
So as technology has evolved as it became more Monetize, the
storage equation. So it's no longer considered bad
practice, adjust or how much ever you want.
But then, I think the Crux of why Big Data makes sense in
conjunction with something like the cloud.
Is that along with the fact that you need to be able to store?
You also need to be able to plug in play, extremely different
types of accessing energy. Yeah, not all forms of accesses
are going to be applicable for During every consumer or every
every user of the data, like some are going to be really
proficient with like, Hands-On programming while some like the
business users will just need like snapshots of what they need
and they'll be people in between right who know some amount of
programming. Let's say a declarative stuff,
like a skewering. Yeah, and they'll always be
some. Some folks who are data,
scientists A machine-learning Engineers or statisticians, who
don't really understand the weeds of like, distributed
programming or anything like that, but they still need to be
able to access the data. So what, Therefore emergent was
this need to unlock agility. Like yep, and the unlocking of
agility first started with Hadoop and Technologies like
that. We should all predominantly on
for my strike in the data center kind of technology is okay.
So a little bit. How do emerge as an onto my
sting? Is it?
It didn't. It's not Cloud native.
I do when it first came in, it was done by Yahoo to basically
do a better job of indexing, and, and retrieving their
internal search data. Yeah, right.
And this was all done on Prem. So in fact at that time, how do
became popular because the old school way of doing things
classical data warehouse, database ways of doing things
were just extremely restrictive, right?
The technology was getting in the way, right?
Yeah. So I took was one of the first
Big Data technology so to speak, which, which didn't get in the
way, you know, in a sense of little of the anybody to just
put all kinds of things on top of it, to go answers and process
and do things with it, right at the - fundamental level that was
a very, very attractive price. So as a result it caught on like
wildfire obviously the commodity pricing model all of that helped
so it caught on like wildfire became extremely popular, but
then it imposed other kinds of costs.
Okay, and those kinds of costs were like, maybe we can discuss
that separately, right? But that's a different examples,
are the costs related to data processing and what does it mean
to reason about data correctness and things that all of that
become became? When problem statements, now,
with Cloud again, we don't journey of cloud started off.
As, you know, let me just run to your virtual machine, but then
as things progressed as Big Data evolved in its parallel track
Cloud started to pick up and say, look, I have all this years
of experience of having done, Big Data, well and understand
understanding all these other ways in which big data is
failing, right? And I know, of course is how the
classical Technologies our fingers That have really black
box. Two people have let me actually
learn from the best of both worlds and bringing those
features and benefits. Right?
And so that's the, I guess the single biggest value proposition
that cloud is brought to the data table, which is that you
get all the editing of variety of different types of access
patterns. And at the same time, if you
need that, if you need for speed, right?
Because of organizing it in the certain way, which is extremely
optimized for it. You can do that as well.
Right, so, Cloud has now gotten the best of both of these
old-school worlds and India, uh, agility.
And at the same time it's gotten his own complexity, right?
Always with Cloud things are not as easy as it seems on the
surface. So that complexity is also
something you have to plan for. So very, I would say, also will
calm, submissive become easier, but there are newer challenges
that have come in because of cloud.
Got it. Okay?
Now basically, I again going back to fundamentals.
I think we'll keep Back to find a bit easier here.
But like, so we are collecting all the data that we can get.
And then we want to store it in a way that is easy to access and
analyze and so on later on. So in your view from what you
have seen, what are the different use cases for which
people want to access data? What are the different kinds of
things that very broadly speaking across domains and
things like that? What are the different kinds of
things that people want to do with data that has been sort of
just collected and dubbed in one place and so on.
Yeah, I think the first thing to do, First thing that everybody
wants across the Enterprise has to reason about what is it
existing there? Which what it exists, right?
Yep. Yes, like the the first use case
and that's it's almost like a like a prerequisite step died.
Even before you start to planning start to plan other
kinds of things. So but classical beginner world
even reasoning about what exists is actually a very hard problem.
The reason being for see that there were the volume of the
data, just So large and secondly, the fact that it's
spread out, and at the end of the day, it's just a file
system. So, whatever you put in, is what
you get. So garbage in garbage out into
those kinds of issues are also there.
So, okay, let's first use case. The Second Use case is the fact
that you now have to start to know what has happened recently.
Yeah, and how is that comparing with what has happened in the
past? Yeah, and that's a very common
question that most decision makers at the business level.
Ask me. That's what we would call, SBI
devoting and things that are right.
Then the third set of use cases is accelerating and your, you
start to have the analytics types who want to look at things
like, okay? Yeah, I have this hypothesis.
Let me go test it out. So, they want to know compare
what happened. Let's say, lastly, while he was
just this, the vale here and let's see, what are the
differences in fraud. Right?
As an example. We have that's very exploratory,
simulations LED, right? Then the fourth set of use
cases, are people who want to learn from data and sort of
predict things in the future. Yep, that's the the data like
this model building here, kind of tight.
Right? Then.
You also have the special functions.
I would say these special functions are monitoring things
like overall risk to the company.
Yeah, right. Overall Financial is and things
that aren't even answering questions post by regulatory
bodies, right? And that's a very common tasks.
You have been you start to get into those kinds of special
requests. So those also are critical.
Use cases that are going to be put as a subset of analytics.
In some sense. This is not as so it.
Actually it's more. It's more a data dump like right
here, right as opposed to as opposed to anything else, right?
That's why it doesn't fall neatly into one of those other
categories here. The example here as a telecom
provider like you could get an F IR to you know to to share data
of 15 listed delcambre numbers, right?
Mobile numbers. Yeah, so that's a glittery.
You have to give it a time or the police authorities.
So there's no analytics. Are you just you just have to
dump it, but that will be from any time in the last seven years
has an example, right? Right.
Yeah. Yeah, and the last use is one of
the very important uses is also worked.
Can I keep improving my internal systems by analyzing the exhaust
that's coming from it. Yeah, and that is also being
stored in the same play. So now, I mean, obviously, I
mean the way I see it and so on, like the way you access data for
defeat, Each of these different things is very different.
So for example, if you were to look at a bi use case, you
probably only only need the last few days of data for some some,
some stuff, some other things. You'll need a longer time
period. So it's almost like you are to
take a little from your reading from there and so on.
So it's a different sort of an accessing things like that for
your special functions thing that you mentioned.
There. It is like going to some
particular place in the past and like and then getting stuff out
of there and it's like going to some unused basement and Finding
some file like they show in the movies and things like that.
I'm guessing so how so I'm assume that the way you need to
store and structure. Your data is very, very
different based on which of these needs are dominant.
So and I assume there's a sort of a trade-off.
So what are the sort of the real trade-offs here?
What are the Technologies? Or I would call it Technologies,
but what are the principles According to which you organize
the data to optimize them for some of these different
combinations? And use cases.
And I really like the word. Do you use to of fundamentals?
Right? Because I think this is exactly
is a fundamental design principles for data
architecture. Yep.
You have to think about all the different use cases that that
are typically existing just from a consumption point of view,
right? And and cater to it with with a
with a very important design principle, which is that what is
the least common denominator? Right?
Which actually serves all of these needs.
And if you take that LCD, kind of an approach, you will see
that - 6 natural into that site, which is that.
Let's just collect all of it together.
I'll beat with some curation. I can't just dump it all and
figure it out later on. You have to put some curation,
put some boundary around it. And those are all not just a
property of the use case, but also just good hygiene, right?
You need to know. You need to know how you're
getting something. Otherwise do not appreciate the
value of getting it right? That's a almost make it a life
principle. Yeah, but If you operate the LCD
LCD, thought process, like Occam's razor, or something like
that. The first thing that happens is,
you know, we built a big kid earlier.
Yeah. Now, once a player is done, then
each of these different use cases.
Actually kind of Branch out into their own storage options and
optimization of things like it. Rightly.
I do too, right. And here, the nature of
Technologies has as completely, you know, I would say evolve
over the last 10 years, okay. It's no longer the case that you
just need one technology that will do all of it.
In fact in the cloud. You will have completely
dedicated Technologies, which are meant to do.
Exactly that one thing and also sort of we have a very good
price point right to make it justifiable for you to kind of
invest in that kind of Technology the first days.
So yeah, if you just want to give you just go use case, but
use case for the vi side, you would absolutely need a way to
curate data even more than what you've done in the Big Data
layer, right? And here, the creation that
you're looking to do is not just reason about where the data came
from. But also reason about how
different representations of the same data look like, right?
And therefore, what do I triangulate and and promote a
steam, the representation that I want, right?
For my business. So that's a crucial task, right?
And then in a conventional sense, it's called as data
integration. But yeah, that's a that's a
crucial topic that you have to address and there are
Technologies which are very good at, you know, optimize can be
very good at doing data integration and here again,
there are two angles to it, which is the fact that many
businesses are, okay? Looking at, you know, data, as
of yesterday, looking back in time.
Yeah, right, but then we always businesses like, for instance.
I guess you don't employ a right who wants always real time data.
Yep, I guess hosting and data of what is happening.
Not is this. Ali analytics and machine
learning but just give me today's date are giving house.
Clothing company. Yeah.
We need to know, like you asked me where your packages for that.
We need to know where each package has real-time and so on.
So we see obviously we need the data.
Exactly. And you know, you want to know
how many packages are being missed because of the storm
happening in Gujarat, right? Exactly.
So those are extremely important real-time data.
And I would say that real-time traffic also used to be a
problem with classical Technologies, but nowadays there
are options available. Even for that in conjunction.
Would be decision-making that needs to happen on the vi side.
So on the pi site, I mean this is this is what one set of
Technologies would be would be doing machine learning.
I mean, obviously there are many many Technologies each having
its own pluses and minuses, but at the heart of it, they all
work off the LCD, which is the beginner storage layer.
Right here, very few people require.
I mean, what if you get assigned some machine learning technology
is really require you to curate in an additional storage layer.
May be there optimizations, which are used Abel like, for
instance, being able to stand up data in memory so that you can
hydrate much faster on it. Yeah, that's a, that's a pretty
useful technology. Some, some technologies have it.
The other angle here is if you want to do really large-scale,
complex data, science occurrences, apply, a team
Learning Network. Then you need other mechanisms
of exchanging data, so that the volume does not, you know, flood
you down and the task actually has a reasonable chance of
completing their. Our technology is available for
that as well. But those are again.
She likes technology is not really none of the men that you
would use on a daily basis. And in some of those
Technologies, obviously the computer, uh, plays a role as
well. Right?
If you would only use, he use whenever you can.
So that thing's gets better past other things that I'm on the
regulatory side, the special function side.
What you're looking for is effectively a way to go, go back
to archives. Like I said, and go look at that
specific needle in the haystack, or, you know, something like a
very specific type of query which is, which is not, which is
not something that, Look at the storage is optimized for so in
those kinds of setups. It's more important that you
give an answer less important that you give an answer
immediately. Of course.
Yep. Yep.
So therefore the trade-offs are different there, right?
Yeah. Okay.
Yeah. So again, I think I'm a little
more into this. It's basically the obviously
like different companies will have like different use cases.
Obviously one company might we may not be doing any machine
learning for example, and so we might have organized our In one
way, so what happens? Let's say because companies also
evolve over time brake. So for example, we might just
start by doing some tactical dashboards and stuff.
Then later on, we decide to add machine learning and then find
that data scientists at the cost of their queries is either is
very high either in terms of time or in terms of dollars or
whatever. And then like, let's see another
day. We start an analytic function
like it's a sort of a typical Evolution that a lot of
companies go through. I mean, not me.
Not be mature companies, but at least a lot of smaller companies
go through this Evolution. So so what do you do?
I mean, like, how do you engineer this?
Do you can't keep the engineering every few years and
it's difficult to predict also, right?
So what do you want? How do companies deal with this?
So, actually, I wouldn't, I would say it's actually par for
the course to re-engineer on. Okay, that happens all the time.
I mean, like I said, most companies evolve organically and
while that happens, the needs do change and needs to change.
The software does need to change, right?
I think that's some, in my view inevitable, but I think we
design principles. They hold the test of time,
right? They hold true.
So you have to think about, and it takes a really present leader
to think about it from day. One that let me actually collect
everything and I'll figure it out later.
Right? So, you take that kind of an
approach then be LCD, which is the big data storage layer that
stands the test of time, right? Because that's going to be to
come. What may then at various points?
You Branch off obviously? And one of the key like
conference is one of the key evolutionary decisions that most
companies go through is. I've always collected data on
that, but certainly now this is have evolved.
So I To get my answers done in real time.
Yep. So what do I need to go back and
change and the answer that is going to change everything from
the source application on words, right here.
Because we are, real time is not just a property of your pipeline
running faster. It's also whether the source can
actually give you the data on the first place in a way that's
much faster than whatever's here.
Is it meant for? So, those are all critical
decisions will have to go through that and that
regenerating is inevitable there.
But the part around, you know, the agility, right?
That's a very key thing because, Was what you don't want to do.
Is get stuck with a technology, especially if its propriety, or
it has two black boxes, right? That prevent you from doing
additional flexible things here, and more importantly that has a
cost of compliance, right? That there is a cost of
technical debt that you incur is because you're not taught
through the consumption pattern and you live with that cost
until the point that you're ready to throw away or until the
point. You're ready to bring in.
Brand-new technology of brand new use case.
So it kind of becomes a case of risk aversion and and cost
optimization than a case of, you know, can I choose the right
technology right now? Right.
So you have to think about that angle is well, right?
Then you can you decide something right now.
Yeah, and so you're saying that re-engineering is fairly common.
Okay. Okay, because I mean, what I
find, is that like sometimes you Certainly said, the some new guy
will come in and he wants to look at the data in a Cell, very
different way, for example, and that will be very different from
what has ever been done in the company and the current
structures and so on Earth, just completely unsuited for that or
impose high cost. So they're even case this new
way is going to be sustainable and so on, you recommend a sort
of a complete re-engineering retooling, kind of a thing
happened. Kathy.
I think that's a very important question to consider.
From multiple angles, right? I think, the first angle is
technology. Naturally.
There's no doubt about that. But the second angle is also the
sustainability of that kind of an approach, right?
So what is the question leaves? And there's a very common
question that lines us being most most Enterprises go
through, right? So then are you stuck with
technology? Which only one person
understands? Yeah, right.
And that's a, that's a pretty common problem in the world of
data processing and software, right?
I think it used to be the case that Ours was extremely popular,
right? And and, and still dislike for
another analysis and statistics in use it, but then now I know
exactly like many people do. And then you now have these
graduates who are just weird-- by python in, right?
For everything. And then they tend to think that
like statistics for instance are all possible just within python
in the world of python itself. Yeah.
Now, if you're a company who which is built up an entire data
science or analytics team that's just based on our and for
whatever. Moved on to their next greener
pastures, right? Yep.
And then fire, the all these pythons graduates in what you
do? With all that our investment
that you have already made. Yep.
You have to think about those pipelines all over again,
especially with a cloud coming in the pace at which the newer
releases of software, like, soften as a general term.
Yep are just data. They're not just data science to
get something done. These releases are just
happening. Almost on a monthly basis,
right? Yep.
So and each one has a like a, like a very good feature and
these are being tested. Third at massive scale
Enterprises much bigger than yours.
So it becomes almost like a no-brainer to adopt the fact
that, you know, like Netflix is doing it.
Let me do it right. Yep.
And then the moment you do that, then you start to realize that
some of these pipelines that have wilted obsolete.
You have to re-engineer them. So, engineering is actually
earned the Mantra in the cloud world.
I would say, I just want to see on some of the, some of the
technical terms, which have around this, which I've never
really understood people talk about, I mean, the common word,
select data warehouse. Then there's Literally, at
people say, Aroma some guy telling me a few years back,
data warehouses are now obsolete, everything.
See the data leak, but the way I understand it, like both of
them, sort of like, coexisting things are so.
So what is the difference between them?
And how do you, what is the? I mean in a Layman's language?
How's the, what is the difference between in terms of
how the data sets depending on how you choose the architecture?
I would say that. The data lake is probably the
least common denominator right wing, but he talking picture.
Yeah. I said, as the name suggests.
It's the, it's the point is the place where data naturally
gravitates towards, right? So here is the analogy is of a
water body and you have all these Rivers like coming in,
rather all these streams and abilities.
And even they would gravitate towards the point where That is
this equilibrium where the water can accumulate here and that
becomes the central place from where, what are the further
disseminates into multiple multiple streams for the
downstream and things that. Right.
So that's the that's the analogy which is quite valid for the
data Lake as well. Now, obviously the data Lake by
its very nature because data is gravitating towards it.
You don't really control typically.
What is the quality of the data? That's gravitating it.
So it's not something that you say.
I'm going to only allow secure your data, right?
Doesn't make sense because we have the lake.
The concept of Purity is a, is a is a Latter-Day construct,
right? You don't put it up, put it up
right up with the the beginning when you're trying to complete
our concept of Purity can change over time.
So what is pure now is not clear tomorrow.
So yeah, and in fact there is Merit in storing your data as
well. Of course, you have the other
angle of it right here. You need to know what improving
things. So now that's the data you then.
Not to W by itself because it cannot guarantee things like
Purity and quality. You need to have mechanisms of
doing that. The moment you have, you put
those mechanisms of guaranteeing Purity and quality you start to
construct more curated data sets.
Yep. Yep, and those security datasets
serves. Let's say a more specialized
needs like one of those use cases that we talked about.
We are another point. You start to think about how
specialized should this be? And what is the property, right?
Property of curation, that that is being critically relied upon
by that the business consumer and so on.
So for instance, if you're basing businesses business
decisions, right? On top of it, you have to
absolutely give the guarantee that the data is correct.
If the data is wrong, then you making wrong decisions in
potentially affect people's lives in some form, right?
We have so that process of creation and the creation of
that curated data set is a is what typically, you know,
classical world would happen in our data.
Our house, okay. Okay data warehouse becomes this
Enterprise void single curated zone of Truth.
Okay. Yep, right.
That's the classical definition of a data warehouse.
The reason it's enterprise-wide is because it is meant to serve
all business functions yet. The 8hr supply chain, like,
payroll sentence, Regulatory Compliance.
And and of course, the Top Line function sales, this Allison.
Things that I'm so because it was enterprise-wide, it, had
this very Grand appeal for the classical ideal 06, cios, and so
on. Who would think of it as a job,
like a multi-month multi, multi your kind of a project to make
sure that let me just build it out once.
And if I figured out once and from that point on, I can, you
know, wash my hands off and then incrementally, just keep
building things. And it will always like, you
know, survive the test of time and you're giving me the right
information. Now while that work wonders for
some of the large mature companies, which have a very
good process of curation. What ended up happening was the
smaller companies became smaller.
I would say, like nimbler companies companies born in the
digital Iran. They didn't need the Enterprise
wide view, right? Because here for them, heads up
with a completely different function from the Top Line
function. So it became more important for
them to build this for the Top Line function, like the business
in the marketing function. And that was a lot more privacy
and fight. Because if we have We happen,
not move. But as he had often could wait,
right? They could still operate out of
excel. Sheets.
Not a problem. Yeah, so then you start to think
about can I get great quraysh directed zones, like this,
specific to a business specific, to a function and that's where
this concept of a data. Mart, came into play.
Right? Okay, um, in the older days
data, Marts were always existing, but in the newer age,
it became much easier to create data Marts because the needs
were all completely disparate. Like it wasn't as though the Is
one person who's responsible for all those needs to be served
together. Each function, started to do its
own thing because we had to build their own agility and
build in own speed of decision making and things that aren't so
he demands became again a very popular activity right now.
So data lake is where all the double scope like and they're
from there you put in some quality checks.
You kind of, make sure that like, there's nothing data is of
the reasonable standard and stuff, then it goes into Data
Warehouse, which and data warehouse.
I am I from my memory predates. Big data and so on the correct
me if I'm wrong. Clean up themselves, very old
concept, I guess. So.
And data warehouse used to be like in traditional.
It's Enterprise while white. But so now data, Marts are like
they serve particular businesses, where does datum,
so, how is this? How does this connection happen?
Is it data Lake State to data Mart, or do, does it flow
through the data warehouse and also like when the data values?
Obviously like, I mean, I think the assumption that you build
it, once it serves you forever. It's pretty much doesn't work in
most places. In a few places it right away in
most places. It doesn't work.
So, does that mean you keep updating the data warehouse, or
after point? You just give up on it and just
let it be where it is. Send, like construct new data,
Marts in things like that. What do you what is, what is the
way to go about it? I mean, I guess it gets a bit
controversial right to see. My view is I don't think I don't
think of data warehouse is make sense anymore.
Okay. Okay, except for very
specialized circumstances, right?
Where it's a very heavily heavily regulated industry.
The extending matured. And therefore there is a bit.
There is more meditation centralizing the view of the
data and then decentralizing. Yeah, but if you look at now,
the modern modern data-driven Enterprise, the Mantra is that
each function is is onto its own, right?
They make their own decisions. They make their own schedules
and Agility work. Yep, and for that to happen, the
dissemination of the Has lot more important than you know,
introducing Layton sees and blockers in order to centralized
data, right? So yeah definitely vision is
already happened in the data lake.
So why create? And yet another centralized own
better you just disseminate as soon as possible and let the
functions themselves build their own views of what they want to
do, right? Yeah, but they live by it and
the debates that means you all of them to happen.
You all of that to happen. So that becomes lot more.
Hopefully, as a mammogram so directly answering your
question. I don't think the warehouse make
sense anymore, the classical way of looking at it.
It makes a lot more sense to just take the data from the
data. They can directly create these
purpose-built data Marts as soon as possible, but isn't the
downside of that they could create silos.
So for example, like for a company, I consulted for like
some 10 years back. There's someone there were two
days data sets which were being one, was maintained by the
finance. My one by the HRT is an external
person got access to both and emerge on that through some
insights, which they couldn't validate because the finance
team couldn't see the HR teams data and vice versa.
So does a data warehouse, I guess is a is one place where
everything is there. But data my tag is like, how do
you solve this problem of silo-based thinking within?
Yeah, and I think there is two or three simple things you can
do, right? One, is that they could be.
Actually, for instance, are like, like a Parent determined
so to speak, right? And wherever you see that there
are multiple functions requiring access to the same view of the
business. It makes sense to just create
one business data Mart, which serves that view.
And then each of these different functions without takes its own
cut. Somebody and that makes sense.
Obviously, right then the other angle here is the data warehouse
could be still very relevant technology, that kind of a setup
when there are many functions which want the same view, but
with the crew, You think the crucial difference from the
classical way of looking at it beam?
You don't need to boil the ocean to build that enterprise-wide
centralization, right? Yeah, you could just start with
something which is very simple, which is just given by these
three functional requirements of these three functions, want you
to look at the same data and as the functions, get on-boarded,
right as more and more people want the data, then you start to
see that there are patterns and then you start to like we've it
back into this form, a layer, like so logically, you could
eventually still end up. Building on like an
enterprise-wide data warehouse, but I guess the, the guts of
what I'm trying to say is you don't need to start by building
it, right? Yeah.
Okay. Yeah, so, that, that completely
makes is I mean, it makes sense to start to the couple of basic
use cases and then incrementally, build it out
rather than they spend a year just building a data.
By which today, I don't think any company has the time for
that in things like that. When you have to keep creating
data, might send you to create maintaining data might say.
So how do you organizationally? How do you see this?
Like we will do, see there's going to be a sort of a central
Data engineering team, which should do this or how, how
should this be sort of like structure?
Yeah, I think being the key is again, what is being guaranteed
to whom right getting? That's the, that's the, that's
the Crux of the requirements, right?
If you want to, just boil it while all that down to one
question. Yeah, which is that, if you're
guaranteeing, that I as a As a as a creator of a data Lake, I'm
giving you the access to any and all data produced in the
company. Like that's a very strong
guarantee. And if you're able to give that
guarantee, then that completely decouples, anybody who wants any
data from having to go talk to the source system directory.
Because the last thing you want is every function, talking to
every source and saying, give me a copy.
Also give you a copy also fight because it becomes like a can
cross him, if that order n square is problems.
So and it's extremely bad, right?
Again of a principal and whatever we do in technology, if
you say it's order, n Square. Imagine if now, people are
talking to each other like that, right?
I guess the night man. Yeah.
So from that standpoint, the creator of the data lake or the
owner of the data lake is actually playing an extremely
crucial goal, because he goes a guarantee.
Now, if you need to talk to the stores anymore, right?
Yeah, I'm talk to me. I'll give you whatever data you
want. I got of a frequency, whatever.
Velocity X attacks occur, right? Yeah.
So that's one. Getting me then.
The next guarantee is that, if you're in the moment, you're
putting a cure. Raishin, what you're doing is
actually also interpreting the data in a certain way and
therefore it's not going to be applicable for all functions
because some of the functions might want to interpret in their
own way, right? Yep.
So the some of the best companies I've seen they tend to
take a decentralized approach when it comes to consumption.
Okay, they don't they don't like the engineering team.
So they just say look, my job is to guarantee that it has
available and I've done that from now on.
If you want to build things on top of it, just go, okay.
I've made these interfaces Democratic, right?
We have anybody can consume and whenever you consume, I'm going
to charge you back, right? It's like a chargeback model.
Yeah, so I'm going to forget and I'm going to charge it back
saying, okay, you consume this much amount of data, but then
whatever you want to do on top of it.
That's up to you. You spin up your own engineering
team and I think that works better because it decouples the
agility from one of wanting from another.
Yep. Yep.
So again, in terms of again, there is a thinking in terms of
organization. They like some organizations
have. I mean, I'm coming to my Pacific
need which is like analytics. Like we're like some company
companies, have one centralized, centralized analytics team,
which takes care of all the analytics needs of the company.
Other companies sort of like mind.
We have multiple analytics, seems there might be some
centralized teams, but also like small analytics team will be a
marketing analytics team. There will be an HR and extreme
and so on. So so in that sense in analytic,
I again, it's a good debate in analytics, but also in terms of
engineering like you think the data engineering again, get
split across Teams like this in terms of especially creating
these data much. So let's assume that data lake
is own centrally and guarantees you.
So the 0 n Square problem is salt.
So the data lake is a good source of Truth for everyone,
but for people to build their own keep this not everyone is
adapted writing SQL queries, especially AI University.
So, how do you how do you sort of the war?
What are the trade-offs? Their eggs?
Warm? Yeah, yeah.
Yeah, I think we the moment you start to think more than one
function, right? And more than one function,
wanting the same curated view of the data.
Yeah, then I think that's the right time to ask this question.
Where should that curation happen?
Okay, where should the team decide who's doing?
The duration? Should the team report to that
one function of the other function or should it should be
a separate team? Which has, which has its own
reporting lines. And that's all, that's an
organizational question. I don't think there is any right
answer to it. But in my view, the data exists
to so business, right? Yeah, that's the fundamental
frequency. So if that's the case, then the
team that works for the data, should also be very closely
aligned with business, not the writing, of course, like so and
so by that extension analytics reporting data science and data
engineering should all be ideally be aligned to the
business. Yeah, not Beyond centralized
thing that's reporting into ID. That's my view.
To be more specific, the engineering that's required on
data. Ultimately comes in the form of
requirements from the, from the analytics team, which in turn
gets the requirements from the business game.
Like that's the logical flow of requirements.
So in which case the data General theme is actually not
operating in silos, right? It's actually operating very
closely aligned with what the business eventually wants to
see. Now I know is it the
enterprise-wide business that we talking about or is it just a
function that functional business?
That's the call that he loved to make and accordingly?
You like to take a poll on whether that team that the, in
fact, set of skill sets to be centralized, one doesn't need to
be aligned to the function or not for it.
And I guess the other variable here is cost as in dollar cost
rate because if you have a lot of data mites, then you might
end up like having the same data, duplicate it in multiple
places, which NF can push up your costs rather than getting
everybody to subscribe to the same data warehouse.
Absolutely. Absolutely.
And I think there is a, it's a very easy way to deal with it.
That's a classical concept has existed from the way in the days
of their housing, which is of course data governance.
Let so you need to have a governance team and the
governance team plays a critical role in ensuring that, you know,
there isn't too much abuse going on of the data right in the form
of even things, like, you know, people using it for nefarious
purposes or purposes beyond what it's intended like especially
with things like gdpr and privacy laws coming into play.
But it also plays this very simple goal, which is the fact
that what data sets exist across the company All right, okay, and
for every way in which people process data, is there, another
copy of it which which is resulting in that practice, so
that interpretation. So those are also actually part
of the remit of the data governance team and that data
governance team should report into the you know, what, in the
classical world would look like a risk function, right?
Yeah, because ultimately you're basing business decisions on
Based on data and the data is wrong.
It amounts to the risk, right? So grip of that data governance
in between as part of the risk of mediator.
Those God or Enterprise risk, and then they are responsible
for ensuring that you know, nothing untoward happens.
Or you know, people are not wasting time or resources or or
misinterpreting data. Okay, so, okay, got it.
Now, let's sort of like, I think we will take another step back.
I think a while back you had or find mention that we will
discuss this later that Hadoop imposed other kinds of costs.
So I'm taking a very big leap from what we were discussing
about how to our The governance and so on.
So what is this other set of cost that Hadoop imposed and
Liz's? It's perfect.
So I think he liked would all Technologies.
There are pros and cons and Nadu had its fair share of cons as
well. Right?
I think the biggest one was the fact that technologically it was
such a complex Beast. Okay?
To manage that, it was not, the didn't have the cyclic form
factor of, you know, let's just deploy something from from the
internet and then we're done, right?
To get right. A variety of different things in
sequence right often times. And if you don't do it, right,
then you have to go back and start again.
Now, the good part is because it's open source, there is
enough and more support that was available, right in the form of
community. Brethren.
People were trying to do the same thing here.
Then that didn't really calls the pain, right?
The pain was the fact that it was just a technologically,
complex thing to manage. That's one thing.
The second thing is that, The problem with the you know open
to anything can access pattern, is that you're not really
optimized for anything either. Yep.
So what this meant is why it's great for Discovery.
Kind of use cases data science. Use cases.
Things are just need to run without requiring to run
immediately, right? The moment you start to impose
business, criticality. Like you have to have the
answer. Now, the answer has to be
correct all the time that kind of thing, right?
Then you start to run into problems of You know that the
technology does not need to be supporting it.
Yeah, and then you have to layer your own Technologies on top of
it. Right?
So what ended up happening was an already complicated Beast.
Now started to have all these platter of an umbrella
technologies that were required in order to serve very specific
access patterns. And guess what?
Because of the, you know, the Landscaping so wide.
At any point in time, you could have a failure.
That one thing was incompatible with Another yes, of course.
Yep, right became a problem of, how do you keep this entire mass
of software certified against each other as everything evolve
in parallel, right? Yep.
Hello as a result. There was a lot of fragmentation
and that fragmentation resulted in enormous, confusion in the
minds of n. And n, The Price is Right what
to think about down like they are fighting to do the same
thing. What do I do like so
architecture became a problem. Like, how do you reason about
something because Was very easy to reason about in the past
noun, you couldn't do it anymore.
You need to have a separate skill set to reason about
things. And I think, one of the last
times last thing I wanted to point out there in the Hadoop
world that came in much later, was that as newer Technologies,
became lot more popular, especially on the cloud because
of the very fact that term, it's community supported the moment,
the community went away. Yep, you know, we're stuck with
something that is not just obsolete from a technology
standpoint. It Dooley didn't have.
Have any kind of support anymore?
Yep, like so what if it's open source, we doesn't solve the
problem. Right?
Just looking P. Inside the code, doesn't mean
your life is any easier? Yes.
Oh, yeah. There was a fallacious amount of
thinking. They're right open source means
white box axis, and I don't need to think.
I don't need to worry about being locked in, but actually
open source. Is also a form of log, n, right.
So then that became a huge challenge for many companies and
then in fact, what couple of the very popular libraries which
actually started off. I do No in a big way, they were
all just recently announced by the Apache Foundation as being
end of life now. Okay?
Okay, so what happened? So so how do pregnant all these
problems that you mentioned? So I'm sure we have figured out
a solution for that. So what what is replaced her?
How do, how did that transition take place in so on?
Yeah, and I was kind of mentioning about this before
Cloud, the cloud players, right? Really were clever about it.
They just took P the best of best of Worlds, right?
So even some Hadoop, they took all the pros of flexibility
agility Open Access patterns and things that.
And then they papered over the corn side of it.
See one of the things about Cloud, right?
It's really driving home. The value proposition that
Administration of software administration.
Of technology is no longer something that any Enterprise
needs to spend money on tight. That's actually a very key value
proposition of cloud. So we have you do not need to
spend it admin time, trying to manage your manager machines,
right? Because it's just manageable
automatically by its own using software using automation Yellow
by extending that logic. They really address this first
pinpoint of Hadoop, which is this technological Beast.
Now, you could argue with the cloud.
Some providers for the cloud is also extremely technologically.
Complex is so many Technologies, so many ways of putting things
together, but the good thing is called also provides the
automation on top of it, right? So you can just fire up readily
available, automation scripts, and then just spin up this
massive Technologies and spin it down as well, keeping control
and cost, keeping control in the complex plane.
So they extended that Paradigm for Hadoop and that solve that
particular problem of, you know, how do I get this entire thing
up and running? So, so now, when Cloud, right?
It's a As simple as just positioning using a button.
Click, of course. Yeah, you get all that.
Pick it up - strike. So that's a big thing.
The second thing is that the fact that Hadoop at this problem
of not being optimized for any specific access pattern there.
The mean the way Cloud Solve It Is by by its fundamental design.
That cloud. Fundamental fundamental design
is hurt. Everything is decoupled and I'm
going to do something that's very specialized for me by
myself. And for everything else, I defer
to somebody else who's better than me.
Me, right? So it's also called this.
It's also called as a micro services, or Services oriented
architecture. Yep.
Cloud is fun. We build on that.
Right? So everything is a service and
if I if one service needs to do something, which is not its core
capability, it will Outsource it to some other service together
delegated to some of the surface to get it done.
And then once that service gets it done, it brings it back in.
So, as an example, if I were to now construct, a big data
pipeline in the cloud storage is a dedicated service.
Yes, like being able to run. Data science model is a
dedicated service like Sage maker on it.
There's an example right now. Once the model is run.
I need the output to be visualized as a, as a, as a
pretty chart. That's a separate service
altogether, right? And the way that all these
things need to talk to each other, the orchestration that's
a separate service and for all the things to work correctly,
like in terms of like failures and things like that.
And alerting logging such a separate service, right?
Yep. So in putting it all together,
there is automation. But automation will say, okay,
all these things need to talk to you.
Juror, and, and if I need to do something very specialized.
I just need to spin up service and add it to my automation.
I'm done. Like, so the access to optimized
ways of looking, at data in the cloud is, as simple as just
configuration. Yep, and I don't need to deal
with this plethora of Open Source software, which I don't
know who's going to support is not going to support that
ability. All of them.
Now, taken care of right? So Cloud solve for that in a way
that is fundamental to the way the cloud is designed.
And by Cloud, I guess you mean companies like Amazon.
I will make this up which provide these big hosted the
clouds are. So let's say AWS manages this
thing about they produce a Jamaica the told you they see to
S3 all those things and they provide you the connections
between them. You can spin up whatever you
want at any point in time and like they manage the whole
thing. So so that's so that's the value
that each. Okay people.
I think we're now tying back all our discussion, right?
Leg in terms of how hard you have you organize the data of
you kind of what big data is like the what are the pros and
cons of Hadoop? How the Technology has evolved
into one now. Suppose.
We are like, let's say there's this new startup.
Okay, which currently doesn't have too much data, but like,
you know, that you're going to be collecting tons of data and
things. As a, how do you kind of go
about architecting your entire the data team for the lack of a
better phrase in terms of like so that it's geared up for
growth but also likes of which includes things like how how you
store your data, how we kind of organize your databases and all
those things. Like how would how should a
company? See that starting of now, look
at it. Yeah, I think we at the first
term choice would be perceived. Just just to decide the cloud or
not, right? Yep.
I think that's the first question to answer in my view.
The cloud is a no-brainer. I don't think any setup would be
wise to not consider the cloud. Even the innovations that are
happening, that, of course. Well, let's assume the answer to
that is yes, right? Yes, Lord.
Yes. Lord is upon me.
Now, if cloud is a must the good part is the Architectural
patterns that that are applicable for constructing a
data estate on the cloud, right? When you can follow the latest a
data platform, whatever it is. Yeah, but the architectural
patterns are or, and in such a way, that it doesn't matter,
whether you're small or large. Okay, I guess we'll start with
the same architectural pattern. And as you grow as a company,
they are the execution of the DACA texture, can just
seamlessly grow along with you like without having to change
anything significantly. Yeah, and that, that's the
biggest benefit of doing. The cloud because each of these
components are individually, elastically scalable, right?
So as an example of a new company, you're collecting
starting to collect data. In fact, I would say data, is
one of the critical modes right for most startups.
Yep, because you need to build that mode over a period of time.
And the best way to build up mode is actually round the data
Lake, right? Yes.
Okay, out of the Crater Lake, just collect everything.
It's Dirt Cheap. It doesn't matter what you put
in, obviously, the more curation you can, Do the better it is for
you in the long run and the simplest form of creation is
that you can do is just track where the data is coming from,
right? They get what is called as you
know traceability you want to be able to trace back like five
years from now that this data was actually collected by that
version of the software that I deployed on that particular
machine. Yep, if you just able to reason
about that, that's more than enough, right?
So the data link is a very good foundational pattern after you
need that and then don't spin up Technologies until the end until
and unless you need them, right? Yeah, I think you can put Mantra
and that matter of holds true for taking for data as well.
Right? You don't need to understand,
sorry over engineered with a warehouse or data Mart and
things like that, maybe to start off with you just have a very
simple open source database like postgres a my SQL, and that will
get you off the door and keep you running for two years and
three years without entering much cost at all.
Yep. And then the more you the more
you play with it. Then you realize that the same
database serving the need for both your source application as
Well as the need for analytics is kind of creating a
bottleneck. So at the point you start to
say, okay, maybe I'll spin up something else using my data
Lake as the source, not my post as a service anymore as a perm,
like a brand-new, like, on Amazon could be like redshift as
an example. Right now.
Let's plug in my the. I do not top of it.
I started to analytics. The moment.
I do this have decoupled down the source application from my
analytics. And then, as you start to build
up on top of it now, see, other thing to keep in mind is that
Most data driven businesses. Ultimately want to take the
output of analytics and tied back to the source application.
If like, as an example, if I want to influence a customer
who's actually using mice by application right now,
obviously, I'm collecting all the data but then the analytics
house as to feedback as a recommendation as an example,
right way. So that ability to close the
loop back to the source application is a is again a
critical capability and that's something that you want to do
from day one because it Alfred you might not actually have any
intelligence, it could just be rule-based, right?
Yes. Again as you mature, as you
change these data pipelines and that becomes a lot more
sophisticated, feel the looping mechanism allows you the
feedback from day one. So that you know, that that
feedback is also something we can keep learning on.
Thank you for listening to data shatter.
If you like this show, please leave a comment, share and
subscribe to the podcast. You can find this podcast on
Apple podcasts Spotify or wherever else you go to get your
podcast. Once again, the staff exciting
one. Thank you.