Now, the role of be either for us evolve into not just
providing data to essentially what I call a identifying the
the needle in the haystack, right?
When you're given a data science is part of an engineering team
or as part of a an end to end product, where you are model is
embedded. You're solving for actions,
whereas it is in bi or solid footballs gravity, and I also
keep encouraging the bi developers in my team.
They don't call yourself a microfiber Jeepers.
Don't call yourself a cab. Don't call yourself about here.
We have our. So you are essentially a bi.
Hello and welcome to data. Shatter the podcast on all
things data. This podcast is a series of
conversations with experts and Industry leaders data.
And each week. We aim to unpack a different
compartment of the data suitcase.
I am your host, Karthik chassis that I want blogger newspaper,
columnist book, author, and a former data Etc conceited.
I currently head analytics and business intelligence for
delivery. One of India's largest logistics
companies. You can follow me on Twitter at
Karthik s that is Kar Phi. K s and read my blog at no
Intruder.com. That is n. 0e n t hu be a.com.
All opinions expressed in this podcast, belong to eat and my
podcast this and I do not reflect the views of any
organizations. We might be Associated, nothing
discussing his podcast, should be taken as Financial or legal
address. I would be lying.
If I were to say that business intelligence does not have a
branding problem. What if period of time, the term
has evolved to mean, a very specific kind of software
engineering, which involves connecting database.
Visualization tools. However, business intelligence
is not necessarily limited to that my teammate delivery.
For example, does it fair amount of statistics and over and
modeling and machine learning and make the odd report?
And we are called the business intelligence team.
And originally if you think about it, business intelligence
was all about making business decisions through the
intelligent use of data. So my guest today also works in
business. Intelligence is name is Balaji
Cooper Swami. And he's director of business,
intelligence products and data infrastructure at YouTube.
He's had a long career in data and analytics and decision.
Support having worked at Captain one McKinsey City Mo Rocca of
Commerce and lows. So you're the director of
bi-products and data infested. What exactly does it mean?
What do you do? Yeah.
So, one of my core responsibilities is to kind of
look at in the diverse sources of data that there is and
connect it to the needs of the business and needs for
decision-making that various stakeholders have beer in terms
of partner management of weed. In terms of advertisers
management. My job is to connect the data to
actions that The they want to take based on the based on
information that they have to ask about.
Generic question. What exactly do you mean by bi?
Because I mean just if you think about it, all the terms related
to data, if you think about it, it means different things to
different people. You say data sends as 10 people
about data science. Until you did ten different
sure. Yeah.
So what is your definition of business intelligence?
Yeah, so when I think of business intelligence, So there
is a saying we call right data's, worth pennies and
actions are good dollars. So, I believe that the role of
business intelligence in an organization is to convert those
pennies into Dollars, which means that there's a ton of
data, especially when you're in a business, which is digitally
focused. There's just so much of user
data being created, so much of operational data that is being
created. How do you kind of take that and
an influence the decisions and actions?
Since that the that variety of leaders, want to take across the
organization V8 starting from say, an analyst where I see to a
manager. We could be a product manager
with an icy. You're all the way up.
We'll see. So that I believe the job of the
bi is to make their lives Easier by providing insights and
actions, that matter, to them based on how they can to solve
for their. Okay.
Ours and to solve for their results.
Okay, and what do you mean, you think the market?
Things VA does this tool? So my I think that doesn't mean
bi is relatively old term. I maybe there's a probably need
to reinvent the terminology of what you call BI, but but be I
started off and like, older days where sap and Oracle it
basically meant data warehousing and Reporting, right?
Okay. Yeah, and that is what generally
people think bi is essentially a job of just collecting all the
information and The spitting it out.
Whereas, whereas I think with the with number one, just the
volume of data exploding it. It would have been fine.
If you had like five metrics that you want to look at.
And that's all you needed to do. But whether demand for data on
the hunger for data increasing across across the across the
organization, as well as the volume of data, also
continuously increasing. Doing the same thing,
essentially is becomes very pointless exercise because there
is too much data to do any analysis.
So, Now the role of B A. Therefore has evolved into not
just providing data to essentially what I call a
identifying the the needle in the haystack, right?
There's a big case type of data, so much.
That aesthetic that how do you identify the needle?
That matters to you? But how do you identify the most
important aspects of the data sets that matter to you?
So that you that can be? That can influence your
decision-making on a daily basis.
So that is how I believe it started off.
As a hey, I'm going to collect all of those information from a
variety of sources into a data warehousing system, which is on
opened some reports. Yep.
To what I believe now is is turning that and using using
algorithms and statistical models to find out what's
actually going on in the inside the data.
And we only surfacing data that matters only surfacing insights
that matters. Okay.
Thanks. Okay, I think Our definition of
bi is broadly sort of aligned with my definition of bi and
what we in my company also believe that he is supposed to
do which is like sort of help Drive data into actions, help
help, help the business. With the sort of intelligence
that can be derived from the data to put it in a very simple
manner. However, I mean, what I find is
that I have a bit of herb marketing problem in.
This is, like, recently. I was hiring, we very quickly
agreed on the sell very quickly and things like that, but the
guy was like, can you I mean something other than bi because
in the market there is a bit of a Down Market, sort of a field
to the word GI and this is something that I've heard from
other people as well. Not necessarily people have been
hiring, but others have also mentioned that like, because bi
has gotten sort of become synonymous with people who, like
they'll ask, how have you worked on that Tableau or quickly
knowing these tools knowing how to build the pipelines and
things like that. So so I mean I find that The
chicken wing ba has got a bit of a marketing problem.
I don't know if you have faced that as well.
Yes, and yes, and no. No, I think it depends about it
depends. But thinking about how the
vision for the, bi is usually pitched to.
And I think I really thought when you thought about bi again,
right? The historical job, that'll be,
I professional used to do this. Is essentially, just a data
collecting Under The Dumping job, which may have, which is
exactly. As you said, right, like a take
build all these data pipelines and put it in this, which I
believe is immensely useful guidance cement Foundation, that
is required to essentially do anything with the data you want
to add. But what from a from.
But I think where the the role formal rule for how it is
evolving, now it is thinking about how to stink like a
product manager, right? So I feel it is more.
Or actually a product rule versus are versus a, just a bi
room, right? So go.
So I also think that there are very specific skill sets.
Rather. There are parting about three,
four layers of skill sets that are required in order to be a
well-rounded bi to form a well-rounded bi team, right?
I don't think I can probably have a one person who does
everything completely. But what I think, two things,
Fit like a product is what is what will Elevate the marketing
for BF. But I don't think people look at
it as a product because the think it's still a data dump,
right? And that is historically, it's
been associated with that fact, but when you start positioning
yourself as a data products team, hey, where your job is to
build scalable Data Solutions. I mean, think about it from a
standpoint, a lot of software as a service business that has been
set up and it really A lot of software service which have now
a i animal tagged to the hair to their names.
In the end. They have a huge component of
their work is bi. It's actually about collecting
data from a variety of different.
Maybe your client, maybe, third-party, consolidating them.
And then starting to surface them in different ways or not in
one form or the other day doing a roll of bi.
So I think, I think the marketing problem exists, but I
have found success in positioning, it more as you're
building. A product and out of the product
is actually understanding the use cases and then connecting
the data back to that, right at the decision that use cases and
connecting the data back to that.
That is how I have found success.
And but when you actually really look at when you split the word
business intelligence, right? That is why I like it is, it is
basically just artificial intelligence, but you're calling
it business intelligence. But yeah in the end of the day,
that's what it is. So so I think there is a stigma
associated with the term. Be II.
But I think we can use the, I think we can use the mahout
where the market is moving with respect to actually building
data products and thinking from a product mindset thinking from
a design mindset and then bringing all the data to solve
for those use cases to essentially, to essentially show
to the candidate or shoe to your team members, how much of a
broad range of skill sets that you can, you need to bring to
the table. And also, you can pick up why?
Does rolled and that is probably the sets you up into a role in
product manager. It sets you up into her role as
a data scientist because you will be doing transporting, it
will be doing anomaly detection that again as follows.
Yep. It sets you up as a role of a
user design because you'll be actually it's like understanding
decision-making. In a business is one of the most
important aspects it out spit. Also it also gets you up to
speed into the most important factors in a business, right?
Like think about it. I always say that.
One of my first jobs and I went no Capital One was to do
monitoring. Okay, and, and within six
months, I knew so much about the business at not even an expert
in our particular field, New because I used to always sit
with the executors. I used to understand how they
look at the business. And yeah and solving for that
also gives you and bi also helps you kind of grow and as a
leader. Because are you starting to look
at the business from top-down also from Bottoms Up, of course,
so I think it just, it just gives you a broad range of skill
sets and obviously the tech skills associated with staying
up-to-date with Streaming data that's data and then injesting
that. So it has a full gamut of
literally what you need to know to be an analytic professional.
You can Branch off into wherever you want to go after that.
Yep. I see.
So I think is a good time to for me to kind of read out one of my
favorite quotes. I think I have, I have a
newsletter on data science which are where I probably use this
code three times already. So this one is it's by this guy
called Kay Kyser Fung. He is a Kaiser.
Adjunct professor at NYU. I like Columbia or NYU.
So on this block he had written this a couple of years back.
'Hey, I'm quoting here. The data science Community is
guilty of talking down on the business intelligence function.
There's a misperception that bi is for Less skilled people doing
boring things. The reality is, there is more
science in bi than, in so-called data science, science of, after
all is about figuring out. Why things are as they are
Engineers by contrast, use our understanding of science.
He's debating touch. Yes, and I think that very, very
relevant to where be, I was thought to be a bias is evolving
or where their vision for being needs to be exciting.
It's very relevant. Yeah.
It's like because you need to kind of like if you think about
it, like it's almost like somebody will be like you could
be building a simple sales dashboard inside.
And somebody could be the question that will be asked, is
why did sales go up this month? Let's say, for example, it's a
good problem to have. But why did the sales goal you
need? Need to be able to intelligently
dig in and the better you can present your information right
in the dashboard. And so on rather than let's say
I mean in some cases I've seen like especially when you have
companies that treat bi as a function to be staffed by people
less skilled people. What, what do you end up
happening? Is there's a constant back and
forth between the business and the bapi saying that be Isis.
Sales went up business, has why did sales go up?
Then the be Isis or sales. Went up in California?
Why did? Is the go up in Calico?
So this is back and forth. So the I think in some sense is
one job, one job of the bi to sort of pre-empting solve this
problem. Right, anticipate?
What questions are going to come and then like present the data
in a more intelligent Manner. And as you said, like, sort of
pick up the needles from the Haystacks in sort of present,
what is actually required for the manager, rather than, like,
just just dumping the whole thing, because that how it was
meant to be done. Yeah, and a lot of times I have
seen demos of dashboards, even from vendors.
They come unto me o hero. Is a top sales and then they
give you an inability to drill down drill down, drill down,
drill down and I keep asking them the question, but why do I
have to do that? Right?
Yeah, like when when I know that the data when I know that the
gold is lying underneath. Why don't you can why can't you
just bring up the gold upfront to me?
And then why do I actually have to do so that is really the role
of Pi? So what's your what's your view
on interactive interactive dashboards ended?
So what do you mean by interactive do interactive
dashboards again, like something like, I mean like where you as a
so. So again the way I look at it
like dashboards can either be like static or interactive.
Static is like I say this is. This is how things are, and I
control the message and you just consuming interactive verdict.
Okay, if you take it for yourself and like you can do the
cuts, you can do your drill Downs, whatever.
And you figure out what you want to do.
So, what's your yes? Yeah.
Yeah.
No, no. God encourage.
So. So I think the road Of bi is
very, this is where it is different from data science,
right? Where you think about data
science and embedded data science, the the their majority
of the time the decisions are already made by the algorithms
are made by the models and you are essentially living with the
the cost of miscalculation versatile model does, right.
So that's part of the life. As you're automating that
decision, yet. The role of business
intelligence do is very different from the standpoint
that there is more or less. Usually a human layer.
For the decision is actually made figure.
And, and to understand that human layer and understand.
How do you solve for that this? How do you solve for?
The right decisions? Made by the human layer is very,
very important as part of when you think about the entire
bi-product. Yep.
Why I say that, bringing it to the context of the two things
that you mentioned writer. Static forces interactive now
now depending upon the user, depending upon the person who is
looking at the dashboard. There are variety of You must be
said that they may want to solve for and one of that could be.
Hey, I want to know the what happened.
There are a few folks who are like, hey were probably
Understand the models to understand the data science is
going behind a day. Say hey, I trust the information
and then I need to move, but there are a few folks who will
be come back and say, hey maybe that job, that pnl there is
their bonuses dependent on it. I cannot take a decision that
your model is putting out me or the what you are showing me as a
as a god-sent answer. I have to understand why that is
there because at the end of the day, it is why my neck is on the
line for the decisions that I'm making.
Yes, I believe that is an art of showing the recommendations
first, but then explaining why is it such a recommendation
happened to an interactive manner.
Is what I think our dashboard should solve right interest
rate. It should not be a less of.
I will give you the word and you can explore it.
No, that is not what it should be.
You have to neither. Here is the recommendation live
or die by it. The second.
I think the answer is little bit between which is to say.
Here are the recommendations and then you should immediately, and
then ate and ate It and it should be interactive enough to
take you through. Why such a recommendation was
made and what is that slice of the data?
Because finally, there is a raw data Trend that you have
captured in the business intelligence that is subsurface
top. And also it is great to actually
have a human layer of validation of that to see whether your
models are acting picking up, the right stuff for Mark.
So I think they're, I think the answer is a little bit both, but
instead of interactive from Top down and started from the below.
So I believe there's a, there's a, there's a marriage that can
be made to both of us. Good.
So, okay. So now we have grabbed the kind
of father. I think you've got the you mean
comparing bi to data science. You've been sort of talking
about how it's slightly different from data science and
so on. So I mean and I think you and I
sort of believe that there is some amount of data science that
goes into bi. But I think the unfortunately,
the ladder Market income statements, that doesn't really
believe that because it is it's about to order the.
So if fix it, if you are a data scientist working in deai, how
does your job apart from accounting for that human
element? The that's there in the
decision-making. How does your job sort of vary
from being a data scientist on the, on the product side?
Where you're sort of, like, where your output is going into
the tech product? And things like that?
Yeah. Yeah.
Now this is very different. I think the very different from
standpoint that when you are, when you are given a data
science as part of an engine. Team or as part of a, an end to
end product, where your model is embedded.
You're solving for actions. Whereas it bi or solvent for
causality, right? So there are two different kind
of problems that you're solving. And there is really I mean, in
fact if you look at it as you said, but of the the big motor
design, I'm going back. What the professor said is
really understanding causality, right?
I think for us as a business intelligence when I think about
it and if you were Ready. Dissenters, right?
I think there are. I see there are three layers,
right? One is getting the raw data.
Is generally done by your date. Somebody who's talented as a
data engineer. Alexander is a sec.
The second layer is what I call the cap.
The metrics. Calculated metal clear.
This is basically you're aligning on cleaning up the
information and making sure that the data and the metrics for
calculating sales, you know that your poxy optimism for the right
sales number and so on. Right?
Yeah, and then the last next layer is actually, I think is a
model modeling layer, the modeling layer there.
Is is about Trend detection, which means that you've got
understand? How to detect Trends, right?
Because you need to have a benchmark need to understand
where you're comparing that. Well, how are you?
What you're comparing it again? So now there could be a variety
of different benchmarks, right? And especially in an environment
that we are in. You cannot use your ear over
here as a as a benchmark. Just live with it right out yet.
So you have to think about ways in which, you can be smarter
ways in which you can actually use machine learning to
identify. What, what should have been
expected and what is the trend? And, and whether or not The
actuals are along those Trends and and there are also ways in
which you can triangulate through through rules based
algorithms as well. Right?
Yeah, in terms of triangulating through year-over-year,
quarter-over-quarter benchmarks, percentile, benchmarks, and
there are so many ways in which you can calculate Trend.
Don't thinking about, how do we do that?
And calculate the trend itself is one of the big on blocks that
you can give to the business that that they don't have to do
it manually. Yeah, then there is an aspect of
as you said, right figure. An hour depending upon the use
case that there could be a normal use that you want to
solve for because there is always something that is broken,
be it in terms of in terms of credits.
It would be it would be a default rates.
There's usually some kind of a classification model angle that
comes into it in almost every aspect of business where you
have to think not only from from a linear Trend standpoint where
you have to catch those catch. Those escalations or cats.
Something that is broken or in the retail World, especially the
pandemic. We all know that the supply
chain went through a huge Crunch and and orders were not being
able to fulfill and they were getting cancelled.
So solving for cancellation. There are different ways in
which you can apply the models to say, how can I prevent
something from happening one? So, let me take a step back.
Right? So one is to say, Hey you cute
is what happened? Yep.
Second is the cutest. What happened, and what?
I say words of what behind a wall, which is yes, a third
needle, the segment, the most important segments, that move
that method really is your just one second.
Yeah. Sorry, it's my alarm going off.
So just going back and saying, hey, there are three aspects of
it to four aspects of it. First is understanding.
What, what is the data? The second is understanding the
segment of the data that really matters that you need to look
at. Yeah, and then the third is what
are the cause of drivers for that segment.
So that is when you think about actually good actually thinking
about not just going drilling down what correlating that with
factors that that you can move. Needles that you can move,
right? And then the fourth is, okay.
How can we get ahead of the problem, right?
Rather than now, even till now we have been detected.
No problem. Yeah, it's job is to how do we
get ahead of the problem and solve the problem before it
actually happens. So, so when you think about
this, Literally, after solving for the data, almost everything
that I explained is generally a data scientist job.
And especially when you're thinking at scale, it is not a
it is not a hundred line axle that you're working on.
You're probably working on gigabytes and terabytes of data
with probably thousands into thousands of Collins to look at.
Yep. So from that standpoint, it
cannot be done by with pivot tables or anything else.
It's got to be solid at scale. It's got to be solved through
through Machinery. Yeah.
Okay. So coming to machinery.
So, I mean let's just go a little deeper here, right?
Like so when you are building a machine that, again, the
question is similar in terms of like, if you're solving the data
science problem that the, you're solving for Action, which goes
into a product versus if you are solving for a sort of a bi
problem, where the where you're solving for to find out
causality and things like that. How do you, how do they ml
models themselves change? Is there a certain kind of
models that that are favored for the data science kind of a thing
and certain kind that you would favor for the I mean in terms of
large classes and so on like that you favor for the bi K Pi
world and so on. Yeah, I mean it's a very
interesting thing to think about, right?
Because when you are very when you have an and this is how I
think about it, right? And again I go back goes back to
why are we building the model for each of those use cases,
right? Yeah.
Building a model in a product to actually make a decision.
So so not to to actually implement the decision that the
Already been, right. So so from that standpoint you
have to, you really have to optimize for, you have to
optimize for the right decision. Right?
Which is BS that verses explain ability.
When I think about how much you have to explain a model versus
how right a decision has to be. Yeah, you solve for the decision
being right versus experiment in bi while.
Many times. I have seen that we end up, we
end up doing classification. Models.
And we end up trying to figure this piece out.
And we and really a NN model comes out as a, as a as the best
model yet. What we end up going with the
logistic regression though. It is less slightly less
performant. I can actually easily explain
the factors that is going behind those logistic regression to the
use to the users of the tool and and therefore can explain to
them what happened and that enables The role of bi, which is
to actually power and decision versus actually just implement,
or just do the decision. Right?
So, I think that is where I see, as a big class of difference
with respect to how a model, how you should think about a model
in a product setting, versus how you should think about a model
from a, from a cause of a risk, especially aspect, you know,
trying to explain setting. That is 1.
Then the second piece also is that when you again, just
because of the fact that there is a human error associated with
that, you got to be very careful with respect to the input
drivers that are going in as well.
Right? Yeah, not be in a several times
in a product said embedded model has seen like people just dump a
huge set of feature set. And then then the model.
Do its job right here. You've got a be in a situation
where you actually have to be very careful in thinking about
what are the factors that opinion?
Because in the end of the day you're gonna surface that you
have to see whether it makes business sense.
So I think there is a, there's a, there's a you go to A little
bit more from a standpoint of actually put yourself more and
empathize with the business user in a bi V than you have to in a
world where you're actually building a data science model
for in a product s. So the oak is likely cycling
back sake. So you seem to have a bit of an
unusual profile for somebody who's working in bi and so on.
So, how did you get it to be? I ache.
What, what you? What was your journey?
Like, I know, you were in Capital One and Mackenzie and so
on but what is your journey like?
Ya know, I think the interest Eating crediting curve, as I
said, initially started off at Capital One, a thing as I told
you one of my first jobs was to essentially own portfolio
monitoring. Right?
And I and I and I saw the power that it had in terms of just
teaching your business and and and almost always I are a
realized that the whenever I could went into new business and
build up seeking a role like that because it immediately
taught me the business very quickly, right?
So so I understood the power of And I went into good and then
and even abstract adults were on the time and when I went into
Mackenzie as well, I started this, that's when you remember
Tableau and all of those things are actually coming out because
all the monitoring I did before was pulling in from from
teradata and then, and then writing me be scripts to
automate it in SF in Excel and all of that stuff, right?
And yeah. And all that's what I used to do
in the past. But then you started seeing the
tools that will coming in like Tableau and other solutions that
were actually the surfacing of. And I built a lot of in fact,
even in the neck. And they did a lot of projects
where, where where I essentially help think about understanding
say, Call Center Performance and things like that through these
trees through these realization to.
So while believe in never Left the concept of doing the bi
work, but at the same time, it was about how do we.
But I, but it in this inner-city, the first phase of
my life. It was about just learning the
technical skills. Improving.
Hey, getting the data together dumping it and figuring out the
trends and so on. Yeah, the second phase of it
mackynzie was about usually I was is the work I did was part
of her strategy product. So it is about how do I how does
that connect with strategy right corner?
Center was an example. Other one was an example where
there was literally, no, the client did not even have a
mechanism in which they could store an information.
So we literally had people track an order and write it down
manually saying, what's actually going on and then we fed the
data it would Excel and then we built out a dashboard to
identify bottlenecks in their processes, right?
So varieties of different ways in which we solve for b, i from
a standpoint, but we realized it was a banker secret strategy.
Then the Phase of my life was at Boomerang Commerce, where it
became where I realize the importance of scale because I
that point I had to think about pricing decision for the entire
world of retailers, right? How do we think about solving
the problem at scale? So so going from Bonnet, going
from understanding the data science of the bi world, to
understanding the Strategic importance of the bi word at
McKinsey, to understand in the technique importance of
technology and platforms of the mechanic at boomerang.
Commerce is kind of what join all the pieces of the puzzle
together in terms of me thinking about continuing in the Journey
of a be in continuing the journey in the sphere of Pi.
Yeah, then lows. And right now at Google I'm kind
of putting together the lessons that I have learned throughout
my career and kind of joining the technology, the data signs
as well as the business decision making to to essentially bring
the insights and actions forward versus say, just just Bir
technology can do by themselves, right?
It's kind of putting them together and getting a hold of
getting a sum that is greater than the sum, of course.
So I guess you're sort of some sort of.
You've done some data science, work in the past and you've done
some strategy work in the past. You've done some dashboard work
in the past. So it's almost like all of this
has sort of come together for you to kind of.
Now, sort of do your bi work to put it.
Yes. Okay.
Awesome. Awesome.
So, I think in some sense, in that sense, It's all about bi in
play. Yeah, I think you described it.
Right? Like it's up.
It's a sort of a product product manager, kind of a thing gets
on. So now, like, I mean, slightly
switching tracks, like look for a lot of people when they think
of bi. They think about it in terms of
tools. So, especially in the market as
it stands right now. If you look at the bi job market
and the other day, I had joined the radicand EI and Reddit
subreddit on pi. And there, it was all about
people were discussing about It is Tableau better than power bi
better than it is all about tools.
And it was about the knowledge of to send someone to question
for. You, is like your Dubai.
I mean, I know you work for Google now, so you kind of might
be, you might be considered to use mostly Google's tools.
But like, are you a sort of, like, have you been sort of
constrained by to sorry. Do you have a favorite tool?
Or like, what is your or in terms of?
How do you how do you look at this entire bi tools universe?
It's a, it's an interesting question because I think it can
its, there's no, there's no straightforward answer to that,
right? When you're solving for, when
you're solving for a success inside an organization, right?
You've got to use the tool. So thinking about and I have
done this test at variety of different places.
Bringing a new to bringing a favorite tool.
M. Versus.
Working with a tool that may not have, all the features that you
have but has the ecosystem that supports it.
Yep. Right.
And I have found always that the tool where the equal there is an
already, an established ecosystem that supports it
almost far out performs in terms of the, in terms of the impact,
overall value created by the thing, don't do it may not be
the best visual tool, though. It may not give as much, you
expects ability. But but the fact is that, if
there is not Of a platform or a set of ecosystem that is
supporting the, a particular tool you will end up becoming
you'll end up basically having something that could shine.
But there is no, there's no Breeze kind of making it Shine,
right? Whereas if somebody even if
something that may not be a shiny, but if there's an entire
team that is lifting at all. I believe that drives Home
Project having said that, having said that I have, I have work.
In technically, I've had now had the opportunity to work and all
three different areas right at the start of that.
I work for, we were an aw shop, right?
And then, so we've had an opportunity to work with work
with red shift, and then Tableau.
And then in between we did had have a snowflake as well as a
computational platform. So, I think that really really
worked well with a We have low steam was embracing and open
source, open source Technologies as much as possible.
So, that is an area where I got an opportunity to work with a
variety of different open source, tools, like like Druid
and, and Kyle, and so on. Then, then the next piece is
now, now in Google. Now, I am in a Google shop.
Make, oh, yeah. So from here, I'm getting, I'm
getting a sense to work with. Not only tools.
That are available in the market, but also a lot of in
build tools that are available for for googlers.
And yep, and I'm able to get to see a lot of the wide variety of
it. And honestly, this is where I
write my opinions are far from the standpoint that there is
actually no best to it's about tools that the tools that that
you are. And I also keep encouraging the
bi developers and my team to say hey don't call yourself a
microfiber is a person. Don't call yourself.
The photographer said, don't call yourself a power bi person
you have our. So you are essentially a bi
person that you've got to be in a position like similar to a
software engineer. They don't go out and stay.
Okay, if a C++, I men if it's not C++ I'm out and you cannot
be in the same word as as that. So you have to you have to
essentially think as a bi professional and then tools are
just tools are just the tools, right?
But you are the one who create in the value.
So a Again, tying it, behave in. I think you are in my definition
of bi is that there's a lot of intelligence.
There's a lot of data science that goes into the into the
computations before let's say the final dashboard or equal to
whatever is developed and so on. So in general how compatible are
the sort of the tools out there in the market to sort of
building in this kind of intelligence.
I mean, I have a confession that I have really despite having
been in sort of broadly, this business for about 10 years.
Now. I have not really taken to any
of these tools and I sort of Do everything from first principles
using our or something like that?
But so how how compatible are there tools to connect?
Because my impression of some of them is that they are just.
You just need to write the pipelines to go from the
database to the thing. I'd like, putting in the
intelligence is, is, I mean, it's very nearly impossible, in
my opinion to put in intelligence into an SQL query,
some sense. So, yeah.
No, I think there are several, there are several ways in which,
I have seen to use these tools and insert intelligence.
So one obviously is to have your pre calculated data sets.
We are feeding into your Tableau, right, which means
you've already done, your models, your scores have been
calculated and if that is the data that was feeding, right?
So so if you can create a separation of data set and
secondly and I believe there are rules are coming up and I think
Booker and Tableau they're creating ways in which you can
actually do detection prediction and a Addiction and even
simulation scenarios inside the inside, the pool itself so that
I think is a so, so the the the software's are essentially
making it easier day by day to do to use the tools that they
have. But again, I think about it for
somebody who wants to start from first principles.
It may not give you as much flexibility in terms of the
modeling techniques that you may want to use.
It may not give you the flexibility in terms of in terms
of precisely the choosing your thresholds and so on as you may
want, but the tools right now are actually capable of giving
you very lightweight ml, in terms of just inserting a
particular algorithm and just finding out an auto a score.
Right? There are the there are tools
that are helping you do that. Even if that Tableau even inside
looker and and I'm assuming property is well, but
irrespective of that. I think the best mechanism Best
way to optimize it is actually pre-calculated very similar to
what you would do, like use our or Python and run.
All your models in in your computational platform.
And then use the visualization tool mostly as a user Discovery.
You user user experience problem versus the computational
platform. But if somebody who's not as
experienced as say, you are an experienced data, scientists are
in terms of building that I believe there are options.
There is options for them to do something like, wait, inside the
tourists, were somewhere in the beginning.
I think you had mentioned that like, he look at bi as a product
where the you have people with different sort of skills coming
together and so on. So let's say somebody is looking
to develop is looking to build a new bi team in some
organizations. Let's assume that for take my
situation where I I joined six months back in until then we
didn't really have a bi function and I have been setting it up
and so on. What's your view on integers?
How do you build a deep? What is sort of skill sets to
look for among the different people.
I am assuming that you need to bring in people from different
sort of backgrounds here because it's not it's not like you can't
bring in let's say 10 DIY professionals to put a team and
so on. I guess you need different skill
set. So what are the skill sets that
you would sort of look for when building a team player.
That's a great question. So so I think again just going
back to their appeal when you think about the layers from
data. A Shinto.
Action generation or inside generation, right?
So I think there is one one team that is required to own the data
and the logic, okay? As 11 basic.
I think there are four layers spheres of ownership.
One team owns data across or or one skill, set the bones data
and of it is not dripping or call it a team.
There's one skill set that bones insights at scale which means
that they are responsible for understanding the power inside.
Generated in the business and also producing them at scale
using data science. There is one team that owns
consumption and user Journey rap.
So say. Hey, now this data is there.
How do you kind of translate all of that into a into a user
journey in terms of how whether they want to consume it as a
dashboard that they want to consume it as an email?
There. Is it an alert?
Or is it like just a written summary of your insights?
Whatever it is? That how do you want to consume
it mean? So that is for third?
And then the foreplay, which is the canvas that is spreading
across as what I call the product management new.
Yep. They open both the users success
as well as team success. Yeah, so so I think these are
the four buckets of talent pool. The needs to be brought in to
solve for a b are. Now if you're starting from
scratch, what is p? 0 data is P0 that stakeholder
management is p 0, which is hey, somebody who kind of understands
the use case, understand solves a problem.
I think that becomes P0 so you've got to start with that,
right? Which probably, Many of the
companies already have invested a lot of time and effort and at
least with the data over the course of the tower over course
of the years, but but I think if you're starting from scratch,
that would be the first thing you start.
Then you, then you would think about adding in bringing in the
data scientists to as part of your scrum teams to think about.
Hey, how do you now look at now? The data is being used.
How do I get one? How are they looking?
However, the business stakeholders looking at, at
what? What, what kind of insights are
they looking for? How are the Actions influence.
How are these houses data influence into action, so that
you can now stream line, the process and favorite inside the
scale, and the third, and I user experience is very important.
Again, if it is an external facing tool, that would probably
also become as p 0. But if you have an internal bi
setting us up for doing an internal decision support, we
can take the luxury of giving them a not perfect you X so that
so that we can continue to build on work on it.
As a P2 and if you are thinking from a standpoint as a bi
leader, but this is what we, this is how I would think of P0
P1 and P2, in terms of how I would set up the set of the
organization and bit of the organization while also creating
value in the same same page. Yeah, and what kind of skills
would you look for any time in the, for the data?
And logic? You'd is it like data Engineers
or what kind of skills. Are you looking for?
Each a buckets? Yeah.
So let's go into the data piece right now.
Now, as I told you a lot of it is dependent.
Pending upon the platform on which you are sitting on, right?
But so so several times. I have seen, you actually need
hardcore data Engineers or if if that entire engineering combine
is made, it is become very easy through some kind of a platform
initiative. You could go with.
I mean, I've seen technical analyst also grip, be pretty
good at managing a chronic datasets, right?
Because there are right now we have tools which are which we
just help you subscribe to streaming batches of data at
Easily. It helps you build your own QA
alert. It helps you were on your own
car, bomb data, consistency checks, without doing anything,
all all by, just all with SQL. You don't have to do anything.
You don't need to know any other language in order to all of this
stuff. Yeah.
So so with that said, there is a team.
Now if the tools are not available, you will end up
needing somebody who is who is much more well versed with
working with data platforms versus a somewhere.
All of that is made easy through the platform.
The second piece is as you said statistical analysis.
So, this is where I mean, this would be your traditional data
scientist data scientist. Rules folks, who have again.
I would look again, my thing here, as you said, right?
It's different from maybe a product data, scientist role,
who may be well versus an engineer.
The, the team, the person here the has to have empathy for the
business, right? And from that standpoint.
So I again, I see a lot of analysts who have behaved in the
past who have experienced in our who have actually probably got
under a master's in data science or analytics seem to be very
good candidates for this particular road because they're
able to do business analysis. Very good.
Well, and also able to nothing, how do I solve that business
analysis at scale using using algorithms efficiently, so that
would be The second please. So user research.
This I will admit is one of my areas where I'm still continuing
to develop and forming my thoughts and opinions on.
But I think you did read a statistic that most of them are
companies between or in 2020. The indeed.
There's they're engineered for ux design.
Ratio was about 20 is to 1, whereas now it's about eight is
21. So so, so the focus on user
journey is obviously a user. Ux design will obviously
increase. Every day.
And, and I am constantly thinking about how how do I
build in that? How do I bring in the right?
Right. And I think the talent is
actually baked is this missing, especially in the bi were,
because again, you have somebody who had who understands the
data, as well as the business to kind of Be Sedated in the
center. So from that perspective, and I
think somebody who's interested in solving for this has a, is,
will probably be in high demand and And, and I'm really looking
for folks to work with folks who are in that area.
And then the last piece is your product managers and
inexperience analytical leaders do really well in this because
they also read because you need to kind of span all the way from
data engineering, understanding the metrics of the logic, to
begin sighs, working with the stakeholders understanding
understanding, how the CEO makes a decision, which is how a
director makes a decision and kind of Geling the all the dots
together. So I have seen several senior
managers of analytics, or managers of analytics with with
good, with good, high ownership of high ownership and
accountability. Right.
But they really become good product managers because they
also know how to lead some time weight and your lead without
Authority as a product manager. And and also they have probably
done many of these jobs that In one form or the other.
And I think. But I like, the clear weakness.
I see in this entire funnel in the talent pool that I see
outside is at the ux. Yep.
Yep, because I think ux you need to connect you to have an
appreciation for data as well. They did, in this case, itís,
not a general. You've exigent aux engineer.
You need to actually have somebody who who can also do
analysis. So it's like, if there are be as
water, bu X, that would be amazing, but you don't see many
people switching from vs. USA.
One of them be to data science of great engineering.
Yep. We are actually I sort of got
lucky that I got a former front-end engineer who wanted to
move to analytics. Exactly right as you.
But why don't sort of? So one other comment that you
sort of made here and there, just like you kept talking about
scrum teams when it when it came to be.
I so scrum. I mean, I don't exactly know
what it means, but I know that it comes from the agile,
methodology and things like that.
So what's your take on the usage of agile in bi?
And like, I'm assuming you have some experience in that Lake?
The pros and cons of that. Yeah. so, So, I firmly believe
that. The benefit that I get some
modules, predict abilities, okay.
All right, because everybody knows what's happening every
week, this predictability and transparency.
Yeah, right? And I think, yeah, I mean you,
so from a standpoint, everybody knows.
So if you think about what is the big detractors of success
for a bi team one? As you keep getting Pink by all
the users saying. When is this going to come at us
that cannot calm? Well, I need this.
I need that, right. So there is that whole list of
priority issue. The second the second issue
comes where, when all of this happens you have to, you
actually have to disturb the team who are actually building
the solution in order to tell them.
Hey Venice is going to come - are going to write.
So that is one. This is the third and then and
may never have seen when you not had an agile is.
There's no clear roles and responsibilities.
Like you sometimes have the D data engine, somebody who's
killed, wants to be a data engineer.
Do we have you work? Whereas somebody who wants to do
statistical analysis? Is is doing is just bending.
I just had like some reports. Yeah, so so so now that you have
secure lab clear roles and responsibilities means that you
are able to actually put the right people in the right, get
the right talent and and fit them in so that they feel happy
about their job and to get job satisfaction.
Yeah. Less disturbance in a, in a
standpoint, right? If you're not releasing for two
bonds, people will always disturb you but if you're
releasing every Two weeks people know exactly what's coming in at
there. Is that level of transparency
that comes in? So given that people don't get
Disturbed, there is a little bit of a cost to that which is that
there are scrum team scrum meeting that happen every day
and we have several terminals that needs to do that.
We need to do there's a cost of it but it's a predictable
coughs, it's not an unpredictable cost, right?
Some suddenly. There is a CEO asking question.
Then you got to drop everything and then you've got to solve for
that and then everything goes Haywire, but here it's a
predictable cost. You know, exactly the product
manager is able to handle. All of that, chaos that can
protect us from Team from, from, from the, from just focusing on
the work. Yep.
Then the last the last piece under the last pieces, I
Traditions obviously, right? I think in a world where we want
to and I'll seem too good. Pros and cons of I trading fast
versus making the product usable really invisible before you
deliver it, and I think, I think that is very fascinating ideas
on both side. And I think it depends upon it
depends upon organization of the culture of the users, as to,
which the way you want to lean on, you want to hydrate very
fast and really something very Scrappy, or do you want to
really make the product usable before your release?
So, so, so, that is so that is also an area where Deep house
because you're able to get feedback that you can go back
to, you don't have to wait, four months or two months to actually
get. So, so yeah, in order to release
every two weeks, you need to actually have, you need to
actually have operate like a factory, the cons of it.
As I said, one. As you said, there is a cost
associated with it and, and sometimes and I know Engineers
hate attending meetings. So there is regular meeting that
was scheduled to people. Keep saying, hey, why do I need
to attend this? Because the sea is calm, most of
the time, there is usually a stop there is sometimes a storm,
right? You're paying a cost during the
when the sea is calm in order to prevent stop.
So Gap. So when you actually don't, when
you don't see the star, people always ask the question as to,
why am I in the meeting yet? When you take away those
meetings and the storm starts hitting then we like people get
let's get that right? So yes, I think that is the so
usually I have seen pushback from engineering to obviously
the second thing is that It also creates a lot of impact with
respect to all the other teams are extremely chronograph occur.
Right? Exactly.
After the most common eating with all your engineers and all
your product managers being in a room and solving for all of that
wood style puts pressure in terms of K.
Maybe somebody taking a night call or somebody's taking D
call. But if you actually have, you
would be it. It qisas that pain.
But apart from that, I have seen generally have seen the
benefits. Outweigh the The costs
associated with that and especially even as a leader.
I get I get a level of transparency of how the scrum
team is progressing, and whether they're working for towards ago,
he has. And if there is a, if there's a
need that I need to enter me intervene to, to course-correct
unable to do so before, like, Things are Slip too far, right?
So I think that's, that's really the help that I get from a shell
as well and whatever the data science part of it.
Like, I mean, if you're in model building and things like that,
the what I would using my old favorite favor, What I would
classify as they start to work where like, you need to sort of
like where it's the development, doesn't happen in a linear
fashion. But like example, is that you
end up getting your insight into only how conducive is that to a
child? Yeah, and I think there are when
you have data science in the team, then there are essentially
again, there are always two parts to it.
Right? As you said right, there is one
gotta go to definitely invest, time into doing invest time into
solving for the big one. This is generally during data
science and also any region, a free ride where you are solving
for a big technical Observer ready, right?
Like that. You gotta solve for our you got,
you have to invest time in your in your Sprint's to do that,
right? While at the same time, you got
also think about what a quick and dirty way in which I can
move the ball forward and not wait for the end of the world
before we actually crafted best model.
Yeah, if it means that you want to solve because many of the
Tanners, you said, right? How can you make the research
usable on a date? Mrs. Slater, that's really,
that's really the role. Where the product manager, the
data scientist needs kind of sit together and say, hey, if you
guys are doing because many times you end up with a lot of
bivariate to multivariate analysis that seeing, if there's
the rule, rule space way in, which I can start getting some
value out of it. Yeah, for before you make,
except the different ways in which we can try to try to use
the research in and monetize that we searched every
throughout, the Sprint's versus just wait for the final result.
Thank you for listening to data shatter.
If you like this show, please leave a comment, share and
subscribe to the podcast. You can find this podcast on
Apple podcasts Spotify or wherever else you go to get.
Podcasts. Once again, this is Karthik
signing off. Thank you.