So the first thing that I put together for them was actually a
mini data, visualization of sorts a scatter plot Matrix,
which of course didn't exist. Even by name at that point.
This was in 2090, personal stuff.
You can do in Excel. Yeah, nine percent of the stuff
you can do in these tools that we are still can't do this one
person. That neither of these can do
one, go to have JavaScript python, whatever you need.
Welcome to data, Shadow the podcast, on all things data.
This podcast is a series of conversations with experts and
Industry leaders in data. And each week.
We aim to unpack a different compartment of the data
suitcase. Your host that the chassis that
I'm a blogger newspaper, columnist, book author, and a
former data and strategy consultant.
I currently heads analytics and business intelligence for
delivery. One of India's largest logistics
companies. You can follow me on Twitter at
act Karthik s and read my blog at no into.com that is noen the
HUD a.com all opinions expressed.
In this podcast belong to me, and the podcast, guess and do
not reflect the views of any organizations.
We may be associated with nothing.
Disgusting, his podcast should be taken as Financial or legal
advice. Now onto Today's Show.
When we think of data, many of us instinctively think of
spreadsheets. And that means, Microsoft exit,
the world's most populous big sheets of metal and the time of
recording at least in some ways, the use of excel in the world of
data analytics is much underappreciated.
Another thing that many of us instinctively, think of when we
think about data is graphics and visualizations bar graphs line,
graphs dashboards and the like in this Not get a lip is sort of
data shatter. We bring together these two
obvious Concepts in analytics. My guest today is s Anand or
stud Anand as he's known in. I am Bangalore circles and is a
co-founder of granular data science company, leads the team
that automates insights from data and narrates, these as
visual data stories is recognized as one of India's top
10 data scientists and is a regular text speak.
Ow, another is a gold medalist at IIM, Bangalore and an alumnus
of IIT Madras London Business School, IBM, enforces Lehman,
Brothers. And BCG.
Most importantly is hand transcript, every Calvin and
Hobbes script ever and dreams of watching every film.
The IMDb top 250. You can follow him on Twitter at
s and 0 that is s a nando64. Oh and his website is It's a
document. That is s Dash DN A and D dot -
you get into this because you would have heiko sadly
consultant at BCG from what I from what I know.
So, how do we go from there to kind of being a hardcore data?
Gay know, the strategy Consulting was the mistake.
I was actually at it again. Okay, so meaning I wanted.
To be in first used computers. When I was, what in class 6.
We had a BBC micro at school, and they taught us programming
and I thought that was a really cool.
That's, very impressive. Comes considering you are much
older than me and I started in class 7, so yeah, 86 is when I
started, okay, and then in 87, my dad bought me a ZX Spectrum.
So initially it was playing games and the learning a little
bit of basic and then slowly it went into generating Fibonacci
series. He's and then slowly went into
fractals creating a mandelbrot set, slightly more efficiently
and creating a 3D rendering with Phong shading and stuff like
that. All of it on a tile is Eric
Spectrum with 48k. So it was fun.
Yeah, and I was hoping to get into a computer science degree,
but pretty much no College. Gave me a computer science
degree the closest I could get to that was tight.
The EM where I say, I basically ticked off every Branch other
than chemical engineering and then the Prof.
Who was supposed to be guiding? He looked at this and said, no,
for your rank. You will get chemical
engineering support to take against chemical engineering.
It separately don't like intentionally.
No, it's okay. Go ahead.
That's what I was. It's so I spent those four years
trying to do anything other than chemical engineering and
computer science department was my retreat.
Okay. So in 96, when I got a job offer
with I've been said, okay great. Good, riddance.
Goodbye, and it Three, good years of thought.
Then then that 99 is where the mistake started.
Actually, the Mystic started in 96 when I cared and didn't get
through. So I was pretty pissed off.
So I decided to write cat again just to show them that I can get
through and I wrote it and I got through and then know this whole
pressure builds up. No, no, you have to get there.
So I got the winter. I am a interviews income.
You have to go to any sense too far away as I am be, that's as
far as I was able to resist or IBM.
Was thinking, I'd be most impacted.
So have you was enough? Okay, okay.
And so it was almost a repeat of 1992.
Again, enough air pressure. So I said, okay, so let's do it.
Yeah. And MBA and then it just
continued after that. Now you have to get into an
investment banking career or Consulting career.
Luckily. It did an internship at Lehman
Brothers and you that your investment banking is nice good
and all that but not for me. So as a chill, let's learn
something that lasted for four years where I dcg.
I basically tried to work my way into every technology.
Project. That was in fact, I want strong
but there was a piece of work. We were going to do for Oracle
and others. The First Technology project
that I come our way. This was based out of Delhi.
I was in Bombay by default, Consultants from Delhi would get
staffed. So I remember calling our
partner James and that was the morning when they gauge meant
was I think about to start our kickoff or something.
I said James. I'm at the airport.
If you tell me, I'll buy a ticket, get on a flight and come
over there. If not, I'll go back.
Yes. He said yet.
If you're that desperate, they're okay to hop on the
flight and come but 2005. I had clear I had Clarity that.
Yeah, I'm going to get back into technology of course having done
strategy Consulting. Nobody really gives me a job
into technology. Yeah, so it was disconcerting.
I'm good. Yeah.
Now you have to say going from McKinsey to Google or Facebook
wherever, right? Now, it's quite a common today.
But yeah, most people said, look, the best I can do is give
you a sales job or a marketing job.
Yes. I want to program.
So Infosys Consulting was the closest where it was a quasi
technology quasi Consulting, kind of a role, and there was
mostly sitting and coding doing stuff like this, IMDb to goofy.
So it wasn't quite a shift away from strategy Consulting.
It was a desperate claw back from 99, 2005 to technology,
which is what I really love. Yep.
Okay, today, this very interesting.
So I probably we should take you here on a slightly alternate
path because I got my first computer when I was 11 or 12 in
a couple of years, learn to code, then started doing these
random graphic things on our 386, 10 things like that.
Then I decided, I wanted to study computer science actually
managed to get computer science in IIT Madras and then like, I
completely fell out of love with computers things.
For the those four years, I completely sort of.
I hated programming. I was reputed to be the best
programmer in my school, but I completely hated programming.
So for me, I am be was a respite.
It was like I want to get away from take so and so I am be
happy. Very similar to you internship
at JPMorgan happened. And I was like, no, I don't want
to do this. Then I joined 80 Connie.
And then I said, I don't want to do this either and I left in
within the within the first few months and then like and then
yeah, like, I mean, I 3500, various places went back to
invest in banking for a bit and so on.
But yeah, you know, it took me about five years after
graduating, from computer science in, IIT Madras to that
liking to code again, ho ho, ho. Wow.
So maybe maybe in some way by not doing computer science and
doing chemical, you manage to sort of retain your love for
coding through your shit. Okay, then I have to thank that
Professor, not curse him. Hey, it's also where it's from
Infinity to the thing you say to grammar or like you you deserve.
Yeah, yeah, 2011, my ex-boss from IBM, ROM.
He reached out and said, took all of us old guys.
We are looking to do something you're interested.
I said was, yeah. Absolutely.
Let's try again. Okay.
Okay, awesome. So, I did like, I mean, you sort
of like, I mean, I also like we sort of bumped into each other
at various places. First, know you as a sort of a
Visualization Guru. So, how did you get into Data
visualization, which is not exactly programming, right?
So, I think there are two parts that let me there.
The earlier one was just an interest in graphics programming
its key. I like playing games.
Yep. The question becomes.
Now, can I create games and for that, I have to learn some of
the basics of Graphics fractals were always an interest and 3D
rendering was always interests. So, and my dad's an architect,
so at his office, I was interning of sorts creating 3D
models for him and sketches and stuff like that.
That was one phase where I was generally interested in
programming stuff that generates visuals, even if it's a simple
matter of the ray tracing a seen, those is software called
all free which I've been using for quite some time.
Of course, autocad's autodesk's 3D Studio was another that I've
been using quite It was on one side, but the data hadn't really
come in at that point data independently.
During my, I am days. I got into mostly playground
with financial data and a bunch of other things.
But during my BCG days. I did get an interest in visual
design, simply in terms of slide design, for example, because I
really sucked at it. So, for many years, I had been
spending time trying to read learn the principles of design.
I'm and honestly, the one book that taught me.
What little I know of design was this book by Robin Williams, not
the actor, called the non-designers design book.
Okay, and so brilliant book. It tells you a few simple
principles that you follow to make sure that you get.
Design, that is not bad, not a sign that is good.
But designer is not back. And while I was doing that since
I maybe was familiar with programming and said, what does
it take to make design programmatic and automate as
much as I could. And since I was also looking at
data left, right and Center, what does it take to present
this in a cleaner nicer? Professional way.
So some of the early data visualizations that I put
together, I distinctly remember One when I was at a Consulting I
was doing this piece of work for S.
Co.com., The British Retailer's website.
So one of the things they had was a 40-page PowerPoint
presentation that talks about what where the weekly statistics
from Google analytics. Okay said what would it take to
create this in the form of an infographic?
A brochure high-quality design your at takes the data from
Google Analytics. It's and programmatically
generate set. So I put something together.
It was an interesting experiment in many ways.
It was very information dense design a dread Edward tufte by
then. So okay had a sense of what kind
of information dense displays one can create put it together.
It was a python script. I still probably have it
somewhere that pulled the data and then I tried an experiment.
Is it what kind of attention would this grab from?
And who does it really attract? So it was late, one, Tuesday
evening. I took color print outs of this
after everybody had left for, which the UK is easy.
So, yes, basically around him or something.
Okay, they worked a little harder.
They're left these printouts on pretty much, everyone's desks.
Okay, with my hypothesis being well, not even hypothetical my
question being Who is likely to pick this up?
Okay, so then came in the next day and I hadn't put my name on
it. I had put anything so it's like
nobody knows where this comes from.
Yeah, so this is support sitting in there and I check
mid-morning. It is interesting.
These are the reports had been picked up by every one of the
exits. Pretty much everyone in the
marketing team had picked it up. Only the head of operations had
picked it up. No one from the vendor's teams
had even touched the report number.
From the front team have touched.
The report was just laying there in from called kind of thing.
So it gave a beautiful perspective of who actually is
interested in this kind of data and force that landed mean,
probably because it was a big Witch Hunt of sorts that started
saying okay, who's this person who seems to have gotten access
to all of our data and it's doing stuff with this scent.
Okay, let's happening. Yeah.
Ended up well, but that was the first time I realized what
happened was then. The head of marketing after I
shared that. Yeah, I could sneak the head of
marketing came over and said, why did you do this?
Say, why do you ask? She said, well, look, this is a
professionally designed report. So I thought we had paid someone
to do this, and I had no clue whom we had paid the same.
Oh, okay. Something that's professional.
Yes, not a bad thing to start off with.
Yeah. That's where the idea hit that.
This could be a thing. Safe before data visualization
was even thing. Yep, which year was 2008 or 2009
so Miranda, so wow. Questions like I mean, so let's
do the difficulty to the static. You said that the marketing
everybody had picked it up operations, only the head had
picked it up and nobody else was interested.
And in the vendor team. Nobody was interested to, why do
you think that happened? And like let's talk about Let's
use this to get into. What kind of visualizations are?
What kind of reports work for? Which kind of people like, based
on what they doin sir. And so one thing I've learned is
finance teams actually do well with numbers and not, that they
can't do poorly with visualizations, but they just so
much better with the numbers and need the numbers so much yet.
We are probably better off staying away from I shouldn't
even say Finance. I should really say account.
Yeah, That's Amore out. Yep.
I learned that the hard way because the purpose of a
visualization is to give an overview to save.
This is bigger than this for the accountant.
It, it's kind of for a manager. Yep.
One cent versus $100 is 10,000 x different to accountant.
A discrepancy is a discrepancy. And that is something that I
missed for quite some time. To be honest.
It in very be visualizations, invariably work for senior
execs, because ultimately, it helps them get a perspective.
What's big, who, what's important?
What should I focus on? Because prioritization is one
thing that we realization helps dramatically in.
And that's one of the key challenges analysts.
I increasingly find are using this because Helps them from an
exploration perspective. Again.
It's a prioritization thing. What should I focus on?
Where should I dive in? Yeah, and that is happening.
Typically at Junior levels when people are saying okay, if I can
use a visualization little help me dive deeper, that works
across Fields, whether its sales and marketing or whether its
operations or admin or even H are now finding that the less
numerically Savvy. The people are the more visually
attuned. They are the think it's simply a
yet Style. Love thinking or working.
So net, net. I have two simple rules of thumb
for 33 on whether a person who will dive into visualization
senior vs. Junior.
Yep, seniors dive in more numerically Savvy versus
numerically non Savvy and the numeric lean on savvy are more
likely to die even more. Yep, and the Actually, I'm just
these two third of the subset of the second.
Okay, so mapping with back to your original example, I guess
the marketing people were more like they were likely to be like
I gets less numerically Savvy and so they all picked it up and
like the report. Is that the no, I know I think
even if I had just the numbers there, they would have picked it
up because this was literally about sales and marketing.
See this case. It will serve domain bias.
I suspect that if I had put together something similar for
operation. It's in Terror operations, team
would have picked it up and only the sales and marketing head
would have picked it up fair enough.
And what explains the windows not taking that they give us
completely relevant to them the window managers even care.
Yes, you did tell me to do this. Yes, sir.
Okay. Okay, you know, we are very
busy. I mean, like, as you might know,
for the last few months, I've been working for delivery taken
up a job after a very long time. So I have a friend in finance.
She was like, can you when will you do some analytics
visualizations for us? I was like, look, I'm Doing
visualizations for you because your needs are too precise for
me like actually, right? Because the finance is like it's
all about like which is what I mean going back 15 years, which
is what I figured out during my little internship in Investment
Banking. As well that click the need for
precision was like, yeah, I'm like you need thousand two
hundred dollars have come 2001. 98 you have word just round it
and you're dead and they're like, no, we need to.
We need to match the last penny and I was like, okay.
Yeah, I guess we're visualization.
Is that, since works for when? You are when you want to produce
the show brought trains, let's say or when you want to Lexa.
So I guess so if you have to present it to finance people or
other otherwise numerically Savvy people.
I guess it's more about yes, give them tables and give them
well formatted ajj table. Exactly in the areas that they
are looking for very refined. Visualizations.
Helpful is our to help them see stuff that they otherwise don't
see to give you an example. You said the two dollar founding
of a new gas. Now their point is that to two
dollars, two thousand dollars. It's all the same and error is
an error in which case the metric that we are tracking
starts becoming different. It's not the magnitude of error.
It's the presence of an error and then we get into something a
lot of roughly along the Realms of data quality.
And now, can we visualize data? Quality can be visualized fraud
but it's data T. We did something interesting
there. Was an issue where they were
getting fraudulent purchases, adulterated matches, and a big
part of the problem was trying to trace where this was coming
from. Okay.
So what they did was they actually took about 100,000
batches and looked at the flow of these hundred thousand
batches from beginning to end. And at the end that these are
batches that they had money, re-inspected and found that a
certain proportion of them where it will treated.
And when they had the whole supply chain here.
Facing the flow of these are all great, advances across the
supply chain, and being able to visualize that now that's an
Auditors Delight. This particular guy.
Oh, I know this guy. He's actually colluding with
this fellow because he's his brother-in-law at this plant.
So I know why this is happening. Ah, yep.
It's a very good. Yeah, which is harder to do with
the numbers and it wasn't something that they had fought
off. So generally we think about
presenting this differences, give them what they know they
want as Numbers. Yeah, them what they didn't even
know. They wanted as visuals.
That works. Really, really.
And, and what do you what do you use to buy?
I mean, now I guess you've come a long way, like, having been
running this company for a decade and things like that.
So, what did you start? In terms of a, you said you
wrote a python script for your original thing when you were
working for Tesco and then like, so what how is your tool Journey
evolved over the course of time? So the one constant in the tool
Journeys? And I'm sure we'd want to talk a
lot more. Of course, of course, of course,
we come to that in a bit. Yeah.
Yeah, but the programmatically so well since I was in Texas
Bay's, I'd always been playing around with Visual Basic.
So that was part of the Toulon since usual, basic actually can
be almost copy pasted into python, which I thought was
remarkable. Maybe.
Yeah, in 32, cam in Python. I've done this.
I've actually recorded macros in Excel and copy pasted them and
with Very few changes, it actually runs in Python.
If you import the winter to come Library.
Wow, so I'm not that surprised, because there was a news, a few
years back that Microsoft was planning to replace Visual Basic
with visual python to allow you to manipulate Excel sheets using
python. So I guess they would have also
seen this similarity in some sense quite possibly.
They've gone the JavaScript route now, which okay, actually,
I'm very glad for because for us from visualization.
Stick to python was the de facto server-side language because it
was really good with data and any language could do
visualizations, but the one disadvantage of any language
except JavaScript is. It doesn't run natively on the
browser, and if you want interactive visualizations, it's
got to work on JavaScript. Yeah.
Overtime. JavaScript became so good and
good. Enough on the server side that
in 2015 or so. We shifted our visualization
tuning from, bye. Ethan to JavaScript, right?
But barring that there hasn't been a major language change on
the library, side on the server side, the python always had the
likes of matplotlib and now Seaborn book Etc, but I've never
really used those. Those are for conventional
charts. And these generally reason can
exist is because of her unconventional visualizations.
It's very doing what these like these can't do.
So we ended up creating svg's using templating languages and
that's in my mind still over the last 10 years, that one
technique of creating svg's using templates is the one that
has led to the proliferation of the kinds of visuals.
We create today and is still arguably the most powerful way
of creating unconventional visual representations, so that
we did in Python and then shifted over to D3 initially,
but the trouble with D3 is it requires too much programming,
too much learning? And Vega was a good mid-level
alternative, higher, than that. And Vega light is a good high
level alternative. On top of that.
I wouldn't say that. The problem is solved.
We still need a certain amount of tooling on top of Vega light
for more people to be able to create it.
Basically between Vega late and Excel, there is still that Gap,
but if that can be bridged and a little more flexibility were
brought into these libraries. I think we could say that the
visualization problem is solved yet.
Probably. Every step of the way there.
Okay, actually, I mean it's interesting that I my journey
has been like a briefly disjoint from this.
I moved directly from Excel to our to ggplot and I got repulsed
by python, because Matlab matplotlib, it's so difficult to
sort of handle back road little Seaborn.
It's like I just couldn't get the hang of it.
So I almost like a, this one assignment.
I was doing it a couple of years back where like the client was
like, can you please give us code in Python?
Because it will be easier for us to Integrate I will I did it for
six months and then I told him. No, I'm shifting to higher
because I'm 10x faster than the are then by.
So fair is like, I guess personal preferences and like
also I'm not a corded corded anymore in some sense.
So right here, we are really looking for the answer more than
to the repeatability of the answer.
And what we're doing really is more helping someone else figure
out the answer. I am less interested in telling
you someone the answer dancing. Here's something that will
Constantly, keep telling you, the answer in the perspectives
are very different. Luckily, for exploration.
Our is forgive much better. Is a much better environment.
And even after we factor in the likes of Jupiter lab and
whatever other advances have come into the ecosystem.
This whole lot easier to it's flat.
It is a whole bunch of these libraries.
Each of these libraries. There, something specific you
go. Look at the documentation.
You try it. It works finished.
Move on. Yep.
Yeah, that is In Python, you make sure that well, it fits
well with all of the other pieces, the beginning and at the
end and it's repeatable. So it's engineered in a very
different way. Yeah.
Well, I think I have sacrificed on engineering which is why I
like I get a lot of sort of benefits on the other side,
right? In terms of how quickly you can
do things, how easily you can do that at how much programming you
need. And all those things though.
Like it's like like the tortoise and I am be a long time but
everything is about it laid off anyway, so let's come to your
favorite topic which is The so how do you feel having?
I would ask you how you got to accept because it as a strategy
consultant. I think that would have been
your 99% of your time, kind of thing.
So and then, when you do start doing sort of protocol is
gymnastics in Excel and like creating all those cool things
in one. You know, I actually don't
remember. I think it must have been during
my IBM days. It is when I learned Visual
Basic or maybe even before that. Even during IIT was playing
around a little bit with Excel and learning a little bit of
Visual Basic. But the fact that you can press
a button and get it to do something cool.
That was always interesting. Yeah, first really interesting
piece that I actually remember doing was at Lehman Brothers.
So there was this guy Scott Tucker.
He had he had the ETA of how the markets where correlated with is
basically had historical performance of the indices.
All the major indices. Mm 40, 50 of them.
So the first thing that I put together for them was actually a
mini data, visualization of sorts a scatter plot Matrix,
which of course didn't exist. Even by name at that point.
This was in 2000. So pretty much did all kinds of
index match and we cup jugglery to take all of those scenes and
create that scatter plot Matrix of the correlations between
those, another one was looking at the US Treasury yields across
time. So take the US Treasury yield on
a given date and it's a line graph.
Yep. So what I did was I created a
slider, which, As you move the slider, it changes the date and
plots a graph on a different day.
So as you drag the slider it smoothly animates.
Yep, the USD and it was possible to see that the treasury yield
was clearly flattening because visually really powerful.
Okay. So among other things that got
me placement of, okay, good. I deserve, I think back when
the, it boom was still on - Ruth, summer of 2000.
Yeah, exactly. Because it was literally, I
think a week before the.com bust.
Yeah. I was in Tokyo at that time.
And in fact, in the Night before it crashed and everyone's like
partying. Things are going great.
And so on wanted to keep your very worried about the bubble.
I've actually everybody was called confident that it was a
boo-boo. But question is, when is it
going to burst? And then next day was oh my God,
the dating field of a bloodbath. I can employ young exactly
around that. Yeah.
Yeah, I okay.
Yeah, and I think the thing with Excel is that like, I mean,
especially once you have learnt Visual Basic, the number of
things that you have, you can do with it, like multiplies, like,
sort of with a large Factor. Absolutely, absolutely.
Arguably. Even before that.
I think in the developer tab, is at least one feature that I
think is as powerful as visual. Eric is simply the ability to
connect the scroll bar to excel As you move.
The scroll bar the value in the cell changes in bicycle.
Yeah. Now that's effectively linking a
visual element with control, effectively slider to a salon.
Once you've got something into a cell then formulas can take care
of all kinds of things. Yeah.
So just with that, and of course, similarly radio buttons
were required and so on. So putting together and
interface with these kinds of control, sometimes even just
drop downs for which you don't even need the developer tab.
I felt that was remarkably powerful.
Even more powerful than the likes of pivot table than
vlookup which in themselves are extraordinarily powerful.
Yep. But yeah, we should base it
takes it a whole realm beyond that but so high that I very
rarely needed to resort to that. So things like for example using
an external service, connect to web application and do some
complex processing on the server or get social data from there.
Yeah. That absolutely needs.
The likes of visual. Basic which is unfortunate
because on Google Sheets, we can just import a Json or XML using
a formula. So those formulas are built in
and it's a Pity that actually doesn't quite well and how it
kind of does. But it did for a very long time.
Have you informed that that could fetch data?
But yeah, outside of that, the kind of stuff that you can do,
especially Excel has the shapes. And the fact that you can start
coloring shapes Based on data is really powerful.
Yes, so for me, the bulk of the Visual Basic that I have written
is largely about mapping shapes two numbers.
Okay? Yep.
Yep.
Okay, interesting. And I think one cool thing that
you are done in Excel. I mean like which I mean again,
I don't work from a very long time because they think you can
create Maps using exit colored mastering.
This is during some election or something.
Some work you some election was you will talk talking about a
new have created colored Maps using Excel.
I mean, that blew my mind. And I hadn't thought that's
possible in Excel. So seriously, the actually, I
don't remember how that came up. But the need was always there.
No matter what kind of format a shape file is in.
It's just a nightmare even today, color a map, it, I can't
do it anymore. I have this with the same.
It's not practical for me. It would take me an hour to
color a map in any way other than on, except if it's there on
Excel, it takes A minute. That's okay.
You should quickly difference that need was always there and
it kept nagging me. So the first experiment was if I
change number, can I change the color on a set and serious?
Yes, with a little bit of Visual Basic.
So once I cracked that, then it got pretty exciting, which is
now can I figure out? Can I take a color scale and
interpolate based on that color scale?
So interpolating colors was a little bit harder to get on
Visual Basic and Visual Basic as a language just sucks.
Coming back to practically any other language data.
Is there an array? I don't even know by and large.
My answer to the question is a featured as a feature exists on
between basic is? I don't know.
I'm going to do a Google search. I'm going to copy paste.
Try it out if it works great. So it's something that I really
don't know Visual Basic at all. Okay, what.
Yeah. Interpolated colors finally
managed to find or write something like the we that does
it, which end up being reasonably compact.
So now we can integrate colors. The toughest part was reading a
shapefile. File and there are drawing it.
So again at the time I wrote it, there wasn't any Library good
enough that could read a shapefile and tell me what the
points. Where, so, the closest I could
get to was using think some JavaScript library that would
render it as SVG, and then read the S EG.
And, for each point in the SVG, draw a free-form in in Excel.
So, the good thing is, this is where pythons ability to work
with Excel comes in. I can be literally anything that
can be done in Visual Basic Works in Python.
So in visual basically, I basically record a macro where
I'm drawing a shape and it gives me the code for it.
Just clean Visual Basic code paper copy paste that into
python just change the points, put it into a loop and that
works. So, we took a whole bunch of
shapefiles, converted them into CG convert them into Excel, put
them online, which will basically at that time the
election of the Parliamentary constituency is the assembly
constituency is for a country for each state exactly, which
makes it a whole lot easier for journalists to just fill in the
numbers publish. It made a huge Larry.
These are still being used. In fact, literally this morning.
I was talking to strike on that collapse who was doing some
support Initiative, for covid, terms of getting beds and to get
the donors to see what areas have less pets more beds.
He said over a shapefile. I could go to that an actual map
and it's a whole lot easier for their team, label them.
However, they want put in arrows.
However, they want, exported it works seamlessly.
Yep. Yep.
So why is it that like, I mean, I have two questions.
Why is it that in general data scientists don't like Excel?
Think it demeans the nobility of their talent.
I'm obviously being ridiculously sarcastic here, of course, but
it'll probably take a few seconds.
Maybe even a minute to work that out of my system.
Yeah. Yeah.
At some level if I'm, I don't know dick close enough to a
real-life problem. But if I am a surgeon and you
come to me for advice, my advice is going to be well, chop it
off. Yeah.
As opposed to a general physician who's going to say,
well look, I don't know what surgery but medicines, might
cure it now, what pays a data scientist more?
Bitin or exit. Let's go even more precise.
What weather data centers Force? I touch or pandas or exit.
And the answer is, probably in orders of magnitude is forgive
me multiples higher. So given that.
Why would I want to do something filling my experience in a tool?
That pays me less? Yeah, and therefore if I have
problems are scarce at Level. So time is scarce certainly.
So if I have a problem at hand and I can choose to use my time,
building my skill in a particular tool, that is going
to pay me more. I'd rather do that if I'm
already really good at and US python by torch and the whole
stack. Let's say, yeah, then I'm going
to look for the next one. That's going to pay me more.
Not the one that I left behind. You're not going to be
programming in Haskell. You need a suit, right?
Even if of course, no Bhaskar nor am I, I think it's exactly
the same thing. The other hand.
I think Excel is the better tool.
Lets, you know, I read that I think one shortcoming that I've
noticed with accelerate, especially I mean over the last
few months that I've been extensively were getting data,
out of databases. Is that with Excel?
Like it's little automating tasks, into end seems a little
more difficult compared to like, with our equation.
So, absolutely. And this is where I just Yeah, I
just hate that Excel Works. Only on Windows and sheets
because Excel is quite automatable.
If you have Python and windows, servers aren't quite as good as
Linux servers. See almost on a Linux machine
practically anything is automatically?
Yeah, heck browsers are automatable to am in with the
chromium and specifically Puppeteer coming in.
Yeah, there's nothing that you can't do on the server side that
you can do the proxy. Literally mimics every action
that is still not true for Excel.
So yeah, I was talking to the visit is be names deeper.
She was talking about how she wanted to build simulations for
classrooms. Okay, and could we build a tool
that will allow students to do it?
The use case is basically Perfect.
For example, you have a set of formulas.
You have linkages, you build a full-fledged model around it.
The student weeks, a few things gets to see the result.
Now if Excel could be exposed, like a little mini web
application, you send an input, it does all the calculations and
sends it back for her. It will be trivial to edit the
model. And for the user, it will be in
just seamless because hundreds of people can use it exploring.
They were then, So I guess yeah, I'm actually I'm doing nothing
more than repeat what you just said.
In many more words. This literally are the lack of
automate ability in Excel such a way.
This is why I mean, in some ways I have not really used Excel for
over a decade. Now, as when I use it.
I always make sure even on a personal computer.
I have a Microsoft 365 subscription just for exit
because I got one without it, but I haven't done any real
work. Work on exit.
Well, I think except release course is in terms of
interaction with business letter.
For example, you have to the business, guys are all very
proficient at Exit. If you have to give them
something like okay, you tweak this then need not even be a
slider. They can they can change the
cells and you change the cells and this is what it pops up.
Excel is so much Superior for them to them, than giving them a
web app or any of the other things like that, crazy, two
clicks, and not just for the spreadsheet.
Except itself, but one of my colleagues to talk was telling
me that we are giving except for one of our clients as the to in
which they related stuff, dynamically generating fairly
complex supports and in fact, models and formulas in Excel.
So because the shortcut keys are so familiar with identification,
they could quickly jump around from section to section and do
stuff. And it's so much more productive
to do that yet. So perhaps this isn't even a
case of this may also be a case of a tool that they are familiar
with. Yep, as much as it is the right
tool for this, for the problem. Forget.
It's a good combination of a tool that they are familiar
with, and which has which is reasonably flexible and
Powerful. So you bring those two together
and like you can create a great interface method.
It was it's just traffic talking, which Simon picking
Jones, who has, in fact, written, few papers talking
about how Excel is actually a turing-complete.
Programming language. It is meets all the criteria.
Yep. Yep.
I have probably come across the existence of the paper, but I
haven't really read. It will put a link here for
people who are interested by the way, they cup.
So see the allocation high-risk. What's your view on Google
Sheets? Even that you are such a big fan
of fix it. Oh, brilliant.
So why would I not use? Google Sheets?
A because it's not on my system. So sometimes when I'm traveling
on a bus car in a flight, whatever I need Excel.
Why else? Because shortcut keys are not
the same as. Yep, but short of that, I use
Google Sheets like crazy. The specific youth.
Basically our website grammar.com is powered by Google
Sheets. Okay.
The content is actually all on
Google Sheets. The marketing team is goes to
some spot and edits, some text and click submit button.
And it refreshes our website and not just that whole bunch of
websites are powered by that. This is not even an uncommon
thing. Hey, the New York Has micro
sites, powered by Google Sheets collaborative editing in a
structured way is just remarkably powerful.
In fact, one of my side project which is on hold for almost a
year now is to come up with so Excel is great with table
structures. It's naturally structured as a
table. Is there a standard by which we
can create a hierarchical structure?
Okay, so I want Inside which I have a table inside which I have
a given potential. Yep.
And the reason for that is one of the cells literally needs to
be table, that kind of a concept.
So, now if I am still working on some kind of a standard by which
we can represent this in spreadsheets in a way that's
possible by systems and humanely, highly readable.
But if that existed, right, our website will become a hell of a
lot managing. Our website will become a hell
of a lot simpler. But put on With Google Sheets is
now the database for our content management system.
Okay, why do I like it? One?
It has its collaborate offices, which way now does provide that,
but super funky very clunky. Yeah.
Exactly. That's not quite the same.
Yeah, s. Its web native in more ways than
Google than Excel is specifically.
I can you know formula import from a webpage just big chunks
and read from it. Third, I can write macros in
JavaScript though. I'm not entirely happy about how
it's been done. Not recall.
I'm not at all. Happy about how it's been done.
But that it's been done is not, I think is a great thing.
But yeah, but it's in descending order fact that it's
collaborative the fact that it's all that native, until I can
write stuff in job. I cannot commit in JavaScript.
These are wide why I choose Google chief.
That accept who I choose Excel or Google Sheets is familiarity
and all climaxes. Whatever term visualization on,
I mean, on Google Sheets. Have the I have never even
really used it for making graphs and things like that.
Not right now. I use it for I use except for
it. Even though if Arabic.
But never Google G2, we are. I haven't really thought about
why either, but you're right. I've never done that hike.
Strings. Yeah, so maybe that's something
that Google really I okay the maybe Google didn't really think
about that as being a use case because I think the
collaborative - itself to cover so much that like they didn't
feel like they needed to offer proper powerful Graphics or
something true. So I'm doing course right now,
with an organization called landmark and set of leadership,
course, they run this course for the whole series of metrics and
these metrics are tricky. Person gets a really, really
sophisticated Google sheet in which data is drawn not just
from across sheets, but from across works, okay, and they've
connected that into Master dashboard for the coaches to get
to see the summary across all of their coaches that holds and
higher level for the organization acts like a network
of Google Sheets. Now, I don't think that can be
done except not to the kind of Fitness.
Yeah. Yeah.
Let's get back to visualization.
I think we have spoken enough over to Excel and like let's get
back to visualization. So what do you what is your
opinion on stuff? Like they say?
If I mean I use are we both agreed that python is not great
for visualization. We both agree that like Excel is
great for visualization and what we can do.
And so on whatever it other your tools, you have all these
dashboarding tools, like Tableau and click View and also have you
used any of them, does your company use them and equality of
view of the graphics element of of that?
So a blow is very so okay. Firstly, all of them are a notch
above Excel. I must be okay.
And come with the associated fiction partly of learning.
Yeah, partly of availability, which is a function of living.
Everybody has, except everybody doesn't have more power, bi or
click. So I can't really speak for
click because I haven't really used not, we have that all use.
Click, but I've been using power bi and Tableau server button for
be a more so than Tableau just a bit more.
All of them offer our more than accept and that's kind of like
saying. 90% of stuff you can do in Excel.
Yeah nine percent of the stuff you can do in these tools that
they are still can't do this one person that neither of these can
do and go to our JavaScript python, whatever you need, for
the 90% of it, still use Excel. Okay, it's really only for that
9%. What I've seen is that the
people that use these tools, the likes of Tableau power, B, Etc.
It's not as much fun. Self-consumption.
It's for production ization, which is a very big thing.
These are supposed to be self-serve tools.
But the majority of the users are part of a reporting team,
who create reports for others, who in theory are supposed to be
able to click explore and so on, and a small fraction of them do
that, but the majority of them, export it to Excel and play
around with it there. Okay?
For a couple of reasons, they have other data in Excel that
they want to link with, or they have a workflow, which fits with
Excel or Because they just more familiar with excess shortcuts
and that's the tooling system. They prefer.
So at some level the promise of self-service bi has been
realized a lot more people can create reports yet the other end
it was promising a need that probably didn't exist in that
self-service bi has existed ever since it's still Excel existing.
So, I believe these tools have a very important role in that nine
percent range, but whatever the quality of the Fix-It that, they
could example, if I see it, I don't know.
I mean, I'm that I'm telling my opinion here, if I see a tableau
table. I'm like, I just want to close
my eyes somehow just like a blue Graphics.
I mean, somehow they just it just doesn't look good.
Like, I don't know what what it is about them.
And like, though, recently. I read that like Tableau
completely subscribes to the. So, you know, the ggplot in our
stance of grammar of Graphics. It was like, Tableau is also a
direct descendant of the grammar of gas.
That has its own. But if I look at the graphics in
the summer, I don't know. It just doesn't by default.
It doesn't look good to you. So think about it, D room for
the gracias reproduce with Excel 97 and Excel 2001.
Yeah. Yeah.
The defaults of pretty terrible there and I think it was Excel
2007 or 30 year, 2007. I think we're through the
quality of the defaults in Excel started.
Improving dramatically 2013, to get a notch higher.
Because of the power, I think in Tableau people create what they
can and leave it at the default. I don't think the quality of
people working on Excel versus startup Loan.
In terms of their design ability is any significantly different?
How come neither have any design ability?
Yes, so they're just going with what's out there.
So at one level, you're right excels managed to improve the
quality of the The default Graphics to a level higher than
what Apple has been able to achieve arguably, Tableau faces,
a tougher challenge in that. There are more things that
people can do and Excel has some minor, but important advantage
of see, just aligning to the Grid in Excel.
You just an ALT drag away. Yeah.
So alignment, which is arguably, the single most important design
aesthetic. If you ask me, it is easier
because somebody says, oh, look, these are in the lines and is
okay, worst case, they don't all drag the manually drag.
But anyone who knows? It's a you just all Dragon snap
it to another ruined. Columbo dancing.
Things are now automatically aligned and it's a resize
columns and other tasty aligned that W doesn't have a grid that
people use by default. Yes.
It supports a great but it's a very different if you don't
think grid first. When you think of Tableau any
more than think of great source when you think of PowerPoint, so
some of these small little things basically mean that
Tableau designs are very, I think Excel designs.
We're in The five, but I think another part of the problem is
the rapid growth now that the eye is big and people are
getting in the volume of work that we're seeing done by people
with less than let's say two years of experience in Tableau.
Is it honest of break somebody who's in 2005?
We had people who use Excel for a decade.
Yep. They're veterans yet.
We haven't yet. Gotten to those Tableau
veterans, who are Aren't really concerned as much about how to
do stuff. They know how to do stuff.
The veteran knows it, and now he's looking at the same.
I don't like it. So they get bored of what they
create that. They start looking for
Aesthetics. That stage hasn't come yet in
bulk attic. Yeah.
Yeah, I'd also like the other
thing is the experience graph. If you look at it, like it's
very bottom heavy in tablet compared to excel at least like
a yeah, even 15 years back. It was in that bottom heavy.
So it was like, people Using it, for various reasons.
And so I took that out their own stuff, I guess.
So, okay. So let's get to the, possibly
the most controversial part of the thing.
I mean like this, a few quick. What do you think of pie chart?
I've probably gone the complete circle on this and from using
them because they look nice to eating them because of the said.
So and I could see why, and going back to sink.
Yeah. They have a place in life.
They are good. Good, they have very limited
data density. So the amount of information
that you can communicate with those pretty small, but the
aesthetic behind them is reversed.
So where do they have that place?
Where would you use a pie chart are not useful.
Well, the classic case which I guess even tufte would agree
with this Harvey balls. So just a matrix of Harvey
balls, is actually a pretty efficient way of communicating.
Okay, this one. Low medium high kind of a dick.
So okay that works and that's effectively a pie chart with two
variables. I would never recommend the pie
chart with more than two variables.
So it's really only proportion. Yeah, the Second Use case is
where you're sharing a proportion and you want to get a
better aesthetic, simply showing something that is different.
It also lends itself to far more info graphic variations.
You can put in a picture into a pie chart that looks What
different. You can shape the pie chart into
a variety of circular shapes or coin of, which is like sort of
buy a pie out of which is slice of the pie.
So many other things that also has value and these things are
at one level. They are charged chunk at
another level chart. Junk has value, the Normans gone
through this cycle, as well from no functional design, in the
Design of Everyday Things too emotional design where he has an
ordering point of view. Aesthetics has its place.
So I've gone to that kind of a journeyman.
So short answer pie, charts. ER, don't use it for anything
more than one variable, or one proportion and use a matrix of
features if you can, but for those cases, yeah, it works.
Oh, wait, I think we're going to be my last question to you.
So, while talking about pie, charts you mentioned update.
And and data density. Can you quickly in maybe five
minutes or so? Like I think most of our
listeners would not have Really, unless the real Vision, they are
real visualization weeks. They would have come after
stuffing. So, can we just talk about this?
Perfect. So, Edward tufte is the god of
father, of modern data visualization, his book.
The visual display of quantitative information is
still the seminal work in the field and was created at the
time when even charts, we're fairly popular.
Let alone data visualization which goes beyond that.
Pretty much all of this. Books are a visual Delight day.
Share some fairly deep with circles.
These are principles that we have.
I believe absorbed and gone beyond the sense.
We know where they work, where they don't work, but it's kind
of like Newton's Laws. Like you don't study quantum
mechanics before you figured out that Newton's Laws, really
absorb them, really applied them and of these principles lay out
the equivalent for data visualization.
One of which is data density. One way he puts it is Data to
Ink ratio. Yeah, whiskey use as little ink
as possible to show something and this is far better seen than
explain, but I'll try and do what I can to explain it.
Supposing you had the sales of beer, shown as a bar chart with
beer bottles on top. It looks nice, but the beer
bottles are not adding anything to the information.
Content knocked him off. Yeah, keep applying that
principle knock off. Whatever you can knock off
without making it unclear. And you may find that in fact
knocking stuff, off makes it clearer.
So use just like, they say it less is more.
Use the fewest words to convey a point.
Edward tufte is data, density principles.
Use the fuel, the least amount of ink to convey a piece of
data. Thank you for listening to data
shatter. If you like this show, please
leave a comment, share and subscribe to the podcast.
You can find this podcast on. The podcast Spotify or wherever
else, you go to get your podcasts.
Once again, this is Karthik signing off.
Thank you.