You always want to look at what is your business need to do
differently or your organization need to do differently than
other organizations. If you can, outsource it.
If it's not something that makes you different, you should use a
service because you will always be asked to do more things that
you can build as a developer that are differentiated, that
are like special to your organization.
So don't add in things that aren't.
Hey everyone, my name is Henry Surya Virawan and you're
listening to the Technically Journal Podcast, the show where
I'll be bringing you the greatest technical leaders,
practitioners and thought leaders in the industry to
discuss about their journey, ideas and practices that we all
can learn and apply to build a highly performing technical team
and to make an impact in your personal work.
So let's dive into our journal. Hello Joe, it's great to have
you here. Welcome to the Technical Journal
Podcast. Great.
Thanks for having me. So Joe, I always love to start
my conversation by asking my guest to actually share a little
bit more about yourself. Maybe if you can share any
highlights or turning points that you think we all can learn
from. I'd be happy to.
You know, I think my main journey started as a tech lead,
and I think the main thing I've learned across my career is how
important it is to understand the business goals and the
business itself that I'm working with or nonprofit organization
or whatever. I think when I started, I really
wanted to work on cool technical things, and that was my focus.
And you know, I realized over time that the quality of the
business, the quality of the organization and the quality of
the leadership and the culture really mattered a lot more than
how cool the technical problem was.
And so I've kind of developed a rubric for myself that it's much
more fun and rewarding to work with people you like working
with in an organization that's grown.
Because the organization that's growing is creating more wealth,
essentially in many different ways that can be shared.
And so it's an organization that there's like a healthiness to it
that makes it a lot more pleasant to work for.
And that that's nothing that I optimize for.
At the beginning of my career I I'm on my 6th, starting my 6th
company that I've been with now for about 5 years.
But every company along the way I've made mistakes around not
thinking deeply enough about how growing and how great is this
garden that I'm working in as opposed to you know, how cool
does it seem? That's the primary piece of
advice I'd love to go back and give myself if I could from
earlier in my career. Hey, thank you for being part of
the technicianal community. This show wouldn't be the same
without your ears, and you are the reason this show exists.
If you're loving TLJ and want to see it keep on growing.
Consider becoming a patron at Techligional dot Dev Patron or
buying me a coffee at Techligional dot Dev Coffee.
Every little bit helps field the research, editing, and sleepless
nights that go into making this show the best it can be.
Thanks for being the best listeners any podcast could ask
for. And now let's get back to our
episode. Yeah.
Thanks for reminding us as well about choosing the right place
to work with. And it's not all about cool
technologies, right? I think there are still some of
us technologies who love to play with the new techs, right?
Especially these days, there are a lot of advancements, new
things all the time. So always understand about the
business, the quality of the business, the people you work
with, the culture. I think that's also important.
And yeah, working in a growing organization, I think that's
also very important. I was just to say, you know, if
you pick a growing organization, there's going to be plenty of
money and advancement and things for everyone.
And so that one I think is a real key.
It is a way to get everything. You have a good culture and it's
not growing and it's not going to create those benefits for
you, but if it's growing, it will.
Yeah, not to mention when the organization grows, there's so
many challenges you can learn from.
I mean like it's a double edged sword.
Yeah, lot more challenges means a lot more headaches probably.
But also at the same time you grow much, much better as a
person or maybe in terms of skill set.
So you wrote a book, Serverless Game Changer, How to Get the
Most out of the cloud. So maybe before we start talking
about your book, tell us a little bit more your
relationship with Serverless. How did you bump into
serverless? And yeah, what made you started
to write this book? Yeah, you know, I've been on a
journey, I think my entire tech career, of trying to leverage
the benefits of advancements in technology.
And one of the things that I've realized over the past 25 years
is that we get really big, important advancements in how to
build software about every 18 months.
Now, every 18 months, it's not so much better that you should,
like, throw away everything you're working on and like, just
change everything. But there's a compounding effect
to this, like every 18 months, big changes.
And this really hit me hard in about 2007.
I had been building software for more than 10 years at that point
in time. And we had someone, I had
started a company and we had a company looking at making an
investment in US and they sent someone to do technical due
diligence on us and was looking at what we had done.
And he said well you know, what are you doing with continuous
integration And we were feel like I don't know what
continuous integration, what are you talking about?
And he recommended the book, the sort of the key book on
continuous integration, to us. And it really changed everything
in terms of how I thought about not only like, wow, we need to
do this, but also wow. I had no way of knowing this,
like in the way I had been practicing.
And so at that time I started asking how do I learn about new
things? How do I go on my own continuing
education journey? And that led me pretty quickly
to a hypothesis of we continue to figure out what parts of
building software are undifferentiated for us and we
continue to be able to hand those to other companies and pay
them and really pay them for use.
Even in 2007, this was true. I mean 2007 we had S3 and was on
S3 as a great, wonderful way to just store data that it would
stay forever. It was very cheap, I mean
relative to anything else at that point in time.
And when EC2 came out the next year, we had a company that
really needed not so much to dynamically scale, but we had a
scheduling problem. We would get all this data in
and need to process it and being able to like flex out to north
number of virtual machines and then close them off when
running, it was a perfect need that we had.
Then Hadoop came out and we saw wow then this all of the stuff
we were building ourselves we could actually like outsource
that even a bit more elastic map produce was Amazon service at
that point in time. And so I saw this trajectory of
I can take these parts of these programs that we're making or
the workloads that we're running and processing for our companies
and we can hand them more and more off to people and we can
break them into chunks. I mean from the beginning was
let's break the application into pieces and then use services for
the pieces. And so when Lambda came out, I
think in 2014, it was just another progression to me of how
do I break an application up and just not have to worry about
running things. And I was already at that point
in time I had started four companies and the amount of time
and pain that we had to spend as a fairly small staff just
keeping things up. I mean we used RDS early, but
like we had cases where the volumes filled up and it took us
down and so there was never a case where those services were
fully like I can sort of just delegate the running of these.
And so really feeling like I can delegate my uptime to a managed
service it, you know, starting in 2014, 2015, I was like, this
is it, this is perfect. But you know, interesting
immediately what I saw was people, in my opinion, designing
things poorly like looking at Lambda and saying, oh, we're
gonna like put every function in a different Lambda, they're
gonna call different lambdas. And we're gonna like take an
application which is here and we're gonna like spread all that
complexity out across like network latency calls to all
these lambdas. And so I started talking.
So I gave a talk at the first serverless conf I think it was
in 2016, 2015, 2016 hosted by A Cloud Guru and started writing
more and started arguing more about not only like this is how
we should build in serverless, also building serverlessly on
Firebase and even building applications like on Google
Sheets and Google Apps Script. Really trying everything and
interacting with a lot of skeptical developers who would
say, oh, that's nice, you get to build toy applications that are
very simple. The real developers don't use
serverless. It doesn't work.
It's too expensive. And so I started this insurance
company. Branch is a full stack insurance
company. We bear the risks.
Sell home auto, renters, condo umbrella insurance.
We're the fastest to be able to sell those bundles by orders of
magnitude over any other carrier.
And I was in all these conversations with people saying
you only know how to build toys, Serverless is just a toy.
And so I said OK, I think I need to address all of those
complaints and skepticisms about building serverlessly and really
write down now like no, we built a full stack in a financial
services carrier that has lots of regulatory compliance
requirements on it. We built it fairly serverlessly.
There are no VMS or no containers.
There's no Kubernetes anywhere. And so how do we think about it?
How do we do it? And why does this actually work?
Why is it not a toy? And so that has been the arc,
but I feel like it's been this sort of constant attempt to take
advantage of We get these new technologies and like they can
really change how we develop software and make it cheaper,
faster and better. And so that's been my focus and
I'm happy to share it in this book.
If you're very skeptical of it, this book is designed for you.
It's not designed to say you believe in serverless.
Here's how to do it. It's a book that's for the
skeptics to help you understand how this actually works.
Yeah, thanks for sharing your interesting story.
So actually when I read the book in preparation of this
conversation, I'm not a skeptic of serverless, but I get more
intrigued by reading what you are trying to explain in your
book, including Branch, right, which I'll probably will talk
more about later. Like what makes Branch unique in
terms of adoption of serverless? But what piqued my interest in
the beginning when you said right, so technology keeps
advancing every 18 months or so, right?
And the gap, I think you mentioned in your book, the gap
between the best software development teams and average
software development teams is getting more enormous.
I think it's related to all these advancement.
Maybe if you can pick a little bit like why you think the gap
is getting much bigger. Yeah.
I mean, I think if you think about like, if there's like a
huge change over 18 months, there's this compounding effect.
And so startups that get to start today get to take
advantage. There's no legacy, right?
They're not relying on any existing technologies.
So they can just use the absolute fastest, best way to
build something today. And if you started a company
like we did five years ago, you're already gonna be
committed to some choices that if you were gonna rebuild it
today, you would build it differently and it would be
cheaper, faster and better, you know, if you had all the
knowledge that you could have. And so we, you know, we look at
that ourselves even. And so in the book, I give an
example of Instagram, when it was acquired had 13 developers
and doing about the same thing roughly Facebook had hundreds of
developers. And then before that, you know
Friendster and Myspace had over 1000 developers.
And so there's. And actually Instagram had 13
employees. I think they might have had like
5 or 6 developers. And if anyone has gone through
the process of like how much can one developer do and then how
much can 4 developers do, There are these step points where you
lose massive efficiency per developer.
I mean, I tend to think of it as kind of like 1 developer, 4
developer, 12 developer, 25 developers, 50 developers, 100
developers. And once you're at 100
developers, there's like a core inefficiency that's in the whole
operation. You know, you're getting an
order of magnitude less productivity than if you have a
solo developer. Now, you can't run everything on
a solo developer. It's not a sustainable thing for
a company. But there's also a reason why
Amazon has these like 2 pizza teams and trying to keep these
services small. It's the only way to keep really
good quality and velocity together.
And so when I say that the top teams are orders of magnitude
better than the average teams, it's largely that the top teams
are leveraging technology so much better than the average
teams that they're just not bogged down.
And I feel that very much at Branch, the company that I was
working on 10 years ago built fax which was fully in the cloud
infrastructure as code, but like running on VMSI mean it's still
probably is. You know, in Amazon load
balancing, we were using right scale at the time to do some of
the orchestration. There was a core inefficiency
that we had in running and a core inefficiency that we had in
building new features. That branch doesn't have
branches, just faster. And so the average developer at
branch is just producing more for Branch and at a higher code
quality than the average developer was at build facts.
Yeah, it's quite astounding when you mentioned about this.
Instagram only have maybe 13 employees or five developers in
total, right? But can build a world kind of
scale, you know, like with users from all around the world.
And I think what you say is right.
So for people like me who has been around in industry for
quite some time as well, right. I mean the reason, yes, you can
actually see that, oh, there are so many technologies that I'm
not even aware of. But actually it's really cool to
probably prototype new applications, right, building
something from scratch and even like for example, you mentioned.
So you can actually build something like only Google or
Amazon can do last time, right? And now you can actually tap
into those technologies to build similar things like them.
And I think what you mentioned as well, right, If the developer
can produce more, that means they're focusing a lot more on
the right thing, right, which you mentioned as differentiated
versus undifferentiated things. I think this concept is very
important for you to actually adopt as technology leader or
technologist in general, right. So tell us more about this
differentiated versus undifferentiated?
Yeah. I think you always want to look
at what does your business need to do differently or your
organization need to do differently than other
organizations. And what are things that it's
fine for your company or organization to do as well as
other companies do, like the best other companies doing it.
And so let me give 2 examples that are both undifferentiated,
but that will generally, in my experience, get very different
reactions from lead developers. So let's start with the simple
one where I think we'll all agree.
So if I talk about sending text messages, sending text messages
is undifferentiated. You don't care if you send text
messages as well as the best other company sends text
messages. Now, by the way, there's
probably some listeners who are going then no, if you don't
register your 10 DLP campaigns, you know, you'll get the
messages rejected. Yeah, yeah, yeah.
But if you do it as well as the top people, you're fine, right?
If you can send text messages as well as you know a quality
things, you will be registering 10 DLP campaigns.
You know you'll have your short codes and you'll use Twilio or
something like Twilio. But Twilio is a great product.
I know it's a little expensive for what it is, but it's a great
product and so I don't run into a lot of lead developers who
will tell me no. I need to buy, you know,
separate boxes and hook up phone lines and have telephony cards
and like run them in a data center.
I don't find that although I did that in 2005.
I ran IV Rs that way so I remember that it was awful.
I would never do that. And so I think everybody agrees
that sending SMS is undifferentiated heavy lifting
and you should use a service like Twilio to do it right?
And in fact, you should probably use a service like Braze or
Iterable or Customer IO to like actually trigger those, cause
those help make it even easier and you can do templating and
have people do that. And if you don't know these
services, you should look at them up because they'll make
your life easier. And the back of my book actually
has a whole appendix on you should know all these services
'cause they're very useful. Now let me get to a
controversial example and I don't know why this is.
I mean, I know sort of why it's controversial.
Most lead developers want to run Elastic Search or like Amazon
Open Search themselves as their search index.
When there is a service called Algolia, they will do all of the
painful stuff for you. I will tell you that running
search index that's performing is undifferentiated.
You do not need to do it better than anybody else.
I mean, unless you're competing with Algolia as a business and
so you can outsource a lot more to Algolia.
It works phenomenally. It's really fast.
Elastic search, open search, they're very finicky services.
You'll have to be responsible for their up time.
Indexing to them is kind of a pain in the butt.
Generally speaking, front ends can't talk directly to them
because the security model is too poor.
So you'll have to build a proxy. You'll have to query through
your back end. It'll slow it down.
It adds more development. But most lead developers will
say, wow, I looked at the pricing for Algolia.
It's really expensive and I'm not quite sure, oh wait, how do
I do it in a dev environment and the right way to save costs?
And no, no, no, it's just safer. I'll run it myself.
I know how that works. We'll run these things, we know
how to run. But like that is classically
undifferentiated heavy lifting. Everything you're choosing to do
there to run open search or elastic search when Algolia
exists is undifferentiated. And if you take an actual cost
of ownership view, like the time that it costs and the people
that are needed internally to run that thing, Al Golia is much
cheaper, just period. But most developers will make
the choice on behalf of their organization to in house this
undifferentiated heavy lifting with something like open search
or elastic search because of honestly this misplaced stance
of how they should be optimizing.
And so that's my favorite example for this
undifferentiated heavy lifting. If you can outsource it, if it's
not something that makes you different, you should use a
service because you will always be asked to do more things that
you can build as a developer that are differentiated, that
are like special to your organization.
So don't add in things that aren't.
Yeah, I like their last line, right.
So you are always being asked to do more things that bring
differentiators to the business, right?
So I think not just Elastic search or such kind of a
technology, people these days run their own logging, you know,
messaging bus, maybe even containers, right on Kubernetes
and all that. So I think that takes a lot of
engineers hours and also effort and not to mention like if you
want to make it production scale with high availability, right,
that takes even much more effort and actually cost.
Yeah, right. Yep.
Yeah. So speaking about cost, right, I
think this is probably where the, I don't know, misconception
or miscalculation, so to speak, right, like why people are still
opting for managing things themselves.
They think, oh, it's open source, I can also run it
myself, you know, maybe build POC, it works pretty simple,
right? And they just start from there.
But I think the most important challenge is to understand the
real cost, right. In your book, you have some
breakdown of cost. Maybe from your advice, how
would you tell us to look at cost differently so that we can
actually see? The differentiated versus
undifferentiated. I mean, I think that you really
need to internally understand how much it costs for you to
develop and run things yourself in terms of your people costs.
And I think you also need to understand and in terms of your
velocity costs. So in general, the more teams
that you have in your company that are required to get code
live, the more inefficient you are.
And it's a compounding effect. In my book and everywhere I
talk, I recommend people read Accelerate and use the Dora
metrics. We use Linear B at Branch as a
way of monitoring the Dora metrics.
The cycle time is critical. And so you know, what I see is
that the more undifferentiated work you do, the longer your
cycle time and especially when the undifferentiated work is
running the things. And so when you use a service
like Algolia versus you are running, you know, your own
Amazon Opensearch or you're running elastic search on
containers and you're building a proxy service in there, those
are orders of magnitude different impact on cycle time
when you release changes to how things go to the index.
And I think if you're honest with yourself, both about the
actual cost of people and about the impact on something like
cycle time or the other dorometrics, it gets really
clear that you just don't want to run.
I mean, ideally use managed services or, you know, don't
write code, use managed services always gives you better
outcomes. Here now always does have an
asterisk. I mean, there are services that
are very expensive, that don't make sense for you to use.
So like, Algolia isn't the solution for everyone all the
time because, you know, if you have an index where you're
searching, you know, billions of records, probably Algolia is not
reasonable. But the way I generally think of
this is there's like 9095% of the time we're all kind of
building the same apps. They don't have like enormous
traffic. By enormous traffic, I mean they
don't have so much traffic that your data transfer costs are
like the biggest part of your bill.
Generally speaking. You know, none of us have more
than, I don't know, 1000 simultaneous users.
I mean, generally speaking, most applications don't have more
than like 15 simultaneous users, truly simultaneous.
But you know, 1000 simultaneous users, most of our database
tables at most are sort of in the 10s of millions at most.
You know, there's a way to set that up.
And so there's sort of a standard 9095% of the time.
And yet most or many lead developers I find always want in
their head to be designing for like huge amount of data
transfer like you know millions of like that's the model of
aspirations like built this like take.
But the problem is like those just need to sit.
Those are all exceptions. And most of the time, actually
looking at Dora metrics, actually looking at cost, you'll
find that managed services that are used by everyone at Twilio
and Algolia, Cloudnary, whatever, are just going to be
any sort of honest and fair comparison of actually taking
into account the cost. And I'm always shocked at how
many people just don't want to even think about how much
developers cost. They want to view them as like
developers are free, and so if you're a lead developer, you
don't know how much your development team costs and
you're making decisions based upon this is more expensive than
that. Like you need to find out how
much the team costs. I mean, like if you don't know,
you obviously can't leave making good decisions on what actually
costs too much or too well. Yeah.
So I think speaking from my experience, what I can see in my
area, right, industry start-ups, a lot of people, especially
during the tough time recently, right, the winter time some
people call it right. So they just look at the bill,
they will see for example all these managed service or
serverless, it costs a lot. The easiest one to reduce is
actually you know those Bill, right.
And since we have already a number of people, right.
OK, maybe let's just switch that to something that we can manage
ourselves. But what you're saying is true,
right? So at the end of the day, the
cost of the people needs to be calculated equally, right?
So you cannot just say I reduced the managed service bill, but
actually that cost is translated to your engineer hours, right.
And also probably headache whenever there's an incident or
trying to scale things, right, the on call thing will create
some kind of a cultural issue within the team as well, right?
So the harmony might not be as good as before.
And speaking about this managed bills, sometimes I find as well
the bill is so high because it's unoptimized, the usage is
unoptimized. So maybe also you can think of
just optimizing rather than switching gear to managing
yourself. So I think that's my experience
as well. We have these built in moments,
right, where every quarter or so we take a look at all the bills
that are more than $50,000 a year or $30,000 a year and we
just go ask you know, what can we do to get them down.
We move services a lot. And one of the benefits of
Serverless is because you have this really nice container,
right? I'm using this service, it's got
a well documented APII know what I'm using in it.
It's all isolated over here. It actually turns out to be much
easier to switch them and move them out than it is if you build
it internally. So we're on our like 4th
referral management service and we used airplane dot dev as an
automation tool and they shut down with 60 days notice.
I think today is the last day they're running and we actually
decided to in house what we had put there.
But it was so much easier to do it because we could just copy
their API and so we were able to take all this stuff we'd written
for airplane and port it over. You know, I mean, I don't want
to understate the developers who've been working on that have
like been working very hard and like week nights and weekends to
like get that done. But to think that we took a
service that we were spending like $60,000 a year on and we
were able to rebuild parts of it like a little Shim
infrastructure for what we were using of it in about in less
than 60 days. Probably more like 35 days was a
result of having used that service, was a result of having
this serverless mindset and of being able to say, OK, now we
understand the problem so well and we isolated it so well that
actually rebuilding it or understanding how to shift it
somewhere else ended up being much easier.
And so I love being in an environment where everything is
kind of contained and small. There's good separation of
concerns, and the best way to do that isn't micro services.
The best way to do that is to use a lot of managed services
because they've already built all those great APIs for you.
And then if, look, the case of it's too expensive or I'm locked
in is you just build it, then. I mean, that thing that's so
wild to me is people saying like serverless is about locking,
It's terrible. It's actually like a very cheap
way to like learn how to build something and then if you're
gonna build it yourself, just copy the parts of it that you
like. It's actually much faster.
Yeah, interesting insights, right.
So I think many people are, I mean do afraid about the lock in
and all that. But I think, hey, if you can get
started very easily and you can use all the power of the people
operating those services, right. So they are very well expert in
running those services, right. You can actually just use that
because, yeah, it will make you start really, really easily.
And I think you mentioned earlier that if you have a
choice of writing code versus picking up managed service, you
should always choose, you know, managed service.
And in your book you mentioned this thing called code is a
liability. So I think some people think
code is asset, right. Some people think code is the
most important thing in your company.
So tell us why code is a liability.
Yeah, I mean code is something you have to maintain and so this
idea doesn't come from me. And so in the book, there are
some good links here. But you should think about
anything that I write is something there that's necessary
for my organization to work properly.
Then I have to make sure has proper testing, like doesn't
regress, keeps working, and so I have to maintain all of that.
That's a liability. You know, the asset is the
experience you're building. And so if I can build something
with 10 lines of code, or I can build it with 1000 lines of
code, I'm gonna be much happier in the long run with the 10
lines of code. It'll be easier to maintain,
it'll have less bugs, it'll be easier to test.
All of those things will be much better.
And so the less code, the better.
At Branch, we have a kind of a waterfall process where when
somebody says, hey, can we build this thing and will you build
this thing? Our first question is actually,
do we need to build anything at all?
And so we do a lot of why don't you use this software as a
service instead? And if we need to make some sort
of integration, we'll do that. But like, let's just just go buy
a software as a service. That's step one.
Step 2 is kind of can we like down scope it?
Because if you buy a software as a service like you get all those
features, those are great. If that won't work, how do we
like take out what you're asking for and make it really small?
One of the things that I've found over my career, what
generally happens in organizations is people say I
want this thing and they usually make it really big because they
know that you're gonna go develop it and as soon as you
put it live, you can't work on it again for a really long
period of time. And so they're gonna pack
everything they can into it because that's the only way they
know how to get. If you reorganize and this is
very much in line with the Dora metrics and the Agile Manifesto,
if you say, look, two things, one, I'm gonna make you down
scope this thing to Tiny. But two, I promise we'll keep
working on it immediately. Like we won't stop working on
it. It's not a limited time window.
Then we work on other projects for a year, keep working on it
as soon as it gets released. It's got to be really small.
Then you get all this benefit of the feedback in the cycle.
So you just squash down what people want to like very tiny,
like just helping them a little bit, and then you get it live
and then they can see it, and then they can make that next
request. If you train an organization
that way. And it's easier as a founder
than as someone who's inside an organization with other
founders. But I can tell you, you can
train an entire organization to think this way, and as long as
you deliver on, we will make updates, you come to us, we'll
go get other small updates in as quickly as possible.
Everything gets more efficient because, you know when you build
these big projects, like they're poorly designed, you put them
live, they don't work right. We realize like, oh, that was a
bad idea. That's a bad interface.
People don't understand it, like, just build it small and
watch what happens and iterate on it.
So we have this whole process of buy SAS downscope, use a managed
service, use open source, everything we can to not write
custom code or to write as little custom code as possible.
Yeah, I think it's a good principle thought process for
anyone who listen, right. So code is a liability.
It's a reminder again. So if you think writing a lot
more code and infrastructure code or configuration also
counts as code, right? So you do remind yourself.
It does although and we don't have a ton of it.
But I do think that, and this may be an unpopular opinion, but
I do find that domain specific languages for infrastructure
like YAML and like cloud formation I do in my head put
that in a different category. And for this reason, my
experience is when we write cloud formation YAML and we get
it right, we never change it and so the maintenance cost of cloud
formation YAML is very low. And so I I don't really worry
about having a lot of that. In contrast, something like
Pulumi or CDK, that's like code. I find it gets a lot edits,
people keep updating it, it's got a lot of maintenance and so
it has a different profile to me.
And so I would view CDK infrastructure, Pulumi
infrastructure as code, as this code that you want to minimize.
I actually think of YAML as more like a YARN lock file or
something like that, Like it can get really big, but it's not a
maintenance headache. So I actually don't find lots of
YAML to have the same liability problem, because generally
speaking you get it up and it works and you don't cheat.
Interesting. So speaking about serverless,
right, I think we have talked a lot about serverless, but I
think it's appropriate to get the definition right.
Some people think serverless is, you know, Lambda.
Some people think it's a function as a service.
So in your view, right? What is your definition of
serverless? Because I think from your book
it encompasses more than just a function as a service.
Yes, and I give I think like 4 definitions in my book of it.
My favorite definition of serverless is it's not my up
time and so it's functionally about not having to run that.
If it goes down and it's my fault, it has to be like I
misconfigured it or I wrote bad code, but otherwise I don't
control the up time. But Ben Keogh also has, you
know, Ben Keogh's definition here is I'm only doing
differentiated coding, right? Everything that's
undifferentiated, I'm handing off and there's a bunch of stuff
about scale to zero and things like that.
If your view of serverless is And I I saw a post on LinkedIn
recently which had somebody saying serverless is expensive
and it'll send you in a ruin and it's too complex and just had
all these diagrams of like lambdas, calling lambdas and
using AWS services. And like I tend to agree that
those are I don't like those infrastructure.
I've never built one of those and I think that they're overly
complex and I I don't think he under.
I mean he's never read any of kind of the serverless
practitioners that I agree with the least about what serverless
is. And so to me Serverless is very
much about asking how do I use. Most of our serverless footprint
at branch is managed services. So it is not.
I mean, we use Lambda, we use Dynamo DB, we use Cognito, we
use AWS App Sync. These are amazing services.
My book explains how we use them and why they're so great.
And so we do use Lambda, but Lambda isn't everything in the
application. It's like where we need to run
some back end code. We run it with Lambda, but most
of the lambdas are getting triggered by calls through app
sync. But if you don't know what app
sync is like, to me it's like a key part of the infrastructure.
I also written a lot with Firebase.
I think Firebase is also wonderful.
I think what in writing kind of hobbyist apps.
I think Firebase is just easier to work with than app sync and
the Amazon infrastructure. So I like that Firebase
infrastructure for simpler applications.
The Amazon one is just much better for like financial
services. At least it's it's more robust
in a bunch of ways, but that's how I think about it is really,
am I responsible for uptime? As long as the code functions
properly, then that's serverless.
And so I often will tell people you know it's serverless to go
build your shopping site on Shopify.
Like that's a serverless architecture for me, and if you
don't think about it that way, then I don't think you're in
line with what serverless actually is and how to get the
business benefits out of it. Right.
And you also mentioned managed service.
I think for some people there is a lot of confusion as well.
What do you mean by managed service?
I think Twilio is the best example for people at a managed
service. It is cloud based API usually
that you pay and they have a well documented API.
They've got a whole operations team, it's multi tenant as a
service and you pay them money and they're just up and they do
things for. You.
Yeah, and I think in your book, if I remember, I also see like
for example, if you have to patch something, right, if you
have to choose the size, if you have to scale it somehow, right.
I think that's not serverless, right?
So. Right, right.
And there are these Gray areas, right?
Because in Lambda you will pick a size and so like I view Lambda
as serverless and I would just say in Lambda just by the
largest size because you get more CPU with it.
At least by, I don't know, 4 gigs, they default to this
minimum size. That's terrible and you
shouldn't use it, but you know, that's a Gray area.
But yeah, I I also think about it like, yes, if you have to
patch it, but also if it runs out of disk space, whose
responsibility is it to manage that?
That's a question I like to ask a lot.
And so if it's your responsibility if you fill up a
disk like, it's not circles, right?
So I think that's a good reminder definitely.
And of course when we talk about serverless, there are many
skeptics like you mentioned, right?
So many objections in the beginning we talked about cost
lock in, some people also think about security, right, Because
if let's say you use managed service for those managed
services like a vendor, third party, right, you don't know
them, your data spreads across different services.
What's your take about all these objections?
Maybe tell us how would you actually, you know, advise
people? Yeah, the most important thing
that you need is good third party risk management in your
organization. And I would actually say one of
the biggest hindrances to serverless adoption
organizations that really want to is that they have very poor
third party risk management programs in their company.
And what they have decided to do in the company, maybe not even
consciously. What they've decided to do in
their company is basically make it really hard to do contracts
with other companies as a way of security.
That's not security, that's just pretending you're secure.
And I recognize a lot of lead developers may not have full
control over this, but it's useful to know It's useful to,
like, have this in your head. Your company should have a
robust way for you to say I would like to use this company
and to be able to review it in a reasonable amount of time and
see if it meets the security guidelines for the information
that that service will get. You should have a good way of
looking at its historic uptime and evaluating whether you think
it's uptime is going to be appropriate for your use for it.
And then you know your finance department or your legal
department need to be able to review the contracts and make
sure that they have an appropriate DPA, for example,
and have other things. And then if they don't meet
those things, you shouldn't do business with them.
But I'll tell you, as a regulated US insurance company,
there are a few companies regulated, you know as heavily
as you as insurance companies. We use lots of these services
and it's not a problem, but it's about having really good third
party risk management. And that third party risk
management understanding that they exist to help us do
contracts and work with other companies because that's a key
differentiator in how fast we can build.
So getting the company aligned with that is a top down goal of
like helping everybody understand we need this function
within our company to be able to do these things.
But that's how you do it. I've been in charge of
information security, you know, many organizations that have had
a lot of compliance requirements and every time I run in, anyone
says, well, we we never pass compliance with that.
I'm just like, no, you just don't know how to do it.
I guess to quote Twitter, that's a skill issue.
It's not difficult, but it does require sometimes creativity and
it does require knowing your stuff.
You do need to understand like, you know, what is our
information security policy? Why do we have these rules?
Why do these regulations exist? And you need to be fluent enough
in it and have a good Chief Information Security Officer who
understands them and cares about productivity.
So you don't have all these things that can be very
difficult. But I can tell you it is so much
easier to be secure with serverless and manage services
than it is with an internal network.
And so like we are fully no internal network, fully zero
trust at Branch and the levels that we can achieve and what we
are able to do on the budget we have which is very low is
absolutely phenomenal and would have been completely
unattainable for me 10 years ago at Guildfax, which honestly has
what would be considered like a best practices cloud
architecture today. I mean it's full infrastructure
as code. It's got great network roles and
great in VPCS and everything, but managing that, proving that
and everything around that is so much harder.
And so, you know, didn't leave a lot of budget for other things.
And at Branch we have managed to make budget for just amazingly
high quality security. I'm very amazed when I, you know
read that branch is a insurance company, highly regulated and
yet you succeed in adopting all these serverless right, because
I think security compliance is always top of mind especially in
Fintech or you know financial services industry, right.
But I think what you mentioned is probably like it's more about
understanding the compliance itself, right, and be creative
in solving it and maybe you put more proper checks and balances
before you actually use those managed service.
And speaking about, you mentioned it's not our up time,
right. So what happens in incidents
where the third party is down, right, I think it's also one of
the most common objection. If those core services is down,
then we are also down and we can't do anything.
So what's your take on this objection?
Well, one thing is we run in US E 1, which is great because if
there are problems in US E one, we know a large chunk of the
Internet's going down as well. And generally speaking, we go
down less than the rest of the Internet.
So I've been in USC Swan since 2008 and this is what I found.
So I remember the outage in like 2012 and there was a Netflix
outage there. I just realized that, you know,
you get a lot of grace when it's a USC Swan problem.
So in the last five years, there was only one significant USC
Swan outage. It was a couple hours.
It didn't take us fully down. It was the one that like
affected Route 53 DNS where it became clear that that service
is bound in USC one and it's not really free from USC One.
And so you know everyone says why would you be in USC one.
But I'll tell you like if USC One has problems, like no one's
going to blame you because every all of their other products
aren't going to work either. And then outside of that, you
know we basically don't have outages.
We did have one in January 2022. So we've been selling insurance
since the middle of 2019. And so I can tell you other than
that couple hour USC one and this January 2022 where this
insurance specific vendor we used and so there aren't a lot
of other choices. This insurance specific vendor
we used went down. Other than that you know there
were kind of partial or degraded services from time to time.
Occasionally something won't be available like we have to pull
motor vehicle records and those come from the States and those
services go down like every other day.
One of those services down for like 15 or 20 minutes and so you
know you just build a lot of alerting around it.
And so our applications are very aware of that.
But in terms of the core infrastructure of app Sync,
Dynamo, Cognito, Lambda and US E One, I don't think those have
really gone down at all ever. Even in that US E one problem
they were all still working. I think Cognito had some issues
but was just slow. So I would say my experience
with it's not my uptime. I hired a company that is very
good at uptime has resulted in we basically don't go down.
We have much better up time than I've ever had running you know
in any of these organizations running my own infrastructure
and so we're running some of the infrastructure.
So I think it's kind of a non issue.
It is certainly possible. I I will give one sort of caveat
here is I did build an application using off 0 for
authentication and that service went down all the time in my
view. Not for long periods of time but
all the time and like I would never.
I mean maybe they fix this. I'm skeptical though after their
acquisition and like everything that's happening with Okta I
would never use off 0 again and I would be very cautious in
authentication services. But I can personally vouch for
Cognito and Firebase off, having been very reliable services.
Yeah. So when you speak about up time
and reliability, right, when you choose serverless, it's not like
any kind of serverless or managed service, right.
You also look at the historical up time you have, Yeah.
Can they guarantee a good SLA, right?
Is there any compensation that they will give or never?
This yeah, I tend to look, SLAS are non compensatory, like
they'll never compensate you. So I tend to like an SLA where
it's punitive to the company. What I'd like to feel is like if
you go down, it hurts EU. And I do think like Amazon use
these outages very personally and reputationally.
And I think it matters to them quite a lot.
And so they're never going to like compensate me appropriately
in the SLA. But I think the historic up time
is the thing to look at and just to make sure you're getting an
honest look at the historic up time because that's the best way
to tell. We do use some services that are
in beta and you know, we just do a lot of testing to get comfort
with them. But there are occasionally times
we'll say, yeah, we're not going to use that yet.
We're going to wait another six months or a year, but we use
Neon really early. Serverless Postgres because we
really wanted it, and Amazon, Serverless, Postgres are not
Serverless, They're terrible. And Neon is a great Serverless
Postgres database. And so we started using them
when they were in private beta. We tested them out.
We were like, how do we feel? And we've just built in some
backup caching in case it wasn't available.
And that's worked out really well.
Leon's been very reliable. Yeah, apart from the historical
up time, I think one thing that is quite, you know, recent
trend, right, people updating postmortem whenever there's an
incident, right. If the company diligently
uploads postmortem and they take it seriously, I think that may
also be an indicator that the company is serious about their
up time, right? Yeah, it's not a reason why
Amazon is so serious about that in a way that Google and
Microsoft are not as serious about it, But I just feel a lot
safer for production workloads in Amazon, although again, I run
them in the other clouds as well.
I think many people here would have been intrigued by Branch,
right? Let's talk a little bit about
Branch. What makes it unique, maybe in
terms of how you run the development team, how you run
the infrastructure? There are a few things that you
mentioned in the book. Maybe let's start with your
primary development principle, right, where you say when we
write code, we optimize for maintainability.
I think this is very interesting.
Maybe can you talk more about it?
Yeah, I'm always surprised at how infrequently development
teams will say what they're optimizing for in terms of what
they're doing. Like, most teams don't have this
principle. But it's very useful to know
when you're like looking at code or when you're sitting down to
write something like what is my guiding principle on how to
write this? My experience is if you write
something for a good organization that is going to
live for a while, you know I wrote code in 1997 that's still
being used widely in Pearl in 1997.
And so I think a lot about what is the most important for the
company. And I don't think it's about
making it speedy, right? I don't think it's about other
things. I think it is.
I want an average developer to be able to understand this and
work on this. I think far too many companies
are like, we've got these 10X ninjas, we write for 10X ninjas,
we're 10X ninjas, we just hire 10X ninjas and all that.
First of all, if you have a bunch of people who think
they're 10X ninjas, one they're probably not, and two, they're
probably writing unmaintainable garbage.
It's a lot better to have people to have the cultural identity of
like, it's important for me to write.
I do like the idea of like I'm writing this and next developer
has to work on this, has a gun to my head like write this thing
so it makes sense, so it's easy to read.
So that the code review that you have one you should have code
review. You know everyone, not senior
developers should have their code reviewed.
Everyone should have their code reviewed.
And we should have you know an understanding that I'm writing
this so that somebody else, an average developer can come along
and understand what the heck I was doing and like let's do that
And it makes code reviews easier to know.
Like this is what I'm optimizing for.
It's like having an elinter or having style rules.
You just like have opinions about things and everything gets
easier. And so that's our primary
principle. And there's a corollary to that,
which is less code in general is more maintainable than more
code. And so there's a lot of debate
about like, should you DRY things up, how do you think
about repetition? I think you can obviously go
overboard with DRY, but I do think in general, you know, it's
very common. We hire a lot of junior
developers at branch. It's very common for junior
developers to have patterns that are too repetitive that are just
much better and much more readable.
These are just everybody knows the pattern of like if I equals
one then you know return one, if I = 2 then return 2 or whatever
like that. Obviously that's a you shouldn't
that's too much repetition. Like, I find more junior
developers tend to not have these like ways of like how do I
abstract this? Like there's a pattern that I'm
repeating here. How do I abstract that.
So in a very normal setting, I find that I do reviews and like
somewhat frequently say, hey, you can get rid of some of this
repetition, like abstract some of this logic.
And, you know, I think that that's the key part of the
principle is to have a less good, but not at the expense of
maintainability. Yeah, speaking about developers,
I think this is also unique in branch, right?
You said that you hire more junior developers and actually
you have less so-called infra developers or engineers, right?
So tell us like, how do you actually run a good insurance
company with mostly junior developers?
Yeah, my hypothesis in starting branch was that, and I had seen
this at a small scale but not at a large scale before, is that if
we build serverlessly then mostly we're going to be
building interfaces. And so my belief was I could
hire front end developers and I could basically make them full
stack developers as long as we were using the same language,
JavaScript, TypeScript on the front end of the back end.
And I thought if it's not our uptime, if other people are
doing the uptime, then I should in theory just be able to hire
front end developers, make them full stack developers and then
that's all we would have. And then everyone would just be
a developer, and then everybody could work on anything.
They could work on the infrastructure, which is in
YAML. So they could work on the phone
and they could work on the back end.
And then we'd have it all in a mono repo.
Oh, and we'd have a mobile app. It would also be using
JavaScript and React as well. And you know, everything would
be kind of the same. It would all be the same
repository. We would deploy it
monolithically. Everyone would have their own
isolated Amazon account to deploy it.
And so this was the vision from the beginning.
And it worked. It worked so well.
And actually, let me take a step back.
So at the beginning I said I can't hire a typical senior
developer because a typical senior, I know what's going to
happen if I hire a typical lead who's listening to this podcast.
If I'd hire them into that, the average one of them would have
said like oh, I know how to build.
It's not like this. We need containers.
We need to go build some containers.
We need to do this thing. We need to do that thing.
And my vision would be eroded. And and I believe in empowering
people I hire. I want to hire someone and I
want to let them do it the way they want to do it.
And so I said I can't do that. So I said, OK, the safest thing
that I can do that is if I hire more junior front end developers
who I feel are capable on the front end side, like I think I
can teach them how I'm thinking about building this and I they
won't know any differently, right?
By the way, it'll be so empowering because instead of
just being a front end developer who's like dependent on other
people, they'll be able to work on everything, They'll work on
infrastructure and everything. And so we did this and it worked
really well. And we said, OK, let's hire
people directly out of boot camps now instead of, like, with
a little experience. And so we did that.
And we realized these boot camps aren't teaching people anything
useful. Like, we're having to, like,
retrain them on everything. I've got lots of thoughts about
boot camps. And so then we said, let's run
our own boot camp because the boot camps aren't useful.
And then we run our own boot camp.
And we did it again. And so you know, my opinion
today is if you set this up correctly and if you hire the
right people who don't have preconceptions about you know
how things should be done, then we have lots of people are very
happy in this environment. By the way, we did a search for
AVP engineering fairly early on, about 3 1/2 years ago and found
this wonderful guy Ivan Herndon, who had been an engineering
manager at Stock X, but mainly front end and but like really
understood everything and had taught at a boot camp.
And he was like fully bought into.
Like this is a great model. I like this, but it took like 8
months to find him. Like it was just like a long
journey, but we had built enough.
Then then we went out and hired more senior developers and we
basically screamed for like who's going to like this, you
know? And we found senior developers
who were like, I love this, this is great.
Like this is exactly what I want.
Like it's so easy and I have so much control.
And like again, when our senior developers are given a feature,
they do all of it. They can do 100% of everything.
The infrastructure, the front of the back, and like, we truly
have full stack developers, but they're full stack developers
because the stack is simple, right?
It's all the same language, It's all in the same repository,
right? It has a monolithic deploy.
And so we can make full stack developers if we make the stack
simpler. Otherwise it's much harder to
have full stack developers because you need to know too
many different technologies. And so it's worked phenomenally
well. We just have developers.
There's no front end, there's no back end, there's no API,
there's no infrastructure people, there's no OPS people
because it's not our Uptown. Yeah, I think it's really
interesting model, right? If people haven't heard about
it, I think this is a very interesting thing that you can
learn from Joe's book, right? So I think how to make it work
is something that maybe you need to work on more, like for
example training people, make sure your code is maintainable.
But I can imagine for example, having everything as a
serverless managed service, right?
Every developer is empowered to not just code and commit their
changes, but also bring it up to the production, right?
And even operating it in a sense because they can just see it end
to end. And many people these days try
to build platform engineering right simply because they cannot
do self-service right. They cannot make deployment
themselves. So I think this is a different
kind of perspective how you run the engineering team.
So if you use more serverless, I think you probably don't need so
much platform engineering effort, right?
So you can do it end to end. And I think there are many other
things in the book that you can learn from branch, like for
example, a few things that I learned, the department head
actually is the one who kind of like negotiate contracts with
the SAS, not like the central team.
And the other thing is like the engineering lead is the one who
configure the cloud or be the DevOps engineer kind of thing,
right. So I think that's also
interesting. Maybe one thing if you can share
right, people might be intrigued like how much Cloud Bills
actually Branch is running. Yeah, I mean I think in the book
I shared. So we have an Amazon account for
every developer and for every environment.
And every developer in every environment has a full copy of
production. And this is cheap when you use
serverless because everything scales to 0.
So like the average developer who's working at least 40 hours
a week doing development, their environment might cost four or
$5 a month because it's just not that much usage.
But yeah, I think I shared a bill where our entire all
developer environments, all everything was about $1000 at
Amazon for that month. We're now significantly bigger.
So you know, that production environment is probably more
like the full bill 7 or $8000 a month right now for branch.
But it's every environment, every developer, everything,
including like all of the continuous integration testing
that we have. So it's so much cheaper.
I mean at build fax in 2015, I think dams on monthly bill was
probably $30,000 a month and that was doing so much less
honestly in that environment, but having to run, you know
those VMS and containers. Wow, 78000 a month for all
environments. That's really all environments.
Yeah. And you are a full running
insurance company as well. So kudos for you to run that.
So I think we reached the end of our conversation.
So before I let you go so to speak, I have one last question
for you, Joe. It's been a pleasant
conversation by the way. So this question I called the
three technical leadership wisdom.
You can think of it just like advice that you want to give to
us to learn from you. Yeah, so I've got 3 here for
you, and I'll start with some things I've said before, which
is one you should have an optimization principle for how
to write code. It's just as important as having
linting rules or style rules. You should also have an opinion
about what you're optimizing for.
So we optimize for maintainability.
I think it would be fine at you know in some places and might be
we're optimizing for like the lowest latency execution or
whatever it is. But you should have that.
You should write it down because it will help solve problems and
it will help resolve questions and doubts when you're not
there. So you should do that and then
you know, my view is optimize for maintainability is a great
default. And the second thing is less
code is more maintainable than more code.
And that is just true outside of like obfuscation contests for
like small lines of code and so you should strive for less code.
Again, the book has sort of an interesting waterfall about how
do I what is a tactic when I'm being asked to build something?
How do I end up with less code there and then?
Finally I have a saying that I give all the time, so I'll give
that as my final one, which is along the same lines.
But it is that it is better to spend 2 weeks researching and
two days developing than it is to spend 2 days researching and
two weeks developing. And I'm always just amazed by
organizations that don't give developers time to research how
they would do something. And I'm actually less amazed at
developers who just want to start building.
It's the hardest thing I know. I mean, I always wanted to start
building too. But the more you can train
yourself and your teams to think about how do I spend a good
period of time where I'm just trying to understand what's out
there, What are the options and how do I make that a good thing.
And like, I'm going to throw away whatever's done at the end
of it. And this really you need
leadership to help you with this.
But I think as a tech lead, you can set this tone.
And I'll give us the example. Our airplane migration, when we
found out essentially like January 5th that we were going
to have 60 days, we actually spent basically all of January
trying to figure out what we were going to do.
We looked at a bunch of other services that were options that
we evaluated them and we talked to their teams and tried to
understand things. We looked at what it would take
to build it ourselves and what that would look like.
And so we spent, I think a lot of people would stay.
You spent too much time, you know, like not acting because
the service was going to shut down and it was critical
production functions. But the reality is you just
can't make the right decision unless you spend time
understanding what's out there. And today, if you're building
new things, one of the things you need to know what's out
there is what are the managed services that could do a big
chunk of this for me. And you're not going to
understand those unless you give yourself the time to play around
with them. And almost all of these
providers will give you free trials.
You may have to talk to someone. And I know a lot of developers
really hate scheduling the meeting and doing the Zoom demo
and stuff. You may have to do that.
I'm sorry, but it's still worth it.
You should do it. Don't say I can't do a self sign
up on this thing. I'm not even going to consider
it because there's no pricing page.
We can't consider it. Don't do that.
Go understand it. Even if you don't use it,
there's actually real important value in understanding what is
the service. Get access to documentation,
understand what that interface is, understand what that price
is. All of that will really help you
understand what you need to do better, even if you don't use
it. And we just don't do that.
We don't do any research and planning in dev.
It's like, oh, I know one way to build that, so I'm just going to
do it that way. And like, don't do that.
Do take weeks to research. It's worth it.
Yeah, not just research in terms of product, but sometimes like
design, right? Like how would you design your
solution? Yeah, And sometimes
understanding the problem itself, right.
Because sometimes. Oh, absolutely, Yeah.
All of them is critical, yeah. Yeah.
So thank you so much, Joe, for this opportunity for this great
talk. So if people love this
conversation, they would want to connect with you or find more
about your resources, your book. Is there a place where they can
reach out online? Yeah, I'm Joe Emerson on
Twitter. Slash X.
And yeah, my book's on Amazon. Serverless as a game changer.
I really highly recommend people to read it if you are into
serverless and if you're skeptics, I think you can also
read this book just for your knowledge, yes.
So thanks again for your time, Joe.
Thank you. Thank you for listening to this
episode and for staying right until the end.
If you highly enjoyed it, I would appreciate if you share it
with your friends and colleagues who you think would also benefit
from listening to this episode. And if you're new to the
podcast, make sure to subscribe and leave me your valuable
review and feedback. It helps me a lot in order to
grow this podcast better. You can also find the full show
notes of this conversation on the episode page at
techlitjournal dot dev website, including the full transcript,
interesting quotes, and links to the resources mentioned from the
conversation. And lastly, make sure to
subscribe to the show's mailing list on techlitjournal dot dev
to get notified for any future episodes.
Stay tuned for the next Techly Journal episode, and until then,
goodbye.