Hi everyone. My name is Patrick Akio and if
you just didn't serve it as Technologies, software
architectures and Fook talk, this episode is for you.
Joining me today is Jung Choi. He's a blogger consultant and an
aw, serverless hero. There's some good perks that
come with that title. I'll put all his socials in the
description below. Check him out.
And with that being said, enjoy the episode.
I mean, I've been to Portugal for the last month and I eat a
lot of foods, okay? I can't handle avocado as much
anymore. Like, it upsets my stomach and
there's one thing I can't handle any, coriander cilantro is not a
thing. Oh, running Portugal.
They throw it on everything. So the first meal I got, they
have this beautiful dish, it's called a cut up Lana.
It's like a fish stew and make this one pot meals.
Like a saucer, and they put on the table in the open.
It's like a whole presentation. And the waiters all thing with
it and some cilantro area, I was like, well, shit, I should have
said something, I guess. And then every single
restaurant, if I would forget that say I can't handle
cilantro, it would just be there and I hope you do that.
That's my fault. That's why it's take anything.
And I never knew that was the thing is, I guess it was similar
to me when I was in the thing, was it that we went to Europe,
right? After what after math?
I got married, we went to Venice for our honeymoon, beautiful.
It was it was great. But then, every single
restaurant goes to every single dish, you have all starts with
Mustard relish cheese and tomatoes.
So out of our street Asus three or four days straight of
everything, just being cheese and tomatoes, we're just the
time for something else to even just like getting a burger or
so, yeah, I'm gonna get that. I mean, I told you, I go
friends, Italian and it's just, they're simple Foods.
I could tell him food is amazing, but they're kind of
basic kind of generic. As you say, you had some bread
Montserrat tomatoes that the most beautiful, tomatoes, and,
yeah, at some point, you want some variety?
Give me something else. Yeah, I can't do it.
I can't talk too much about Italian food and, and bad in the
same clothes, variety. And yeah.
Okay. So something that's not so
tomatoey, that's hilarious, man. That's actually what I ate a lot
in Portugal as well because they do have amazing Tomatoes, but we
would switch it up like we live together during covid.
We moved in together and I came like before then I was living
with my parents, so I didn't really cook Couldn't cook
because my mom would cook for the whole family, then I live
with her and she knows how to cook pasta and I was usually
mostly a. So then we had to invent.
Okay, what else are we gonna eat?
Because I would otherwise, I would have pasta for lunch and
pasta for dinner, and it will be the same pass.
That would be also with tomatoes and mozzarella or eggplant pasta
or called Alfredo, which is the cold version of pasta.
And what's your Le? So, yeah, but then finally, I
got some, my mom's recipes, my parents are both Turkish, and
now we're starting to make Make more Turkish meals especially
now because we've been living together for like three years
now, eat a lot less pasta which to my surprise, she actually
likes as well and enjoys. Okay, that's good.
Yeah, our parents are going to come over and of this week.
So it's gonna be a long probably forehead foods and cultures.
Like I didn't know it was this much of a thing, but two,
Italian people food is very, very important where we eat.
And I mean, in Portugal. For example, she researches all
the restaurants that are there. And then we pick the Swans.
And if we have to wait in line for half an hour, she's like,
yes, there's a line that means there's good food, and I like
know there's a line. I don't want to wait.
So here we find a balance within there somewhere.
You should do end up waiting. And the foods really good
though, let's get the iodine. If I appreciate that, you know,
food culture. It's also very big we on, from
as well, originally from China. Yeah.
And they say, yeah, like, we're at the city, where I came from
the number one hobby that everyone have a, which is that
talk about new restaurants as self and yeah, even which ones
are good. It's almost like almost like a
like a Prestige thing that knowing the good restaurants
that the job just opened that no one's known and knows about yet.
That's exactly. Yeah.
Feels like yeah. I mean in Amsterdam, me of the
same thing. I don't know if you know there's
a new Pizzeria it's called Michaela's pizza and that one
originally was in the UK. That was the only Kerr, the only
place they had my girlfriend used to study there and I used
to travel there, come visit her, and we will get pizza there
because it was one of the best pizzas we had right now.
It opened up a joint in Amsterdam and it was a whole
thing. And apparently On nuke is there
was a huge line on opening day way too many people way too
little staff, so it was actually me quite disappointing, the
pizzas and stuff like that. Some people will get the wrong
order and just was a bit of a mess and now they're rolling a
bit better. But yeah, my girlfriend knew her
sister note, knew it was a whole thing.
They would like their opening on.
I don't know. March 13th.
And I'm like, how the hell do you know, if any everything's on
Instagram? Yeah, that's funny.
Yes, very funny. So I said this, there's a lot.
A burger restaurant. Well, but the joints in the
Amsterdam is so, and I've basically tried almost every
single one of the so-called best perk in Amsterdam and the one,
the best one, I found was actually, it's like a tiny
little joint just in the east and west of Amsterdam code that
had to have people. They do this really nice.
So medium-rare burger and it just blows away.
All the other ones, the so-called realized.
Yeah best book in the go there then.
Okay yeah check it out. Happy Bull and I think they only
open about five o'clock onwards. Okay.
So you can't just go there for lunch and fortunately, for the
weekend, but yeah, I said, I do a lot of research as well, in
terms of just know, we're which restaurant to go to.
Yeah, one of the things I noticed as well in Amsterdam or
maybe just Netherlands in general, eating out is quite
expensive. It is a to London.
Yeah. Like especially for Asian food.
You know, we go to some of the restaurants in the new center of
Amsterdam and, you know, it Every dish is partner 17 Euros,
27 euros and the same food in soft.
Chinatown in the in London would be 10 10 pounds or maybe even
less. So it's actually found it a lot
more expensive to eat out over here then compared to London,
which is quite surprising to me. Yeah, I agree.
I mean I've cleaned so long and quite often and I agree with
you. I'm stirring prices are high and
they keep Rising. Like, I've just come back and
I'm just going to be Dreadful. Because Portugal, everything was
cheap. Like, I will get a whole chicken
and so we don't cook them was 750.
I'm like, what the hell? Now, we get a salad.
Like, not even a big salad. Like, we're talking cucumber
tomato, onion, and that was also 7 57.
Mm, like, what am I gonna get the solid, but this chicken and
actually in language because I stayed in language of the month.
They had a place and it had the top 50 best burgers at how do
you say that? It was listed on the top 50,
best burgers in the world. Now, I don't know what listing
that exactly was, but I believe that like some marketing thing,
I'll go there and there. But that burger was delicious,
actually have pineapple and just are all the standards of a
burger, but it was really, really good.
Pineapple. It has a interesting ground.
Yeah, I also thought so, but man, that Burgers delicious.
I went twice smaller restaurants.
I went twice to it. I was like, I want, I want that
burger again. Yeah, this is good.
Food, thing I found was that that is very hard to find the
lamb over here. Even if I go to keep up over
the, Donald keep up, is its beef.
Yeah. And that's really strange as
well because of coming from UK, you know, doner kebab.
There's always lamb and get a lot of lives of lamb dishes in
general. But over here are fun and really
difficult to find the lamb tissues apart from say, like
gettin Curry or something like that.
You can get some lamb Curry but a lot of restaurants don't serve
lamb now in any reason why that's the case, no clue, no
clue. Like I make a lot of lamb dishes
myself. I go to the Butcher and get it
but our restaurant, I don't see it.
You're right. Yeah like up behind the no think
they got lots of beef and local roads of a pork.
But just, you know, very few, very few places you can actually
can lamb. I guess, I can, I haven't tried
butcher yet, maybe was the to the next time.
Yeah, I always go to the Turkish Butcher and there they have it
because I make a good Steward and especially right, sometimes
I love a good stew, but yeah, you're right.
I haven't seen that much. The only place where we get lamb
is from this thing. Is there like a Turkish butcher?
Course Sahand Mark Supermarket. Okay, at this, one in the East
Amsterdam, East News the park, that's the one.
I know that's where we used to live.
And we still get the motor push it.
There is a lot of really good you know, oxtail and things like
that and Lamb Chops but outside of that is we can almost never
find them anywhere. Now I would have to wait.
Yeah, I don't know like I don't like any of the Turkish
restaurants that are in Amsterdam and then like I've
tried a few and I'm like, it's just not as good and maybe I'm
biased because my mom's a great cook, but still, I'm like, I can
pick a better meal than this. So we don't really go too many
turkeys restaurants but I know like, It should be a lot of lamb
dishes in the Turkish Cuisine. There's a lot of eggplant for
sure like everything everywhere you go you'll find that plan on
on a Turkish Cuisine yeah we have lots of really good Turkish
restaurants in. London just lots of ideas all
over the place and there's always you can always get a
really good you know, Donna keep up or something like that.
Some no or lamb skewers or things like that.
Yeah, that's one of things I really miss for sure.
I went to one in London and then I was my girlfriend booked this
trip to Istanbul. For my birthday.
We went to a Whoa, and I found the same place.
So apparently the original one was in Istanbul eyes, and then
you had the one in London as well.
It was really good, it's really good.
I like it when food is cheap, when going out is cheap, and it
is not cheap here in Amsterdam. No, that's that's quite sad
actually. I guess food prices in general
are going up. I heard like, I mean, I've been
away for a month, but I've heard prices are even more increasing
now. And I guess, if you're here more
on a day-to-day you just don't notice it.
You build just keeps increasing slightly, but I've looked at the
supermarket. And I'm like yeah, everything's
like five percent more expensive again.
And I also guessed that they also making portions smaller as
well. Are there.
Yeah, I've had quite a few cases.
We have seen, you know, start the packaging spot the same but
then it's the actual portion that you get.
It's smaller. Yeah.
So instead of a raising prices or maybe as far as raising
prices, they're also making the portion smaller as well.
You also get the inflation and the string string cajón or
something like that, what does that there?
So take the string. The size is, but I think the
price is same. Yeah.
Exactly. And sir it's a different way of
doing the same cost-cutting solve a cost-cutting, I guess.
And the from what I heard London you can generally support even
worse in terms of the inflation and the supermarket prices going
up. Maybe you're more so than the
Amsterdam. Yeah.
A ballistics Energy, prices are coming down with been to keep an
eye, on my, have my energy bill, and the nice kind of back to
what I was saying. Before the, the hopes of energy
energy prices a crisis happened was started.
Yeah, I don't know energy bills. Just something that escapes me,
I looked at it and I was like, I think I should be paying more on
a month-to-month basis. Like I don't have a great
contract, but it was like your usage is less than a one-person
household. Right?
And Mike. How's that possible?
Like our apartment is quite old. So I thought if we heat up
something, it should take a lot of like heating to do that.
If I heat up the apartment where with two people, so we use a lot
of electricity, we use all the gas and at the end of the year,
they were like, no, no, you're going to get money back and I
was like, I don't know how I got into this position, but my
electricity bills were actually quite decent this year and it's
pretty cool. Yeah, so I'm happy with that
kicking myself that when I signed up with a budget that I
didn't say I didn't lock it in for three years.
Oh yeah, everyone lucky for like one year and then so just as the
prices go out, I was only 15. Prices for about two months and
then the 0 or suddenly, you know, you go from paying, I
don't know, 30, 40 euros a month to yeah 120.
Yeah. I was like crazy.
I thought I was going to live in this apartment for a year so I
was like, I don't want to lock in for anything anything more
than a year. So I was like, I'll just I'll
just do the bare minimum. I think I locked in for half a
year and then like three months extensions and then I was like,
oh my God, I made a mistake but apparently the bill was fine.
All right guess we're lucky with that one.
Yeah I mean I I looked you up online.
And, and there's a title that I always think, is interesting
because your says, 80 is served as hero.
Can you touch upon how you got that tight on the first place?
So it depends runs these Heroes program similar to the Microsoft
for MEP program. So is the recognition of people
that are contributing to the community in some way.
I think initially the program was targeting mostly people that
runs Community meetups event organizers, having said that,
but then they They decided to expand the test of Heroes
program to have a most of people that are not just all good, even
organizers. But just people that are
contributing to the to the community in some way.
And they wanted to I think have the separate several categories
for machine learning for developer tools for serve less
for containers. So the heroes program is so
nominated by internal what people internal to AWS. so, I've
been blogging for Hold on about maybe 12 13 years now and I've
been doing it for a long time, the Fairly consistently and I
guess I was one of the earlier, you know, served as adopters and
there's no on the earlier people does was really sharing a lot of
the stuff that was learning and to figuring out along the way.
And so when they broadened the heroes program, they serve set
up the the containers and and service category.
And I guess I was one of the people that was Nominated to be
the first batch of people that they invited into the service
Heroes program. And so that's how you get a
title. And this year they've started to
see, require you to have consistent one, more activities
so that if we don't have people who got the title and then stop
doing things and then they were still, because they as a hero of
in, they want to just make sure that they give a title to people
who are still continuing a contract.
To the active, and helping a community in some way, that's
also parallel to the heroes program.
There's also a Community Builders program, okay?
Which is a program that you cannot nominate yourself and
then they have, I guess some committee internally that the
sides are we are. Okay.
You get the, you get to get to be inviting to the program and
you don't have things like that, the heroes program.
I guess the biggest perk you get as part of the heroes program is
that you get A free free trip to reinvent so they pay for the
flight, that's a hotel. Yeah, and the conference take it
as well. So, and also, during the keynote
that they reserve the first, I think the first row of something
for the, for the heroes. So, so, yeah, that that's
probably the nicest perk that you get besides that, you also
get regular, I guess like no sessions with oedipus's service
teams so they give View like a road map of what they are
thinking about and occasionally, you get access to private beta
for new services or new features.
You can try them out. But usually when you, when they
get to that point where they engage the heroes and the it's
already quite late in the cycle. So there's not other things that
they can actually change. They would just get some
feedback. Yeah, and that's what I think
that actually, you know, I was hoping that they would maybe
change so that get people involved with the earlier, get
the feedback beer. And I'm seeing some of that
already. In terms of, you know, is the
acting team, for instance, getting feedback from people not
just from the heroes program before customers that's all.
Yeah, for like strings have been working on like the JavaScript
resolver things that released last year and as well as the new
merch API think that they feel released just a couple weeks
ago, she has it. So hopefully that's going to be
more of a common thing you see across the, it was a things but
obviously everything's different.
Different. Yeah, it's no one company with
like hundreds and hundreds of first Hops and so we're within
the within the same company. So it's just a lot of
inconsistencies in terms of how different teams that work.
But yeah, the heroes program is it's been it's been really good
for me, just in terms of you're giving me some, I guess the
authority as a consultant that know that really helps even
though, I guess, the program doesn't really recognize the
competence is not a competence program.
It's not Is a community contribution.
Yes. Like a visibility thing.
Yeah, but I still good thing for visibility.
Yeah, for surely. I mean the, the perky touched
upon going to reinvent, like Activia we get like a study
budget and it's rather substantial budget and people
save that budget until the end of the year.
When we invented the book, The Flame, the hotel, and the whole
conference, and they save up, and they spend all of that on
that conference. So, a lot of people will be
jealous if they heard. That's that's what you get.
Yeah, that's awesome. Probably not the best use of a
study. But they're making use of it and
making use of, it's good for this for me.
Now, the most efficient use of high school because he is quite
expensive and honestly, I mean, the last couple years, I know
last time, the last couple times I've been to reinvent, I haven't
really been to any sessions because all of them are
recorded. Another breakout sessions
anyway. Yeah, you go to the there, was
it a chalk talks. Those are more interactive and
the workshops are pretty good as well.
And spend most of my time in bed.
Is just no kachunk people. Because that's the only time
when everybody around the world hadn't just get it together.
Yeah. And the law people that are only
ever talked to online. Finally, the house to be my one
chance every year to see them in person.
And yeah, and this kind of know have that Network and after
socializing I which is what I tend to do.
Most my time with the reinvent, what a breakout sessions are
pretty much available the next day on YouTube.
So anything that's good that you can always just watch it on
YouTube and your own. Time and also at 1.5 speaker is
faster more efficient through exactly what it sounds like an
awesome experience that guy is so, but if I hear people and
talk about the stories that they have on there and just the
experiences and meeting people, it does feel like I'm missing
out so maybe I should. Maybe I should go.
Yeah. It's just that now you've ever
been before? I definitely definitely try to
go at least once especially now that the person events are kind
of back no back. I think the last week Was like a
mix of in-person and virtual. So from what I've heard from a
lot of people that when is stupid weird stupid, they're
awkward. But I think this year is going
to be fully in person. So so yeah maybe save up
yourself. Yeah, good stuff.
I mean, just to preface this. I'm a fan of service
Technologies. I haven't been using them as
long as you have, but from what I've seen is just I love getting
fast up and running getting out of proof of concept as fast as
possible. A low overhead with regards to
operations and think about that and just really focusing on
delivering value again. As fast as possible.
Is that also an especially early on what you saw where the
service technology was going. Yeah, yeah so I guess the, I
said, my first exposure to some Lambda with what 2015, I think
that was just before they introduced the Apex Gateway
integration. So at that point we're still
feels like a toy. We know.
Does the S3 integration there? The Canisius integration, which
is okay for some stuff. But then you can't really build
the entire application entire system on top on using just
Lambda and things like that. But then once introduced API
Gateway that became okay. Now, you can actually build a
rest API using API Gabriel Lambda and don't have to manage
any infrastructure and I haven't been on not been using any basis
since 2010. So I went through the whole
journey of using ec2 and the containers And then Lambda and
the remember back in the day I was just, no I was building very
different things but then you kind of do the same thing over
and over. Every time you got a new project
which is okay, if you know you decide your web framework,
whatever you need right application, this bunch of a
boilerplate stuff. You got to do it before.
You can write your first on the business logic.
Yeah, after that. Okay, there's a lot of the
polling place. Have you got to do around
infrastructure, you got to configure the networking which
requires a premature hold University creatures to
understand how different things you got to do.
Doris Auto scaling or the packaging machine images.
And also, don't forget, you have to do this every couple weeks,
every couple months to make this an issue that maintainer.
You've got the latest security updates and all of that stuff.
Which again, you know, just things that you don't have to do
once you give that responsibility to a debeers.
And I remember spending spending spending about a 70% of my time,
just doing a lot of that, you know, stuff, so that I can spend
30% of my time doing the stuff I should need to do.
Which I mean you still quite fast compared to what I was
doing before because I started my career in banking and you
know we were waiting for I don't know about 69 months to get a
new server into our farm and, you know, you you have this a
crazy meetings with 20 people talking about, oh yeah.
I need to have this thing installed on the new server and
it have that thing installed on the new server.
Yeah, and once installed everyone memorizes, the login
credentials, and an IP address because you going to be logging
onto the server, pretty, pretty frequently.
So that's like a whole year long project.
Again, you serve into your farm and then going to a DPS is a
five-minute thing. Again, I get a new server, so
that's already revolutionary, but then, you know, there's
still a lot of boilerplate stuff.
You got to do is especially as a DPS, adding more and more
control. You can have around the
networking and also scaling all of that stuff which is great.
But at the same time and if I just want to write some web app,
I don't really need all of that. And then unless I'm dealing with
a super high, a scale super high, throughput basic box than
the thing that works for. 90% of cases is just fine for 90% of
our cases. Yeah, and so no one slam, the
claims around and became more capable in terms of being able
to build API rest, apis with it and build it.
All these other things with it and it just became a no-brainer
for me to use it. And I haven't looked back since
just because I know how much work it takes to actually do
easy to is a containers and yeah now don't want to spend all of
that effort. Don don don things does not
going to add any value to my customers and also doing a
project for a client. I think about 23 years ago you
know the client had an idea for a social network.
The first server template is that they hired a guy to do the
build it in PHP and the they do something pretty quickly
together and then they launched it and launch, it just kind of
fell apart. They had two servers, I just
couldn't handle the number of people.
That's trying to login. Trying to sign up.
So it was kind of a kind of embarrassing, how how bad they
went. And so they got some more
funding and they had a second attempt at the second chance and
they really wanted to just make sure that they get it done,
right. It's get done quickly as well
because it's a there behind. Yeah, it's a social network for
University student. So I have to be tied to
semesters know. So, you know, so, you know, may
or September when the next semester starts, everybody's
going to be. That's when One's going to stop
start to create an account so that they can book a sporting
sessions with the University's the gym or something like that.
Okay. And so now that's a pretty short
deadline to get it done and the Indigent done.
And also, make sure that, you know, whatever they end up with
is going to be able to handle some kind of scale, which is
this is not like heavy around load, but it just in talking
about now, tens or hundreds people are signing up and okay,
him? The use the app the same time,
Which is fine. And so, yeah, we did with app
sink and the Lambda functions and things like that and that we
got to having done with in before weeks.
And I was the back when I was the one man that can team was
built a whole thing, including standing up the entire Oedipus
organization, multiple accounts. Following a lot of the best
practices around the security and an account management and
then build the application. But we also build a CMS as well.
Again, all that stuff, it just so easy and so quickly.
So quick to build So that we are able to get it the get it done
with before a deadline and also you know launched a lost people
signed up film, we've got 15,000 people signed up in the first
week or two. Everything's went smoothly.
There's no errors and no report for many customers.
So the client was really happy I can imagine.
Yeah, it's the sort of thing that we have taken like whole
team much longer to do if they have to worry about all these
other infrastructure stuff, worry about.
All of these things that besides what the customer is actually
asking you to build, Old because you do those things to be able
to you know scale to demand not have any problems when the
fitness you got more than a handful people standing at the
same time and things like that and we've served as you got a
really good. So Baseline for a resilience and
for scalability and for security and of course you know if you
have full control of your environment that, you know, very
well what you're doing and dedicated to doing specific
things that really well, you can do better than what you get,
real amateur but for 90% of cases, you don't need better.
You just need good enough and the world and the keeps you out
of the box is good enough for even better than most people can
do on their own because they have to worry about so many
other things besides the network security.
Yeah. Actually going to worry about
actually building the application that the Quran is
not your users want besides just, not focusing on security,
and anybody can dedicate the entire teams of people just
aren't looking after your network environment and you can
do much better job than most people can.
Yeah, then it's not your problem anymore.
It's not your problem anymore. And When there's a downtime me
show, you still have to worry about the resilience and things
like that and you get much better software based on
resilience just because you get multi AZ out of the box when
using things like Lambda and dynamodb and API, Gateway, where
you have to provision servers in multiple regions yourself, you
could work out how much capacity these have in each region.
So that if one region goes down, sir, wonder when a z goes down,
you still can cover all the traffic you've got, and you've
got to set up a load balancing and all these things like that.
Which not besides the fact that you more work to do but you're
also paying for a lot of redundancy that you're not going
to be using. Yeah.
And so for a lot about a lot of the applications, we see out
there unless they're doing really high.
Throughput Lambda is pretty going to be cheaper, and safer
and faster and better than to run your own infrastructure.
Yeah, of course, you got examples like the Prime video
where they got the sounds like, a very specialized workload, but
have to cater for quite High throughput, which they didn't
foresee at a time. So it's intense out there, okay?
Because of the traffic that they weren't ended up having to serve
to serve it made more sense to move the workload to containers.
And that's not that's one of the things that were once you get to
certain sources report and scale, using service component
is going to be more expensive because you're paying for that
premium per request. So if you have a really high
number of requests consistently, then the that premiums going to
be more expensive than say if you were to hire someone to come
in and On managing infrastructure and that's the
cost that a lot of people don't see when you think about.
Alright, I can run containers for another 100 bucks a month
versus running. This training on a peg where
Lambda would cost me, 150. Let's just run those containers.
Oh but wait, we need someone to let you do this thing can I hire
guy? Yeah.
Sure that'd be 5,000 bucks a month.
So your actual savings $50 versus you.
Actually finish a person you hire, which is Usually much more
expensive but no because it depends on how much internal
skills you have people that you. Can you say, okay, you're doing
containers for other teams. Already just no help us as well
which could be the case for large Enterprises, which has got
dedicated teams that it's going to be supported larger
infrastructure footprint. They have for lots of different
assets, but the for lots more companies.
He just doesn't make sense to hire someone to help look after
your infrastructure. When you don't have that
throughput to No to to make up the difference in terms of the
operational cost on a DPS versus the hiring someone to do the
same job. Yeah.
And I mean I've been a bit on Twitter as of late a little bit
more sense of doing this since doing this podcast.
But when the vine video example, for example, happened, people
love, shitting, all over there. Like either it's not good on a
cloud or they shouldn't use Lambda 2 should have used it
from the start like or microservices wasn't a good
option anymore. They're like look this is a
prime example. They go Haywire, they go all
over the place and I'm like, okay.
Well, when you start, you probably don't know what your
end situation is going to really help like you iterate, you
improve and this is a big Improvement for them, right?
And this is Prime video, we're talking about billions of people
using the service watching videos, right?
That's high throughput. Anyway, that's actually the
interesting thing, it wasn't Prime video.
It was the one team within Prime video.
Does that build a monitoring? And and what do people like a
tool that goes through the frames and the think they look
for some problems, you encoding or something like that?
Is one team. Not just not the whole Prime
video. It's know.
You also get to know Amazon is very big company.
Prime video is also not small team.
Yeah, so one team may be different may have decided to
do. Something doesn't mean that the
entire company has made way for a specific technology.
Yeah. And also even read a blog post
is that you're very very reasonable what it talks about.
You know you talked about how they didn't add it, is this
going to be this already existing to?
They're just building a new version of it when used by that
many people. So they build it didn't think
it's going to be high throughput.
But in terms of the to was such a quiet good.
So more and more team started to adopt the tool that the field is
internal tool steel. But I guess it's the, the scale
comes from the number of videos you have to analyze the number
of streams have to processing in parallel.
So and they could be a lot. It could be a lot of, right?
Because there's a lot of videos that you did that so very
important Prime video and so and so you know they they they they
hit a skinning problem that they didn't foresee which you know,
you can't blame anyone for not being the predict the future.
And so when the the context changed a refines the the The
review, the architectural decisions.
And also after having run this system for, well they can
specify they can spot the inefficiencies which may be
easier to build a shelter at the time but now becomes less
efficient. Once you have to operate a
certain scale. So what they're doing you know,
is textbook evolutionary architecture where you adapt and
evolve as your contractor needs and your contacts change and is
the absolute. It's absolutely fine.
But people just not read the text with a specific assumption
in mind. The thing that they wanted to
kind of look out for and they did the need, the nitpicking
small bits of of context out and taking it out of context.
And just all, this is the mic. I'm a sense.
Moving away from microservices, the can't make sense of
microservices find their own truth.
Yeah. Yeah.
So they say, yeah, it's very frustrating.
And anyone who's such a rate that the post for what it is,
with, with objective mindset that we see, okay?
This isn't it. This is normal.
Yeah, this is what I expect people to do to do.
I think based on what they know at a time and then and then
review and the evolved architecture.
When the when the change and one of small things that people
didn't so pick up on as well, is the fact that there's that one
paragraph that talks about how when they migrated from things
that functions to just basically have like a orchestration, run
it inside one container or something like that.
They didn't have to change much code because, you know, most of
the code is the same. Yeah, I'm so that talks about,
okay, people that taught you to talk.
About how vendor lock-in you know once you use Lambda you can
move out. It's just cold.
You know they were able to take the same code, maybe do some
small read write and put into containers and they go, right?
So it's it's it's also one of the things that talks about the
people don't didn't see the fact that okay, kind of, just can't
debunk, a lot of the so vendor lock-in are such a buzz term
when walking such lazy term as well.
Most of the time, just two people trying to so Russian
eyes. Why they don't want to do this?
Don't want to learn this thing was the time.
I think it's just because we're pumping too.
Identities that people have built with around the tooling
that use. Yeah, in terms of containers, I
mean, I've learned, I spent all his time learning containers,
you could be an atheist and now you tell me to learn something
else. Of course, I'm going to be well,
we've invested in this case of the I know.
Yeah. And and there's no Samantha
pushing a different sort of Paradigm, different
Technologies, kind of invalid in my skill set, my effort that
I've put into learning this other thing.
And okay, what's the, what's the easiest argument?
I can use a stir around the, which is, when the Locking,
which is the brief. Strating, because I've talked to
a lot of people that time to throw the argument at me, but
then they can never quite substantiate exactly what are
talking about me. Yeah.
But it's actually mean, and I mean, if he were the same thing,
the same sort of lock in exist with pretty much any position
utility take along the way. You know, if you use this a rest
of our now, I want to use the Expression as well, guess what?
You clarify the application. If I want to use say, MySQL now
wanted use postgres. Well, guess what?
A lot of your codes. Got a good because we've written
a lot of your work is Kirby has been redone.
The same thing with the moving technology from say containers
to Lambda. And that's a lot of different
ways. You can actually make that
portability a lot easier. There's the things, like, Hassan
architectures and modularity that you can do at your code
level, or that is just not quite easy to.
And we saw in the Prime video example that, hey, this is like
a real world example, some some, some of whom Of their workload
from using cell functions and Lambda 2.
Moving containers and didn't have to rewrite entire
applications. Just the orchestration layer.
Just some parts of the application.
Yes you'd expect because of the decisions they made.
Yeah. And I've had in the past that
projects, where, you know, we weren't quite sure.
What one, our previous jobs, full-time jobs was working for a
company called the zone is the is the the Netflix of a sport
streaming. So, you know, they do, they
stream live sporting events and pick the head like it.
Two million concurrent concurrent viewers when I was
there so bad. After I left this for you know
three million or whatever by now but you know it really high
throughput very spiky traffic. So certain parts of the of the
subsystem we just say you know you can't use Lambda because
we're hitting skidding limits that Lambda has around the
sudden burst of going from 0 to say how what was the picture we
can go? No, we've immediately instantly
and after that they can only go up by five hundred concurrent
execution. Superb.
Minute. So we are waiting that upper
bounds of weird sub L skin ability of Lambda so start to
which certain limit. Yeah, but for all of our apis
other desktop not on the critical path of user coming in
login, and start watching, say, maybe 90 playing my city for the
FA Cup. Final things like that, we can
know we use Lambda and functions a lot for love the payment
processing and other things as well.
Yeah, and I forgot where I was going with this photographer but
for years You know, it's, there's no technology that is,
that's the perfect fit for every single scenario and your
contacts is not always the same. So it's a contest change.
You need to know you can and you should evolve and review your
decisions and in terms of the, the riot are and everyone.
I was always going to go with that story.
I've said some of our apis that we want to show as the time I'm
right. Hit that if he's going to be end
up if it's going to end up running in containers or Lambda
functions. So we would never write it so
that using Ideas from a second architectures, where the Core
Business logic is wrapped inside, its own modules.
Yeah. And then we have wrappers for
say if know that we can use inside the express tree sap or
inside Lambda function that calls into our domain logic.
So that way we were able to really easily.
So just take the API and move into a container and running the
expression years or putting to Lambda functions.
So that probability is actually really easy to do.
Just a very simple code pattern that you can employ.
Where was the other thing I was going to say with the with the,
with the Zone example was that? Yeah.
So, you know for different apis. You know you're going to have
shit. Look at your different
requirements in terms of both what you need to do the
functional requirements but also your non-functional
requirements. Like what other soft scalability
you're going to need? If it's something that's for
example, for the Social Network that was users are used to
worked on, they don't have super snow, they don't have zero to
two million people. Requirement these times.
Yeah, that kind of piqued throughput.
But they do happen is lot of a really spiky traffic pattern
just because a lot of the traffic is driven by say,
influences doing something. So these are the social network
is built. When forward one previous jobs
are in back in 2016. That was warm, I guess, my
first, so Venture into service and we remove the work with the
team to move a social network to run, pretty much entirely on
servlets having Was running on containers before, we are lots
of problems around scaling and other things.
So we started to react attacked and so, fixed things that one
one piece at a time. And we thought the service was
really good fit for us. Yeah, and so we use the word and
I'll be using this triangular pattern to move things.
Okay, one vertical at a time into those interests different
services. And as we do, that rewrite, we
and we architecture, we will also read enough to use the
rewrite, the code into Lambda functions instead, and One of
things we saw was that the traffic pattern was raised
spiky. Sometimes you get no 20x the
spikes just because an influence, the post is
something. And so for those kind of spiky
traffic, that's an unpredictable Lambda is Lambda is really good
because you got a lot of scalability out of the box, but
with this is own example. The problem is that we hitting
upper bound of how far Lambda is allowed allows you to scale in
one jump. Yeah.
So, the certain cases where, you know the scalability, there's
like a range in which There's pretty good compared to
containers in the zones example, because a lot of that traffic
spikes are predictable is because there's a match on at
7:00 and everybody at logs in at the 659.
Yeah, so we go from like almost maybe a few hundred requests per
second to know, tens of thousand requests per second with in like
a twenty second, twenty second window.
So, that's kind of spike is predictable when it's going to
happen, but this is the spike is so high that it is so big that
we We were hitting limits on the what you can do with Lambda sirs
first capacity. Yeah, so forth.
That's why, you know, we made the decisions that for anything
that's going to be on that critical part of and it's going
to experience that kind of traffic Spike.
We're not going to put stuff in Lambda function, going to use
containers instead, we have a bill control around the scaling
just based on timing based on our schedule, for when the
matches are going to be on. But for others, no things that
worked on. Like The Social Network example,
where the traffic spikes are unpredictable, but then The not
so big that they were going to hit any sort of skinny limits
around Lambda. So Lambda is really good fit for
that. Yeah, that makes a lot of sense.
Did you work with area, Weinberger by, by any chance at
the Zone? I didn't want with him directly
but I was a, but I know of him and we had some conversations
afterwards as well. I think he also did a course on,
I think next JS or something like that or you need to me.
I think the quite well on the Summit is courses.
Yeah, I had some conversation with him but I never really
worried. Um direct okay, cool.
Yeah, he's come on the the podcast as well.
Okay I did a job interview this was like a year and a half ago
and he was sitting across from me actually okay on the other
side of the table and I remember he worked at the zone as well.
Some some interesting challenges with regards to traffic and
stuff there. Yeah.
So the zone of us are really good.
They're range think a technical Challenge and I think this this
it was so there's also I don't know if he spoke with many many
other at his own people at there are some really interesting
stories around how it is related to a officer was put together.
The haven't heard anything about that.
Okay, yeah, so he was put together very quickly.
I would hand out to people but then the high enough managers.
So there was some initial problems in terms of just this,
like maybe like a bit of a lack of Direction.
I guess that because you have too many people, but then
there's not enough managers and not enough for support structure
around them. Don't happen often those it?
Well, but yeah, it was. So, yeah, that there's a really
interesting sort of a learning experience for me, in terms of
just The scaling and organization very, very quickly
and the putting a lot of things that into place things like
platform teams and the hiring practices, and yeah, interview
practices and things like that. Awesome.
I'm thinking back because again, I'm a fan of service
Technologies. I'm a fan of this movement that
allows people to go to the cloud and deliver value faster.
And I don't care what service technology it is that much.
Because as you said, you have to be evolutionary with diversity
or technology, right? So you use what's most apt Or
whichever use case you have and not everyone is a Prime video,
so speak traffic like Peak traffic, you have to figure out
what your own Peaks are going to be, and if they hit higher than
you expect, then you adapt accordingly, right?
So, the normal process, the same is with your skill set.
If you've learned Docker and kubernetes, you still have those
skills, you can still try other things.
They'll be afraid to adopt new technologies in that way.
But then still in this era I'm again, going to reference an
article. I saw there was this article and
I think it was a few months ago where there Was this guy and
they said we're going to save like Millions, probably in the
five or six millions by moving off the cloud this time and
there was a whole post about. Okay, this specific part of our
infrastructure and disturbing. I think it was of images,
specifically don't know if it was images but in any case he
laid out this whole thing about the costs and the benefits of
running his own racks and knowing exactly what to do, but
for me, exactly the part that wasn't there was the operational
cost of the people that was going to maintain this part.
And I don't know if it's going to add up to the Millions.
But obviously people jumped on that the same with the Prime
video example, where they said, okay maybe we shouldn't put
everything on the cloud. Maybe in this case, wasn't such
a good idea. Do you remember the article?
I'm talking about. Don't stop that.
There were a couple. There was one from.
Think th the guy that founded the boot camp, and he was
talking about the hater cam. I think I that's the one, that's
the one, that's the one, right? Yeah, there's another one from
another company that's think they do SEO.
I've got a name tag. A now, but they were talking
about how they were going to stay.
Would have saved. They will have spent 400 million
or something like that on a TBS if they had one there's that the
stack on the database instead. Yeah, that was crazy.
Yeah, it's also completely wrong as well because I wasn't there
for that for the estimate. I think he just looked at just
this, some background back of the napkin calculations and what
car we are. This this this is how much we're
paying now and based on the cost of the, the CPU door.
And I'm going to get on a TBS surface of liquid equivalent
service on the hourly rate is going to how much you end up
spending. Yeah.
So very black and white. Yeah and also when you consider
the fact that I think we're also striking at the the amount of
demand money that Netflix spent on ec2 and use use, use it was
less than the amount that they they reckoned.
They was spent on a database for easy to and when you consider
that, this is SEO company. Not not.
Netflix, some pretty sure that they wouldn't have spent any
idiot nearly as much as they did.
They said. But, but, yeah.
And the, the a chick's article remember that, as well, he said,
he's, I don't know, he's the old one, he sir, he's a busy, no
quiet, very smart guy, and share a lot of things in his career,
but then the he say he's totally against his throat against the
microservices. And so the in the cloud I don't
really know where that's coming from.
The same time, I mean, I guess some of the same arguments you
see talks about how in some socket know.
Microsoft microservice is not right for everyone, but nothing
is the rifle. Exactly.
And I'm personally, I still think that, you know, monolith
is is nice. Good choice for a lot of use
cases. Even when doing Lambda function
meetings with served as you know I still quite prefer no booty
monoliths especially when I when I just put in the first version
of something, I'm the only person working on it, it doesn't
mean that they have to have. Of to have just one Lambda
function or something like that because Lambda function is not a
microservice know. It's just a component that's
part of your system. I'm only fair.
You know, I've got one code base, one repo, one deployment
for the once of system, and that system happened to be my graph
qap. I'm running an app sing with
bunch of different Lambda functions and down, TV tables.
That's the very prolific to me. So when I want to change
something, that's one, place to go, and one Department that
deploys, everything that's to me, that's not a micro services
and When I've got a company with no 500 people working on the
same system, I want to have microservices because the allows
me to scale the team so that individual developers only touch
one small part of the system. And the only need to understand
how that particular system. Works independently
independently and importantly, also, if they mess up, or
something happens to the system, is not going to bring down the
entire application, the entire system, right?
So microservices is, is really, really powerful tool for scaling
the Ation understand that they know for hate account.
Maybe they don't have hundreds of hundreds of Engineers that
the necessitates that micro services.
And also I appreciate that a lot of the one of the most common
misconceptions about microservices is that people
tell me this all the time is that all we are is there to
scale. Your application is absolutely
not necessary application. If you don't know how to scale,
I'm on the left. You're not going to know how to
scale. A micro service architecture.
Yeah, and one of the third, the most scalable model if I know is
I used to work for Company and they would take the gold bought
by a supercell and the from that I need and I don't know.
I've spoken with a lot of Superstar engineers and the
softener got a glimpse of their stack which is a very very
simple is one big jaw, Java application jar, they deployed
to easy to read in a horizontal scaled and hold on his skin and
the shore on his shot at the mySQL database.
Very simple. I to Street here application,
but I did they handle millions of millions of controlling users
just by scaling the model a very simple way.
Horizontally. Ali.
And if you don't know, those are skating patterns and apply to
Individual Services because we're looking the video services
within microservices is the same scaling challenge.
Yes, getting model. If a sponsor and a one service
one system. And so now is absolutely not
about scaling your make your application sparse, killing your
organization so that you can have the hundreds hundreds of
people working on the same project.
The same time. If you look at no construction
projects, we have 3,000 people working on the same site but
that if they always just hammering the same there, You're
not gonna be able to have that many people working on the same
project at the same time, right? So our service is the same way,
is that it allows you to segregate your bigger building
site into lots of different small area.
So that different people can just work on, you know, one
part, one floor, maybe a one part of the of the job like
mixing the the, the material. So I don't know.
Nailing the, the boards or something like that so that you
can have people working on individual tasks within the
sport smaller software domain, So they can become easier to
unborn new people as well. So yeah, microservices is good.
For some cases may be more around your organization.
Attend, the technical challenge? Are you trying to solve?
Yeah, I came into a project because I mean, work at a
consultancy company as well. And there they started off with
the micro Service Act architecture.
And I was like, okay, interesting because I wouldn't
start off with it like that, right?
And then very soon, we notice that two of the services were
very into dependent on Each other, and I was, the killer
makes sense, because we just decided those were going to be
separate, microservices. But when we needed to change
one, we also need to change another, and sometimes, even a
third one. So then, we're like, okay, these
are not really an interdependent.
So, we just made this kind of distributed system, and it
didn't make sense anymore. So, we were like, okay, wipe
that off the table, we're going to start with the monolith and
then we're going to see, okay, what parts need to actually be
separate from different teams. And we haven't even gotten to
that point. Yet is the same scale where
Justice that we're even faster because we don't have to go to
different. Repositories change different
things, it's all in the same spot and now when there is going
to be a second separate team because that's going to happen
pretty soon actually, then we're going to be like, okay, these
things were going to split off and that makes sense for that
team. That's how we're going to grow
organically, not by deciding. Okay, this is what it's going to
be in state and then working like that because that just, it
really diminished our speed. I noticed and it was a real hard
argument. Because the person sitting
across from me, was like, okay, it's really hard to go from a
monolith to microservices and I'm like, no.
The way we have a Now is really hard to operate.
It and it's way easier to transition when it actually
needs to happen. But that argument.
Yeah, we had to, we had to learn how to trust each other, and how
to actually evolve the system and wipe some things off the
table and redo, it just for the fact that we were losing a lot
of speed. I felt like yeah, after seeing
the quite a few times. In fact, that one of the things
that if he now you've listened to a lot of The microservices
Advocate, when I first thing to say is that don't start with
Wanna Live. Yeah.
Until you know the system, you know where the microservices?
Yeah. I start start with us.
Start with monolith. Yeah.
Until you you know, where the discussed the separations are,
and then then then you can break our things into microservices.
But yeah, the micro service is active Advocates, won't know.
They don't tell you to start with microservices.
Now they tell you to start with monolith, learn about system and
way it makes sense to split. Then you split it.
Ya like it says, no, it's not. I mean, I just go instead
straight away unless unless you're moving from a say, like
is an existing system. Where you already been running
this thing for a while and that's just that's the case with
the Zone. Where, you know, if they the
first version of it was built by a third-party third-party
company. Was one big one, the left.
So, by the time I joined a company and started hiring
hundreds, hundreds of ha of Engineers.
We already know about a system and whether the software where
the separation should be. Yeah.
So we had a pretty good idea in terms of how to split the team
teams at the time, were reasonably argue into an
identity, in terms of how the stability and the Zone also uses
a microphone as well, which is a microservices applied to the
front end, which is quite interesting.
But yeah. So a lot of that is because, you
know, we ended up with we're going to have a company with
hundreds of Engineers. We're going to want them
interview with to be able to own specific part of the system and
be able to iterate independently.
And also if one team have a problem with their services,
don't bring down the host, the whole system, which is actually
something that you have to invest in the so build with that
in mind. Yeah.
Because we Is really easy to build those into servicer, the
appendices that's the pipe coupling that one service goes
down, you have this recorded a domino effect, cascaded Cascade
failures. So you have to build those, your
system with that in mind and that's one of things that I
really love about. Event-driven architecture is
it's really a very natural way to to remove that tight coupling
between different Services is by depending on event from each
other as opposed to relying on synchronous API calls to
different server. Opportunities and services.
Yeah I think the for me the hard part and I'm thinking back to
this argument is experiences what drives people, right?
So if you've seen something work well in the past you're going to
lean on top of that experience and be like, this is what worked
well in the past. And if you saw something that
didn't work so well, in the past, it's hard to be like
objectively, look at, okay. Why was it that?
It didn't work that well, and why in this case, can it work
well, can it work better? What can we do differently?
You can still make it work. I feel like that really drives a
lot of arguments and in this Ace maybe even with the example of
interlocking, it really drives a lot of arguments and then it
fails or some people fail to look at it objectively and be
like, okay, maybe in this case we can do differently, maybe we
can do the same, but have it be a success instead of a failure
in the past? Yeah, I think that's one thing
that's missing from a lot of the online content is the people
give you the the hell. I know, something happened.
So we change but like I was right?
Look at it. Yeah.
But I didn't give you the context of why?
Yeah. And, and that's the one that has
the task does a bit. I should look out for that.
Know, why did that work for you and not for somebody else?
Because something always works for somebody enough for somebody
else, you know, medications, the same thing.
You know, I think we talked about intermittent fasting.
I can do one meal a day for a 10 ounce.
Maybe you can, or maybe maybe I can eat.
Avocado, you can't. Yeah.
So it's nothing is always going to work for some people and not
others. So that context is is how we go.
How you go beyond the surface or just Pattern surface that you
know, that naive fantasize that this thing doesn't work.
Yeah to say okay understand when these Frameworks and that's
something that I think is missing from a lot of the
content out there and that's what I want to things that I've
had a few colleagues and Friends cannot come to me and asked me
to serve, you know, review their blog post.
If that's one of the things I was always so kind of feedback
to be ghetto. People who are starting to to
block is going to the explain why explain the context, I like
that. A lot of people they go straight
into The solution to say, here's why I did hear some know.
Here's the, my beautiful elegant solution and one of the first
feedback I often, get give, you sir, sell the problem first, and
you people care about the problem, you know, you're trying
to trying to solve. They're not going to really care
about your solution or they're not gonna understand why your
solution works and for you but not for them when they tried to
copy it. Yeah.
And you always sort of brings me back to that that that that
scene from Wolf of Wall Street where you know his name, The
Punisher actor. Who's he He's got a, he was that
they were all sitting at a table and and DiCaprio's character
asked him to know. Sell me a pen.
He doesn't isn't going to describe how to pan is awesome.
You know, tells you oh yeah, what forgot, what were your
first is that? It was a whole story where the
end he actually needed the pan and then this was the pain.
Yeah, he created a need for a pen.
I think something right down like a check or something like
that. I'll give you a check but
independent writer, something like that.
Yeah, so he created need for the pain and then he starts out of
the pen and that's something you need to do for, you know, your
content as well. Create a need for running for
for your solution before you try to sell it to solution to your
readers. I mean, it's something that's
missing it at from a lot of content out there but waiting to
see them that they know what they're really good at.
Okay, explains, here's my see, here's my situation.
He's my context. And this is the problem that we
had and the solution. Here's our solution and it works
for a context because we are large company.
Mike Services works for us because we need to be able to
allow people to come in and just work on one part of system
without having to know everybody and everything else that's
happening in the system and be able to have an application that
is going to be resilient to small failures, too small part
of the seat of the application and the actual, some nice
examples, as well from what happened.
Wait, don't do that properly. One of the two companies I used
to work for and they have this very day went through, kind of
the microservices. So Revolution against large and
large company not Enterprise, but not okay, amateurs, stop
with lots of users and the last Engineers, they kind of did
their microservices thing on their service boundaries but
then they were still using a shared database.
And so there was one time when the Sphinx, someone from the
customer support team did something in, like, internal to,
which ran a really heavy query. No, took down a database.
Yeah. And what because they didn't
have a read replica for for the, for the custom support to so.
And then that because everybody, a lot of different Services was
using the same shared database. So you've got these tributed
model if. Yeah.
So it took down the entire system for the whole weekend.
The funny thing is, you know, these a food delivery service.
So you know the cost of moving for customers is really low.
So all the customers went to another food delivery service
and funny thing is the other services also wasn't doing very
good. In terms of architecture.
So it took down the outlet that other services as well.
So I guess the farm business point of view.
They didn't really suffer too much from that outage but it was
enough to for them to rethink a lot about the architecture and
the actually hire a really good person.
I know into the company just to fix that.
A lot of that problems that they didn't the that it had
displaying for. But yeah.
So yes again, that's kind of shows, you know, one of the
problems that you have with know when people I don't really
understand what microservices and just doing this of surface
level, you know, we just had a service but they're not really
following through and understanding that.
Okay? We have microservices Services,
should be autonomous, they should be independent.
Yeah they shouldn't be sharing the same database.
Otherwise you can create this implicit coupling and the single
point of failure which is one of the yeah.
Really. I sister but maybe not nicer but
very interesting story that I've had to have formed from that.
Yeah, I can imagine I mean it's the Same with people looking for
Solutions, right? Or explain the solution more so
than explaining the context is also.
Okay, I can put a micro servers up but it's really needed.
Like why do I need this micro service in the first place,
right? And do I understand why
microservices or how they work in the interdependently from
each other? What a, well, well laid out
architecture would be if someone runs a query on the database is
going to Cascade and it's going to like, flatten my whole system
or do we have a replica for that?
For example, you have to understand why and when to use
WhatsApp But that's not the fastest way because the fastest
way it's just the use the solution you think is right?
Yeah. But I can have severe
consequences. Yeah.
So that's the so that's that's something that's hard to
integrate integrate with were all of my yourself a Content
just know making sure that I always explain the context and
try to do as much work as good a job as I can to explain the why
the solution works. And we know when the solution
works because stuff, man, this was a lot of fun.
Yeah, I'm gonna round it off here.
Um thank you so much for coming on.
I'm going to put all yawns socials and links to his blogs
in the description below. Check them out.
They'll know you came from a show and with that being said,
thank you for listening. We'll see you on the next one.
Thank,