I mentioned LinkedIn had all of these issues.
We had issues delivering our software, right?
It doesn't matter if you're an app developer, if we can't ship
your code, it's not going to provide any value.
So we got to fix how we ship code.
What's the smallest amount I can do that gives me the benefits I
want while minimizing the drawbacks?
Your customers do not care what technologies you're using.
None of them care if you're using Kubernetes or service
meshes or whatever. Like they just don't care.
It doesn't matter. Honestly, your investors
probably don't care, they just want something that works.
The better approach is to use actual incrementalism.
And the key is you don't just chop it up into pieces.
But each of those pieces must be something you can deliver and
get value from by itself, even if the other pieces never
happened. That is real incrementalism.
If I was a betting man, I think the future is going to look a
lot, lot more like serverless than it does like Kubernetes.
I don't think Kubernetes is the long term direction of the
industry right now. I would argue that the state of
software delivery, DevOps, etcetera is not secure by
default. You usually you build something
and like all the networking is wide open.
Nothing's encrypted. If you have third party
dependencies, nobody verifies, nobody's keeping them up to
date, there's no monitoring, and maybe worst of all is a lot of
vendors charge extra money for the more secure things.
Like using single sign on usually is only in the expensive
enterprise plan, so we're the exact opposite of safe by
default. Hello everyone, welcome back to
another new episode of the Technician podcast.
Today I have with me Yefgeny Brickman.
So some of you from the infra world probably knows Yefgeny.
Yefgeny is a Co founder of Groundwork.
So if you are into the infrastructures code, you might
have heard about Terra Ground or Terraform.
He's the author of Terraform up and running.
He's coming up with a new book which is currently in progress
titled The fundamentals of DevOps and Software Delivery.
So today we're going to cover topics from the book, right?
And I'm happy to have a chat with you.
If Guinea, welcome to the show. Thanks so much for having me.
Right if Guinea, I always love to start my conversation by
asking you to share a little bit about your career.
Any turning points that you think we can learn from that
journey? Sure.
So really quick kind of big picture on career.
I started my career as on the app development side of the
house, software engineering, worked at Cisco Systems,
TripAdvisor, LinkedIn, and then started grunt work.
And there's been a lot of interesting twists and turns and
things like that throughout the career.
But maybe the biggest thing to share, and I share this with
pretty much everyone I talked to and shared in most of my books
as well, is just this idea of, of taking some time to learn
every day or every week. But basically what I generally
see is, you know, we go to school, some of us go to
college. And so we spend this whole time
doing this very deliberate learning.
And then you get a job and then you stop and people kind of just
stagnate. And so I've written a few books
and something that's really sad to me is, and I don't know how
accurate these statistics are, maybe they're dated, but I
remember hearing somewhere that people read something like 4
books a year and in the tech industry, less than one book a
year on the tech industry itself, which makes me really
sad. And you know, books aren't the
only way to learn, but I think the general trend is people
don't invest enough in just continuing to learn new things.
And that for me has been the single biggest thing that has
turned my career and, and let it grow, is just forcing myself to
spend a little bit of time every week doing some sort of
deliberate learning. And it's always fun, right?
It's always something interesting.
It it's not like I'm forcing myself to learn something I
don't care about. I just go and pursue the things
that kind of are intriguing. Oh, what's this technology that
I read about over here? What's this kind of cool thing
over there? And that can be by reading
books, that can be by playing with code.
In fact, it's better if you do a Little Mix of both.
And just learning without doing doesn't quite work.
And that's kind of the basis of most of my books.
As you learn and you do, there's tons of like hands on examples.
So that's probably the biggest thing is I would encourage folks
it's just carve out somehow find the time to carve out every week
to learn something. And I'll give just a couple
examples of how that's had this just profound impact on my own
career. One thing that I was just
randomly curious about in that must have been like 2007, 2008,
I started playing with CSSA lot at the time for building
websites and decorating them. And I got curious and I'm going
to skip all the technical details of it, but basically I
ended up building in my spare time a little tool that had to
do with like creating CSS sprites and things like that.
There wasn't really good tooling around that, and I just did it
because I thought it was interesting.
There's no other reason I didn't do anything with it.
It was just like a fun little hobby thing.
As it happens, about a year later, I had an interview and
questions around the stuff came up and I was literally able in
the interview to bring up this tool to talk about it, to share
with it, which was a big part of the reason I got my job at
LinkedIn, which had a pretty profound impact on my career.
And then I ended up meeting some of the folks that built some of
the open source tooling like around Compass CSS, if anybody
remembers that from back in the day.
So just this random little project literally helped me get
a job. Fast forward another five years
or so working at LinkedIn and something that caught my
curiosity at that time that I spend it just a bunch of time
playing around. Honestly, that's what it feels
like. It doesn't feel like learning.
It feels like playing around with this stuff.
I started looking at all sorts of web frameworks.
I was looking at Ruby on Rails, and at the time there was a
Groovy on Rails and all these different things.
Kind of became knowledgeable on these things.
And lo and behold, LinkedIn goes through this big dev OPS
transformation, something maybe we'll chat about later.
And one of the big things we end up doing there is changing the
web framework that we use within the company.
And I end up leading that whole project just because I had been
playing with these things and I knew about more about it than
most other folks. That project got me from kind of
the application work that I've been doing as a software
engineer into this DevOps infrastructure, soft software
delivery side of the house, which then led to me starting a
company in that space and writing books in that space.
So these little things that are just kind of fun, but you know,
I had to take time to do them, have completely caused my career
to to change direction in in very positive ways.
That's probably the biggest thing I can share with anybody.
Take the time to learn. Don't stop learning as soon as
you graduate from school or from college.
Do you need to scale your engineering team or find
specialized talent quickly? Finding good and experienced
developers is hard, even for seasoned engineering leaders,
and I've been there myself. The usual hiring cycles can take
months while critical features sit in the backlog waiting to be
implemented. Here's an alternative for you.
Check out Lemon dot IO. Lemon dot IO is the go to
platform for hiring top tier developers from Europe and Latin
America. They can match you with
experienced developers in just 48 hours or less, offering a
fast, reliable and efficient hiring process.
Lemon dot IO vets all their engineers through a rigorous
process that involves 100% human check on every stage.
And that's why only 1.2% of all candidates are on boarded and
listed on the platform. And here's the interesting part,
you work with lemon dot IO on a monthly basis with no pressure
to commit for long term contracts.
So go to lemon dot IO to find your experienced developers.
Now you can search by role or by skill and they will find a match
for you from within their thousands of vetted developers.
And you can get 15% off for the first four weeks of work by
mentioning Technical Journal when you reach out to them.
So stop burning money. Hire developers smarter with
lemon dot IO. Very interesting inside from
your turning points in your career, right?
So I think it's really key for everyone in the tech to actually
spend time to learn, right, Because obviously these days
tech moves so fast and especially there are so many
plenty of technologies being invented being, you know, I
don't know, maybe reinvented all over, you know, the time, right?
So I guess without learning continuous learning, I think
it's pretty hard to actually catch up.
Maybe, I know it's probably to generalize, right?
Maybe if you can give some tips for the techies here.
How do you actually spend your time learning?
Is it like every day you spend, I don't know, like focus time to
actually read books? Or is there any kind of a tips
that you would want to advise people to, you know, start their
own learning so they can be able to maybe, I don't know, like
improve themselves in their career?
Yeah, there's two things that I do and exactly how I implement
them has varied throughout my life.
It'll be different for everybody.
One, I used to have just a little tradition where like I'm
a night owl, naturally. So this is something that has
changed in my life. But for most of my life, I'd be
up really, really late. And so one of the things I used
to do is after everybody else around me, my girlfriend,
everybody went to bed, I would carve out like 20 minutes to
just either read something or watch an interesting video and
then play with the technology. And So what, what this ends up
looking like is I find that you learn really well if you both
have a little bit of like theoretical conceptual learning
and then you get to practice it on something very concrete.
So I'd usually find some little project that I just wanted to
do. And you don't have to like, no
one has to see this thing. I have like dozens and dozens of
little code snippets that will never see the light of day.
And that's fine. Some of them, a handful of them
made it through and I'll open source them or whatever else.
But that, that doesn't have to be the goal.
They're just something I wanted to try and play with.
So I mentioned that CSS tool that was just purely a side
project that never really saw the light of day.
Whereas the web framework stuff, I actually built a little
project using several different web frameworks and then actually
ended up putting it in production with one of them,
which was a Ruby on Rails thing. I put it on Heroku.
But the whole point of that project was I wanted to do
something, build something, but I didn't do it using stuff I
already knew. I explicitly built it with
things I hadn't used before as the way to learn them.
So then I'd watch a video on Ruby on Rails and then I'd go
and try to build something with Ruby on Rails.
I'd watch a video on something or read a book on something and
then go try to build something. So I think that's one of the key
ingredients is just find that it's and it's not a lot of time.
So this is the thing that really blows my mind is the difference
between essentially 0 deliberate learning, which is I think where
the average person is, versus somebody who just carves out an
hour a week. You know, just find 20 minutes
three times in a week sometime. The difference is tremendous
because that stuff really, really adds up.
It doesn't sound like much, but there's fifty weeks in a year.
The years go by and all of a sudden you've spent hundreds or
thousands of hours doing something that no one else
around you is doing. And you look like you're some
kind of amazing expert and you're like, I don't know, I was
like in my pyjamas like like 2:00 in the morning, like
playing with this thing. But you still are far ahead of
everybody who isn't doing that. So that's one piece is to carve
out that little bit of time when it is really various.
I used to do it late at night. I'm no longer a night owl with
kind of rework my schedule. Now I do it kind of a different
time of day. It doesn't matter.
Just find some little time. The one other thing that I would
suggest that really makes this effective is to also, if you
can, and this is a little bit of a bigger ask, but it has, it's
even higher leverage in my opinion, Share what you learned
with others. And that can be in the form of
writing blog posts, that can be in the form of giving talks,
doing podcasts, writing books, whatever you're able to do,
share it. And this is something, I think
it was a Steve Yegi blog post I read years and years ago.
What are the biggest fears when I tell people, you know, start
blogging, you know, start sharing what you know, is they
say, but who's going to read my stuff?
No ones going to want to read what I write, right?
I'm, I'm not good at it. No one cares.
Yeah, yeah, yeah. So first of all, that doesn't
matter. You're sharing this stuff
primarily for yourself. The act of trying to write down
and share something that you learned will make you learn it
better. It makes it stick better.
It makes you, you know, you'll do a little bit of extra
research. You'll connect it to some other
concept you hadn't thought about.
So the primary audience really is yourself.
So even if no one reads it, that doesn't matter.
But the reality is almost anybody can write something that
will find at least a small audience.
And this is this Steve Yankee blog post where he talks about
this idea where we picture in our heads that other people know
more than us, right? They have this giant mass of
knowledge. And I only know this tiny little
bit of the world. And that's not really what
reality looks like. What reality looks like is
everybody has these random, like a Venn diagram, random little
circles of the world that they know, and then these huge, vast
areas that we know nothing about.
And everyone, everyone's collection of these little
circles of what they know is different.
And everyone has a different path for how they navigate
through it. So if you start sharing what you
know, you are going to find someone else out there who's on
a similar path to you or on a path that benefits from what you
know, but might not benefit from others writing because it's a
different path. So you probably will find an
audience, and by writing you will learn this stuff a lot
better. And the other thing is you will
again, get these really random, serendipitous, completely
unpredictable impacts on your life, on your career.
And I'll give again a couple examples from my own career.
I'll give just one example because it's such a bizarre and
unexpected 1. While I was at LinkedIn working
on some of this web framework stuff, redoing Linkedin's
infrastructure, I decided to start blogging about that on
Linkedin's engineering blog. And so I was just writing and
kind of sharing what I learned and it found a small audience of
people that found it valuable. That led to a few talks, which
was kind of cool. Got invited to some conferences.
But the really unexpected 1 is I was at work one day and this guy
who I've never seen before just comes up to my desk and he says,
hey, you're, you've got any brickman, right?
I'm like, yeah, you know, who are you?
He's like, oh, I found your blog posts on this framework stuff
and I read them and I really liked them.
And I thought, oh, this is so cool.
LinkedIn is doing this kind of cutting edge, interesting stuff.
And that's what made me decide to be OK with LinkedIn acquiring
my company. So this was a guy who had
started a little startup and they got acquired by LinkedIn.
And obviously this wasn't the only factor, but like, it kind
of shared the culture and it made him excited.
So I got to know the guy who's a really smart guy, really
interesting. So first of all, that was just
one random thing where I just connected with him.
And believe it or not, a few years later, I leave LinkedIn, I
start grunt work. And one of my absolute first
customers is this guy. He had moved on to a different
company. They needed some infrastructure
work and they ended up hiring grunt work.
And it's one of the customers that helped us get off the
ground, honestly, as a company, all because and and he knew he
could trust me and he met me and all of this all cuz like years
and years ago, I'd taken a few hours to write a couple blog
posts on some stuff I learned. So you never know where the
stuff will go, but it probably will go somewhere.
Almost everyone I talk to who regularly shares their work.
And I'm sure you've seen this in your own career.
It, it just has this weird, unexpected, profound impact.
So yeah, I guess in terms of how to learn, find a little bit of
time and then if you can, share what you learned with others.
Thank you for sharing your interesting personal story,
right? I think I can relate to some of
what you've shared just now, right?
So I think the first thing is cover time to actually do some
kind of learning or maybe do some project, right?
You can do some mini project even like I had one guest
before, John Cricket coding challenges, right?
He advised people to actually do some kind of mini project to
actually get hands on experience and learn some fundamental
concepts. And sometimes actually
initially, if we cover up maybe 20 minutes, if you like that
kind of project, you'll easily find time to actually do more
because simply you enjoy what you do, right?
And I think sharing definitely, yes, I find also the great tips
just now you mentioned, right? Don't share it for other people.
Share it for yourself first. So I think make yourself as the
first audience. So one thing that I also pick
interest from what you shared in the beginning, right?
So you came from the application development background and then
gradually, slowly moving into the infrastructure and you know,
DevOps kind of world. I find some engineers actually
are interested in doing this, although some people actually
find it really hard because they are typically two different
worlds all together, right? One is more like application
design, system design and things like that, while infrastructure,
maybe things like cloud, you know, data centers, you know,
servers and things like that. So maybe from your experience as
well, how do you actually bridge the gap between, you know, like
knowing application development and then moving into
infrastructure world? Because for some people, this is
typically a hard thing to do. Yeah, that's a good question.
So one thing that I'll say part of the reason I ended up on the
infrastructure side of the world was I mentioned LinkedIn had all
of these issues. Honestly with all of its, the,
the term DevOps didn't really exist or had just appeared on
the scene. So we didn't call it that.
But we had issues delivering our software, right.
We had a bunch of app developers who wrote code and built this
stuff and then getting it out to users and running it and
maintaining it and scaling it and securing it.
Those were really, really, really tough.
We struggle with that enormously, and so part of what
got me in there was kind of being thrown into the middle of
this and it's like, well, we got to fix it.
Doesn't matter if you're an app developer, if we can't ship your
code, it's not going to provide any value.
So we got to fix how we ship code.
In terms of learning it, I personally struggled
tremendously to learn it. I found the space, especially
back when I was starting to do this stuff, like 2011, that's
roughly the time frame. So it's a little different now.
But back then it was just legit hard.
It seemed like there was this app development side of the
world, which like takes a while to learn, but there seemed to be
a lot of resources for understanding it and learning
it. And then there was the other
side of the house, which was just felt like magic, right?
Like there's just these people that like literally knew these
little magic incantations in Linux or in DNS or in whatever
else. And then they would wave their
wand and suddenly your app was running and then it would crash.
And they're like out on vacation.
You have no idea what to do about this.
And it was, it was very frustrating, to be quite honest.
So I kind of learned it the hard way.
And I think a lot of people learn it the hard way.
That is something breaks. You go through a tremendous
amount of pain. You maybe fix it.
You probably fixed it the wrong way.
It'll probably break again. You'll be awake again at 4:00 in
the morning. You'll fix it.
And eventually you kind of figure out better and better
ways to do it. But it was a slow and painful
process, to be just honest, that one of the reasons I built grunt
work is 'cause this stuff was driving me nuts.
Like, it's a little weird to say, but I started a company to
work on something that I just really didn't like.
I just really did not like this aspect of software development
where I'd have a great idea, I'd build it, I'd design it, we'd
all be excited. And then you're like, oh, God,
how do we get it out there in a way that's like not a security
nightmare and like is maintainable, right?
You can like, deploy it, but then how do you update it
without breaking everything? It was just really, really tough
space. And that's part of the
motivation for building grunt work was all I could think to
myself was there must be an easier way to do this.
And this is also the motivation for writing this new book is I
couldn't find a single kind of comprehensive resource for
learning this stuff. As I said, I had to learn it the
hard way. Piece it a little bit from here
and a little bit from there, some from doing it wrong.
OOP, I found a book that accidentally mentioned, oh, now
I understand how to do this. This guy over here knows how to
do it. This person over there knows how
to do it. So I tried to, in this book,
kind of bring together. A very hands on guide to the
most common things you have to deal with to deliver software.
There are a lot of resources these days on the DevOps side
that will teach you the cultural aspects of DevOps, and I think
that's also incredibly valuable and I do recommend reading
those. At the end of the book, I have a
whole set of recommended reading on a variety of topics including
that. And so things like, you know,
how do you structure your teams or how do you do a post mortem
or how do you do things like error budgets and SLA's and
SLO's and all of these things. And I think those are really
valuable. But as far as I know, I still
haven't found a single guide that's like, hey, you built an
app, you know, here's your little Java app, your Ruby on
Rails app, What's next? How do you put it on
www.something.com so everyone can use it and it scales and
it's secure, etcetera. So that was a big motivation for
the book as well. So I guess my answer now in 2024
for how to learn this stuff is, you know, get my book,
obviously. And beyond that, there's a few
other things that are quite valuable.
I list, again, a whole bunch of them in the back of the book.
There are a few guides, the cultural aspects of DevOps that
I think are very solid, and there are a few online courses
that do a decent job walking through things.
Other than that, do it. Just practice it.
Find some kind of a side project, find something at work,
find some way where you need to deploy an application in front
of other human beings and just start practicing it.
And I think that'll make all these concepts kind of come
together. So it's still not easy to learn.
It's a little easier. And maybe with this book it can
be a whole lot easier. And one of the things I write
right in the preface of the book is I'm really, really hopeful
that we're going to have a generation of application
developers and infrastructure developers who can learn this
stuff not the hard way. Because here's the other truth.
When you have bugs in your application code, occasionally
there's some pretty severe bugs. And we all have, you know,
occasional nightmare stories, but usually they're minor
things, they're annoying, and you kind of fix them.
When you have bugs on the software delivery side, you tend
to like, just take everything down.
You tend to like, accidentally delete your production database.
You tend to have a security breach, right?
Like these are serious problems. So learning the stuff the hard
way is not only just painful and unpleasant for you, it's also
legit harmful for the software industry in general, right?
Like we're building software in a way that isn't very secure,
that isn't very reliable, that isn't very stable because people
don't know how to do it. So I'm really hopeful that the
next generation of folks that comes along will benefit from
some of the stuff and maybe learn it just a little better
than than what I had to do. Thanks for a good segue to
actually explain the motivation behind your book, right?
So I find also like for application developers, right?
It is definitely very, very valuable if you also know the
infrastructure or software delivery side, right?
Because you cannot just write code and let someone else ship
it, you know, like making it into production, right?
If you actually know end to end how you can deliver your
software to production and to users, even better, right?
If you know how to scale it, if you know how to make it secure,
right? And making sure how you can, you
know, update, rollback and things like that.
I think that will make you as a very, very valuable engineer.
And those skills is really valuable in the industry.
So I think you mentioned something about software
delivery that is a hard thing to learn.
You typically learn from your mistakes.
And there are so many concepts, especially now you know, you
have cloud, you have different products, different paradigms,
like for example, VMS, containers, serverless and all
that. I think there's just too many
permutations for some people, right?
And I think I find your book is pretty one of the rarest
resources that you can actually use to learn because it's
there's a mix of infrastructure related and there's a mix of
application development and software delivery practices as
well. So I highly encourage people who
are interested in learning this to actually pick up your book
and actually try to learn from there.
So you mentioned about the story of LinkedIn, right?
So initially they were struggling, you know, to deliver
great software, right, in terms of, you know, scaling and all
that. So what makes these kind of
skills really valuable? So maybe if you can tell us why
someone should learn how to deliver their software in a much
better way. Yeah.
So I'll add a little bit of context on this LinkedIn story
and that that might help answer this question.
So back in, it was around 2011 to some extent, LinkedIn was
looking great. We had just had our initial
public offering. The stock price was going way
up. The product was doing great.
We were adding something like 2 new members to the website every
single second of the day. So it was just crazy hyper
growth. It also sometimes felt like we
were like hiring 2 employees every single second of the day
because the company internally was also growth through hyper
growth. So things seemed really, really
good. Revenue was way up etcetera.
But the reality was we got to the point with our software
delivery practices where we could not deploy.
And I don't mean hypothetically, I mean like in actual practice,
what we were doing back then is there'd be a deployment once
every two weeks. And so this was like this
release train model. You know, if the train leaves
the station, either you're on it or you got to wait for the next
1-2 weeks from now. And so we had one of these.
We did all the stuff, the release branch, etcetera, and
then we went to deploy it and we rolled out a bunch of code and a
whole bunch of things died and broke and crashed and things
were pretty bad. So then we spent a bunch of time
bringing everything back up and trying to fix the bugs and
rolled out new code and that broke more stuff.
And then we rolled out new code to fix those issues and that
broke more stuff. And it went on for a day and
into a second day. And we just could not stabilize
the new code. And we literally just have to
roll everything back and kind of patch things up and mostly get
things back to working before the deployment started.
So we literally could not deploy.
Like we were completely, totally stuck.
And so the company had no option.
At this point, a new VP of engineering came in and he
basically saw no other option. And we had to put all product
development on hold completely. This was something called
Project Inversion. You can Google it if you want a
lot more of the details, but literally we just said no more
product development. Every single person in the
company, whether you're an app engineer, whether you're a
designer, marketing, whatever, is going to go and work on
internal stuff and tooling. And again, there was no DevOps
term at the time, but that's basically what we're trying to
figure out. It's like DevOps processes and
software delivery practices until we could reliably and
quickly and effectively ship code.
So maybe the very first reason to learn software delivery is so
you don't get into that state. That's not a good place to be.
That's the kind of thing that put the entire company at risk.
And we got through it and you know, we went from deploying
once every two weeks or or not as the case may be, to deploying
hundreds of times per day with far fewer outages and issues.
So it ended up being a success story.
A lot of good technology came out of that work, like Apache
Kafka was developed at LinkedIn during this kind of period.
It worked out OK, but it could have gone very, very poorly.
And so that, that's one reason that it doesn't matter if you're
app engineer or whatever your career is understanding some of
these basics of software delivery, you, you don't want to
go there. You don't want to get to the
kind of the darkest long nights that you can get to. 2nd reason
to learn it is honestly to be independent.
There are companies where you can work, and I early in my
career did this as well, where you're the app engineer, you
build a thing and then you kind of toss it over the wall and
again, somebody magically deploys it for you.
That's actually more rare these days.
But even if you are today at a company where that's the case,
there's a real good chance you won't be tomorrow.
One of those reasons is you might want to start your own
company or a side project or something for fun.
And so understanding how to take your app and do something useful
with it so it works in production in a reasonable
capacity is incredibly liberating.
This is one of my favorite kind of discoveries as part of this
process is all of a sudden I could take any idea I had and
ship it, right? This is one of the things that
gave me confidence to like start a company and also do a lot of
other projects even before that. It's just understanding some of
these real basic things. So being independent and these
days more than ever, I think these days it's hard to get a
startup noticed. So the marketing thing is still
really, really hard, but oh man, is it easier to start a startup
now than ever, ever before, because of the cloud, because of
the technologies out there, because of the knowledge that's
out there, because of the reach through the Internet, through
mobile apps or things like this. It is easier than ever to start
your own thing. But if you don't know how to
deploy things, you're not going to go very far.
So that's I think a real good second reason to learn this
stuff. The third one is even if you're
in a place where somebody else does the DevOps E software
delivery stuff, for you to call it out frankly, and I can say
this 'cause this was me early in my career, you are not a
particularly good app developer if you don't understand how the
stuff gets deployed and maintained on the site.
The code you write is probably terrible.
And again, I can say this because the code I was writing
was terrible. I see that now.
And you understand that once you understand things like you have
to understand the architecture of your site, right?
That is kind of the one of these places where app development and
infrastructure development overlap.
It's the architecture, right? Is there one copy of your app
running in the world or are there 200 copies on different
servers? Are you a single monolith or are
you 127 micro services that communicate over the network?
These have profound impacts on how you build your application.
Security is another huge one. The amount of insecure code out
there that exists just because people don't have the concepts
of what is really happening on these servers is kind of
terrifying for our industry. So that I think is a huge reason
is you don't have to be an expert in this stuff.
You don't have to go super, super deep like the people that
are handling this. But if you are completely
unaware of how code is deployed, scaled, you know what high
availability means, things like CAP theorem, etcetera.
If you're completely blind to those, you're going to be
writing terrible, terrible application code and your career
probably won't go very far. Your bosses won't be happy, your
customers won't be happy. So I think that's a very
compelling reason for everybody to learn at least a little bit
from the other side of the house.
I highly agree for your last point there, right.
So if an application developer doesn't know how the code gets
shipped skilled and being secured, right?
So I think typically the the code that they wrote won't be
that great simply because they don't know how to support the
system later on, how what kind of issues that typically happen.
And worse, they don't actually have ownership because they
think their job is just to write code and somebody will operate
that. So I think if you want to be a
good software engineer, right, So make sure that you know how
to deliver the software. And thank you for sharing your
LinkedIn story as well. I find it really fascinating,
right? So almost every company that
goes big, right or whatever scale up out there, right, would
face this challenge at, I don't know, a certain point in their,
you know, life, right? So it typically they couldn't
deploy, they couldn't scale when there are so many traffic and
demands, they couldn't release new features or new updates or
it takes a long while to actually deploy their features.
So I think typically these are the turning points where company
would invest a lot more in doing software delivery.
And I'm sure we don't want to wait until that time before it
actually happened, right? So I think the other thing that
you mentioned in the book is something that I find really,
really important to discuss here because typically when people
talk about software delivery these days, there are so many
technologies. OK, let's use Kubernetes, let's
use, you know, serverless, let's use containers, let's use
whatever, whatever technologies that are available out there.
And they start, you know, even though it's a simple problem,
they start with, you know, all these different technologies and
techniques available. In your book, you mentioned this
concept called minimum dose, right?
Minimum kind of like MVP style, you know, like the the minimum
thing that you need. So maybe tell us why this is
important to actually think of software delivery in this
manner. So one of the things that our
industry is very much guilty of is following, we're like the
fashion industry, right? Somebody puts A blog post out
there that's like, look at this micro service architecture that
Netflix moved to. And now you have thousands of
developers that are like, we must do micro services like
Netflix and then somebody else put something out about service
meshes. And now everybody rushes to do
service meshes. They're the hot new thing.
And we kind of are following these trends, these things that
sound cool and sexy and this like resume driven development,
right? Oh, I have to put Kubernetes on
my resume, right, to get hired. So we're all going to use
Kubernetes. And I think what people really
miss out on is context, right? Netflix, just to use them as an
example, you know, when they move to microservices, there's a
certain context in which that decision makes sense.
And that context is typically in, in the sense that they have
like thousands of developers working on just millions and
millions of lines of code to serve hundreds of millions of
customers to serve I don't know how many petabytes of data going
through their systems and things like that.
And within that context, the technical decisions that they're
making are a good fit. If you are a three person
startup trying to get off the ground and you have 0 customers,
you don't have that context. And if you try to apply their
solutions to this wrong context, it will be actively harmful.
It'll be a really, really poor fit.
And I've seen this again and again, the classic one is again,
this like three person startup. They've 3 developers and they
have like 37 microservices to manage and like a service mesh
and they're using Kubernetes and all these fancy things.
It's just not the right fit. Like the amount of complexity,
like these things have a cost. They all have overhead, they all
have drawbacks. Every technology, right?
That's almost like the hallmark of a senior engineer.
A junior engineer will tell you what's cool.
The the senior engineer will tell you all the drawbacks to
every single thing because all of them have drawbacks.
Every single thing you use has advantages and drawbacks.
And those drawbacks are worth it in a certain context like micro
services have. And in the book I have like, I
don't know how many pages of drawbacks to using microservices
architectures. There's a huge number of
drawbacks, but those are worth it in a certain context, you
know, a la Netflix or Google or LinkedIn.
They don't make sense in all contexts.
So one of the things I talk about is you typically want the
minimum effective dose. This is a term from medicine
where in the world of medicine, if you take some kind of a pill,
the reality is most medicines at the wrong dose will kill you.
That's a reality. They'll be really bad for you.
So you usually aim for the minimum effective dose.
What's the smallest amount of this medicine I can take that
gives me the benefits I'm looking for without all the
drawbacks, without killing me, essentially.
And you want the same thing with almost any technology choice,
and that includes all of these dev OPS and software delivery
processes. What's the smallest amount I can
do that gives me the benefits I want while minimizing the
drawbacks? So jumping to like a giant ultra
super fancy microservices architecture, that's probably
not the minimum effective dose, right?
Sometimes you need to look and ask like, what's the concrete
problem we are actually facing? So for example, if the problem
is that you have an outage every time you do a deployment, maybe
moving to a microservices architecture will help that.
But maybe all you need is to automate your deployment, right?
Maybe you need a little bit of infrastructure as code or some
scripting or things like that, right?
Maybe you need a better CICD pipeline.
Maybe you just need better automated testing in your
existing code. And the key thing is you want to
invest as little as you can in the stuff and still get the
benefits and minimize the drawbacks.
Because here's the other thing I can tell you as a startup
founder and someone that's, you know, seen the business world,
your customers do not care what technologies you're using.
None of them care if you're using Kubernetes or service
meshes or whatever. Like they just don't care.
It doesn't matter. Honestly, your investors
probably don't care. Nobody cares.
They just want something that works.
And so every single minute that you invest in the under the hood
stuff that isn't absolutely necessary is probably wasteful.
Like most companies live or die based on the product, based on
their ability to reach customers.
So product I'm using is a broad umbrella for marketing, sales
product, right? That's the stuff that matters.
And if you get to the scale of a Netflix, yes, you're going to
need to do a lot of crazy stuff in your architecture to make
that product work. But it's the product.
Like making that work is the real goal, not the under the
hood craziness, the under the hood craziness.
It's a cost. It's the cost you pay to support
that product. So if you don't need to pay that
cost, don't. Absolutely don't.
So yeah, I talk about that quite in the book.
I talk about how architectures and software delivery processes
tend to evolve within companies because there's some very, very
common patterns, right? Almost everybody starts with
like the single monolithic app and maybe it's connected to a
little database and that's it. And that's fine.
That's actually the minimum effective dose to start out.
There are tremendous businesses that use that architecture very
successfully and there's not one compelling reason to make that
any more complicated if they don't need to.
But if you have to, right, If you now have millions of users,
well, now you start to make it a little more complicated and you
have some caches and you have some this.
And eventually, usually not from scale in terms of customers, but
from scale from the number of teams internally, then you might
break the monolith into some services and then you could do
service ownership. But you want to align your
company to the right kind of stage in this evolutionary
process. And it really, it does feel like
evolution. It feels like growth.
Every company just, we don't design, build our technology,
right, It grows. And if you try to design and
build it and jump 10 steps ahead, it usually doesn't work.
And then you have to go back, you know, go back to the simple
thing and let it grow. So yeah, think of it as the
minimum effect of dosed. Think of it as incrementalism,
which I can talk about at some point, maybe today, as the way
you want to approach these things.
Don't go for the technology because you read A blog post.
That's not the best way to design your architecture, right?
I think this is gold, right? For everyone who just listened
to the explanation, right? So I think context is king,
right? Whatever technologies that you
read or whatever technologies that you learn, right?
So obviously they will paint all the great stuff, but sometimes
it has a cost associated behind it, be it for example,
maintaining it, the skill set that you need to learn.
Don't forget about that. And typically, right, it's not
just, OK, I managed to run it the first, right?
So there are many other things that you need to take care
about, like for example, the administrative stuff, for
example, if there are massive scales suddenly coming, do you
know how to operate that in a most efficient manner?
And also the security part, right?
How do you secure the actual infrastructure and the thing
that you use, right? I think that's also another
thing that you should not forget and the people aspect don't
forget when you operate in multiple microservices, you will
probably need more people, right to actually help you maintain
that. So definitely context is king
and I really agree with you. Fully agree.
If you're a small startup, don't start with the fancy kind of
technologies. So I think definitely stage also
matters because I think in most of the stories that I heard or
read as well, there will be some time where you will need to
split rewrite re architect, right?
But that is associated with the growth or the success of the
company, right? People like to kind of like go
play their solution, right? They think, OK, we will succeed,
but don't actually assume that you will reach that stage,
right? So I think just wait until the
growth happens and then you can adapt accordingly.
So you mentioned about minimum effective dose.
From your experience working with many infrastructure staff
or DevOps, is there any other anti patterns or pitfall that
you want to advise people to avoid?
There's a whole bunch, and again, they're going to often be
context specific. One of the ones that comes up
quite often. I think you actually touched on
it a little bit. I've seen this especially with
the infrastructure's code space, but it comes up just about
everywhere is somebody gets excited about a technology that
they ostensibly adopted in the company, but they didn't
actually do the legwork to get everyone bought in and to get
everyone the time to learn and really master this technology.
So in the infrastructures code space, this happens a lot.
Somebody gets really excited about Terraform, open Tofu,
Palumi, whatever kind of tech. And the most common pattern is
there's just the one guy at the company who just loves it, and
he brings it in and he writes tons and tons of code.
And it's like, it could be amazing code and it's lovely and
it's beautiful, but the rest of the team has no idea what that
guy's doing. They haven't been given the time
to learn. It's not something they've used
before. And inevitably what happens is
there's some sort of problem, an outage, something crashed, they
need to fix it, and they don't know how to use the code to fix
it. So what do they do?
They fix it manually. Well, the thing with
infrastructure's code and a lot of these tools is if you go and
do something manual over here, well, now your code doesn't
really reflect that. And the next time you try to use
the code, it's going to run into issues because of this mismatch.
So now the code doesn't work. So that even if somebody tries
to do it the right way and use the code, they can't.
They hit a bug, they hit a problem.
So then they go and do some more stuff by hand, manually click
UPS essentially, which makes the code even more problematic.
And you know, rinse and repeat a couple times and suddenly the
code doesn't work at all. And all of this work that this
one guy did is essentially just throw it away and they have to
start from scratch or do something different.
And that's wasteful. And it's a shame.
And people of course blame the infrastructure as code tool, but
the reality is it's not really the tool.
It's just that the team never had the time to adopt it, to buy
into it, because it is. It's a genuine change in your
process, right? You're taking folks that are
used to like SSH ING to a server and running a bunch of commands
as the way to do things. And you're asking them to like,
go check out a repo, open it in an editor, find some piece of
code, maybe run some tests, commit the code, maybe some more
tests, run. Then there's some automated
process, right? Like it's a completely different
way of doing things and it has advantages.
But if the team's not bought in, if they don't have the time to
learn it, you're not going to gain those advantages,
essentially. So that that has been something
I've seen again and again and again, just not getting that
full buy in and adoption. And that sometimes happens
because there's one person excited.
That sometimes happens because somebody at the top just
mandates something, just marching orders.
We will use technology X and everyone below them kind of
shrugs and says, OK, but you know, they're not really bought
in, so they don't do it. So that's usually, that's a
very, very common pain point. And honestly, I would say you're
better off not using the quote UN quote better technology if
you don't have time to to do it properly.
Like if you're going to do it, you need to do it the right way.
Otherwise use your time for something else that's more
valuable. Again, minimum effective dose.
The second, I think closely related issue.
And this also really, really ties into the minimum effective
dose concept. And I mentioned incrementalism,
so probably a good time to talk about it is the attempt to do
The Big Bang migration, right. This usually happens from
somebody above says we will move out of our on Prem data center
in six months. And also we're going to adopt
DevOps, whatever that means. And we're going to use
Kubernetes and we're going to be cloud native.
And it's like buzzword, buzzword, buzzword, buzzword,
long list of things. And we have to do all the things
all at once. And if you have any degree of
complexity, right? Like in mostly, these are
companies that have been around for a decade or two and they
have, you know, millions of lines of code and lots of
customers. Like you can't do that stuff
quickly. And you don't even want to try
to do it as one giant kind of ball of mud essentially is what
it turns into trying to do everything at once.
Trying to do like a massive rewrite usually ends poorly.
And I would say usually, But if I'm being completely honest, I
would say 100% of the time, every single one of these large
companies, and I've had the privilege to work at this point
with probably over 1000 different companies, I've not
seen a single one of these big projects completed on time, on
budget, or usually at all. What I typically recommend is to
do things incrementally. And the best way to understand
the word incrementally, because it's this is one that's very
easy to misinterpret. A lot of people here
incrementally and they just think, OK, chop it up into small
pieces and that's not really what it means.
So to understand what incrementally means, the most
useful way to do it is to look at the opposite, which is false
incrementalism. False incrementalism is where
you take a project and you slice it up into small pieces, but
until the very last item is delivered, the project does not
provide any value whatsoever. So if you're doing this big
rewrite where you got to rewrite, maybe the way your
front end app works and the way the back end app works, and the
way you deliver these apps and the way you secure these apps.
And until every, and even though you could do these as one piece,
and that's what a lot of people think of with incrementalism,
but until they're all done, you can't actually ship it live.
You can't put any production traffic in it.
That's not incrementalism. That is a Big Bang migration
that you just happen to do in little pieces, but you got 0
value until the very last item is delivered.
And that's a really, really risky way to approach a project.
Cause in the tech industry, large projects almost always
fail. They just, they don't get
completed. And there's a million reasons
for that. We're not going to have the time
to dig into them. But the most common one honestly
is people just lose patience. The CEO says, yes, you can do
that rewrite, but after 18 months of waiting for your
rewrite to complete, they're like, no, we got a business to
run. Your rewrite has been cancelled,
right? And if at 18 months is that, you
know, that person upstairs says you're done, you're out of time.
If you haven't gotten any value from your work, well,
congratulations, you just threw away 18 months and you got
nothing in return. That's the worst possible
outcome. So the better approach is to use
actual incrementalism. And the key is you don't just
chop it up into pieces, but each of those pieces must be
something you can deliver and get value from by itself, even
if the other pieces never happen.
That is real incrementalism. So you're able to do 1 little
thing, and if the rest of the project is cancelled, well, that
little thing was still worth doing.
It's still made your company better in some way.
That's the way to approach this, to approach these things that
actually works in the real world.
And in practice, what that translates into is you look for
concrete, specific pain points that your company is facing.
You don't look to do DevOps, you don't look to do cloud
migrations. These are solutions.
You look for problems. The problems might be outages.
The problems might be security problems.
The problems might be that your team is really slow, like you're
not agile or you're not delivering software as quickly,
right? And you fix 1, whatever that
most painful problem is, you fix one at a time and you fix it all
the way. And then you go to the next
highest priority problem and you pick the minimum effective dose
to solve each of these problems. So if the team is slow, maybe
what you need to do is to deploy more often, or maybe you need to
test in production, or maybe you need to change the web
framework, right? And you have to find out why is
the team slow, which is the most important thing you can ask.
The answer isn't just do DevOps, right?
Like there's a cause and you want to fix that very specific
cause. So that's the approach I
recommend. Go find very, very specific
problems, fix those very, very specific problems, and then just
repeat the process over and over and over.
And eventually you'll have something where you maybe are
using the cloud or infrastructure coders, whatever,
but you get there in a way where every single step has delivered
a little bit of value, which makes the person upstairs happy,
which makes you happy and actually makes your company more
successful, even if eventually somebody runs out of patience
and you're not allowed to do any more of those.
Well, I think those are really great learning points, right?
The first is about bringing people in, you know, getting the
buy in, making sure they follow the journey, right?
And not something that just few people want to champion.
But then it dies off after some time, right?
Incrementalism. Thanks for explaining the
concept, right, the false incrementalism and the true
incrementalism. And the last thing is probably
find from the pinpoints first the problems rather than, you
know, implementing from buzzwords or resume driven
development. So speaking about, you know,
some of these evolution, right? So I know that recently we have
this AI and so many other new cool technologies coming.
What do you think is the future of dev OPS and software
delivery? Maybe you have some crystal ball
here that you can share. Yeah, the very last chapter in
the book. I try to take some guesses at
some interesting trends that are coming.
I'll put a big caveat here. Anytime you try to guess the
future like you're hilariously wrong and that's OK.
I don't mind being wrong. I think this is just interesting
to look at and kind of what what are the kind of cool trends that
are coming out? And my guess is a few of these
will be kind of interesting and then it really what'll happen is
something none of us saw coming will be the big thing and that's
fine. So what are some of the trends
that I, I've been seeing? I'll list a few of them,
probably won't have time to get into all of them.
But the first one, and I think it's the pattern that kind of is
we're going to see through all the trends is this idea towards
a move to higher and higher level abstractions.
The analogy that I use is if you look at the history of
programming languages, what we've seen is a move to higher
and higher level programming languages.
So you started with, you know, binary machine code, you moved
on to assembly languages, you moved on to, you know, things
like C, eventually Java, and then, you know, these days Scala
and even more modern languages over the last few years.
And at each level, a couple things happen. 1, you give up
some degree of low level control and power, right?
And we can all hear like the C programmers yelling at Java
programmers, you know, hey, I can't control memory the way I
want to. And, you know, I need this
control. And they're right, right.
For some percentage of applications, if you need that
control, yeah, stick with a lower language.
But in reality, the vast majority of use cases don't need
the lower level control. And so the second thing that
happens with these higher level languages is they make it
easier, they make it faster, they make it more productive to
do what you need to do when you don't need the lower level
control over things. And so that makes programming
more accessible to more people and that lets us build things
faster. And so that's why the whole
industry gradually is shifting towards higher level languages.
Never 100%, you know, there's still code being written in C
and machine code and assembly, and there will be for a long
time, but the majority of developers gradually move to
these higher level languages. I think the same thing is
happening and will continue to happen in the DevOps and
software delivery space we moved from.
You know, I need to buy a physical server and shove it in
a rack and deploy my software on it and hook up cables and all
this stuff to the cloud using virtual machines to the cloud
using containers to now serverless where it's not even
containers. It's kind of this little
deployment package and I think there's going to be, we're going
to keep climbing that abstraction ladder.
I don't have a good name for it. I kind of pitched this idea of
infrastructureless, which is the next evolution from serverless.
So serverless today is pretty close.
And if I was a betting man, and again, I could be hilariously
wrong here, but if I was a betting man, I think the future
is going to look a lot, lot more like serverless than it does
like Kubernetes, for example. I don't think Kubernetes is the
long term direction of the industry.
And when I say serverless, you know what, what we have today is
you kind of hand it a deployment package.
Here's a little piece of code, sometimes a function with an
entry point and you say please run it when a certain trigger
happens, like an HTTP request comes in.
So that's pretty high level, right?
Here's code, you go figure out how to run it.
I don't want to be bothered with the details, but in practice,
I'm still bothered with an awful lot of details with serverless.
I still have to think about, you know, if you're using, for
example, Lambda in AWS, you still have to think about AWS
accounts. You still have to think about
the networking aspects. You still think have to think
about provisioning, concurrency. If you need to talk to a
database, you have to think about how to do long running
connections with server. So there's just a bunch of
infrastructure stuff that is still left.
I think when we go from serverless to the concept of
infrastructure list, and just as a reminder, serverless doesn't
mean there aren't servers, There's obviously still servers
and infrastructure list doesn't mean there isn't infrastructure.
It's still there. It's just not something you have
to think about as a developer on a regular basis.
You give up some control, right? I can't control what's happening
under the hood, and in certain use cases that'll be a problem
and I won't be able to use it, and that's OK.
But for the majority of use cases, I don't need that
control. I instead just want to hand you
some code and say you go figure out how to run it, how to scale
it, how to secure it, how to handle a lot of these details.
So I think that's what we're going to, I think serverless is
like a very early glimpse of what that can look like.
And you can tell it's not a very mature technology yet.
So there's all these like problems and drawbacks to using
it. But it really it like, to me, it
smells like the future. And I think if you extrapolate
it out, things are going to look much more like that and kind of
turn into this infrastructureless thing for the
most companies, not everyone. There will always be companies
that need the lower level control and they'll stick with
either running their own servers or something a little lower
level, but most use cases will move on.
So I think that's one interesting pattern.
A second one I'll talk about 3, I guess a second one that I
think is interesting, and I get questions about this all the
time, is what's the impact of AI on the infrastructure and DevOps
world? And when they talk about AI,
they're usually referring to these large language models,
LLMS, the things like ChatGPT and technologies of that sort.
I'm not an expert on this AI stuff, so I will put that right
out there. My experience is with using a
whole bunch of them and usually tearing out a lot of hair with
it. I think there's some early
enthusiasm from folks that are saying, OK, AI is going to
replace us. It's going to write all the code
for us. It's so productive to me.
Again, could be hilariously wrong, but that strikes me as
less likely to be our future, and especially in the DevOps and
infrastructure world. The reason I say that is the
things that matter more than almost anything else on the
infrastructure side of the house are things like reliability,
reproducibility, security, predictability, right?
Like, it's kind of the boring stuff, but that's what matters
in the DevOps world. And the problem with at least
all the LLM things that I've seen to the state is those are
the exact areas where they're really, really, really weak,
right? LLMS are notorious for just
hallucinating things, right? Just making things up.
And that doesn't seem like a small bug of one LLM
implementation. I think that's just core to the
infrastructure, to the way that they're designed.
They're also kind of random. Like literally they have a
random seed built in and the tiniest little change to a
prompt can profoundly impact the results, right?
You change like 2 words like a pronoun here and all of a sudden
you get a completely different response.
So it's really hard to get like repeatable, dependable results.
And the last thing I want is like the security of my company
being dependent upon an AI that will like hallucinate an answer
or will give me a slightly different answer.
You know, it used to work, but now today they changed something
in the model or I changed my prompt and now I get something
that's not actually secure. So I'm not very bullish on the
AI is going to replace this angle that strikes me at least
with what I'm seeing today as not super plausible.
I think the place where AI can get interesting on the DevOps
world is what's called retrieval augmented generation.
RAGI don't know if you're supposed to pronounce it as rag
or not. You know, I don't know if this
is like the GIF GIF debate, but anyway, I'll call it rag.
And the idea here is you take one of these large language
models and then what you add is some additional context, usually
some sort of database with extra up to date information about
your very specific use case. And so this thing can combine
these two pieces of information, the model it was trained on and
your up to date info to give you hopefully much better responses
about your specific context. And I think there's two ways
that that can be interesting. The basic 1, and that's the one
we do see today, and there's a bunch of tools trying to do this
is you feed it the information about your infrastructure as it
is today. Here's how I deploy things.
Here's all my metrics, here's all my structured events going
through my systems. And now that's the additional
context that I give these RAG tools.
Now they can answer questions for me about my own
infrastructure. So if I have an outage, I can
say what changed? What did we just deploy?
What happened, what metrics changed, right?
And so we're starting to see that, you know, there's like a
tool called Honeycomb and it's starting to use AI to kind of
intelligently help you debug outages, for example.
And we're seeing that in some monitoring tools where they are
trying to predict you're about to have an outage based on some
kind of metrics. I think that stuff is pretty
interesting. And I, I do think eventually
it'll be reliable enough that you can really use it and really
accelerate either preventing outages or debugging outages and
under even just navigating and understanding how your
infrastructure works. I think this could be very
useful. The one that I'm more excited
about, but I don't know if we can do it, is if I can feed in
not just my context, but the context of 1000 other companies
and now these AI models. No, can see these common
patterns amongst all the companies.
So when there's another security vulnerability, it's not me
having to go in there and say I need to update my code.
It's this model can say, hey, I saw that 957 other companies
just updated their open SSL library.
Here's a patch to update yours and I've already rolled it out
into your test environment. Does this look good?
Should we push to prod? Right?
It's the ability to see patterns that everyone is facing and to
extrapolate those into my case. I think that would be an
incredible thing. Because that's the reality,
right? Like there are 1000 DevOps
engineers at 1000 companies doing the same stupid thing over
and over and over again. But we don't get to leverage a
lot of that work, so maybe it'll help.
The challenge as far as I can tell is how you get one of these
large language models to expose information about other
companies without leaking that company's proprietary data
secret sauce. Can those models tell the
difference and not hallucinate and not, you know, get it wrong
between what should be kept private and what's OK to share.
I think that's going to be a challenge.
But if we can pull it off, I think that could be a really
profound acceleration to how we build software.
So that's the second item. Third one that I'll mention a
little bit tied to that, but I think it's it's own thing is.
The idea of secure by default, and this is something I'm
starting to see a little bit of it, and some of it is like
wishful thinking. And maybe if I say it enough,
maybe people in the industry will start going in this
direction. So what do I mean by this?
The analogy I use actually has to do with elevators.
There's this really famous demonstration, and there's a
little bit of controversy of whether it actually happened
this way, but it makes for a great story.
So back in the early 20th century, we wanted to make our
cities taller, you know, taller and taller buildings.
And we had elevators, but people were terrified of the cable
snapping and plunging you to your death.
And so this Elijah Otis guy of Otis Elevator comes along and he
does this amazing demonstration in front of huge crowds where he
has this open elevator shaft and he has this assistant lift him
up. And he's Elijah's standing on
the elevator as it's raised to like the fifth story or 60 feet
up or something like that. And then the assistant comes up
with a knife and cuts the cable of the elevator while Elijah's
standing on it. And the elevator drops about an
inch and then immediately stops. And Elijah's like all safe, all
safe gentlemen, everything's fine.
And it's a really cool demonstration.
And the way that it works is incredibly clever.
The basic idea with what's called the safety elevator, this
thing that Elijah came up with and has been improved on since,
is that if you look at the outside of the elevator, it has
these metal hooks that stick out into the elevator shaft.
And the elevator shaft has a bunch of metal teeth running
along the side of it. So these hooks stick out and
they grab onto those teeth and the elevator can't move at all.
And that's its default state. It cannot move.
The only way to pull these hooks in is if there's an intact
elevator cable. If there's an intact cable, it
kind of pulls up and this little springs come in and these little
hooks come into the elevator. Now the elevator can move and as
soon as the cable is not intact, if somebody cuts it, they spring
right back out into the shaft and the elevator can't move.
Why do I bring this up? What's so clever about this
design is that it is secure by default.
The default state of the elevator is safe.
It cannot move, It cannot fall. And the only way it moves is
when something else proves that it's in a safe state, which is,
you know, the cable. And so that made people
confident enough to ride in elevators, transform cities,
allowed us to build tall buildings.
Really cool stuff. So right now I would argue that
the state of software delivery, DevOps, etcetera is not secure
by default. We are very much in that early
20th century state of like, we want to build fancy tall
buildings and infrastructure, but we can't because we're
afraid of plunging to our deaths.
It feels like all the defaults are not secure, right?
Usually you build something and like all the networking is wide
open. Nothing's encrypted.
If you have third party dependencies, nobody verifies
you know where the hell they came from.
Nobody's keeping them up to date.
There's no monitoring. And maybe worst of all is a lot
of vendors charge extra money for the more secure things.
Like using single sign on usually is only in the expensive
enterprise plan. So we're like we're, we're the
exact opposite of safe by default.
We're like horrifically dangerous and deadly by default.
And if you pay us enough, maybe we'll make you more secure.
So that's bad. The good news is there are some
signs. The industry is starting to move
towards a more secure by default state.
And there's a whole bunch of little signs.
And hopefully we'll get more. And I'll mention just a few of
these. One that's been appearing more
and more recently is this concept of shift left.
And the idea here is if you kind of look at like a, a diagram of
how something goes from code to being deployed on the site, you
know, from left to right, there's like write the code,
test the code, deployment, etcetera.
The idea with shift left is to move security testing further
and further left. In other words, closer and
closer to the development of the code.
So you catch security issues as early in the life cycle as
possible, which is when they're easiest to fix and is the most
secure. You certainly don't want them
all the way on the right when you're already in production and
you've been hacked. That would be the catching it on
the very right side. So there's a whole bunch of
these shift left tools from things that allow you to enforce
policies around your code to all sorts of automated testing
tools. So those are really nice to see.
Basically thinking about security really early on.
A second pattern is you're going to hear a lot more, I think, in
the future about supply chain security.
So in the world of software, your supply chain is basically
all the software you depend on, right?
All of your open source libraries that you use, all the
vendor libraries, even the cloud that you deploy into, those are
all software written and maintained by somebody else.
They're part of your supply chain.
And the reality is if somebody compromises any part of that,
they can do some amazing, amazing damage.
And hackers have caught on to this and they're trying real
hard. I think these days, I forget the
exact number, but something like 70% of the code that a typical
company deploys is not written by that company.
And I guess that's an underestimate that.
I think that's just the open source portion.
I think if you factor in things like, you know, the cloud that
you're running on and the Linux operating system and all the
tools on, there is probably like 99% of the code that you rely
on. You didn't have anything to do
with it. And hackers know that.
So if they can go into some little library hidden somewhere
in Linux and put a little back door in there, well, they can
take over everybody's software, right?
And everybody's hardware. So this is a big battle that's
happening right now is how do you secure the supply chain?
How can you be confident that all the software you depend on
is the software you think it is and is secure?
And maybe they shifted left and has been tested.
And so there's a lot of interesting technologies
emerging in that space. I think it's super, super early
days. I think our answers there are
still really weak, but I think I'm, I'm so happy to see someone
who's thinking about it at least.
So supply chain security is going to be, I think a big, big
deal. Another trend with secure by
default is a push to move to memory safe languages.
Again, this is moving up the abstraction ladder.
As I mentioned before, it turns out that something like 70%,
again, I don't remember the exact number, but very, very
high percentage of bugs, security bugs specifically, are
due to memory safety issues. You take a language like C where
you have, you know, pointer arithmetic and things like that,
and you put a little bug in there.
You didn't quite do the arithmetic and somebody can pass
in some little input and makes you read or write to memory
they're not supposed to. It turns out 70% of the security
issues we face as an industry, 70% are due to that issue alone.
And those bugs do not exist for the most part in memory safe
languages. You just can't do them in higher
level languages where memory is managed automatically.
So if we switch programming languages, 70% of our security
issues go away. Like that's a huge, huge deal.
And so you're starting to see that the US government is now
pushing to move away from languages like C and on to
languages like Rust and Go and things that are more memory
safe. And I think that's a really good
thing. That's a really hard thing
because like all of our operating systems are written in
not memory safe language. So there's a tremendous amount
of work to do there. But if we can do it, we will
make things vastly, vastly, vastly more secure by default.
And then the final one, another pattern that's appearing more
and more is this idea of 0 trust networking.
I talk a lot about this in the book.
The kind of the older way of doing things is the Moat and
castle approach. The idea is you create this
really secure perimeter around your infrastructure, a little
bit like a castle with a Moat, right?
You have a really tall wall and you have a Moat with alligators.
So the perimeter is really hard to is really secure, really hard
to get through. But once you're inside that
perimeter, you can do whatever you want.
And the equivalent of that in the software world, in the
networking world, is you have really, really strong firewalls
and things like that on the edge of your network.
But once you're in, if you somehow found a way in, well,
now you can access the wiki page over there, and you can access
that service over there, and you can access that database and do
whatever you want. You know you have free rein once
you're inside. Turns out that approach used to
make sense. When everybody was in an office
owned by the company, All the infrastructure was in probably
the same building owned by the company.
All the computers are used were in that office owned by the
company, right? It kind of worked.
That's not the world we live in, right?
We all now work from home. The network now extends into
like, coffee shops and libraries and coworking spaces.
Your infrastructure isn't in your office, it's in somebody
else's cloud. The devices you're using are
these smartphones, right? That are not particularly
secured. So it doesn't really make sense
to use the Moat and Castle model anymore.
The new model IS0 trust networking, and the idea is your
location in the network. The fact that you're in the
network doesn't give you any special privileges of any kind.
Every single request, every connection must be
authenticated, authorized and encrypted, every single thing.
So just because I'm able to access the wiki page does not
give me any access to that database or any access to that
issue tracker or anything else at all.
Every single thing has to authenticate, authorize, and
encrypt separately. And we're starting to see that
appearing from vendors. It's not the default, It's very
far from being the default, but things like service meshes and
and tools of that sort are starting to at least make it a
little easier to do it because it's really hard to actually to
do zero trust networking properly.
So we're at least making it more accessible and maybe at some
point in the future, it'll become the default, which I
think will be a tremendous, tremendous improvement to
security. So yeah, those are the three
patterns that I'm seeing in the future.
Higher level abstractions some role for generative AI,
especially RAG, and this move towards secure by default.
Wow, really fascinating just hearing, you know, all the
predictions and you know, up and coming technologies and
techniques that you just mentioned and shared, right.
So I think definitely we are optimistic to the future, right?
So higher level abstractions, memory safe language, secure by
default, Hopefully software gets easier and easier to deliver,
but at the same time also hearing about, you know, this
supply chain attack and all that, sometimes we have to be
paranoid as well to what we use from the open source world.
Speaking about supply chain attack, right?
And I know one other trend that happens quite recently is the
changes in the open source licensing.
And I know you are one of the core founding member of open
Tofu, right? So the alternative to Terraform,
maybe also tell us a little bit about your, I don't know,
prediction about what actually happening in the open source
licensing. And what should we all know
about, you know, as a software developers, what should we know
about in terms of how do we protect ourselves from all these
open source changes? Yeah.
So to add a little bit of context for folks that haven't
been following along specifically with like the
Terraform open tofu thing, the context there is Terraform and a
bunch of other Hashicorp tools were open source under fairly
permissive license, the MPL license for ten years plus for a
long time and built up huge communities, tons of
contributors, third parties, etcetera.
Recently Mahashi Corp made a change from this open source
license, shifted almost everything they have to a what's
called this business source license, which isn't really an
open source license. The short version, as it says
you can use this stuff unless you're competitive with Hashi
Corp and that's problematic for a whole bunch of reasons, but
Hash of Corp is not the only one.
Many, many other companies across the industry have been
changing from permissive open source licenses to some flavor
of these kind of business friendly licenses that basically
say you can use this stuff as long as you don't compete with
us or various other restrictions.
I have a lot of thoughts on this.
I'll, I'll try to keep it short. I, I'll say first, I am a huge,
huge believer in actual open source, proper open source, not
these business licenses, but you know, these permissive MIT
Apache, those types of licenses, I think they are one of the most
important and valuable things we've done as an industry.
I think they're one of the biggest accelerators to all of
software. And I personally find it
devastating to see companies doing anything to hurt trust in
open source. I think that is going to have
profound negative consequences on the industry because as I
said, there's 1000 DevOps engineers and 1000 companies
doing the same thing. And that's true of every kind of
software development. We are often doing the exact
same things and sometimes we can share that stuff commercially,
but we've seen that the open source stuff, especially for
things like infrastructure for these building blocks of the
entire Internet and our operating systems, etc.
They need to be open source, truly open source to be
reliable. So to an extent, I'm personally
just devastated to see the industry again and again move
away from these licenses. I'm hopeful there are a lot of
people like me. And so now we can ask, well, how
do we solve this and, and what led to this?
I don't have any internal visibility into the companies
that that made these license changes.
So this is me speculating. I'm sure they can tell you, you
know, a more accurate story. But what I've generally seen as
a pattern is the things that were open sourced by companies
that are trying to do hyper growth.
Typically these are companies that are venture backed, either
trying to be public or just became public.
And their goal isn't to just make a profit, it's to grow
massively huge to produce tremendous investment returns.
What I've generally seen is these are the companies that are
doing these license shifts on us and it makes sense, right?
They that's literally why they took venture capital.
It's to, you know, produce that kind of growth.
So in a sense that was part of the contract.
But we, I don't think those sorts of license shifts have
happened in cases where someone isn't trying for that growth,
right? It's rare to see somebody shift
license just because like they can't afford to pay 2
developers, right? Like that's it happens and we
need a better way to do that. But it's mostly the VCBF company
that isn't happy with making only hundreds of millions of
dollars. They need to be making billions
of dollars. And if you need to be making
billions of dollars, yeah, you got to find every little lever
and advantage and eliminate competition and basically build
a monopoly, hence the license change to be not competitive
with them. I think there's a few key
takeaways from that. I think if I was picking
technology for my company to depend on, especially like
really important foundational technologies, that would be very
painful to swap out. First of all, I would only pick
things that at least as of today are on some sort of truly open
source license. Chat with your lawyers about
what that means, but it's usually things like MIT, Apache,
MPL and handful of others, Berkeley, etcetera.
Second of all, I would look very, very carefully at the
company's behind those projects. If there's a company, some
projects are just an individual developer, some are built by
foundations, which I think is a much more secure thing to do.
But basically look who is behind this project.
If it's owned and primarily operated by a venture backed
company, you know somebody that wants to go hyper growth,
another pattern might be somebody that got acquired by
private equity firms. That's a very similar kind of
set of incentives. Be really, really cautious.
I think at this point you just need to think twice if that's a
good idea to adopt or if you're better.
Not because eventually the incentives are going to push
them to change the license. That's just what we're seeing.
And instead of venture backed companies try to pick tools.
The best version I think is if it's something that's managed by
Foundation, Linux Foundation, CNCF, things like that, because
those are specifically designed so no one company, due to
whatever incentives they have, can go and mess with the
license. That's the best option.
Second best is going to be either like solo developer,
which has the advantage of, well, if they can make enough
money or have enough spare time, they can make it work.
And they're not likely to chase, you know, venture back returns
or possibly something that's backed by a large company where
they're not trying to make money off of this piece of open
source, right? You know, Google open source and
stuff, Facebook open source and stuff.
They're not trying to sell a lot of that stuff.
Those can be reasonably safe bets as well.
But the Foundation 1 is the best.
So that's, I guess in the short term what I would really look
for. There's a lot of other things
you need to look for when picking open source libraries or
anything else, you know, part of supply chain security.
But at least from a licensing perspective, that's kind of the
hard lesson learned for me is be really wary of who is behind the
project and what license it has. I think in the longer term, this
is something we need to grapple with as an industry.
What does open source mean? Do we need to start creating?
And maybe there are some out there licenses that basically
say, hey, this is not only open source, but I also pledge to
never swap this thing. You know, I legally prevent
myself from swapping this to some sort of not open source
license in the future. Maybe we need some sort of
guarantees along those lines. But even more generally, you
know, how do we fund open source?
And this is something a lot of people have been thinking about.
There are some interesting directions that I've seen.
I haven't seen anything that seems like a great solution, but
there are at least some positive trends for solo developers or
really small teams where you can make like a good salary to live
on and to be able to afford to focus on open source.
And I think that's great. And so really what we're going
to have to reckon with as an industry is do we trust
companies with hyper growth as an incentive to release
something open source that is core to their business?
So I should put that as a big asterisk.
Again, it's with Hashicorp, you know, Terraform is what they
make money on with Mongo DB. Mongo DB was what they make
money on, right? So if it's the thing they're
trying to monetize and some aspect of it is open source,
should we really trust that? Or is that just like clearly an
anti pattern in our industry? Is there a way we can build
trust around that? I think that's something that
smarter people than me will have to look at and figure out.
Yeah, so I guess that's that's where my mind goes with that
stuff today. And the final thing is if you do
pick projects that have truly open source licenses, Apache,
MIT, etcetera, then if the worst thing happens and somebody does
change the license or in any way move the project in a direction
that you really don't agree with.
Well, the whole point of open source is that you have the
right, honestly even the responsibility to fork it.
And you know, you still can use the code and you're not
completely stuck. So that is what we're seeing
happening with a lot of these projects.
We are seeing major forks being developed, open Tofu for
Terraform as an example, Open Bow for Vault and many others,
open search or Elasticsearch. So that's kind of the fall back.
But if nobody wants to fork things, right, I, we, we ended
up forking Terraform. It's not great.
We asked Hashicorp to not change the license.
I think that's the best option. You want to have one joint
strong community. That's the best option for
everyone. And that can only happen around
a truly open source license. But if things happen and you
don't have any other choice, well, make sure that you at
least have the option to then fork the code and do what you
need to do as the backup. Thank you for expanding the
intricacies of these open source changes, right?
Because I find that many engineers just use open source
because it's so simple, you know, like, I don't know, MPM
install whatever install, right? But they don't actually
understand the true intricacies behind open source and
especially these licensing changes, right?
Because sometimes it can affect, you know, the software that you
are delivering, right? Like you mentioned 70% to maybe
90% things that we use now relying on open source
technology and dependencies. So if something changes,
definitely it can break your software altogether, right?
Or you have to. Find a way to rewrite or change
some parts of your code base, which is not easy to do and
typically can be risky. And I personally would like to
thank you as well for championing, you know, this open
tofu, you know, making sure that open source stays truly open
source. I think it's really, really
important, as you mentioned, right?
Open source has been accelerating the progress in the
world today. So many technologies, so many
apps being built simply because of the existence of open source
technologies. So Yvgeni, thank you so much for
your time today. I think we cover a lot of things
and I'm sure listeners here are quite fascinated by some of the
things you mentioned. Unfortunately, due to time, we
have to, you know, wrap it up soon.
I have one last question which I always ask my guests, what I
call the three technical leadership wisdom.
You can think of it just like advice.
Maybe if you can share something for us to learn, maybe from your
journey or from your experience, what would they be?
I'll keep it really, really short.
Just repeat a few of the points we talked about earlier.
One, never stop learning. Always, always, always carve out
time for learning. Two, share what you learned.
And three, do things incrementally and iterate,
iterate and iterate. That's it.
Beautiful. So I think that kind of like
sums up pretty nice the conversation that we did today.
So if many people want to reach out to you or speak to you, ask
you more questions, is there a place where they can find you
online? Yeah, I'm on most of the typical
social media things. I'm on Twitter, LinkedIn,
Facebook, etcetera. And my homepage is Y
brickman.com. If you Google my name, anything
else, one of those things will come up.
I'm happy to chat on on any of those.
All right, so thank you so much, you know, for speaking with me
today. I hope you good luck for the
writing the last part of the book.
So looking forward for the release of the book.
So thank you so much for your time if Guinea.
Thanks for having me, it was fun.