If it hurts, do it more often, right?
Continuous deployment is the perfect example of applying this
principle and putting it into practice, because to me there is
absolutely nothing I can think of that is more painful than
deploying to production for things, right.
So in the past we used to have like this manual approval stage
where, you know, somebody will decide when we actually deploy
to production. So tell us why it is very, very
important to actually remove this last critical piece.
So to explain why it is important, I'm going to connect
to the practice of Lean manual factory, right?
And you also got Dave Farley to write the forward of the book.
Write one of the co-authors of the You Know Seminole book
Continuous Delivery. Every single aspect of your work
needs to be thought as a life change in a production
environment. So it feels much more like being
an electrician working on an office building that is
currently in use, right? Well, I think it's a very nice
explanation. So I think also maybe for some
people who are still not yet in this kind of a working mode,
right? What do you see?
Some challenges. Surprisingly, there's a lot of
situations where you think you might not be able to do it, such
as regulated industries, but in reality you're able to.
Continuous deployment doesn't mean that you just deploy
straight away. So tell us what is the minimum
thing that you should have in a continuous deployment and so?
Basically, there are a certain minimum amount of practices that
you should definitely make sure that you have.
Hello everyone, welcome back to another new episode of the
Techni General Podcast. Today I have with me a lead
software developer from Thor Works and also the author of a
book titled Continuous Deployment.
So as you can tell, we are going to discuss about continuous
deployment. How does it differ from, you
know what we know, CICD continuous integration,
continuous delivery. And yeah, happy to have you,
Valentina, in the show. Hi Henry, happy to be here.
Right, Valentina, maybe before we go dive deep into continuous
deployment and all that, maybe let's talk a little bit more
about yourself. Maybe if you can share any
career turning points that you think we can learn from you.
So well, you already gave a bit of an introduction.
As you said, my name is Valentina Servila and I am
currently at Teched at Dotworks. I'm Italian, as you can tell
from my accent, but I'm actually working in total Spain.
But I did start my career in Italy.
So basically I began by joining a very small agile consultancy.
That was my very first job and I had a very amazing training.
So all of the juniors that joined this consultancy
basically were told to study for three months.
As soon as I began on my first day, I got told to read the
Agile Manifesto, which is something that I still really
believe in, you know, all of the agile principles and eventually
led to me writing a book about one of those.
So yeah, maybe this is a very good reminder to know, pay
attention to how you're training your juniors because it really
does pay off. I think if I can use my
examples, three years later I joined the ThoughtWorks in Spain
and I started working with much bigger companies.
I started working with more enterprise type clients and you
know, being a consultant in both occasions I was able to jump
from client to client to client and see a wide variety of ways
of working of different team compositions, of different
practices. And I think that gave me a
really good perspective on all of the technical practices that
teams can adopt to be successful or not to be successful.
I also in between that managed to go on a long term assignment
in Asia. So I was stationed in Bangkok
for about a couple of years. And so it was really nice as
well to see perspectives of how organisations work on the other
side of the world in my case. But yeah, now I am back in
Spain. I was back in Spain for a while.
Now I wrote this book and let's see what's next.
This episode is brought to you by Swim dot IO and I'm excited
to have its CTO and Co founder Omar Rosenbaum with me today to
tell you more about Swim. Hi Henry, very nice to meet you
and thank you for having me. So tell us a little bit more,
what is swim dot IO? At Swim, we want to help
companies understand their code bases.
We combine static code analysis with generative AI to create
comprehensive documents that help you navigate the code base.
As an engineer myself, I wouldn't want 10 years to spend
so much time understanding existing code.
I would want them to spend time creating and building new stuff.
When you have code that has accumulated over decades, and
especially in legacy languages that not many people are adapted
nowadays, then the problem is even bigger.
Swim dot IO is specializing into helping mainframe developers to
understand their copies. Why mainframes?
We actually didn't start there. COBOL had been by some people
obsolete for a few years, and I discovered that it's not really
obsolete, not at all. There are more than 800 billion
lines of COBOL code that are in production and they drive lots
of the business in the world and we got more and more requests.
From. Customers to help them
understand the legacy code business that was written
decades ago and get accumulated over a very long period of time.
So from your customers so far, what are the some of the success
stories that you can share? So we worked with an analyst who
shared with us that it took them a year to document a single
mainframe application, and using SWIM they were able to document
a similar application in a matter of hours.
So saving that amount of time enables them to focus on other
tasks. Thanks Amir for sharing with us
about SWIM today. To learn more about SWIM, check
out their website at swim dot IO.
Wow, thank you for sharing your story.
I think one thing that you mentioned earlier, right?
So train your juniors. Well, I think maybe not all
juniors are fortunate enough to have this kind of good training,
good principles being taught since the very early of their
career. So maybe for those who are maybe
not so fortunate like you or maybe some of us.
So what advice would you give to them such that even though they
don't get the good training from the company or from the seniors
they work with, how can they still learn a good principles at
the start of their career? I'm going to say, well, first of
all, try to look at the literature because that has been
really, really helpful for me. You know, in terms of good
practices, in terms of how to be a better programmer.
The literature has so much insight.
Even if you cannot get it from your seniors or from your
mentors, you can always fall back to that.
Just don't be afraid to ask questions and be annoying, you
know, try to be the one with the keyboard when you're pair
programming, for example. Yes, it will be a bit slower.
Yes, you will have more questions than someone else
would and hit more blockers than someone else would.
But make the best of your mentors being there and don't be
afraid to annoy them because that's the only way that you get
to learn, you know, being hands on.
I like your term be the keyboard, right.
So sometimes we may not be comfortable asking questions to
senior and become like, I don't know, some people think they
might be a hindrance to the discussion and all that.
But I think, yeah, beginner's mind, I think that's always a
very good skills to have in any kind of team.
Yeah, I was going to say, and it definitely helps the team as
well. You know, you might think that
you're annoying everyone by asking questions, but a lot of
times it actually uncovers something more to the
discussion, even what might be a really obvious question.
So yeah, definitely don't feel like an annoyance.
What you're bringing is really, really valuable, which is a
fresh perspective. Right, I think that's really
great tips, right? So for all the beginners, even
maybe for seniors, right, Sometimes don't be afraid to ask
like so-called stupid questions. So let's go dive into your book,
right? Continuous deployment.
For me, it's almost like natural.
One day somebody will write about this continuous
deployment, even though you know, like CICD has been, you
know, around for, I don't know, more than a decade, I guess.
So maybe in the 1st place. Tell us your story, the
background, What inspired you to actually write this book?
Yeah, And actually I brought the book.
It's right here, continues deployment.
You can get the physical copy on Amazon, of course.
So what inspired me to write it, basically, as you said, it
sounded like a really obvious gap in the literature that
somehow was missing. I don't know how nobody had
thought about writing this book yet, but it's definitely a gap
that I felt when I was working because being a consultant, I
was really lucky to be part of some teams at some clients that
were already doing continuous deployment.
They were doing it for many, many years, but sometimes we
were on boarding more junior people or consultants that
didn't have this experience. And even when they were really
familiar with other great practices like continuous
delivery, continuous integration develops, there was still a bit
of a learning curve. So I found myself basically in a
situation where I wanted to communicate what is the
difference between stopping your commits at staging and every
commit going to prod. And it turns out there was quite
a lot to communicate about it. And I had basically no
literature that I could just send them or articles that I
could just send them to, you know, just read up about it and
be ready the next day. And so, yeah, based on the
missing this kind of content, I decided to create it.
So I started the by writing some blog posts and doing some talks
about the subject. And yeah, then it got picked up
by Riley and here it is up. So hopefully this will help
other people who have been in my situations, for example, in
teams that are already practicing continuous deployment
and you want to ember someone else or if you're one of the
people joining such a team. So this is a pretty condensed
version, even though it's 400 pages long, of everything that
you might ever need to know about it.
Yeah, and he also got Dave Farley to write the forward of
the book, right, one of the co-authors of the, you know, I
would say Seminole book continuous delivery, right.
So I think maybe let's start from there, right?
The definition it's always very confusing when we talk about,
you know, CICD in the industry, right?
Some people even already associate CD with continuous
deployment. I guess maybe a little bit off
back to the fundamentals, if you can elaborate what as continuous
integration, continuous delivery and continuous deployment.
Thank you. So this is a topic where the
terminology has undergone a lot of semantic diffusion, as Martin
Fowler calls it. And a lot of people in the
industry refer to CI and or CD as the pipeline, effectively
forgetting that in reality all of these things started as
practices. So the pipeline is just an
artefact that helps you realise the practice, but there is quite
a lot more to it. So starting with some
definitions, the first practice that came along was continuous
integration as part of extreme programming.
So the practice behind that outside of the pipeline tool was
basically the act of integrating your changes into a mainline
shared branch really, really frequently.
This had the purpose of always having all of the changes
bundled into a buildable artefact that could be validated
by tests for example, and so that the team could always show
their progress. So it came with the practices
such as nobody pushes in a broken build, we have to
integrate changes frequently, not hold changes in really long
lived feature branches and so on and so forth.
Then the pipeline, which is how implementation looks like or can
look like of continuous integration, takes care of
taking this mainline branch, building an artifact from it and
running some automated tests. And that was a really big change
at the time because that wasn't a particular way of working that
existed so much before. And suddenly it was really,
really easy to show progress and say, look, this artefact
represents what we have been working on so far.
And it is buildable, it is testable, and it can be deployed
because it contains every change that has happened, say in the
past day or in the past week. That made the integration much
more frequent and introduced the concept of the pipeline.
But then continuous delivery came along and it brought even
more to the conversation. So continuous delivery added on
top of continuous integration, the fact that not only your
changes should be frequently integrated, frequently tested,
frequently built, but also frequently deployed.
And every single change that goes through the automated
pipeline should be verified to such a greater confidence level
that should be considered deployable to production.
So basically this was a really big mindset shift because then
it came with other practices, say from the BOPS such as
infrastructure is called and automating the deployments
themselves, which didn't have to be manual anymore.
But yeah, it was also a practice before it was the automated
pipeline too. Then a further next step on top
of continuous delivery, again to me and to many other people has
been continuous deployment. Where not only every change that
goes through the pipeline should be considered the deployment to
production candidate, but it should be actually deployed to
production. So basically this to me was a
culmination of this concept of the automated deployment
pipeline, where every single step between development and
deployment to production has been automated.
In the traditional implementation of continuous
delivery for example, you might see that many many teams deploy
automatically with their pipeline to some pre production
environments, say diverse aging. But the deployment to
production, while it can be done on demand, is still a manual
action. So it's the push of a button to
be done only when the team feels confident enough.
Let's say continuous deployment is pushing that limit away and
removing that final manual gate in the value stream of building
software and deploying it to prod.
And so, yeah, it is definitely a combination of a journey that
has maybe taken I'd say 20 years since Extreme programming and
continues integration came along and is bringing automation all
the way to the end of the value stream.
I like that you brought up the historical aspect of how all
this combination happened, right?
So from things like extreme programming, you know, DevOps,
Agile and things like that. And I like the specific thing
that you mentioned, right? It's not all about pipeline,
right? Because people associate maybe
CICD with pipeline concept, even tools like Jenkins, maybe no
GitHub Actions and things like that.
But it's about the practices, right?
The practices that actually lead us to use some of the tools that
could help us actually adopt those practices, maybe slightly
a little bit more about definition as well.
People associate CIS less CD, right?
They think they are kind of like a different thing.
And now what is the term that you're gonna use for continuous
deployment? Is it also the same CD or is
there any other term? Actually, that's not really set
in stone yet. I've been debating it with my
colleagues, some of them abbreviated to C deck just to
distinguish it from continuous delivery.
So I if you find it helpful, I encourage you to continue on
that. But yeah, otherwise you can just
spell it out. Continuous deployment, just not
too confusing. So I think 1 aspect that is
very, very important that I would like you to also try to
explain, right? This thing in the extreme
programming, right? If it hurts, do it more often,
right? So I think this is maybe one of
the key reason why continuous delivery was kind of like
invented in the 1st place. Maybe tell us why this is
important for us, especially for those teams who are still kind
of like fearful whenever they make code changes and they want
to deploy even to production. So I love that principle from
extreme Programming. To me, that is the essence of
extreme programming and most of agile practices themselves to be
honest, is software has many more parts to it than just
writing it. There's integration, there's
testing, there's deployment and all of those other parts before
XP used to be pretty painful. So, you know, in an old fashion
waterfall process, you will maybe have your giant
development phase, then a big integration phase, then a BQA
phase and so on and so forth. Now that used to be pretty
painful because it was done infrequently, right?
So the principle of extreme programming basically says if
something is painful, such as these phases of writing
software, you should do them more often because doing them
more often encourages their automation, which is a great
thing. And in and on itself, automation
allows you to first of all document and also make a process
which can be replicated. It is not relying on someone's
knowledge on doing things by hand, but also more crucially,
if you do things more often, like integrating software,
testing software, there is less software to integrate and test
and deploy each time. So when your batches are
smaller, and here maybe I'll talk later about lean
manufacturing, suddenly all of those phases become much less
painful. And that is why this advice
seems counterintuitive, but actually it allows to feel less
pain in performing all of those phases.
That said, I believe that continuous deployment is the
perfect example of applying this principle and putting it into
practice. Because to me, there is
absolutely nothing I can think of that is more painful than
deploying to production for teams, right?
That is so much more painful than resolving some git
conflicts. So during integration or
spending a lot of time on testing, the deployment to
production is when the website can go down, your app might stop
working, you could scrap your database.
So because it is much more scary and much more painful, that's
why I recommend to do it as often as you can, and in this
case automatically at every commit.
Yeah. So I think in the past,
traditionally, I also used to experience those Big Bang
release, right, where you kind of like have a certain window,
maybe quarterly, whatever that is.
And you kind of like build a ceremony around production
deployment, right? Kind of like fearful whenever
the new change going to be used by the real users and whether
there's going to be a big impact or buck happening in the
production. So I think the spirit here, if
it hurts right, do it more often so that your production
deployment becomes like a more boring kind of event.
I was going to say whole ceremony about deploying to
production, the whole do not deploy on Fridays.
That's not something that you should be prideful about.
That's actually a smell that your process might not really be
so great. If that deployment is that
risky, it means maybe there's something going on that you need
to address. Perhaps do it a bit more
frequently and split it in smaller smaller deployments.
Yeah. Speaking about the difference
between continuous delivery and continuous deployment, I mean
one big aspect that I see is the big differentiator is what you
call the still kind of like the manual deployment production,
right. So in the past, we used to have
like this manual approval stage where you know somebody will
decide when we actually deploy to production.
So tell us why it is very, very important to actually remove
this last critical piece, because for some stakeholders,
maybe they think having that is kind of like a guardrails to
ensure things are still going to be safe.
It's kind of like verified before things are deployed to
production. So to explain why it is
important to remove these, which might seem like a small piece
and a very important guardly, I'm going to connect to the
practice of lean manufacturing, right?
To me, really the most foundational practice of Lean is
removing inventory from the value stream of software.
So what we see as the founding principle of Lean is that
instead of having big batches of inventory going in between
workstations, we should try to reduce the flow of inventory
that is going through the system to basically be only one piece
at a time. This allows to do some crucial
things, which is to, for example, identify the
immediately as they flow through the value stream, rather than
waiting for an entire batch of things to be processed and going
to the next workstation. But also, even more crucially,
it allows to drastically reduce the cycle time, so the time it
takes for any piece of work to flow through the entire stream
and be in the hands of customers.
Now, how does this translate to software?
This is a principle that was very successful when applied to
manufacturing of physical goods. But of course it's very hard to
define inventory and software. So what I think of inventory and
software is pretty much every intellectual artefact that is
included in software manufacturing.
So this for example, could be user stories, it could be
architecture diagrams, or it could be code.
So I'm going to focus mostly about code here.
How to define a batch of software?
For me, it needs to depend on how to identify a unit of
software. Defining a batch of software
basically has to depend on defining a unit of software.
So what is really a unit of software like?
When do we know that we have a big batch of inventory?
A unit cannot be really defined as a single instruction in code.
I don't feel like because that's not self consistent.
Instead, I define a unit of code as a commit because that is the
smallest unit that we can have flow through that value stream.
In this case, the value stream could be represented by our
pipeline. So a commit is the smallest
possible unit by which we can review software, test software,
and eventually deploy software. So to me, a batch becomes when
commits accumulate during this value stream, right?
So when commits accumulate in your pipeline and that is when
you have some inventory that is stuck at a specific phase of
your value stream of code and that is where the effects can
accumulate. That is where your cycle time is
slowing down and that is where all of the anti patterns of not
doing lean start to emerge. So bringing this back to
continuous deployment, you can think of continuous integration
and continuous delivery as the gradual transition from an old
batch and queue processes such as Waterfall, for example, where
there is a lot of inventory moving really, really slowly in
between workstation. Continuous integration and
continuous delivery represented a shift from that.
So with continuous integration, you're starting to have what
looks like a one piece flow from the development phase at least
to the integration phase because it says integrate frequently.
So how frequently can you integrate at every commit of
course, and how frequently can you test at every commit?
So you have commit suddenly flowing one by one through at
least some of the earlier phases of software manufacturing.
And so your inventory there is checked with the maximum
granularity that it can be checked, of course, later than
those phases. So say the deployment phase
wasn't yet automated in continuous integration.
So that's where commits or inventory could accumulate right
before a deployment. And basically it would go
through the rest of the process again in batches.
Now with continuous delivery, the deployment phases following
that have been automated. So again, this one piece flow of
inventory had been extended, but still with most implementations
of continuous integration commits still accumulating
preproduction or in staging. So that's where basically,
again, your defects are showing up and your commits are
accumulating right before production.
Now this makes deployments to productions riskier because of
course you have a bigger batch. And so I mean the fact that has
accumulated in that batch is going to show up upon
deployment. So I think if you want to
realize this one piece flow all the way end to end, you
shouldn't have any manual points in the pipeline where these
commits can accumulate and become stale.
Wow. I think it's a very nice
explanation why we should work towards this manner, right?
This practice, right? So one piece flow, I think
that's a very important thing from lean principles, right?
So I think also maybe for some people who are still not yet in
this kind of working mode, right?
What do you see some challenges maybe from your consulting,
maybe from, you know, having the chance working with enterprise
mostly I believe they will have a lot of money shifts that need
to happen. What will be some common
challenges? Because deploying every commit
straight into production is not something that all the team can
aspire to do, right? Some might have regulations,
some might have compliance checks, some might have any
gatekeeping inside the software delivery.
So what do you think are some common challenges that people
need to really start to move away?
Yeah. And what you just said is
exactly why I have at least two chapters of disclaimers around
the when can you really do continuous deployment, When can
you? Not surprisingly, there's a lot
of situations where you think you might not be able to do it,
such as regulated industries, but in reality you're able to.
It depends really on the framework and regulation.
For example, in my case studies, I have a new bank called N 26,
which is subject to basically all core banking regulations,
but they were able to still use a continuous deployment workflow
because they have an automated pipeline that basically
satisfies all of the auditability requirements of the
process. And as they finally says in his
excellent article, continuous compliance, an automated
pipeline is an absolute gold mine for auditors.
Basically, it has so much information on which changes
were applied to production, when, by who, for what reason,
because it will be related to a certain ticket, what tests were
they performed on them, how did these tests prove that they were
successful, and so on and so forth.
So that already really helps, and it's just a matter of
figuring out what parts of your process can really apply to
satisfy this regulation. At the same time, it's very
common in regulated environments to have authorized rules where
nobody can really put a commit in production alone.
That is something that also this bank was able to satisfy because
they found a way by doing a combination of pair programming
and micro branches with like immediate pull requests approved
by a pair. That was their way to prove that
a commit was seen by at least two people, the person that
authored the commits and the person that merged.
It might sound counterintuitive, but there are ways to still use
modern practices in regulated industries.
Not because these practices are less safe, but because they are
more safe than manual approvals and waterfall.
And we just need to find a way to articulate that and really
explain that. Other situations where it might
be difficult to perform continuous deployment are
usually more by technical challenges rather than
regulatory ones, in my experience.
So for example, something that is really common that happens to
teams that maybe want to move from continuous delivery to
continuous deployment is when you start doing deployments
really, really frequently, you might find out that your system
is more sensitive to deployments than you might think.
And so this might affect things like your reliance on caches.
Suddenly when you deploy many different versions all in the
same day, your caches, if you have in memory ones, are going
to be mostly cold most of the day.
So that's when you need to find maybe architectural workarounds
such as moving your caches externally and so on.
Also, this applies to any kind of in memory state that you
might have in your application. So if your user flow depends on
certain state being stored in memory for whatever reason, that
is something that the frequent deployments are going to be
disrupting really, really frequently.
So you need to also fix those kinds of scenarios before you
can really move on to this workflow.
Another really common scenario where people have trouble, and
maybe this one is not so easily fixable, is when your production
environment that you want to continuously deploy to doesn't
belong to you. So after we think of production
as a server that is in our hands and it is in our capability to
deploy to, but what about, for example, mobile apps or IoT
devices or on premises infrastructure that belongs to
someone else, right? In that case, it's a bit more of
a grey area. What does it even mean to be
done with the deployment to production?
Are you done with your deployment when the first user
device has been updated or when the last one has been updated?
So there, that's when probably continuous deployment is not so
easy to implement. But again, there are workarounds
that you can use. So for example, you can use Pwas
to move control back to the server web views.
But yeah, that's when you may struggle a little bit.
Yeah. Thanks for the overview of some
of the challenges which I assume some teams would have, right,
whenever they want to try to implement this continuous
deployment. But having said that, there's
also another team which probably claimed that they do continuous
deployment, which I want you to kind of like explain because
they may not have so many processes.
They just have a commit and they maybe do something in the build
pipeline, right? Maybe creating the artefact and
just deployed production. I'm sure this is not something
that you are advocating, right? Continuous deployment doesn't
mean that you just deploy straight away, but there are
certain processes in place. So tell us what is the minimum
thing that you should have in the continuous deployment?
Yeah. And thank you so much for asking
that question because I don't want anybody to take as a take
away from this interview that whatever your the state of your
automation now you can just get rid of any manual approval and
go straight to prod and nothing will happen.
So basically there are a certain minimum amount of practices that
you should definitely make sure that you have.
So again, a pipeline that is fully automated, but that
pipeline needs to contain tests and a lot of them.
So that's what I go back to recommending, you know, the
typical testing pyramid, which means you should have a pretty,
pretty good unit test coverage, a pretty good coverage of
automated tests both before and after deployments, if that is
possible for you. There are practices that can
maximize this coverage. And the ones that I use most
often are the ones that basically allow you to write
your test first. So TDD is my prime example and
outside in TDD, if you use this, you know that your coverage is
growing together with your code in the same commit which is done
going through the pipeline. So those can make you pretty
safe. Then alongside tests, which are
great for functional changes, I always recommend some forms of
static code analysis to be in your pipeline, such as security
scanning, performance scanning. If you use a tool like Sonar
Cube, that is going to be really effective at catching most
regressions that maybe might be missed by your tests.
And that's really important, right?
Because if you're publishing every commit, then every single
commit might affect the performance of production or
open a security vulnerability. So static scanning can really,
really catch all of those regressions and that really more
granularity. Then of course, another question
that I get really often about what are the prerequisites is
how do you hide your work in progress if you're just
publishing every commit to production?
Because you say you shouldn't have long Fisher branches where
you hide your changes, but also everything on master needs to go
to prod immediately. So we need to also be
comfortable with our team to use patterns like feature flags and
the typical expand and contract pattern or parallel change or
branch by extraction, whatever you call it.
All of the those patterns can help you hide any new change
that you don't want your users to see, even though that change
is in production. So I highly recommend to be
familiar with those. Then finally, of course, you
need to have 0 downtime deployments as a minimum
practice. I think most teams do these
days, but it's always best to check so you can achieve those
with rolling updates. You can achieve it with
blue-green deployments or red, black.
You can also perform cannery deployments with tools like
Spinacer, where basically the tool is going to be connected to
your observability tool and observe for example, the error
rate of your application and roll back immediately if that
error rate becomes worse. So observability and alerting is
maybe the final absolute prerequisite that I recommend
before turning on, you know, the fully automated pipeline,
removing the last gate to prod. Because as employments get more
frequent, there is also less oversight on them.
As you said, they become just a boring event that happens
multiple times a day. So we need to be proactively
notified in the very rare event that something does go wrong.
And that's where I recommend having a really, really sturdy
dashboard, keeping track of all your metrics.
And most of all, take care of your alerting based on your
metrics, because you definitely don't want your team to be used
to seeing a lot of alerts and not having to react to them.
So keeping those really tidy and only keeping alerts that are
useful and require any sort of action that's going to really
help. Thanks for the clarification.
I mean just to re emphasize one more time, continuous deployment
doesn't mean that any kind of commit you just deploy to
production straight away, right? So there are certain practices,
certain minimum that you need to achieve, right.
And Valentin, I just mentioned, you know several of them, right,
which I believe are like some core practices that every
engineering team should aspire to do.
I think 1 aspect of continuous delivery that is also very
important is that everything that you prepare to go to
production, right, must be verifiable and you have high
degree of confidence that it will work in production as well.
So it's not something that you kind of like Yolo, you know, you
just wait for the users to report the bugs to you.
And I think one other aspect is about backward compatibility,
right? So whenever you make so many
changes, right, I'm sure some users, maybe some servers are
not taking the changes straight away, right?
So you need to make sure that some backward compatibility must
be in place, which brings me to this thing that you mentioned,
right? Hiding work in progress,
definitely. If every commit must go to
production, we cannot finish a new feature just within one
commit, right? So you mentioned about feature
flag or maybe expand and contract.
Tell us more a little bit about this practice because I'm sure
some people might still think, oh, I cannot work this way
simply because I have some work that I can't finish.
So tell us how we can use those techniques to do this.
Yeah, thank you for mentioning those.
They are the ones that I rely on most heavily throughout the book
as well. So I think it's definitely worth
to take the time and understand them.
So let's start with the feature flags.
So I recommend using feature flags whenever there are visible
changes to the code base and the application that if deploy to
production, will alter what the user experiencing and the
observable behaviour of the system.
Most people are used to hiding such changes in version control
branches and then the act of publishing them will be to merge
the version control branch into main and finally let's main go
to production. Using a feature flag instead
still uses a kind of branch, but it is not a branch that belongs
in the version control system, it is a branch that belongs to
the tree of classes and functions within your
application. So I call it an execution branch
for that reason, and that will do exactly the same job of
Aversion control branch, except it's going to still allow you to
integrate your code frequently and deploy your code to
production frequently. So during development, you will
basically put a feature flag as the very first step of beginning
any new feature, and the feature flag will be off until that
feature is released to the users in production.
I think this is really powerful because it also allows you to
express the conditional logic of when you want your feature to be
seen or not seen, for example, for a reason, in a way that is
much more expressive than just the textual control that version
control gives you. So I really rely on feature
flags a lot for that reason. Finally, for expand and
contract, that is a pattern that I tend to use when I'm making
not so visible changes, but changes that I still I'm not
feeling super confident in, or that I want to split into
smaller and smaller increments. An example of this could be
refactoring a really big subsystem of your code base, or
for example an entire endpoint of an API.
That's when I would basically duplicate the flow completely,
making sure that the interface of the old flow and the
interface of the new flow is respected equally.
That allows you to basically commit at any point, because you
can migrate the clients of the old flow to the new flow 1 by 1
and still completely respect the functionality of the system,
which your regression tests should be able to catch if you
broke that. So yeah, you can read more about
this in the book. I have a lot of concrete code
examples. Yeah, I highly recommend reading
this part because I think this is one key practice that is very
important in continuous delivery and continuous deployment,
right? So for those of you who also
have not heard about expand and contract before, some people
call it branch by abstraction. Maybe if you have like OOP, you
have interface implementation, different abstract class and
different implementation. This is also one aspect that is
kind of like similar in terms of implementation.
And I think this is also a good time to actually explain the
difference between what we call deployment versus release,
right? Because with things like feature
flag, it allows us to kind of like control the time to release
a particular feature. So tell us about this
difference. Yeah, and this is the reason why
I love feature flags, right? So with continuous delivery, it
was still true that you could use feature flags and have a
separation of deployment and release.
But with continuous deployment it becomes mandatory and the
user feature flags becomes mandatory because your pipeline
is not gonna stop anything visible from going to prod if
the tests are green. Because those practices are
mandatory, the distinction between deployment and release
becomes much more well marked and much more important for
teams. So a deployment in itself during
continuous deployment is any roll out of new conversions into
production. But a deployment in and of
itself should never alter the visible behavior of the system.
At the same time, a release usually becomes the act of
turning on a feature flag, which again, if you have a good
feature flag framework, should never require a deployment
because you can do so at runtime.
So there is a very, very clear distinction in the timing and
how these two things happen. A deployment becomes just an
engineering concern and release becomes a product concern.
It's like, when is there a time to turn on this feature?
Do we want? The entire user base to see it?
Or do we want to run a 5050 test first?
Or do we want to do a gradual ramp up?
How does product feel like this would be best released?
And this also brings us to a really great situation where a
deployment being purely an engineering concern can be done
without the input of products. So we should never be in the
situation, say, where I as a developer get told please do not
deploy any new version because we are in the peak season for
example, for our business and the product wants to wait.
At the same time, I should never be in the situation where as a
developer I have to tell product that feature cannot be released
because as there is a pending deployment because all the code
should have been in production in the 1st place.
So yeah, definitely these two things kind of take a life of
their own with the use of feature flags.
And that is why I consider them such a strong, strong
prerequisite for any modern pipeline, be it continuous
deployment or also continuous delivery.
Yeah, I think that's a good distinction, right?
Deployment should be the technical or engineering
concern, while release should be the product or stakeholders
position, right. If the team can separate these
kind of A2 important activities, I think that's a really great
shape. And I think feature flag, so
many tools these days, I'm sure having a platform that can allow
you to do this behavior is something that is not impossible
anymore. Another important thing that I
think what to call out here is about slicing the work, right?
Because if let's say I want to build a new feature, sometimes a
feature is not like very simple like change a color of something
or create a button or something like that.
But it's more like a maybe like workflow or something like that.
How do you actually advise us to slice the work right?
Because making sure that the small batch committed up to
production is something that maybe some of us are not used
to. That's a great question.
So there's always been that eternal battle between slicing
work by code base and slicing work more vertically by feature.
I'm going to say with this type of workflow, continuous
deployment is even more important than it was before to
start thinking of slicing work vertically by feature instead of
by code base. And it's for the very simple
reason that as I mentioned, we have practices like feature
flags and especially expand and contract that require multiple
small changes to production on different systems a lot of the
time to guarantee backwards compatibility until the feature
is completed. So for example, in order to add
the some new functionality to my system, I might need to alter my
front end and back end code base and my DB.
Right now, because I'm doing continuous deployment, I want to
maybe build it slowly and deploy multiple commits one at a time
into production to guarantee for example backwards compatibility
of my back end, say with my database schema.
Maybe I need to create a new table, migrate something in my
back end to go to this new table instead of the old one that's a
new deployment on the back end. And then maybe I need to
redeploy my database to clean up the old table, let me
synchronize some data. So if I slice these tasks
horizontally, I have to create so many tasks just to jump
around on which change do I need to make now on this system and
on this system, And they're all gonna be blocking each other.
So it's not a really efficient way to work with these ways of
working. Instead, if I slice vertically,
I can split my task, my user story into multiple commits, but
I have the freedom of relating them all to that ticket or that
user story and to really plan out within the story what
commits do I really need to do in which system and in which
order to achieve that functionality at the very end.
So yeah, it becomes much, much more important to slice
vertically. And it also allows you really to
leverage the granularity of continuous deployment as a
practice because the fact that everything is in production all
the time, it can make emerge some situations where actually
you're ready to release a feature in a state where you
previously thought you wouldn't. So say after maybe only two or
three user stories instead of after all 10 user stories,
because perhaps product wants to run an IB test a bit earlier.
And so not only becomes important for technical reasons,
but also it's like you really leverage the speed of continuous
deployment. Yeah.
So I think vertical slicing is the key here, right?
It also goes back to how you slice your story, right?
So a feature should be sliced into a different shape of
stories. And if you can shape it more
vertically, that brings value like any kind of value to the
users, maybe some other third parties, right?
I think that will be the key. Talking about third parties, I
know these days people are working in a kind of like micro
service or service oriented architecture, continuous
deployment also have this challenge of having to
coordinate your changes with other third parties APIs,
whatever that is, right? So tell us how can we do this
better? Because obviously you can't
control how the other team is ready or taking changes from
your new version, right? So how would you advise us to do
this? It depends.
There's third parties and third parties, right?
If you're working in a team that takes care of micro service
within a larger ecosystem, another team within the same
organization will be a third party to you.
But also it could be that you have to communicate with a
vendor system which is in a completely separate organization
and they have a completely different change management
process. So I'm gonna say there is much
more flexibility when you're dealing with other teams as
third parties because you can coordinate much more frequently
on what new version of what is going live at which point.
And you might even be able to pair with someone from another
team to always guarantee synchronisation between the
systems changing. On the other hand, if you're
dealing with third party vendors, that's when you need
some stronger practices that are much more formal to manage the
contracts in between systems. And that's where I usually fall
back to things like API versioning, which is something
that I wouldn't do necessarily if I have an API just for
internal usage. I can very easily coordinate
with other teams and say, hey, please change your usage like
this and we'll tidy up without the need to maintain 1000
different versions of some API. That is going to be a lot of
maintenance, but with an extended vendor where I have no
control or what they're doing, that's when I would pay the
price of versioning an API, for example.
Same thing with contract testing.
I'm going to have much stronger contract testing and
requirements if I am consuming or producing an API for a third
party vendor than for a team within my same organization.
So that's how I would handle it. And of course within the
organization, that's what practices like expanding
contract and feature flags get even more useful them within
just a singular system because they not only allow to hide the
work in progress within one system, but also across several
system. So they can also be used to
guarantee backwards compatibility of systems that
are independently deployed and where therefore you cannot bump
everything to the next version exactly at the same time.
So I encourage teams to look into these practices as well for
working with micro services and testing integration with other
micro services. Yeah.
And I would like to again emphasize backward
compatibility, always ensure that you support a few versions
behind, right, simply because you can't control the other
teams whether they are taking a changes straight away or they
will take some time before they actually apply those changes,
right. So I think you mentioned also
contract testing and things like that definitely is a good
practice as well. I just wanted to add as well
that this is not just something that needs to be taken care of
when you have a micro service environment.
The concept of backwards compatibility applies to any
system which is distributed, even if at a glance it looks
like a monolithic system. In my experience any system that
is interesting rarely is really monolithic right?
For example, even if you have just one simple website with one
database, you already have 3 distributed components that you
need to keep backwards compatible at all times.
You have your front end which is executed on the user browser and
it's not going to receive changes at the same time your
server does. Then you have your server, and
then your database is a completely separate executable
and most often deployed on a different machine as well.
So the thing you really need to take into consideration is inter
process communication. Whenever there is such a thing,
which is almost always, you would probably want to rely on
things like feature flags or expand and contract at a
minimum. Continuous deployment just makes
it much more evident because with that it happens all the
time. You do deployments all the time
and usually micro service oriented environments.
But this is not a new problem. This is a problem that we should
have been solving already much before.
And you know, when deployments are infrequent, sometimes it's
easy to ignore if there's some deployment glitches.
Whereas with continuous deployment you can't ignore them
anymore because they just happen every single day.
Yeah. And this goes back again to the
principle. If it hurts, do it more often.
You should not have. If it hurts, then you stop doing
it, or you kind of like reduce the frequency of doing it,
right. So I think the more you do it,
the more you figure it out how to make it safer, how to make
it, you know, smoother. Another thing, apart from
technical practices, are there cultural aspects or maybe you
know, like onset shift that people should know about before
they can actually implement this continuous deployment?
Yeah, for sure. And that's why I wrote a book
about it as well. I think a lot of it has to be a
mindset shift change, especially for developers.
First and foremost, imagine that anytime you do git push origin
main because you're working in a trunk based development fashion,
of course you're not doing PRS right?
Every time you do that seven times a day, your users are on
that live system that is going to receive that update 20
minutes later. Suddenly, if you were using
continuous delivery before, it can be very paralyzing to think
about what you're putting in the pipeline at any given time.
But it doesn't have to be, right?
So there needs to to be this mindset shift of putting
production as a first class citizen every time you're
writing code and really think through before you pick up a new
task or a new user story. What commits am I going to put
in my pipeline that are going to make that happen?
And which subsystem do I need to start from to not break
production at any point? For example, if I have a
database back in the front end, that is going to be probably the
database system that I'm starting from if I'm adding
something new. Otherwise, if I deploy the front
end 1st and that's what I start developing, there's nothing
underneath to make the feature work.
It's all going to be broken. Every single aspect of your work
needs to be thought as a live change in a production
environment rather than a plan for a change that might happen
somewhere later and might be deployed by someone else at some
specified point in time. So it feels much more like being
an electrician working on an office building that is
currently in use, right? You have to always be careful of
which diversions you're installing, how you're going to
maintain compatibility with your system, to not disrupt this
living beast that is production. And that also includes not only
functional changes, but also security performance.
There's going to be a lot that you you were used to thinking
about maybe when your changes are in staging and you have to
now anticipate all that thought during development, which to me
is actually really positive. It can be overwhelming, but once
you're used to it, it means that you've fully shifted left.
All of those quality aspects, all of those cross functional
requirements for example, that maybe used to be relegated more
towards the end of development. Yeah.
So I like the concept that you mentioned.
Always make sure that deploying to production is the first class
citizen, right? So ensure that you think
whenever you make change, it's going to be straight away go to
production, right? So sometimes this requires you
to carefully think how you actually implement the changes
and things like that, including the shift left.
I think shift left still kind of like the correct mindset for
implementing this, right? As much as possible automate as
well those checks so that you know the pipeline can just
automatically deploy the production once everything is
verified working. So 1 aspect that is very trendy
these days is about AI. Anything related to AI that can
help continuous deployment practice.
I don't know whether it can help, maybe it's more something
to be aware of and slightly concerned about.
So especially for teams that use continuous deployment today, I
see that it is very common right now to just adopt Copilot.
I'm one of the culprits of this by the way.
I've recently installed it in my Intellij.
I just have it generate some code for me from time to time
when I can't be bothered to look up some configuration or maybe I
can't be bothered to do some test coverage.
I ask it to give me a nice suite of tests for a certain function.
But yeah, of course, if you're continuously deploying, it means
you have to double check, triple check, quadruple check anything
that the AI is giving you in order to be fairly sure that is
going through the pipeline. So when you re stick that you
could use this, for example, perhaps the AI could generate
the production code, but maybe you should be the one writing
your tests so that at least worst case scenario your
pipeline will fail. And in general, again, all the
practices that I've recommended before as base prerequisites,
such as test coverage, static code analysis, observability,
those are also going to be a really good safety net.
And yeah, you you should really double triple check that your
pipeline is really going to catch any and all types of
regressions if your team is relying on AI really heavily.
That said, it's a great tool. It can speed you up even more,
but please be careful with it. Take it with a grain of salt
whatever is spitting out on your screen because it is very easy
to make mistakes if you're just get used to not double checking
it and just trusting it blindly. Yeah, I think that's a really
great advice, right? Because AI, one thing for sure,
it can help us turn out more code.
But more code doesn't always mean good, right?
Sometimes you have to check, especially if you don't have
robust pipeline, robust automation tests and things like
that, right? So always ensure that the
principles actually the one that matters the most rather than
churning out the code. So Valentina, it's been a
pleasure talking to you. As we reach the end of our
conversation, I have one last question I'd like to ask you,
which I call the three technical leadership wisdom.
Just think of it like an advice you want to give to the
listeners here. Maybe if you can share your
version? Sure.
So I thought a bit about it. I would say especially relevant
to today's episode, we've been talking about like really modern
practices, I think continuous deployment.
There's nothing faster and more modern than that that I know of
in software development as like ways of working goes, one of the
main pieces of advice that I like to give is many tech leads
and developers tend to get sad if they can't use all the latest
and greatest in terms of ways of working tech.
And it's really sad because I believe it's not necessarily a
must have, right? Having everything perfect all
the time. I think as long as the current
practices of the organization are improving, that is a good
thing in and of itself. So don't be worried so much
about the status quo, be more worried about the trend.
Is the trend an improvement or are we getting worse?
And that's also true for people who are doing continuous
deployment and really nice and modern things.
You know, are you getting better at it every day or are you
slowly getting worse even though you've enabled it some time ago?
So worry about the trend would be my first piece of advice.
The second one is sometimes as developers and tech leads, we
get emotionally attached to solutions.
Again, I think of this is a bit of a pitfall and I believe our
job is maybe more about informing.
So informing of all different options and then let's take
holders determine what is their preferred one.
As long as they are aware of the pros and cons and they
acknowledge any risks, let them do it.
Don't take it too personally. And finally, yeah, my main piece
of advice is just if you're a tech leader like me, just trust
the people that you're working with.
Trust them to have thought about the edge cases, trust them to do
their spikes, trust them that they're working and doing the
best they can. So don't micromanage if you're a
tech kid, would be my final piece of advice.
Really beautiful advice, right? So I, I particularly like the
first one, worry about the trend, right?
The direction that you're going, right?
Sometimes you can take some more gradual steps rather than a big
bangs approach, right? But still, if you're going to
the right direction, I think that's the important thing.
So Valentina, if people love this conversation, they want to
check out more your resources. Is there a place where they can
find you online? Yeah, so I'm mostly posting on
LinkedIn, so you can follow me there.
Valentina Servila, feel free to write.
I used to be on Twitter slash X, but now I've left, so I migrated
to Blue Sky. You can find me there, but I
don't post as much as on LinkedIn.
So yeah, mainly LinkedIn, maybe Blue Sky or just drop me an
e-mail. You'll find the address in the
book. Thank you so much for your time.
I'm sure people learn a lot about continuous deployment
today, so hopefully people get to adopt these practices more
and more. Thank you.