When you work professionally as software engineer, this is not
practicing a hobby. You need to have numbers right,
not just fluffy words. What was the actual landed
impact on the business? Sometimes solving business
problems means building software that is good enough.
Good enough for today. I just want things to be simple.
Simple is complicated enough, especially at scale.
System design, a skill that separates good engineers from
great ones. So if you want to level up as
software engineer, this conversation is for you.
Joining me today is Bassem Dakaidi, senior software
engineer over at GitHub and especially at GitHub, they
operate at scale. He shares how they design their
software and solve specific problems in a way I didn't
expect. So enjoy from an outside
service. I feel, I feel like I've shared
this many times on the podcast, but I feel like the tech
community is quite unique. People create software, they
create products and they put them out in the open for free.
And GitHub is like the backbone that enables a lot of these
things. I feel like I don't see another
industry and it's it's kind of typical for me to say that
because I'm in this industry, but I don't see any similar
other industry where this is possible.
Right. Precisely.
And not only that, like the learning, the teaching aspect
that we have, how openly we share what we learn and and we,
we have this status associated with people who actually teach
in our industry, which I don't often see elsewhere.
A lot of industries, and I worked in other industries, like
I've worked in container terminals for example, for a
while we were building the whole infrastructure and what not.
It's very rare to find information about how a
container terminal operates. Unless you've like went to a
specific program that teaches that or taken specialized
courses, you're never going to get that information anywhere.
Like there are no blog posts that describe how things work.
You just got to learn it either from people who have been doing
or practicing this industry for a while or you go do a
specialized program. So I I definitely relate to what
you're saying and I think it's pretty unique, which is awesome.
There are some topics that I have in mind and especially that
I think are going to be a bigger focal point going towards the
future. Something like system design
where there's a lot of theory out there and then actually
putting it in practice. You can understand the the
theory, but then putting it to practice is still very
challenging, especially in what type of organization you are.
How, How far in the future are you going to design?
I feel like it's challenging bridging the gap between theory
and practice for that aspect A. 100% And I was thinking about
this topic earlier today, right? I was just reading a, a, a post
on X that went viral where a start up was talking about the
migration to Kubernetes and then it almost cost them their start
up. Like the start up could have
almost failed because of that particular migration.
And the owner was listing the the the reasons why they went
for the migrations and none of them were technical like the
reasons were. Their VCs were pushing for cloud
native, their engineers wanted the Kubernetes experience and
they thought that they can deploy faster, bigger scale
better. But what they ended up with is
just a fatter AWS bill and then the much more complication
feature shipment crawled to a halt.
They had to dedicate certain engineers for this.
And I think this stems from the fact that a lot of people want
the status associated with fancy architectures and they want the
experience of working with fancy architectures and not
necessarily solving problems with fancy architectures or
growing organically a certain architecture to solve a
particular problem. And the problem with system
design is, again, a lot of people want the fancy stuff, but
they the boundaries for when you need to scale are very blurry.
There are no concrete empirical numbers that tell you like when
you hit this particular scale you need to do this, and when
you hit this particular number you need to do that.
So people assume that as soon as they hit on their service or API
or whatever, like 1000 requests per second, that's a lot.
So now they need to build for the next evolution of the
system. And often that's not really the
case. At GitHub, for example, I've
built services that handle millions of requests per second
that are just simply running on a 5 or 6 containers and a very
tiny Kubernetes cluster. Somewhere that's not what I
would expect right at all. Precisely.
So there's a lot. There's a lot you can do with
very little and you can even run entire services on very basic
VMS and scale some stuff around it.
And the way we approach problems of scale, we'd never design or
over engineer. So when I designed certain
things, I recently did a lot of that.
We were rewriting some parts of actions, GitHub Actions.
That's what I work on. So we're going to talk more
about that in engineering blog posts and whatnot.
This is like, in my opinion, a case study for decades to come
because this doesn't often happen.
That's awesome. Yeah.
So when we were designing this, we were designing for scale.
But we know already we have established scale.
So we know that we've hit the limit of what the previous
architecture could do. And now is the point where we
need to squeeze more out of a different tech stack to get to
that next level. And we know that we're going to
get to that next level because we are.
We see the demand, we see how it's growing, we see the curve,
we see how where it's going to be in five years from now,
right. So we solved for that.
We didn't prematurely scale. We just went all the way to the
end with the existing architecture, and that's when we
made the decision to rewrite some components and redesign
certain aspects of it and make tradeoffs based on data we had
and based on projections that we're going to attain or get.
Is this then your advice as well?
So to stick with what you have, start with an initial design
that is simple, and then stick with it until you actually reach
certain limits and you can kind of based on your usage, forecast
what the next iteration would. Precisely, If I was a start up
founder or ACTO of a start up right, I would never go and
start building for 100 X scale. Even if I'm like a firm believer
in my idea and know that it's going to reach the masses and
it's going to be global, I would never design for that.
I would design for getting 100 users or maybe even 1000, right?
And I would just run everything on a single VM, to be honest.
Maybe 2, make it a bit more available.
I would not go and build massive database clusters.
Just start with one, maybe 2 nodes pops replication, that's
it. And this will like carry you
really, really far. To be honest, once I attain or
I'm get, I'm like close to 80% of these limits of what my
resources can do. And by the way, vertical scaling
can take you a long way. Like you have no idea.
We have my SQL nodes that have hundreds of CP us on them.
So you don't really need to go and start sharding and doing all
sorts of fancy stuff when vertical scaling can still carry
you. And right now with cloud native
solutions, you can go pretty far.
Like you can get a one VM with 120 CP cores, right?
You can get terabytes of memory, Yeah.
And that can do a lot for you. You don't need to start doing
horizontal scaling before you get you really start hitting
these limits. Interesting.
See, I, I feel like I'm not great at system design.
It's just not something that I specifically put my thoughts to,
not something I prepared like from an interview style.
And then I actually did a system design interview and my
architecture was, it was very simple and I wanted it to be
simple because I don't like complexity when we don't need
it. And it was also for the use
case. I was like, this is, this is the
simple aspect. Then the question was, OK, but
what if we hit X amount of scale and then we need, indeed, we
need high availability and we need charting and we need
consistency. And then I was like, but the
like the argument that I had is in this case it it actually
doesn't matter for me. The part of the benefit that we
have a lot of information out there, people can self educate
is not also turning into this pitfall where complex system
design is almost I think in most industries where we don't
actually have high availability across the globe is not needed.
Yeah, people see that and they look up to big organizations
like GitHub where they say, OK, this is what GitHub has.
So we're going to do the same. We're going to build for the
future. 100% And this is in my opinion, again, going back to
the original thought that people chase status.
So, and this was honestly also very badly perpetuated by big
tech companies, right? And a lot of people followed
like in an interview for, for, for.
If you're interviewing for for a place like GitHub, for example,
it's understandable that they would want you or that they want
to explore problems of scale with you just because from day
one you're going to be exposed to these problems.
So what we are, what we are looking for typically is people
who have that experience who are going to come in and actually
use it on day one. Not that we might use it in a
few years time. We just hire for what we need
today. Because let's say you're
shipping, you're shipping features to a service and a lot
of our services have hundreds of millions of requests in short
periods of time. So you cannot really do a lot of
the you. You need to think about a lot of
things, right? You need to think about caching
from the from day zero. You need to think about your
queries someday. You need to think about your
data access patterns. You need to think about non
relational databases and non relational solutions for
whatever you're building. You need to think about how
you're going to gradually deploy this.
You need to think about feature flags, how they're going to play
in this game. You're going to think about
authorisation, authentication, all of the security problems
that might happen when you have introduced a security problem at
some at a place like GitHub, It's, it's going to be on the
news in a week or or less, right?
So you cannot make these mistakes.
It's very tricky. So we look for experience in
these domains so that people they still have there's room to
learn, but they need to deliver from very early on, right.
For smaller companies, they interview for the same, but they
don't need it. And that's where the problems,
that's where the problem stands. If I was in a start up and I'm
I'm doing system design, I would interview.
I would want to hear from a person who has exactly what we
need for today. Forget to 100 million tomorrow.
Sure. We'll start hiring for tomorrow
when we get to it, to be honest. And with the churn rate in our
industry being two or three years, like people don't stay in
the same job for more than that. Why would you, you know, hire
for the next 10 years a person? Yeah.
I feel like people really underestimate like how much they
should or overestimate how much they should design for the
future. Like I'm in a consultancy right
now. I go from assignment to
assignment and some assignments that I've been in, I'm like, the
system is way too complex for what we had.
We used to have 30 engineers. Now we have 8.
And the system we architected that was maintained by 30 people
and now we have a problem because we only have 8 because
budget is running out. We're behind on KPIs because
likely we we worked on a big rocket ship, but we actually
needed a car. We needed to drive and actually
make meters and. And I talk about this stuff in
my system design course because in consulting and when you sell
a solution to a company, the company wants to pay for that
solution. Once they don't understand that
software is evolved, it's not built right.
And the the term here is evolving it.
So software needs to continuously be evolved and it
needs to continuously fight entropy in the sense that you
continuously need to maintain it.
That is a fixed or running cost for maintenance of software that
you don't necessarily find in other industries.
When you buy a car, there is a cost for maintaining it, but
it's not very regular or as regular as software, right.
In our industry, the maintenance cost is is pretty high.
So companies want to pay once for a solution and they wanted
to scale to to the next 10X100X. And that's not very realistic in
our industry at all. We need to build literally for
the next order of of magnitude. So if we are at 0 today, just
build for 10X or 100X. If we're at 10X or 100X, we
build for the other order, right, 1001 million and so on
and so. Forth, what is then your
perspective on building systems that can evolve?
Because some people, you can say, OK, this is the system
design that we have and we're going to continue on this until
we reach a certain level of scale and then we're going to
revamp and say this would be the next iteration.
But some people also say, OK, we have a system now and this is
evolvable to whatever skill we need.
And then they put in the extra effort to make sure that that is
there from the beginning. In our industry, very few
technologies last for like a decade plus and the rest is
changing often. And that's why I personally
don't invest or I wouldn't design a solution that will last
like 1020 thirty years, unlike what we did in the 80s or 90s.
That's why that's how they that was the design mentality.
But things were way less complicated than they are today
and the scale was different. So I would always come with the
mindset that I'm just going to design for the next order of
magnitude and then I'm going to evolve it from there.
And that's going to require another round of investment.
There's going to be continual investments in this and that.
Business people are not going to love this because that makes
accounting much harder, That makes strategic finance much
harder because they cannot predict or anticipate the costs
that might come in the future. They cannot extrapolate from
today. You know they don't like this
fuzziness of software. It's all risk.
Yes. So and and also like they cannot
calculate like their bottom line or projections for the future,
which are things that are very realistic.
Like public, publicly traded companies want to be able to do
these projections and they want to be able to say that, hey,
we're not going to be spending more here in on evolving our
software, but we're going to make much more revenue and
increase our bottom line and margin.
That's what businesses look for. So it's understandable.
But from a software perspective, there's needs to be regular
investments. And I think the only
approachable way, which is something we do at GitHub, is to
continuously revisit these investments on a quarterly by
quarterly, half a year or a year basis, right?
We can make projections for the next 3-4, five years, but the
confidence in these projections is less and less and less the
more time grows. And I think a lot more companies
need to start doing the same. When you say we will then design
for the next level the order of magnitude for the next level,
how far in the horizon do you look at the with the GitHub
Actions example that you had, you said, OK, five years, but
it's that like an arbitrary thing or how it?
Depends on the curve, right? It depends on the curve like
GitHub Actions has been growing massively over since we, we, we
brought it up at the beginning and we're introducing more and
more features that are increasing the adoption.
So the curve, we can see how it looks like, but it might change
at any point in time, right? It can be disrupted, goes to 0,
or it can continue growing exponentially.
So often the hockey stick that a lot of tech start-ups talk about
like exponential growth, it can appear and you can start seeing
it. And when you start seeing it,
you know, if you have a solution like actions, companies that
migrate to it are not going to flip flop in very short periods
of time. So you can have some confidence
that, OK, this trend is going to last for a year or two or three
or four, right? Nothing lasts forever,
obviously, but you can start thinking in that direction and
then based on that you can make investment decisions.
Got you. I've had many discussions also
in the past when we're talking about our software architecture
in something that can be very specific and also purposeful
built versus setting it up in a generic way potentially towards
the future because we have ambitions and hopes that we have
X amount of solutions that are similar.
So we make a generic right now. What is your perspective on
making something specific versus generic in the first place?
I like solving specific problems.
I don't like solving generic problems because how can you
make trade-offs then, right? What?
In software, there's always
trade-offs. So how do you choose these
trade-offs? You can build something super
generic, sure. That basically is what building
a framework that can be a toolbox and give you all sorts
of solutions to all sorts of hypothetical problems that you
might or might not have. That's not the North Star I
follow. When I design software at GitHub
with my team, we are very specific about not over
engineering things and we solve for the problems we have today.
So for example, we'd never introduce caching until we
really need it and we never over optimize until we start seeing
problems sometimes, Sometimes we we know that we're going to hit
this problem in a few months and we're not going to get the
investment. So we solved it right now
because it makes sense. But we don't over engineer
solutions. We don't start jumping into
things, for example, we don't necessarily jump to no SQL at
the beginning. We just start regularly
relational databases. You know, we build regular
tables if we need, if we need actually relational content.
Sometimes relational databases can always get so far.
So we start needing to rethink our solutions here and we need
to start going towards non relational.
Even if we need the relational consistency or the consistency
of relational databases, we need to start thinking about how can
we move that to the application layer and start using other
solutions that persist data that can help us scale.
There are some efforts, for example where we start using
Cosmos or Cosmos DB and things like that, but we never start
exploring these until we hit really massive problems with our
database clusters like our primaries cannot keep up.
The IO is pretty high, write operations are substantially
big. We cannot scale these anymore.
Sharding is not an option. Application layer sharding is
not an option as well. When it becomes not an option,
sometimes it is. So we start thinking about how
can we redesign this and how can we migrate the data.
And migrations, as much as they are painful, they are part of
the job. Yeah, interesting.
And we do go through them. I'm wondering when we're talking
about system design and then specifically practical
experience, what your perspective is on someone to
really get a better understanding of the theory but
also be able to put it in practice.
You you already mentioned that for example, like GitHub, you
need to have a certain level of experience under your belt and
then you can work at GitHub where we have orders of
magnitude with regards to scale. This is the vicious cycle,
right? So do you, how do you, how do
you build experience that is required by this type of
companies, but there's no other type of company that's going to
give you that experience. Unfortunately, there's that's
that's the reason why there is a proliferation of a lot of system
design courses and things online.
And quite honestly, passing that into passing the interview
threshold for companies like GitHub that is achievable by
learning theoretically these concepts OK and not necessarily
having hands on experience with them.
The bar feels high, but right now it's been sort of exploited
to the point where a lot of the tricks and tips and what not
already available very much online.
Like hello interview guys, for example, they have this YouTube
channel. They have pretty much covered a
lot of the topics that you might get asked in a system design
interview and they go deep. Like they explain what Kafka is,
what event streaming is. They explain right as the
internals how it works, when do you use it, when not to use it.
They explain the differences between relational and non
relational databases. They explain caching, how to
implement caching in specific scenarios, and they give
practical hands on problems where you can actually learn how
to effectively utilise all of these different components to
come up with a relatively scalable solution.
And they give you tips also on how to approach the discussion
of scale within a system design interview.
And I think they do a pretty good job at it.
And I think that's enough because a lot of people who go
to big tech will go to Google MO, they don't necessarily have
that hands on experience beforehand, even though the
interview is looking for that hands on experience.
But you know, with these, because we hacked the people
hacked the game. So the signal is given to the
interviewer. And then people when they have
these, this foundation, they can actually do a relatively OK job
on when they join. And they're going to obviously
aren't going to be alone, right? So you're never going to join a
team and start building Greenfield stuff from day zero.
It rarely happens. You're going to join teams that
already established. So there's going to be people
with experience more senior than you were going to give you
guidance and review your solutions, your design.
And so on and so forth. And that's why people make it
right. But if you are interviewed for
experience and you join a Greenfield project, you, you're
going to own your mistakes. And people do make mistakes
often. Yeah.
When it comes to these solutions, not everything big
tech ships is a great feature or operates well in the world.
And often we discover that oh, this was a mistake and it was
actually very costly and some companies fail because of these.
Gotcha, right? Any failures or mistakes that
are top of mind for you that you can share for other people to
learn? Tricky, no?
I want to keep my job so. That's fair, that's fair.
For me, looking at what we do nowadays, I feel with AI tools,
I'm way more productive in actually writing code.
And I feel like when you amplify the amount of code that you
write, you might also write yourself into a, a bottleneck
basically. So I feel like software
architecture and specifically system design from the start,
having a better understanding, especially I feel like like I do
now, I need to grow there. And I feel like it's going to be
a more important skill to indeed figure out what are we designing
for, what makes sense, what would be the next order of
magnitude after and actually make sure to convince people and
have a certain level of buy in. We're always working in teams.
Also from a business standpoint, how would I, for example, say
we're going to design for this until we reach scale and then
we're going to need to revamp. And this is what that's going to
entail for people that don't understand how to build
software. Understand the business
constraints. So you're never going to be able
to convince the business folks about of something technically.
They're just. It's too abstract.
Even if it feels too intuitive for us, for them it's.
Magic. Yeah.
And even if they try to understand how things work,
they're never going to comprehend the magnitude of
problems and they're never going to have have this intuition
about how software is and how not hard, you know how how
evolvable it is. So it's our job to get closer to
their side. Some people say, no, we want the
business to understand they're not gonna it's trickier.
It's easier for you to acquire or assimilate some of the not
everything, some of the business constraints and discuss them
from that perspective. And that's why a lot of
engineers who grow are very good at understanding the business
constraints and then discussing the technical solutions within
those constraints. So for example, I used to work
in the container terminal industry for a while.
We built a huge container terminal.
We furnished everything from the fibre optics layer all the way
to the three tier data centre with all of the enterprise
applications running, running on it and the operating system for
the container terminal. And it's one of the most massive
operations you can ever imagine, one of the most hazardous as
well. So the folks there, they're as
far away from tech as possible. You're not never going to find
the terminal operator who's going to understand what it
means that your database is running hot and you cannot
support their queries. All they care about is I have a
ship here who's docked. There's like 10,000 containers
on it. I need to empty them into my
yard. I have a very fixed window of
time and if I don't do it in that window of time, this is
going to cost me money. We were operating at 12%
capacity. That terminal was generating
about $50,000 per hour, so 12%. So you can imagine if we get
delayed, every minute of delay is actually money that's
disappearing. So this is how you need to
discuss problems with and and investments and solutions.
You need to understand what the impact is, what the impact is on
the operators. And sometimes it's life
threatening impact, right, If your software fails when ASTS,
when the crane is offloading a container, this can be a very
life threatening situation. You cannot really have your
software malfunction at these points.
So snappiness of the software, sub millisecond latencies, all
of these things matter, right? So you need to approach the
discussions with the stakeholders by understanding
them first. So what I've done for example,
in that industry is I've literally sat on a truck and
went there and I sat with every single individual who works in
that terminal to really understand what they go through
on a daily basis. So a truck driver who's driving
a truck for a six 7-8 hours and it was in Saudi Arabia in
scorching heat, it's not going to understand why the button
that they clicked is just is so slow.
Doesn't matter. Doesn't he doesn't care, right?
He just wants it to just work. So you got to understand all of
these perspectives and then when you do, you can start
approaching discussions from that vantage point.
And this applies to banks, this applies to hospitals, to any
other industry. So understand the industry,
understand the business constraints, and then any
technical solution you want to discuss an investment, you can
approach it from that vantage point and you're going to have
much more success in these conversations.
Yeah, interesting. I feel like the the opposite
then also holds true. So if you do understand the
business context and you don't have a case for why you need to
work on tech, that why you need to scale, why you need to fix
some of the issues, then there might not be an actual case.
And right now is not the right time to do so.
I do feel like it's, it is a lot of prep work that you have to
do. It's way easier to talk about
the technical things, right? Because that's what you see,
that's what you perceive latency, numbers, storage,
scalability, that's what you look at.
And then reframing it and saying, OK, what is the the
impact of the business? What is the cost per feature?
And what if we have slower delivery rate compared to a
faster and a new future? Talking that language is very
different from what you're used to.
Absolutely. And the trickiest part in my
opinion, is when the stakeholders are not transparent
about their numbers. So they're never going to tell
you like, oh, we're making this $10 million investment today in
this piece of software. Because the investor, one of the
biggest investors we have, is willing to put that money in
right now, but nobody has the guts to go tell him that, no, we
need another 10 million in 10 years, right?
And nobody's going to come in and tell you that.
But you need to also think about it, try to extract that
information, try to understand who, who's funding this, What is
funding this? What is critical is, is, is the
business in jeopardy? Is it right now under a lot of
scrutiny? Is the bottom line not working?
Is the margin of profit not that high?
What, what is driving this particular investment right now?
Is it more strategic? So I worked in building banks,
for example, in the in the Gulf, I worked as a consultant as
well, right? So there were banks that were
wanted to make a one shot investment into what they
thought it was their future. And they had a lump sum of money
that they wanted to put down once and build software that
will last them for 10 years. They're not going to.
They, nobody has the guts to come in and tell somebody, an
executive who has no time whatsoever that hey, well, this
investment is going to last you maybe like 2-3, four years, but
you need to put in also another million every year to maintain
it and maybe even another 10 million in 5-6 years because you
need to keep up with their new trends.
Nobody has the guts to come in and say that at the beginning
because then the decision maker might say, Oh well this is not
worth it for me anymore. We might just scratch it
completely. Precisely.
What I've also seen is that some organisations operate with a
number of engineers and then from an outside standpoint I
have no clue what those engineers are doing.
Like if I look at a specific e-commerce website, they might
have 2000 engineers where kind of a similar one with a similar
level of skill might have a few 100.
And then I feel like the few 100 engineers, they need to have a
bigger landscape of ownership. Understanding what they do in
the context on day-to-day on a business side is easier for them
than being part of a group of 2000 engineers that kind of does
the same scale. Feel like if you're a cog, it's
very hard to argue what kind of optimizing your little cog is
going to do for the greater good.
One thing I, I one thing a lot of people hate, but I think is
very functional and a lot of the companies that operate based on
the San Francisco Silicon Valley model is the ruthlessness in
terms of optimizing how engineers work and layoffs.
So in Europe, layoffs are not very famous or very well
received and I don't think they are very well received in the US
as well. But they are a if they are an
effective mechanism, even though it's dehumanizing, but it is an
effective mechanism to recalibrate how people or where
people put their attention and what they are focusing on.
So in a lot of situations when layoffs happen, there's a huge
product features that are cut entirely like we don't need this
anymore. And it's it's more expensive to
repurpose the skills of those same engineers in other areas.
So let's just cut it off immediately.
Hire new engineers that are more experienced in that particular
domain, get them to focus on this right away.
So this rapid flexibility and moving rotating resources or
people basically allows these companies to be much more nimble
and effectively or more effectively allocate engineers
to particular problems that matter today.
The second thing I also think is a good that separates a lot of
the big tech companies from other companies is your work or
your reward bonuses, growth, whatever is associated with your
business impact, not your software engineering impact.
So you can come in and say, oh, I built the most, the biggest
Kubernetes cluster you know, known to the planet.
It's beautiful. It's amazing.
You're not going to get any of that or any recognition for it
unless you justify why that was important.
What was the actual landed impact on the business?
So what did it contribute to in terms of our revenue?
Did it help us grow? Did it help us attract more
customers? Did it solve a very hard problem
for us? Did it affect developer
experience, our people more? And you need to have numbers,
not just fluffy words. So I think that's also another
very effective way to reallocate attention of software engineers.
And it's very easy for us, also software engineers, to drift
into the comfort zones. So I'm building something.
I'm comfortable in it. Yeah.
And nobody's questioning me on it.
Nobody understands what I'm doing.
I just keep doing the same thing everyday.
Yeah, I feel like I've also shared this previously on the
podcast, some engineers that I've met and they love the
technical side of things. And when you let them go wild
when there's no business context or direction, they will build
you a rocket ship and they will really enjoy doing it.
There is a crafts, the craftsmanship aspect of software
engineering. It's still there, you know,
there there is an aesthetic to it.
There is like it's fun, it's enjoyable.
But I always in my even my social media stuff like I always
strive to communicate that when you work professionally as a
software engineer, this is not practising a hobby.
And I think a lot of this craftsmanship thinking it comes
from the fact that a lot of people went into software
engineering because programming was a hobby for them, myself
included. I started writing code when I
was 12. I thought that this is amazing
if I can make it my job perfect. But the job is very different
than practicing code for the sake of writing code and the
aesthetics and the fun of it. Sometimes you are lucky to be
able to bring that into your work, but most often you're not
really solving for the aesthetic, you're solving a
business problem. And you need to keep that in
mind always and often. It's not about the beauty of
yourself. You can, if you can build a
beautiful solution and solve the business problem effectively,
fantastic. That's the cherry on top.
But the priority is for the business problem.
And sometimes solving business problems means building software
that is good enough. Good enough for today, right?
Doesn't have to be good enough for tomorrow.
Maybe it's horrible for tomorrow, but it's just keeps on
functioning. And this investment we did and
the good enough, it's going to give us a a certain runway and
we should be OK with that. I think the we should be OK with
that. That's what I've seen where it's
challenging for people where they say, OK, I'm a little bit
of a perfectionist. I value quality in what I do on
a day-to-day. My job is basically part of who
I am. And if I don't deliver the
highest quality that I can in this aspect, like I'm doing
myself with this justice. And it's very like, if I lay it
out like that, it might sound selfish, but those people have
no clue that what they're saying is selfish.
They genuinely think it's best for the organization.
There's just some misinformation and misunderstanding.
Because there's a dogmatic aspect of software engineering
and it was perpetuated in the previous era into two thousands.
2000 I when I ventured and working professionally at
software engineering 2007 8-9, there was a lot of dogma.
Like there's a lot of this is how you do things and this is
how you should do things. And if you do things any other
way, you're wrong. Yeah, and part of that is
elegance of code, code cleanliness, code quality.
We're all raised on this these these concepts, right?
And they're important. And everybody always complained
about bad code and horrible code because it was difficult to
maintain. Nobody enjoys that.
You're going to make it very difficult for the future
engineers to solve these problems.
But sometimes, you know, you don't have future engineers to
solve these problems. Sometimes nobody's going to
touch this code for decades to come.
It's OK if it's just, it just works.
And then we ended up on the complete other end of the
spectrum where we try to clean up everything that we started
adopting all sorts of design overcomplicated abstractions
that are are just horrible to read, horrible to reason about.
It takes you decades to become proficient in just reading that
type of code. And then when you write it, it
gets even more complicated and it's very difficult to maintain.
And big Tech, and I always tell my colleagues and the juniors
that report to me, simple is complicated enough, especially
at scale. So just find it in the dumbest,
most simple way possible. This is why I also love Go, for
example, as a language, right? It's amazing.
It's dumb. You don't have to think too
much. It's just all in your face.
Whenever you want to understand how something is built, just
trace it around. It's there, it's there.
Fantastic. I love that.
And when the situations where we're building at scale, I don't
want to reason about abstractions, I don't want to
think about edge cases, I don't want to think about complicated
memory locations and complicated memory structures.
You know, I just want things to be simple.
Memory leaks happen often and they are very catastrophic at
scale. I don't want to think about
these things. I don't want to think about how
my garbage collector is going to, you know, shut down the
whole world and start causing multi second latency on my API
requests, which at scale again become a gigantic problem for
me. I just want things to be simple,
easy to reason about and that's it.
Yeah, Yeah. Interesting.
I also feel like we've, we are now in a trend where there is
this conversation happening with people that are, first of all,
vibe coding, loving it, being effective in what they do,
solving a problem through software and they don't care how
it looks. And then you have the older
part, let's say existing software engineers that look at
this and say both from a craft perspective and from an
execution perspective, how does this work, right?
This is kind of really coming into my territory.
I feel uncomfortable. Do I jump on this?
Do I actually like it? Or is this going to take away
what I love dearly, what I used to do day-to-day?
I think in essence, things are evolving for me.
That's a fact. The question is, where is it
going? And I'm curious to see your
perspective on that. It's funny it happens in in
every community to be honest. So I'm starting my and
motorcycle riding classes and in in motorcycle riding, a lot of
the more advanced motorcycle motorcyclists and they love
manual, the clutch. They love, you know, changing
gears. They love the, the controlling
the, the, the motorcycle in every single aspect, especially
when they attain that level of proficiency.
So a lot of them are now resisting a lot of the
motorcycle companies introducing E clutches or clutches that just
operate automatically. They hate that.
I don't want this like I want to control every single aspect of
my bike, right? And I feel this is pretty much
the same thing that what's happening in in software.
A lot of the folks who have attained a high level of
proficiency and they enjoy the craftsmanship aspect of software
development. They love the coding part.
They love reasoning about code structure, syntax.
They get a kick out of coming up with a smarter solution where
they utilize A syntactical feature in a programming
language to do something that was not maybe designed for or do
something smart with it. So if you take that away from
them, you're basically introducing, you know, E
clutches to software development and they're not going to react
positively to it, even though it clutches help like when for
example, in when you're stopping at A at the lights signal, you
don't have to worry about stalling and you don't have to
worry about these things. It just makes your life much
easier. It's fun to ride the bicycle,
the motorcycle, and it's just easier to control it and I
think. It's.
Yeah, more accessible. So bike coding is pretty much
the same thing. I hate the term bike coding
because right now at GitHub, for example, I literally, and this
was part of my presentation at Universe, 90% of my code is
written by agents. So I use the VS Code agent mode
all the time and I've been shipping features to production
with it all the time. But that's not software
engineering is not just writing code right.
There's a lot of other stuff that come to it.
And I now shift my focus towards the operational problems.
I shift my focus towards making sure that we never go down.
I shift my focus towards making sure that I don't introduce bugs
that are unnecessary and 'cause people to have because GitHub
has become part of the infrastructure of the Internet
now. So any problem that I cause has
great triple effects. So that's where I put my focus
on right now. Coding fine, let the agent write
it and I will instruct it to write it in a certain way.
And if there's a syntactical approach that I prefer better, I
still push it to to do that. And if something looks funky to
me, and I also prompted to change the direction, right?
But at the same time, one of the problems I was solving was
recently was a caching problem, right?
And I had different ways to implement the cache key and each
way forced a certain data access pattern because of how the key
is structured. And it had a different impact on
Redis based on how the key is structured.
So what I've done with the Escode agent is I prompted it to
write different benchmarks. Yeah.
And I told, hey, this pattern, this pattern and this pattern,
go write 3 distinct benchmarks, run them, give me the results,
analyse them and give me the output.
That's work that could have been done in what, two or three days?
Maybe that was done in 20 minutes.
Why not? Fantastic.
It's amazing. Yeah.
So. You've been programming, you
said hands on coding when you were 12.
Have you always had this mindset to kind of let go of more of the
craft things and look at what it is from an execution standpoint?
Because I hear you from now and it feels like you're very much
on board and making things more effective, understanding the
business context. It's not just the craft.
Because no, I was never like that.
So when I was younger, I was all about the craft.
And in I, my early mentors, I fought with them a lot on this,
right? And I was lucky to have mentors,
right? I was lucky to have great
mentors at the beginning of my career, but they never were able
to explain to me why their perspective of the business
matters. For me, business was too
abstract, like I don't understand the numbers, I don't
understand accounting. I was not accountable for the
business outcomes. I was just asked to solve the
problem. And I thought I had infinite
business resources. But when I realized that my
mentor sometimes. Took an extra mortgage on their
house just to pay the payroll for their employees.
That makes a different problem, right?
That gives perspective to the solutions that you're going to
come up with. These are real people.
Company owners are not just all millionaires and all, you know,
people who are not investing. Some people are investing their
own money in building businesses, right?
So that matters. So when you understand this and
when you become your business owner yourself, you start
thinking about these things more.
And now I approach these problems from that vantage point
as well. That doesn't mean that again,
people might. I always get this argument when
I post about this stuff online. People think, oh, you write
dirty code. No, I don't.
But there's a certain level of the 20% that is required to
reach that highest level of aesthetics.
It's not worth it, no. So I drop that.
Gotcha, As a final thought, there's many people listening
that learn about software engineering and software design
in the first place that either are early in career, mid career,
are really just looking to level up within their career.
What is your advice for people that really want to become great
software engineers? What do they need to do?
Increase your breadth. There's a lot of focus on the
depth early on in your career, which is important.
And there's an advice that perpetuates, which says focus,
true focus. But at the same time, don't
dismiss everything else that's happening.
Increase your breadth. Personally, I optimized one
thing that really got me far, which is having the ability to
learn effectively in very short periods of time.
And that's why whenever I tackle a problem, I don't have a
problem digging really deep into anything you I encounter from
quantum mechanics all the way to any technical problem that I
face on a day-to-day basis. I'm a, you know, I'm a
university drop out. I never finished my computer
science education, but that did not deter me from picking up
that education myself and really digging deep into all sorts of
theoretical topics, but also practical areas and topics.
And I'm a perpetual learner. So in my opinion, the next phase
where we're heading, it's going to require people to be able to
learn really fast, get out of their comfort zone really fast,
dig deep really fast and attain high levels of functional
proficiency in any new area. And then attaining mastery is
not always required in everything.
So just pick a few topics where you attain mastery in, but also
increase your breadth. Got you.
I think this is going to be very important for the next phase.
I love that. How do you get comfortable with
even saying? I've gotten really good at
picking up new things and learning from quantum mechanics
to anything technical or job related, whatever.
I have a lot of interests and I see how fast I can get to a
working proficiency with these interests.
I play music. I'm learning to ride a
motorcycle soon, right? So I never let if I'm curious
about something, I go and pursue it and I explore it.
And I never let how complicated learning it is determine from
pursuing it. And I've become very comfortable
with being uncomfortable learning something new.
And actually that gives me a joy and kick to the point where it
became sort of an addiction. But you know, I keep it under
control sometimes. But I do enjoy the process of
and exploring something entirely different and new.
And in my opinion, that has given me so much breadth and the
ability to talk about all sorts of things.
But what I love the most about it is how I can cross different
ideas together from different disciplines and have a much more
diversified view of the world. And that gives me the ability to
be able to empathize also with all sorts of people and
different people, right, And understand their vantage point.
So when someone's arguing with me, I take a pause and I try to
reason about where they're coming from, what's their
interest, how they got to that idea.
But I draw from my own experiences also and learning
about all sorts of stuff, including business and finance
and investments and all of sorts of all of these things, right?
So I think that's richness and yeah, I would definitely love
for people to be able to have that.
Me too. I think curiosity is a beautiful
thing. When people say I really don't
have a, a drive to learn something or to explore
something, I think it's a shame. And I honestly don't know how to
solve that for people to make sure they have that kind of love
for something or curiosity for something.
Because I feel like that fuels Dr. and that makes you move.
Absolutely. I, I don't know how to, I don't
know if we really should make people do anything.
I would love for people to have it.
If they don't, you can live a pretty, you know, interesting
life. Still, some people have other
commitments as well. Families, kids, less time,
complicated relationships, and life happens.
It's understandable. And that doesn't mean that I'm
also 100% always on the, you know, on the burner, learning
every single minute of the day. No, I have my life as well.
I have the complications of my life too, so balance is
important, but I feel also curiosity is something that you
need to develop. It's like a it's like a muscle
you need to grow as well. And approaching the world from
the curious vantage point is way more interesting and more
effective than being dogmatic or stubborn or hard headed about
certain things. You can have strong opinions,
but not before you actually did your homework.
Yeah, absolutely. Thanks so much for coming on and
sharing. Man, this was a blast.
Absolutely my pleasure making going for hours.
Me too. We're going to round it off
here. If you're still with us, leave a
comment in the comment section. Let us know what you thought of
this episode and we'll see you on the next one.