Hi everyone, My name is Patrick Akil and joining me today is
Mauricio Aniche, previously professor and researcher over at
Delft University of Technology and now Tech lead for testing
enablement over at ATM, an organization with over 1000
engineers strong, quite a large group actually.
So we talk about the testing strategy in an organization like
that, how you enable testing with developer tooling and
platform engineering, and how AI has been influencing testing and
will influence testing with LLMS now introduced in the picture.
Lots of fun this episode. But before we go there, I want
to do a big, big thank you to everyone, each and every single
one of you, because we are 30,000 strong now, Big 30.
And this is also the year that I'll be turning 30.
So it's a big gift to me. 30s and 30, everything coming
together, a lot of fun. Enjoy the episode.
As as a developer in industry, you are more constrained in the
set of problems you can work on your day-to-day because at the
end of the day, you have to make an impact on, on the business of
the company, right? So they have a smaller set of
problems that you can tackle. But but when you do, the impact
is right away, right? You ship software, you're
changing people's lives on a daily basis.
As an academic, you have the opposite.
You have the full freedom so they can pick any problem in the
plan on the planet to work on right it and and that is
amazing. But at the same time, for you to
truly see your research having an impact on the real world that
that takes too long, right? I would spend like a year
working on a project and then writing a paper and then going
through all, you know, the peer reviewed publication.
That is quite tough. And then I had a paper and and I
was like, now what? How does this get to the real
world, right? And so it, it's a 2 sided knife
in a way, right? That you have all the freedom in
the world, but it's so hard for you to, to identify the right
problem to work on. And then and then having this,
your idea caught up by industry and then people seeing it
applied. And for me it was always about
seeing my stuff being applied. So I couldn't just be a
theoretical guy that, you know, in 50 years from now my work
will be spoken about. That's not really for me and if
you like, but but academia also allows you to really think
deeply about a problem, right? As an engineer, what I see is
you have a lot of very complex problems that you could come up
with amazing, beautiful tools to solve the problem, but you
usually do the smallest that you can to solve the problem so that
the business benefits from it. And then you move to sort of the
next thing that the business wants.
And that's just life. But in, in academia, you can
look at a problem and say, I'm going to go as deep as humans
can, can call on this problem, right?
And then and, and, and nothing is for there's no business
pressure for you to Hey, just move to another problem.
So I know people like the people that I admire the most in
academia, those, those people are working for 1520 years on
the same problem, right? And that's, that's truly
amazing. Not really for me, but that's,
that's really amazing, right? So, and I think that's why both
complement each other because you need people solving problems
right now. So society benefits right now of
small improvements, but you also need people doing moon shots,
right? That maybe 10 years, 15 years
from now, they will pay off and then we can use their
technology. It's it's a nice balance I feel.
Yeah, I I love that perspective and I didn't think about it from
that side. Like the freedom to explore a
problem and to kind of, I mean, you can time box and you can
self manage, but to actually have the freedom to do and go in
depth as as deep as you want. I know a lot of engineers that
go really, really deep in some of the problems and indeed they
do it more so on their own time from an educational standpoint,
from a curiosity standpoint. Some people can't just drop it
at a surface level. They truly need to understand
before they actually execute, and they usually do that in
their own time. So having the freedom to do that
then in academia and work on problems for let's say our
future selves or our future planet sounds amazing.
I do wonder though, like this length that you said, OK, from
idea to execution to paper and then people actually adopting
it. It sounds like quite a long
cycle. And then controversy on the
business side. Sometimes we're too short.
Everything is deadline, fast-paced, too many problems to
solve. Solve it in the quickest way and
move on to the next. It's like we need to find a
balance. I feel like in in both sides.
Yeah, for sure. Yeah.
But then this fast-paced industry world, that's where
you're in now. What kind of problems are you
solving that you can share with us?
So I'm currently at Agin, right? So the famous financial
technology, it started as a payments company for those that
don't know the company. And now Agin offers plethora of
financial services to large companies like Meta, Uber,
Spotify, Netflix and the like. And I'm part of the platform
engineering. So we have a big group that does
platform right? And the idea is let's just give
our engineers a platform so that they can be more productive in
delivering business value financial products to the
market. Right.
Is it self developed the platform?
Yes, OK. Yes, yeah, we rely on open
source. Of course, we have some
foundation on open source systems, but we have lots of
layers that were done by us and my team in particular.
We do everything related to testing.
So we we build and maintain the tools that our agent engineers
use to write tests for your systems.
So think of from, you know, very small, you know, maybe helpers
or layers on top of J units so people can write simple unit
tests up to the CICD running complex integration tests,
static analysis. So we are the ones that sort of
control the static analysis tools that we have in house.
And then sometimes we write new static analysis rules ourselves
for specific problems that engineers have.
So sort of everything that is related to quality, my team sort
of owns it and which back to the academic side matches my profile
perfectly because it can work on developer tools.
We do have some freedom to try to go deeper on some of the
problems. Yeah, and that's nice.
So that's the perfect combination.
That's awesome. Like for the context and for my
kind of idea of the scale of things.
I recently had an app a person on that is specialises in
platform engineering and internal developer tools and we
had a conversation when platform engineering makes sense.
You mentioned that when an organization reaches a tipping
point with regards to engineering capacity, that's
when a team that really kind of owns this platform as a product
makes a lot of sense. What is like the scale of the
organization with regards to engineering capacity?
Yeah, good question. So and I indeed agree.
And at some point when you're too big, yeah, you start to
notice a lot of repetition, a lot of duplicated work.
Things that could be solved once are now solved 10 times across
the organization so that you can be more productive there.
Agin is at this moment a 4000 people company, yeah, out of
which maybe 1000 or maybe a little bit more than that are
engineers. Well, and engineers of all
types, right. So we have back Enders, we have
mobile developers, we have embedded developers because we
also, you know, we have some C code in there.
We have data people front Enders.
It's it's we have sort of everything.
Yeah, my team focuses more on the back end part of agent, so
where all the financial logic happens.
And so it's a a Java super large code base.
Yeah, and it's a good question. How many out of these 1000
engineers, how many of them are Java engineers?
I don't even should I guess my my guess would be like maybe a
500 something like this, but I might be wrong.
But it's it's on the scale of hundreds.
For sure. Yeah.
I, I talked to I think two distinct people that I still
remember, 1 was more involved in eBay and Mark Plant's kind of
that ecosystem from Alavinta. And then one was more involved
more so on the booking side. And we both talked about testing
and quality and when your engineering capacity reaches
this scale, right, thousand engineers, maybe 5, like 50% of
that specific on back end with a specific language code gets
produced like it's nothing basically.
But the funny part is all of that code has a cost of
ownership. So code that is maybe not
effective and especially when we're talking about a test suite
that needs to be fast with regards to feedback and
iterative can be quite compromising when that goes the
wrong way. I feel like how are you trying
to manage or even educate the people that create the code?
Yeah. So we try to to approach testing
from different angles. 1 is and the one that I mentioned is
tooling. But we also try to do not
education per SE. We call it strategy.
We try to to guide teams on, on how they can write better and
more effective tests, right. So maybe 3 1/2, four years ago,
we actually formalized this in a document.
We call this the testing strategy.
And the idea was, you know, this is how we see the testing
pyramid or so. And this is when we believe you
should write a Unity test and you should mock stuff.
This is when we believe you should go a bit bigger, maybe
write a component test, and then you test your SQL together with
your code, so on and so forth. So we try to be more
prescriptive. Yeah, at the end, trying to help
the engineer to reduce the friction of the engineer, right.
How do I test this? Should I go for unit test or a
component test or integration test, whatever.
So we, we do have this part where we try to say to people,
this is how they should write your tests.
And then those are the tools we give to you so you can write
your tests. And if you use these tools,
well, you benefit from, you know, it's sort of stable,
right? It works.
It probably it solves your problem faster because you can
reuse a lot of stuff. Yeah, that's, that's how we've
been. We've been trying to to combine.
And it's, it's funny because sometimes we focus more on
tooling and then we sort of notice, hey, now we need to, to
push people towards, you know, a better strategy on, on how to
write tests. And then when we focus more on
this part then hey, but now we're missing tooling so that
you sort of walk hand in hand in the way we we are doing it.
Yeah, yeah. I could imagine something like
tooling and especially for something that's like very
day-to-day nowadays in an engineer's tool belt with
regards to testing can be quite crucial, right?
Having it the right way is going to be very effective, is going
to make people more productive. Having it even slightly not
great with just uphold people from writing tests or from doing
it right. I like Go as a programming
language. That's like how I grew up in the
industry and testing is very much like ingrained in the
culture of how you write software.
Also how you structure Go code. I mean, there's not many ways to
do the same thing, which is also kind of their vision to keep
things simple. But testing is a first class
citizen. And I I also do trainings with
regards to Go. I'm going to give one tomorrow
and the day after. And that's always been a lot of
fun to talk about testing, designing for testability in the
1st place as well and then executing on that.
And The funny thing is if you design your code the right way,
that testing is easy in the 1st place.
If you design it in a way that it's not easily testable, then
it's very hard. So software design has always
been like a proper principle that you need to adhere to.
I'm wondering what you think of kind of designing for
testability, because I see a lot of people kind of a small niche
advocating for test first development and also another
kind of part of that smaller subset that is very dogmatic
about it. Write your test first, then do
everything basically. And I am a bit more flexible
with regards to that. I'm curious to hear your
perspective. Yes, and I'm also quite more
flexible when it comes to test driven development or so, but
I'm a big fan of. You know you should design your
code in a way that facilitates testing afterwards, right?
Because testing is very important and maybe you have to
change one or two design decisions here so that your code
is more easily testable. Just just do it right.
Make your code easily easier to be tested.
Now some people advocate that you have to do test driven
development. So you have you have to write a
test first so then you can see precisely what your code should
look like. I don't truly believe that
that's the only way. Maybe it works for you.
Go for it. It worked for me for, for many
years, to be honest, maybe the 1st 5 or 6 years of my career, I
was quite influenced by test driven development and I was
doing this a lot. But I don't do test driven
development today, right? Maybe because all those patterns
are now maybe more natural. I can just, I can just see them.
But, but, but I feel like it doesn't matter if, if you write
the tests before or if you write after, as long as you're writing
tests timely, right? A little bit of production code,
a little bit of testing or vice versa.
And your code and and and and and and.
It's easily testable. You don't have to do weird
tricks to test your codes. I think you're fine.
Yeah. Now I think one of the the
biggest challenges in such a big, large, complex and old
organization is that it's impossible to only have
beautiful code, right? And in organisations where code
dates back to 20 or 15 years, you know, back then, we knew way
less. The tooling was really bad.
Let's be well, not really bad. It was the best that we could
have at that moment, right? Yeah, So it all those large
organisations, they suffer from code that is, that is not truly
easily testable. You know, it's not easy.
If you want to mock something, you're going to suffer.
If you want to test a piece of code disconnected to the rest,
it's also hard. So on and so forth.
And if you part of our exercise there at the beginning was like,
sure, if you're writing new code, try your best, you know,
you know, the best practices you understand about design for
testability, dependence inversion, blah, blah, blah, all
those cool stuff in Java. Well, Java or other languages,
whatever. But you know, you're going to
have to handle this legacy and you also have to test it.
Yeah, right. So how can we facilitate your
life? So, so you can write a test for
it. So we, we sort of created a
guidelines on, on how you can, you can test this, this legacy
stuff. It, it, it is hard for me to
give details because it's really focused on the way we coded for
20 years or so, right. But I think that the gist is if
you're in such a large organization, you have two
fronts. 1 is make sure new codes is born with the practices we
know today and node codes, you find ways for engineers to still
be able to test it in a productive way.
Right. When people say people don't
want to test the legacy, it's not that they don't want to.
Even if they want to, they're not going to be able to write.
And maybe they they are, they're going to try and they're going
to spend one or two or three days to write one or two or
three tests. Of course, at some point the
engineer will say, I'm going to stop this.
I have something more useful to do, more impactful to do, right.
But if you reduce the cost a little bit, things get a little
bit better. Yeah.
So I think and that's why a a platform team can can benefit
because can you, you can then sort of see what are the common
patterns or bad patterns that happen in the code base and what
can you do to reduce their impact on untestability?
Yeah. Yeah, I, I was thinking
something that you mentioned kind of triggered me to think
about my own journey with testing.
And I think it goes a lot with regards to maturity as well,
right? If I, I also see people now
having no testing background and starting to test, they're like,
OK, I should test every single function that I create.
And then we're like, OK, but that's not really the point,
right? You test from an outcome
perspective. You have a certain input and a
certain outcome. And it doesn't have to be every
function in between a chain, but it does have to be a chain
because that's the most valuable, right?
And then it's a matter of if you have so many engineers writing
tests and their maturity level varies, you're going to have
pretty strong tests and you're going to have weak tests.
And I wonder how you clean up or at least have a test suite that
is of high quality. Because I feel like if you go
wide and you're like, OK, every test is going to be valuable,
Then at some point your test suite is just going to run hours
on end and it's not going to be effective for people to iterate
quickly and to cycle through their changes with regards to
feedback anymore. How do you make sure that your
test suite is actually of high quality as well?
I think you may get it in in the sense of I feel like at an
engineer level, one of the hardest decisions is at what
level should I test what I'm coding, right?
Should I go full unit test and, and exercise this function alone
from the rest? Or should I go a bit a bit
bigger and then exercise maybe the entire flow right?
And I have a, a funny story at Ajin about this.
A couple years ago, we wanted to to organise the work of teams
when it comes to testing their legacies.
So we said to them, you can create test plans and those test
plans will contain stuff tests you want to write.
So you sort of feel more confident when when shipping the
legacy to production, right? And then, and the idea was
because you're going to formalize this, then product
managers can understand and they can prioritize this and they
understand the importance. So you make the work visible so
people can actually work on it, right?
Instead of just saying, hey, engineer, right, test for your
legacy, right? It's not that simple, right?
And we gave this test to the teams and teams assigned
engineers to, to sort of create those test plans.
And as a retrospective, what we noticed was a bunch of those,
those test plans, they, they contained like, let's just unit
test this function or, you know, let's test this class or so.
And at the end they put a lot of effort in, for example, unit
testing a lot of stuff. But then at the end they were
like, do do you trust now on your test suite?
If, if it's green, do you feel comfortable in shipping?
And some things are like, no, not yet.
Like why is that right? And that was just because they
sort of picked the wrong level to test what they needed to
test. And it's been maybe a bit less
than a year. We released this new tool
because then what we noticed was sure this decision, should I
unit test or should I test larger?
It is a tricky one, but you should never forget the
business, right? And I think you mentioned this,
right, focusing on the business when writing the test, you said
something like this. We created this tool Italian,
where product managers can can in an easy way say those are,
you know, the main maybe critical flows of our app.
And then engineers can see this and this is just written in
English, right? Very high level.
And the engineers can look at it and then write an automated test
for that. And we're still working towards
more adoption of this tool. But our premise is that if it is
very clear about what are the main flaws you have to test,
then you can just write maybe more, you know, larger tests for
this. As an engineer, when coding the
feature, you can still write your unit tests, right?
And that will give you fast feedback that your function,
your new function works. And so, but you also have the
safety of a test suite that focuses on the business.
So we wanted to combine both, you know, like the Super
technical aspect of writing tests that developers are
usually quite good at it and, and the business perspective on
it. So at some point people were
like, is this BDD or so are you bringing a behaviour driven
development to IGN? And we're like, we're, we're not
giving names. We, we just want to make sure
that you, that you understand from a business perspective what
you're testing in an automated fashion.
And I feel this is so important if you're on a complex software
system, right? Because in this type of systems,
no engineers truly understand what's happening, right?
It's, it's so big, it touches so many systems and modules that no
one truly has this, this big vision.
So it's, it's natural that if you're just coding a small
change in some function, you don't see the big picture.
So I think combining both the business perspective and then
the Super low level perspective of testing is very important.
Yeah, You can see that especially from the business
side, covering those critical paths are like a must have,
right? And allowing those people that
have a bit more business context maybe are a bit less technical
to at least try and vocalize what needs to be covered.
Like at a minimum, especially in financial services, there's a
lot of stuff that needs to be covered at a minimum, I feel
like and in whatever chain you're working with, especially
as your engineering capacity grows.
Your engineering capacity grows because you want to go faster,
but that also means that separate teams are going to be
part of one chain in the end. And indeed, having an overview
of that chain, maybe even product managers can't have that
because they are responsible for their own teams.
Maybe overarching product leads have that, but it's more and
more abstract, right? The further away you are from
execution, the further away you know what's actually happening
and what's being executed. So this is like a safety net
that you can't go without, I feel like.
And the funny part was because in this first part where we told
them to, to come up with test plans, my audience was
engineers, right? So I was either talking to an
engineer and I see individual contributor or talking to, to
the team lead to the technical team leads, right?
And the product was like team lead, you talk to your product
person, I don't care, right? But then in this second version
where we created this product, this product where you put the
critical flows or so I decided to speak directly to product
managers and they felt so nice to be included in the quality
assurance process, right? So for the three 4-5 people that
I spoke directly about this, they just, they just, you know,
it's nice to be useful in all ways possible, right?
And they, they, and I feel like they didn't have a clear way to
to make sure that that the product worked.
And now they have. And that was such a click for me
that, you know, yeah, you see, it's not that hard to because
people are usually like, yeah, but you have to convince product
that you have to write tests, right?
But for this, from this experience now of course it is
hard because their agendas, their backlogs are full, right?
And there's business pressure for them to deliver more product
stuff. But once they understand that
they can be part of making sure that the product is reliable,
they a lot of them are taking this opportunity right.
Yeah, yeah. I, I don't know, maybe it's
because I like, I, I've done a year of product management now
and my view on product management, it's like it's
product management and product ownership, right?
Own your product and quality is a huge part.
Doesn't matter if your backlogs fall, there will always be stuff
to add on top of things. But to make sure that what your
build is of high quality and is resilient, I feel like is a bare
minimum. So it should be a non negotiable
that engineers indeed get the capacity to put in the quality
that's necessary. It's just a matter of also
having an engineering team that is mature enough to decide, OK,
these are the critical paths. This is actually not going to
benefit us at all if I test on the small of a scale because the
income and the the input and output is never going to change.
So adding this test, it's just more cost of ownership.
So yeah, it's, it's very difficult, I feel like to manage
that or to have a person that really owns that and understands
it. Also from a technical level, I
feel like it's, it's growing closer though, because from an
education standpoint, especially now with AI educating yourself
on different domains that you're unaware of, I feel like
knowledge is closer and closer, right?
Used to be a Google search. Now I feel like I have a trusted
advisor and whatever question I ask, I get pretty good level of
detail on at least something that either confirms my
assumptions or give me stuff to kind of research and explore
more, which enables me from a product side.
And also, I think from a technical side to either
understand tech or to understand business better, which hopefully
reaches us to kind of make more informed decisions in the end.
I'm wondering though, what your perspective is on kind of the
code generation and code automation part of it.
Because if we're having a context where business people
are kind of defining their critical paths and we're saying,
OK, these need to be covered with regards to a test suite,
that does sound like something that can be automated from a
code perspective. If we're in a mature phase with
regards to code automation. And then conversely, if we have
such a requirement or to such detail what needs to be done,
then I feel like code can also be automated.
What's your perspective on that? Yeah, good question.
So I think my opinion has been changing before LL, like I think
there's a clear division like before LLMS and after LLMS,
right, Because AI for for software engineering is not a
new topic, especially in academia that people have been
working on this for for years. And before the power of LLMS, my
feeling was we need both. We need humans writing tests
that makes sense from a business point of view, that makes sense
to another human. And so, and we need AI to
explore more paths in the application because as a human,
you're sort of limited, right? And complex applications have so
many different possibilities. Then you can have AI to, to help
you there. And I think a prime example for
me was a tool that started at University College London and
then Facebook bought it back then it was called Sapiens.
You can you can Google about it. And this was just a sort of a
smartish tool that would randomly walk in the mobile
application. So think of Instagram, WhatsApp
and so just clicking randomly, doing random actions on on on on
the app. From a user.
From a user perspective, so this was sort of end to end indeed.
But you can do this at whatever level you want, right?
And at some point, if the application crashed, then the AI
would consider, you know, I'm successful, I found a crash,
right? It would generate the report to
developers. And, and the power was this AI
could do this much faster than you, right?
And, and it started, of course, sort of naively random, random
walks on the application. And then the people there
started to make improvements on the algorithm.
So it is for those that understand about AI, it's, it's
basically a search based algorithm, right?
So it's AI, but not as fancy as, as today.
And so for me it was the division was clear humans
writing tests, AI complementing tests for things we will never
find. You know, like maybe another
example that I that I used to see a lot testing highly complex
distributed systems, right? And then think of systems like
Hadoop and this type of stuff, right, highly distributed
systems. What is the other example that
people do a lot in papers like those distributed databases,
whatever. And how do you make sure that,
you know, you, you send messages to those systems and, and maybe
there's a bug that happens because 2 messages gets one
millisecond after each other or five milliseconds.
And as a human, you cannot write a test, you know, that explores
this space and the AI can just keep randomly trying stuff.
So, but then LLMS came and they are super powerful and you can
just give a piece of code and then ask the LLM to write tests
for you. And my opinion right now is you
should certainly use LLMSI use a chat TPT or, or our internal LLM
every day, basically, right? I delegate a lot of, you know,
simple tasks to my LLM when it comes to writing tests.
I feel like we're going to get there, but we're not there yet,
right? So if, if, if you try even chat
TPT, if you, if you give a simple function to chat TPT, it
will generate beautiful tests for you that may even compile
right away. But as soon as you give a more
complex class, which is what we basically have in a complex
system, it gets lost on the dependencies, right?
It, it doesn't know if if it should mock or not mock
something. It doesn't understand how your
database works because maybe your test involves database.
And so it, it gets confused really quickly.
But I feel like it's more about have keeping iterating on the
tool so that at some point it will get there.
Now what what I see is it, it's so my book in, in my testing
book in chapter 1, I make a division there 1 is testing from
the developer's point of view. And you know, as a developer,
you just want to have a bunch of tests.
So if you change something, the test breaks and you know you
broke something, right? But you also want to do testing
from perspective of let me just explore this program looking for
a bug, right? Looking for something that you
didn't see in the implementation, you didn't see
in the tests. So there are these two
perspectives. And if you before LLMS, all the
research was focused on let's use AI to find bugs you as a
human could not see. Now with LLMS, a few people are
more exploring from this. Let's just write tests for an
existing piece of code. And as from let's use the the
capability of the LLM to find something that I didn't think
about. So there's so much space to be
still explored, but I'm, I'm, I'm, I'm very positive.
I feel like we will eventually catch up and we will be able to
delegate this to the LLM and say write tests for me, you know?
Did I miss something in this type of stuff?
I like that a lot. I, I think it's going to get
there at some point as well. And I feel like it's going to be
a tool in a person's toolbox to ensure quality and to ensure
kind of resilience because that's what in the end that's a
means to an end, right? Make sure that we have that test
suite. So wherever in whatever chain in
this engineering capacity, something changes, it doesn't
kind of crush a critical path because that is at the end going
to be very costly, especially if you're in a bigger and bigger
organization. So having something that is then
automated for me is, is going to be quite powerful.
I wonder though, because we as humans, because we carry this
software and we carry the weight of code, we are very conscious
of adding code, not even from a testing perspective, but just in
general, especially if I'm in a small team, I'm like, OK, do we
need this? Can we afford this?
Do we need to outsource this or can we own this like those are
continuous conversations because that's what you need to have.
You can't just work on the coding side of things and you
can't just add things from a product perspective.
We also know that because adding features is not going to make
the best product, you make sure that whatever features you add
are going to be crucial. Small bets, high usage, high
impact and high value and keep it simple, right?
Don't create a, a toolbox for everything in a product that you
have. If you can find a niche and you
can make it very small, then be successful and expand in that
way. With automation and kind of code
generation, especially what LLMS bring now, I feel like generated
code is going to be easier. So we do it more often, which
means we carry this weight of code more and more.
And that's the only concern I have right now.
But I don't know if it needs to be a concern, because if we
don't necessarily care what's being generated, as long as it
can compile fast and it doesn't kind of hit our iteration cycle
and it doesn't slow us down, might not be a problem.
Yeah, two years ago or so, I was in a workshop on software
engineering education. This year, 2025, it's actually
going to happen in the Netherlands, by the way.
And one of the big discussions we had there was that we usually
train engineers to be very good at writing code.
Yeah, we don't train them so much on reading code, right.
Take, take a look at your undergrads, right?
You're mostly writing what, what, what lectures did you have
of, of like, let's just read these codes and with LLMS and I
agreed with the discussion in the workshop that was with in in
the LLM world, the LLM will vomit a lot of codes.
And you as an engineer, you have to be able to read that code to
first make sure it does what you want.
And also if it does in the most effective way that it can do
right. So I get myself looking at the
code from chat TPT and say, very nice, but you can simplify this.
And then chat TPT then rewrites the simpler version of that code
because it just, you know, it, it, yeah, it, it takes the, this
first decision, it, it can see sort of right.
And then, and then I, I caught myself now really doing thorough
code reviews in, in a chat TPTS code.
And I feel like that's at least for now, while the technology is
still maturing maybe 10 years from now, we don't have to think
about this, right? But now I think it's part of the
job to be able to read, chat TPTS code and then help it
simplify as much as as it can be simplified.
Because at the end of the day, for now, no applications are
written, or at least that I that have been part of 100% of the,
the code written by LLMS. We just, it's bits and pieces,
right. And then so I feel like still
having a human reviewing and then making sure that this can
be pasted in the code base and developers will understand that
makes a lot of sense to me. Yeah.
Now it's so hard to make predictions.
As I said, maybe 10 years from now that's not a problem, but
today I think it is more than more often than not I I find the
code from chat TPT to be very good, but it could be just a bit
better, right? And I iterate in my prompts to
make it better. I, I'm very curious because I've
had this conversation with other people and they don't have the
same background in academia as you have.
From my perspective, if, if indeed we focus right now in
academia on writing code. And if you grow out of
university and you go into the industry, writing code all of a
sudden is automated, like your skill set is going to be kind of
outdated. I feel like.
I like that you highlight that reading code is going to be more
and more crucial, I feel like. But it also requires a certain
level of maturity to be able to be prompt and then say and
understand, this is simple, this is maintainable, this is what we
can adopt. And all the other stuff is like
garbage or we have to iterate on top of that.
It needs that level of maturity. And then there's this kind of
chicken egg problem where new people need to have a certain
level of maturity to refine code automation, but they don't have
that maturity and writing code is all of a sudden not a task
for them anymore because that's automated.
I don't know how to solve that yet.
Yeah, so, so I, I, I feel like today, I don't think a is are at
least in 2025 replacing engineers and some people have
been predicting like a week ago or so.
Right. But but, but I do feel if, if
you don't know how to use this assistant, you have to know how
to use this assistant in order to be productive, right?
And I think all your background as as a programmer helps you
because, for example, computational thinking, right?
If, if, if, if you can truly give the task to chat TPT in a,
in a sort of a more structurized way, it will just produce
better, right? So sometimes I see people
writing prompts to chat TPT and this prompt is super generic,
like do this for me and then chat TPT produces what it can
based on the information. But if you give a more specific
prompt, like write this Python program for me that receives
this as an input and then does this as an output.
And then for you to solve this problem, take this, this
alternative, this path and show then the code is quite good.
Yeah. And then for me, it feels like
I'm, I'm still coding it. I, I just, I'm not doing the
boring part of coding ifs and in loops.
And so I, I delegate that to chat TPT and then I also do the
part of let me review the code to see if it works.
Is this testable? Does this fit with my code base?
Does this do what my, my, the business wants?
So at this point, I don't feel like my engineering skills are
being now thrown away because chat TPT can write codes.
I think chat TPT is augmenting my powers because it can deliver
much faster. I don't, you know, usually when
people, when you, you're usually coding, you have this abstract
thinking and then you sort of understand what you want to do
and then you just go to coding mode and sometimes in 10 minutes
you find the solution for a very hard problem and then you spend
2 hours coding the solution. This part, this two hours just
gets what they become 10 minutes because chat TPT will give you a
super nice head start, right? So it's just a superpower.
I think the challenge is I feel like the more senior you are,
the better you can use chat TPT because you have more maturity.
You know more about a lot of stuff, design, coding, etcetera.
The junior you are, the harder it is for you to use chat TPT.
So as, as educators in general, I feel like we have to make sure
that the new engineers, they learn how to use chat TPT.
Otherwise it's going to be just harder for them to, to, to catch
up with the senior folks because it as if I gave this super
powerful weapon to seniors. So they, they now just it's very
fast, but the juniors there, they cannot use the weapon this,
this tool with the same power as seniors.
So I feel like that's where I think, I feel like the gap that
we need to cover in 2024 truly is.
Yeah, I I can see that as well. I wonder where it's going to be
covered because from a perspective of a business, I see
less and less roles opening up for juniors, which is a sad
thing to see because that means that, OK, they're looking for
people that have experience that can go fast, they can iterate,
etcetera, etcetera. But at some point there will be
a new generation coming from academia.
I think it's still a beautiful field to be in.
The fact that you can go to conferences, you have an open
community, knowledge is being shared and it's not you don't
have to pay for it at all. Like things are out there in the
open for everyone to learn it to be better.
I think it's a beautiful environment and a lot of people
want to learn and a lot of people are interested, right?
If you grow up with an iPhone or an iPad, you have that technical
technology aspect. If you see a game, you might be
want to become a game developer. If you found a really cool
website with a really cool design, it might kind of
intrinsically motivate you to think about what makes it take
or from a design side, what can I do to influence that?
So a lot of people are fondling in.
It's just from the business side.
I do think there needs to be more room for people to end,
kind of educate themselves and grow.
I think with tooling, educating is going to be more crucial and
the people that can learn the fastest probably do have or can
fight for a spot for themselves. But I see it being more and more
limited, which is concerning. Yeah, Yeah.
It's been a tough market right in, in the past years.
So I feel like a lot of companies hire juniors because
they cannot afford hiring seniors at the speed that they
need. So they have to hire juniors.
They are they are more available in the market.
And and because companies naturally started to hire less,
this make it harder for juniors. But I'm hopeful that, you know,
the economy will get better, all the variables there will get
better and then we will be able to start or we as a society,
right, will be able to to start hiring more juniors.
But indeed, being a general, you're not on a tough spot,
especially because you also need to master more stuff.
I think though, because seniors sometimes are a bit more
apprehensive, like I've seen some people say I don't use it,
I don't trust it, I've used it years ago and it's rubbish.
And I'm like, yeah, well, things are changing faster.
The new generation is kind of more on top of tooling that
comes out and has the room to experiment so that I feel I can
be a differentiating factor. If you learn faster how to be or
how to excel with tooling that's available, that can definitely
give you an edge compared to the people that don't.
Yeah, for sure. I think if you're a senior and
you're not getting up to date, yeah, at some point the market
will catch you. Dinosaur, yeah.
I'm very curious because since you mentioned you are in a
platform engineering squad and also in more of a leadership
type role. Platform engineering for me is
fascinating, kind of a product with developers as users.
I can just see a lot of priorities and then decision
making, kind of managing your own agenda and still executing
and iterating becomes very crucial.
How do you decide on kind of what to execute on or as a team,
what to solve first? Because I feel like you can go a
long way in a lot of ways basically.
Yeah, for sure. I, I can talk a lot about how my
team has been doing and then maybe I can see a few words
about how the organization is doing.
Because of course I'm, I'm less involved in, in the whole
platform engineering planning in my team, which is focused on
test tooling. We try as much as possible to be
closer to the engineers. So at aging that's easy because
there are no hidden secrets. All channels are sort of open.
We have public channels with all developers.
We have a channel only about testing.
So people know if they have a question about testing, they can
post there. So we sort of understand, we're
very close to them to understand where they are and what types of
pains they have. And this gives us a lot of
ideas. Yeah.
So we also from times to times go to teams and, and try to
really spend a day or a week with them to understand
precisely. I I do that a lot, by the way, I
love to be closer to teams. So, and especially as soon as
you make some friends, it's very easy.
Hey, let me spend one day with you, right.
And that type of stuff. So I think being very close to
them helps a lot in terms of ideas.
I think the challenge then comes in prioritizing because there
are just too many ideas. Oh yeah, right.
We see too many things to be done.
And, and how do you pick the ones that will have the most
impact? And I feel like that's where we
are as, as, as a not only agent, but as, as, as a community.
We're trying to get better. For example, try to have more
concrete metrics that you can measure to truly see the impact
that we're having, right. So you see the emergence of
things like. The, the X core, whatever
metrics that people are now trying to collect to be more
able to, to understand where you are in terms of software
engineering evolution, so to say, are you effective?
Are you not effective? And so, so I feel like that's
been also for us a path of how can we stop trusting only on our
feelings and our feelings are good.
Our, our guesses are very good because we're very close to
people. But can we somehow automate
numbers, get numbers? So maybe an example we, I gave
this talk in November on on the Amsterdam Java user group that
was about the two that we have at agent that does test
selection. OK, so instead of running all
the tests in the code base, whenever someone opens a pull
request, we via an algorithm, we try to to get a smaller set.
So the build is just faster. Nice, right?
And we're on the scale of imagine hundreds of thousands of
unit tests. So if you can reduce the
execution, if if you can skip a couple 100,000 unit tests, the
build is just faster, right? So not bad.
Not bad. So, so that's what we wanted to
do. And then at the very beginning
we had this this hunch that we are running way more tests than
needed, but how much? Very hard to know.
We started to make Pocs and at some point the POC was like,
yeah, the the two works. We, we can select A smaller
subset, but how good is this going to be?
And then we sort of stopped for like 3 weeks or so.
We didn't have a tool to collect data.
So we truly got a lot of open pool requests.
Yeah, we, and we ran the pool request with and without our
tooling. And we did this.
I, I wrote this in the blog post, I don't remember, maybe
around 100 or so. And this was very boring, you
know, semi automated work and we had to wait for the CI and then
sometimes it just crashes, right?
Because that's how it works. And then you run again.
Yeah, I can see that. So we could get numbers on what
type of benefits do we expect from these two.
And and then we saw, yeah, those numbers look nice.
Let's then invest and make it into a tool.
So I feel we're all getting better as, as the day, as we
progress as a team to, to find the priorities.
Yesterday we had this huge two hours meeting on what the cool
stuff we want to build now. And then we, we, we got to a
point there and we were like, this is cool and this is cool
and this is cool. And then at the end of these two
hours, the outcome was like, OK, we just learned that there's
some impact somewhere. We just have to find out
precisely what, right? So, and I think that's the
hardest part. Building is always the funny
part. You have a lot of fun.
You're coding something challenging, deciding what to
build is. And then that's why we're now
getting more product people to help us on those decisions
Culture, right? Because I do feel like now we we
sort of need someone that spends a good part of their day
thinking about this. We cannot just do this on
hunches anymore, right? So I think product in platform
engineering is growing everywhere, OK, right.
And and I feel that's that's, that's the way.
For me, that's a lot of fun. Like I've I always wanted to do
product management. I've not done it as a a year and
it came from, OK, we have this priorities and priorities are
set or indeed we're trying to figure it out ourselves.
But decision making is hard and having one person to be
responsible and to also communicate the why, for me,
that was always really important.
So that's why I'm now doing it myself kind of more and more.
And I don't know if I'm very biased because indeed I have
like a technical software engineering background.
I look at very much functionally what's going to be the value.
How can we measure that? I look at small bets versus long
bets, what is clear versus unclear, even though priorities
might sometimes come from a business side.
If we're doing this for the future and we can do things to
reap the benefits now, like those are all the thoughts that
come into that. That's why I think it's
fascinating from a product engineer or platform engineering
standpoint because your platform is your product and your users
are developers and developers making them more effective.
If we're talking about the scale of 1000 engineers, engineering
capacity in any in any way, I talked to a guy at Miro and
they're really measuring everything because for them a
marginal change is going to have a business impact financially,
cost reduction, which is substantial.
So everything from test automation to enablement with
regards to productivity is going to be crucial.
That's why I think platform engineering is more and more
fascinating. And from a product perspective,
those user metrics that you touched on are going to be
crucial in decision making. For sure and if you like some,
some problems that we are solving, you can clearly see
metrics. So let's say build time, right
easy to see if you make it smaller, it's better.
No developers will complain right of a faster CI.
But when it comes to testing, some metrics are just very hard.
How do you how do you measure if the tests of a specific team is
is good? Yeah, I mean, coverage is a
proxy, but well, it's also a flawed metric, right?
Or we on the static analysis tools, how do you measure the
impact of those tools if the tool finds an issue, linear
code, is this good or is this bad?
How how do I measure the impact of like we use sonar cube spot
bugs. How do I measure the impact of
those tools? This is truly hard, right?
And, and I think for me, sometimes people come with with
metrics and then my sort of scientific background, when, you
know, it starts to scream like, yes, but this is totally
unsound. We can measure this, but this is
totally unsound, right? The other day.
So last year we worked on putting new law way at agent.
So new law way is this tool from Uber that helps you, you want to
teach your Java code a little bit and it at compile time helps
you to find possible new pointer exceptions, right?
Yeah. And then, and then the question
is how do you truly measure the impact of developers using NULL
OA? And sure, one can say you can
now measure the number of new pointer exceptions before and
after in production, but it's, it's just way more complex than
that, right? It's, it's sort of a it, it
works as a proxy metric, but it's also not perfect.
So it's for some products in platform engineering.
And I guess I'm biased, right? I don't know other types of
products, but I can imagine you have some things that are very
easy to be measured and you can see metrics right away and some
things you can't. And then what do you do with
those, right. And I was talking to a guy, he
leads platform engineering. I'm not going to mention the
name of the company, a Unicorn in Brazil.
And, and, and this guy was telling me that they, they,
they, they separate things into some things are just what was
the word he used? Like something like hygiene, you
know, like, for example, improving the builds or like
some, some stuff you got to do and you, you don't have to
measure so much. Like, for example, helping teams
to write more tests. You don't have to be super
precise. You sort of know that more tests
sort of relates to to being good or reducing view time is relates
to to being good. And then other things do have to
be truly measured. If if you're just going to build
a complex project product, you have to be careful about
metrics, but some stuff you don't have to.
And I was like, that makes sense.
That simplifies a lot because then you can have some axioms in
your company, right? Tests are good.
Yeah, small build times are good.
And then you don't have to think so much if you're working
towards those things and numbers are getting smaller than but but
it's, it's such a such a cool problem to work like metrics,
product metrics for platform engineering, so much to be done
there. Yeah, Yeah, I'm thinking of more
and more you have an, you have certain outcomes as goals,
right, as a platform team. And even though you're looking
at it from a tools and metrics standpoint, I do think it's a
lot of also qualitative data that you're going to look at
because developers like or not black and white thinking
there's, they're still human and humans have psychology and
sometimes an emotional connection to a tool also really
helps. So usage and ease of use and
also kind of that buy in that you need to have and people that
actually find adoption. There are maybe even product
champions for whatever you're building.
Those all mean that it's going to be good for your your
outcome. So then all those key
contributors, all those key results are going to kind of
benefit your outcome. And indeed, if something
contributes more or less, it's going to be very hard because
that's going to be nuanced. I love the example of kind of
the implementation on a code level that you gave, because
indeed it might reduce bugs in production, but all our efforts
will reduce bugs in production. So how can we quantify this
small change that we do? Yeah, at the end we cannot.
So we have to trust, I think, based on our gut feeling more
than anything. Yeah, yeah.
And I think that's where things get super funny, right?
Because it's gut feeling plus quantitative plus qualitative,
Yeah. So that you can take a decision
and then learn from the decision.
So such a beautiful space. Yeah, right.
Yeah, as a, as a last thought, so we, we touched on kind of
testing and I think resilience more than anything, I'm always
of the mindset and I think you, I'm curious to hear what you
think as well as that bugs will happen, right?
We do everything we can. We try and prevent our critical
path from breaking. But for example, on LinkedIn, I
saw GitHub was down for a few hours.
Like, I don't know on the scale of GitHub what went wrong.
But a lot of people that host their code on GitHub are
impacted, right? No build times, no changes
pending pull requests, maybe something crucial in another
business that needed a release, right?
GitHub being down cannot be prevented.
It will happen and it has happened.
That's what we see. So then how do you recover from
it? How do you educate or have that
level of maturity in your team that know consciously things
will break and when they break? This is what we do.
Yeah. So I think all complex systems
out there will break and they break, right.
Maybe you don't see it, but they they are breaking on the
basically on a daily basis, right.
Yeah, it, it's just too complex for for anyone to be able to to
see those problems beforehand, right.
So, and usually, and maybe a parenthesis here.
And I felt like sometimes I hear people working in such complex
systems and they say we don't need tests just because we
cannot truly prevent the real incident, right.
And I sort of see the point, but I think the way I see it is in
such a complex system, you still have to write a lot of tests
because you then as a developer, you can open a pull request and
then you can know only for 10 new lines of code work, right?
So you have this small safety net that tells you if if
something is working, but you know that you will never, you
know, be able to truly test a problem that happens because
there was a networking issue and then whatever Kubernetes issue
and the combination of those things.
So those things will just happen in live and you just have to
sort of understand they will happen and then plan so that the
impact is very small, right? And so at this, at this level,
thinking of observability, thinking of releasing not for
your whole audience, but in small bits, right feature flags
and other type of stuff. So you have to do engineering to
reduce the impacts. If something goes wrong, you can
quickly rollback. Yeah.
And you don't affect your, your entire code base, right.
So at agent, for example, we, we, my team bothers people to
write tests, so on and so forth. But we have another team that
focuses on our Sr. ES and our feature teams.
And like, if you're pushing something to production, it has
to be gated. Yeah, right.
So we don't have to deploy to disable your feature.
We just press a button, change a flag here on the UI and then
suddenly your feature is off and and back to to what was working.
So if you all of those things they complement and they have to
be part of the quality assurance of a complex system.
Yes, yeah, I think all. And then of course, and I
remember now what I wanted to talk, and then you see sort of
companies that are doing state-of-the-art software
engineering. They do testing for from a
developer's perspective, they do observability, Robex, blah,
blah, blah, because they know life is hard.
And they also start to have tooling to explore and maybe
hopefully identify the problems before.
Like, for example, I think the famous example is Netflix doing
Chaos Monkey, right? They purposely break stuff to
see how the infrastructure will will behave, right?
Uber wrote a blog poster maybe a year and a half, two years ago
that they have this testing platform where you as a
developer, let's say you, you change one service, you can
deploy this one service, but then the whole flow happens
through production and then product.
But production knows it's a testing flow.
It will switch the flow just so that it hits your test container
and then the flow continues, right?
So sort of creating stuff so you can get as closer to production
as possible to explore problems in different ways or teams doing
fuzzing testing. You know, people are now trying,
OK, we got the testing developer testing part covered.
What else can we add? So hopefully we find bugs in in
advance. Yeah.
So I feel like that's the third wave now because we not all know
testing works. We all know we have to pay
attention to monitoring, etcetera.
And now the 3rd wave that you see the big companies doing is
let's just create crazy tools that will try to simulate
problems that only happen in production, right?
Yeah. So the more we learn, the more
we are able to build tools. And I think we are now on this
third wave as as a society, right?
Yeah. I'm very curious to see what
from that third kind of pillar is going to stick, right?
Because you have whatever test, whatever software we create, we
know is of high quality. It's going to be covered with
regards to a certain level of test suite.
And then indeed, in production, we make sure our observability
is there. And when we have an issue, we
know what to do. It's like education and tooling
to enable. And then indeed, what then
remains? It's like the things that we
don't know. You give the example of AI kind
of browsing through the application, finding issues that
a human might not be able to find.
Or indeed, when I remove certain suites of code, does my tests?
Do my tests still work or is there missing coverage?
Or does my infrastructure break when I disable a component?
Or what happens? Indeed, I'm wondering what
practice from that third pillar like the chaos side.
I feel like the remainder is going to stick and then also be
kind of a first class citizen in a developer's toolkit.
And for me, it's funny because it feels like that's the one
problem we don't understand so much.
And that's why you don't have good abstractions, right?
So if you think of developer testing, you have J unit, and
all Java companies in the world can use J unit to build tests,
right? And then for observability, you
also have a a million observability companies that,
that work for, for everyone. And that's because we understand
the problem. This third one, what you see is
companies creating their own tooling and not really open
sourcing them because they also don't see like a common layer
that they can, they can open to the rest of the world.
And to me that indicates we're still learning about the
problem. Once we know very well, you will
start to see tools that are generic enough so industry can
adopt in large scale. But I only see now and, and they
are not even writing so much about it in, in the academic
world. You see Facebook and, and and
and Google Uber writing papers about this.
So it's still, they're targeting academic conferences.
It's not even for the whole, you know, normal industry
conferences. So this, this to me says we're
still learning, right? But we'll get there.
That's, I love that observation. I really do.
And I love this conversation so far.
It's been really fun picking your brain on kind of leadership
in testing, enabling people with regards to testing and design,
also from a strategic and education standpoint.
And I love the knowledge you have from academia as well and
what's coming. Is there anything that's missing
that you still wanted to share? No, I think we covered a lot.
Yeah, that was a fun conversation for me as well.
Thank you. Thank you so much for coming on
and I'll round it off here then if you're still here, let us
know in the comments section what you think.
Like if you liked the episode and otherwise, we'll see you on
the next one.