As a developer it's your responsibility to make sure that
what you do works in. The only way to do this is by
writing tests that prove to you and to your team that your code
works and automated testing is just as such an easy and cheap
way of doing it. And that's why I say in the
book, that's effective developer is someone that effectively
tests what they do, Hey everyone.
My name is Henry Surya with Robin.
And you're listening to the technology, you know, podcast
the show where I'll be bringing you the greatest technical
leaders practitioners and thought leaders in the industry
to discuss about their Journey ideas and practices that we all
can learn and apply to build a highly performing technical team
and to make an impact in your personal work.
So let's dive into our Journal. Hi, everyone.
Welcome to the Chocolate, you know, podcast the podcast where
you can learn about technical leadership and Excellence from
my conversations, with great thought leaders in the tech
industry. If you haven't, please follow
the show on your podcast app and social media on LinkedIn,
Twitter and Instagram. And for video contents package.
You know, is also available on YouTube and Tick-Tock.
And if you are willing to support my work, please buy me a
coffee at technology. Know that deaths.
Last tip or subscribe as a patron at technology.
Arnold Dev slash Patron. My guess what?
Today's episode is Mauricio a niche, Mauricio is the author of
effective software testing in this episode.
He explained how to become a more effective software
developer by using effective and systematic software, testing,
approaches. We discussed several such
testing techniques such as testing pyramid, specification
based testing boundary testing structural testing or code
coverage, mutation testing and property testing Mauricio also
shared his interesting. You about test-driven
development or also known as TD, which you may find quite
surprising and towards the end, he suggested the one area we can
do to improve our test maintainability.
I hope you enjoy listening to this episode and learning a lot
of things about effective software, testing and several
different systematic testing techniques.
It will be really great if you share this with your colleagues,
your friends and communities, and leave a five-star rating and
review on a podcast and Spotify, your small help will help me a
lot in getting more people discover and listen to the
podcast. So let's go to my conversation
with Mauricio after a few words from our sponsor.
Are you looking for a new cool swag pack, leader, No, no
office. You some swags that you can
purchase online. These wax are printed on demand
based on your preference and will be delivered safely to you
all over the world where shipping is available.
Check out all the cool tracks available by visiting
technology, you know, the death / shop and don't forget to break
yourself. Once you receive any of those
tracks, hey everyone, welcome to another new show of the
technician our podcast today. I have with me Mauricio an inch
He's the author of a book titled effective software testing as
the title says, we'll be talking a lot about software testing
today. How to do it effectively and how
to do it properly. So our Niche is the technique at
the end a company in Netherlands, I believe.
And I think, what is really cool is a niche, is also a lecturer
and he has won a computer science teacher of the year in
2021. I think that sounds really
exciting. Maybe you can share a little bit
more about that. So thank you so much for this
opportunity. I'm really looking forward to
discuss about software testing. Today, thanks Herring party,
invites Marisha. I always like to ask my guests
to share, maybe a little bit more about yourself telling us,
your highlights or any turning points in your career.
De interesting for us to learn from.
Yeah, cool. Yeah.
So, my name is Mauricio, I work as a software developer for
almost 20 years. I had a little break when I went
fully to Academia. So, I spent five or six years as
an assistant professor in software, engineering doing
research in software engineering.
So I wasn't coding professionally for the period.
And my career was always about, or my passion, in my career was
always about software, design, and software testing.
So, how to model your code in a way that's easy to maintain and
how to write tests for it. And at the very beginning of my
book, I tell this story that one of my first projects in my
career, as a team leads, we coded some software that was
supposed to run on the hardware. And then we spent six months
coding version. Number one, we flew to another
country, we installed it, it didn't work for 24 hours.
There was Super big bug and that was super sad that my first
software didn't work for a day. And that was sort of the cake
for me to start focusing on testing.
So since then, and that was in 2006, so it's been a while ago.
I've been trying to write tests for everything that I do, and I
think the book was just a consequence of me putting those
ideas on paper. Well, it always amazed me like,
your first experience in your career can actually leaves you a
lot of impact to your career, right?
So you mentioned that your first project maybe it didn't work as
much as you want. Is a team and that actually let
you two have more passion in testing and I think let you do
also writing this book. So maybe tell us a little bit
more about what if you can reflect back on that experience?
What actually probably some of the root causes of why the
project failed and it didn't last for a day, even?
Yeah, lack of testing was the main reason we were doing a lot
of testing for sure but manually, we had beautiful
Excel, spreadsheets full of things that we Test.
We even create a simulator because in that software, we
would talk to an external party via serial Port.
So, we even create a simulator, so that we could test a little
bit more like the real world, but those tests are never
automated. And what happens is as a
developer, you change something, you test the happy path of your
change, and that's it. And yeah, then we had a bug and
a bug that was totally preventable by very basic
automated testing. So there was a click for me, as
I said. Yeah.
And also you set you go back to Academia, right?
So you spend five six years and I think that also lets you to
win this award. Maybe tell us a little bit more
about it. What made you?
Leave a lasting impression to the students I guess?
Yeah. So it's very hard for me to
decide if I like working in Industry where I can write
software and deliver value right away to people or if I like
Academia where I can just sit and reflect about how to make
software Engineers better at what they do.
So my life was always a little bit like this and it was working
as a developer. I did my masters and then I
said, oh, that looks cool. I did it.
HD. I finish my PhD.
I said, well, maybe I want to try this a little bit more.
I did a postdoc. So, I took a postdoc position
and they're like, yeah, it's amazing to the research, right?
And then let me try being a professor full-time.
And what usually assistant professors do is they do
research and they also do teaching and I started to teach
software testing here at the Delta University of Technology,
a very important Technical University in the Netherlands
and first time I gave this course was 2017 and I've been
doing this course until today day and I think teaching testing
has been also an amazing experience for me because it
made me just understand way more about everything that I was
doing. I had to formalize my thoughts
so that I could pass it to people.
And I think, throughout the years, I've been doing lots of
changes, may be a big one was during Corona, because when
Corona came and then we suddenly switch to online.
I didn't want my students to be in front of the camera as a
compulsory activity, you know, because people had their own
challenges. So I decided to open my lecture
notes and I remember I just got it.
Clean assistant here and then I said, those are my lecture
notes, just make it a little bit more.
Beautiful publish it in a website and that's how we're
going to do. The course, people can read my
lecture notes and to my surprise and then they started to get
emails from people from other universities.
Hey I'm reading your lecture notes, this is very cool.
Do you have exercises that I can use?
We have slides and then same thing in your number 2 and then
this is started to grow. And at some point said, well
maybe I need to turn this into a book and I contacted many, I
submitted a proposal. They accepted it.
My book went through. They are very thorough review.
So the book just got way better and I think in the end this
teaching award that I got. So Delta has a ward every year
that they give to teachers that are doing something impactful,
and this is based on the surveys from students and Etc.
I think a lot of it was because of this first switch from,
instead of me lecturing, I just gave them a book.
That is easy to read. That is very practical, and in,
creating the book itself, I think students, like very much
the content of the course. So my course is far from Those
theoretical courses usually see about software testing that you
never see source code, it's more practical.
So all of these combined gave me this award that I'm actually
super proud of and even today. So I'm giving this course right
now, actually in this quarter students are reading my book.
And again, the feedback has been super nice.
One of the students came to me this year and said, you know
what, this is the first book in my bachelor so far that I read
cover to cover or back to cover. So yeah, that's the story.
Well, thank you for sharing, such a beautiful story and I
think, yeah, it always gives you a very proud moment to hear such
impact, right? Especially when someone reads
your book and do and especially it's a technical book.
I must admit myself. I rarely finish technical books
and to end because there are hundreds of pages, sometimes
it's dry. But I think, looking at your
book, I think I can see. There are practical things.
They are sample code, they are things that guides people to
actually understand your thought process, which is a good segue
for us to start talking about your book effective software
testing, I think one key. Key sentence that I pick when I
read your book is that you mentioned to Be an Effective
developer. You must become an effective
software tester, but in Practical World, sometimes
developer, and tester, there are two different roles.
So tell us more about this. Why do you say that to become an
effective software developer, you need to become an effective
software tester. Thanks for the question.
I think it's, because as a developer, it's your
responsibility to make sure that what you do works in.
The only way to do this is by writing tests that prove to you
and To your team that your code Works show indeed before in a
even worked in environments like this where I would just focus on
coding and then I would send my software to another company.
This company will do the testing for me, they would send me a
report I would fix the bug so on and so forth, and that's just
not productive back and forth between these two things is too
big and I think again, it's our responsibility to make sure that
things work. And automated testing is just as
such easy and cheap way of doing it.
And that's why I say in the book that Effective.
Developer is someone that effectively test is what they
do, right? And I think, I mean, I don't
know, I was a developer last time, sometimes developers can
be very confident type of person, right?
So we rewrite the code, we test a little bit during the local
development. We think it works.
We always think it works. Many developers probably do not
enjoy writing test, for some reasons.
May be apart from you neither sometimes because it's very
close to their workflow. So, how would you actually
invite most of the developers to actually have more?
Passion or energy to start writing that if they have the
perception that I'm super confident, my code.
I don't want to spend more time to write automated tests and
things like that. Yeah, that's a very interesting
question. So the first part was you
mentioned about developers being confident, right?
And what I always say is you never know, you don't know how
many times a day you make a mistake.
So you program something wrong and you only know this once you
have tests because if you don't have tests everything that your
code looks perfect, right? To just believe it works.
As soon as you have test then you see, you know, oh my God,
I'm ready. He might test 50 times a day.
And then you quickly realize how much you need them because we
break stuff all the time. Now, how to convince developers
to write text. That's a very deep question.
I think there are many angles to this one is to show to them and
to have them practice so that they don't see writing tests as
something that is a burden. They just get used to it, they
get proficient with writing tests, not only with the testing
Frameworks, but on writing code, that is easy to be tested on.
You look at a program and the test case is emerging your head
and all you need to do this too. Them in your programming
language. So there's this aspect of just
practicing and getting better at test itself.
The second one if you are in a very large and complex Software
System, things are too complex, right?
And then for you to have this pleasure back in writing tests,
you need to make sure it's easy to write tests.
And this means as a company, as a team, you have to invest in a
small infrastructure. That facilitates the act of
writing tests. I was discussing this yesterday,
actually not here at the company, but in another
University, It was yesterday and I was saying if you're working
on very complex Software System to test something, maybe you
need to know to instantiate 10 entities or have data in 30
different tables in your database.
And that's just because your business is complex and you have
to have some something in there that makes this easy for you
that in a bunch of couple lines of code.
You can set up the scenario you want to test and then you can do
the test so those are the two perspectives.
I see, one is again getting good at testing itself and the second
one is having the proper plan. Form so that you can write it,
this totally makes sense for me. So, from my point of view also
sometimes we also need to maybe introduce back like, what you
mentioned? The beginning of the pride,
right? So how can you tell that your
code works now and also in the future?
There's no other way to do it other than having some automated
tests that can actually prove that your work is always passed.
Write that in terms of test cases, and I think many
developers also think, probably writing, more code means like
doubling their effort, right? So I think this must be changed
as well. In terms of perspective.
Active so that you don't finish writing the code.
If you don't have the test, maybe just some perspective from
me. Let's move to the thing that you
mentioned in the book, right? So you mention about effective
and there's another key word that actually you mentioned in
the book, which is systematic. So, I find these two things are
very interesting, the way you explained in the book.
Maybe, if you can explain here as well, what would be your
advice for people to be a more effective software tester and
also to be systematic in coming up with those tests?
Yeah, indeed. So they make a very strong point
that Maybe you can be a little bit more systematic.
When we approach testing, we wrote a paper a couple of years
ago. And in this paper we asked
developers to write tests. So we gave them a program and
write tests and we observe them and they were thinking aloud, so
that we could see how they were coming up with test cases, and
we observed a couple of things. Maybe the most important one for
this conversation is that developers were never
systematic. They were always, you know, just
following their hearts. They would look at the code is
this shows like the next thing I want to Test.
They would go and they would write the test and then they
would repeat, what is the next thing I want to test?
So, they would just follow their hearts and follow.
Clear experience, and feelings are very important when it comes
to testing. But it also opens up space for
mistakes, maybe you're not in a good day.
And then you forget something second.
If you're always using all your brain power to come up with test
cases, even the basic ones, you're not saving energy to
think of complex cases. So when you're more systematic
in terms of testing, that means No, just follow some sort of
cake recipe. That helps you to come up with a
bunch of test cases that are quite easy to see.
You can put them on a checklist, basically, and then it is your
brain power to focus on test cases that really requires human
Mark, people thinking about it. And the fun part is that are
lots of techniques like this. If you look at academic books on
testing, even the books from the 80s, they are go back.
Then was to come up with a recipe that teaches you how to
write perfect tests and what are lots of good ideas there.
And you can think of, I'm not going to give it 30 hero.
Of course, but you can think of basic things like, for example,
if your method receives a list as an input, there are a bunch
of interesting cases that always make sense.
So, for example, what happens if the list is empty, what happens
if the list is new? But if there is no linear
programming language, what happens if the list has just one
element, right? Because sometimes your algorithm
changes if there are multiple elements were just 14, there are
a couple of days. It's stuff that you don't even
have to think and you can write those tests and they are usually
good enough to get you to a very high coverage already and then
leave Space for you to then focus on Boundary testing, you
know, Corner cases in this type of stuff.
And being systematic, is something that you seen other
engineering Fields, right? And I think I said in my book,
there's this very famous book called The checklist, Manifesto,
and there's a story they're backed up by research.
That shows that medical doctors, that use a checklist before some
surgery, they make less mistakes that medical doctors that don't
and you don't have to be ashamed of following a checklist.
It's goods and I think that's the point of my When I say we
can be a little bit more systematic.
Of course, we don't have to be systematic all the time, it's
just too expensive. Maybe too complex to be
systematic all the time, but identifying, like I'm testing a
complex method here, let me be a little bit more systematic.
So that's the whole idea of being systematic when it comes
to testing. Yeah, I think your point is
Right, sometimes we have typical use cases, just like list or
maybe you'll have an API failure cases, success cases.
So, all this can be made as a checklist.
I've heard about checklist, Manifesto as well.
I think the author named are together.
Day and I think becoming systematic in solving.
This kind of a testing problem is really crucial, because in
your book, you mentioned that, if you are systematic, you can
assign any developers to the same problem.
They will most likely come up with the same test Suite, right?
How cool is that? Because sometimes we feel that
we only need a certain test, has to come up with a comprehensive
amount of number of tests, but I think the systematic way is
actually to have any developers working on the same problem, but
they can come up with the same test Suite that I think is
really, really powerful concept and I think it also becoming
effective. Means that you write the right
test because I think it never ends, right?
You can come up with any number of tests as you long as you can
produce the test inputs, right? And I think to come up with the
certain right. Amount of test is very
important, which brings us to some of the techniques later on
one of the techniques that is commonly.
I don't know discuss in the software world is about test
pyramid. Maybe let's start from their
first. Do you think this is a
systematic way to think about producing test and what is your
view about this testing pyramid? I think the testing pyramids.
Helps you, too. Pragmatic when writing those
tests because one thing is to come up with a test case, it's
right. So those are the inputs.
I want to give to my program and this is the expected output.
The other one is writing this in code and making sure that this
works nicely in your, let's say, development process in your CI
and Etc. So for example, if you just
write end to end tests, maybe your test Suite will cost you
too much to execute. And then at some point this
becomes a bottleneck, right? So I think the testing pyramids
gives this pragmatic point of view on, hey we're going to have
to automate the Stats and we're going to have to maintain those
tensor and we're going to run them the whole day and I like
the idea of the testing pyramid very much on the idea is that
the bottom you have unit tests, right?
And why is it at the bottom? Because they are usually cheaper
to rights. They are cheaper to run.
They tend to be more robust. They did not do really fail.
They are not flaking General and then you go up the pyramid.
You have integration. And then to intestine, you still
have to do them, right? You have to, but you do them a
little bit less, maybe focus a little bit more thin than you.
That's maybe it's okay to maybe have some duplicated test series
for ensure. Are you covering the same thing
or not? I think it's okay to make this
sort of mistakes in the unit. That's what it integration and
end-to-end tests. You want to prioritize a little
bit more. And I like this idea of the
pyramids. Although, you know, in the
software engineering at Google book, I think from 2018, I think
they bring A New Perspective that I also appreciate very
much. That is forget about unit,
testing and integration testing, right?
Just separate your test Suite between is this one fast enough
that I can. Actually run it during a pre
merge together with your merge request, or pull request.
Or is it super slow? That I will have to run in a
separate machine and etcetera. So separating, the death sweet
infesting slow, makes a lot of sense to me because if you
remember back in the 2000's, our big discussions were what is the
unit test? If I'm testing two classes
together is this to unit test or not right.
Then, I think in 2023, we know that this doesn't matter at all
as long as it runs fast and it gives you proper feedback.
That's good. So, I like this new way of
seeing things and it brings way. This is space for this type of
discussions, right? Because if you no one can argue
that a slow test is better than a fastest, you can argue that it
may be like integration test more than unit tests, but no one
can say a slow test is better than a fastest, right?
So I like this new way of phrasing.
This yeah, let's go to the point, you mentioned just now
because there are some discussions and debates about
people preferring more about integration tests rather than
unit as something that doing unit test because you probably
most of the things they feel is less valuable.
So what is your view about this? I know it's probably a little
bit more hot topics to discuss, but what is your view on this?
I think I have a pretty clear opinion on this, right?
And I think walking can be dangerous for sure because if
you can walk too much then at the end you're not testing
anything so you want to test as much as possible real Behavior
or behavior that will look like in production.
But at the same time you don't want to be slowed down by thanks
that you don't control so much. For example, let's say you're
writing a piece of code that makes a call to web service that
is developed by another thing. And suddenly You to really run
this test, you need that web service available to you, right?
And this becomes very quickly, a pain.
So in this case is smoking, makes a lot of sense and the web
service you don't control so much.
That's given an example of something, you control a
database. Should I mock my database or
not? To be honest.
I believe in my opinion is that you should not walk your
database because today you can make tests with database so fast
that you can actually run them during your pre merge time and
you get feedback. They're very fast.
So why would you walk something that is not really preventing
you from, right? In the test, in an easy way into
running too fast. So I think that's the trade-off
that needs to happen. I think it's not about writing
integration tests or unit tests, but it's about writing.
Lots of tests are fasting the end and that you can have full
control over and those are the things you should mock, right?
Things that you don't have control over and then of course
if go back to the testing pyramid, let's say you're
mocking this web service so you can write lots of unit tests you
know Superfast tests that just mark this web service but you
can still have one or two or three integration tests that are
a bit more expensive that make real calls.
To this web service. So you see the things work, when
you put all the components together.
But you know, something that I always say is you should not
write the integration test that could have been a unit test,
right? You don't need an integration
test exercise. An if statement in your code,
just write a unit test for it, leaving integration tests for
what it really pays off. That is to find the integration
works. I really love your pragmatic
approach to answering this question so I fully agree as
well in terms of control, right? So first, think about the you
control that and especially is not just the external service or
infrastructure related time. Is also, probably one thing that
you want to consider, right time is something you cannot control
by itself, but you can actually create like a fixed system or
Market sometimes. So I really love the way you
explained about this. So let's move on to maybe some
of the other techniques, right? We have talked about unit
testing. As integration tests.
And to end tests in your book, you actually mentioned a couple
of testing techniques. So let's start with the first
one. Which is called the
specification based testing and it's actually very interesting
the way you start by expanding this.
I would assume like some people would start with unit tests but
you start with specification based testing.
So maybe tell us more what is specification, based testing why
you prioritize that as the first who?
Yeah. So what is testing, right?
So best thing is about trying to find bugs and how do you do
this? Because you compare what
Expected program to do with what the program does, right?
And for you to do this, you need to know what the program should
do. Where is this information in the
requirements? Then I'm not saying like a Word
document that contains the requirements or uml, it doesn't
matter. The requirements can be in your
mind's at some pointers is notion of what the program
should do. And specification based
techniques are the ones that help you look at the
requirements and identify interesting test cases.
This is not new again this dates from the 80s and we became very
good at coming up. With techniques.
And basically what I want to show you, my book is to look at
the requirements, it's quite easy to see, you know, those are
the inputs of the program. This is sort of what the program
is to do in the output. Those are the different paths
that the program may take, so on and so forth, and you can look
to all of this and then get inspiration to write your tests.
And the one technique of showing my book is actually a very basic
one. That basically says, look at the
inputs of your program for the message or testing or the
component, whatever, or the inputs separate them, one by
one. Your function receives three
inputs and integer is string that represents whatever and the
list. Look at them separately and
explore their domain. So let's say this is string.
What are the possible strings that can come here?
Are there any special strings that would make the program to
do something different? You do this / inputs, then when
you do the separately because it's just much easier for our
brains to process small things, right?
So you do each one of them separately, then you try to look
at all of them together you look for other possible Corner cases
and so that might be explicit on the documentation or the
requirements and then and only then you come up with test cases
and that is sort of the idea of specification based testing that
you start your tests from what the program should do, and not
from the implementation. And I actually like this very
much, because as a developer, right, the person writing the
codes that I'm about to test. That gives me the opportunity to
disconnect from implementation. Really focus.
Hey, this is the input that I'm going to pass the program.
This is the expected output. So this is my suggestion.
If you really want to be my little bit more systematic when
it comes to testing, His first step is start, dragging tests,
or creating test cases from the specs from the requirements.
So when we talk about specification based testing, I
think people always, I mean, not always often associate with bdd
right Behavior driven design and sometimes it's like gherkin type
of language and things like that.
What's your view is of this, do you always have to come up with
bdd type of specification test? Or is there some other types of
tests that we can also use? Yeah very good question.
And maybe that's a difference between my book and No other
books. Usually when people talk about
specification based testing, they are focusing on a very
coarse grain feature from the point of view of user.
Why should I test in my book? I show that you can do this at
method level, for example or a class level because as a
developer, that sort of the unit's, you're always handling
with, right? And I think, to me, really makes
more sense when you're looking at the big picture and looking
at the whole functionality from really, the point of view of the
final user and the specification based techniques you can apply
in both now, Should you write tests in a bdd style?
I think that's really a matter of taste.
I am personally not a bdd fan, so I don't like tests using vdd
style. I don't use cucumber.
That's just not my thing. But I see where this comes from,
right. And I don't think, you know, in
practice, this is two different. It's really telling you to focus
on the specs, on the behavior of the program, right?
And coming up with the best cases tooling is a matter of
personal taste in the end, right?
As long as you're looking at the behavior of the program, what
you expect from it? I think this is a great Step
towards a good testing. Right.
You also mentioned just now in your explanation about Corner
cases, right? Sometimes when you have
requirements, most often, it's just the normal case, right?
I expect the program to work like this.
So first of all like who will come up with this corner cases
and how would you actually invite people to come up with
more creative Corner cases? And I know you mentioned about
boundary testing. Is it also one way to actually
generate Corner cases. So tell us more about this.
Yes indeed. So Corner cases are the cool.
Sprite and Empirical research actually shows that bugs, love
boundaries because we are very good at implementing happy
paths, the bugs, then they start to cluster on things that were
not so good at that. Is to handle Corner cases,
right? And coming up with Corner cases,
is very hard because we develop complex software systems, but
one way to get started is to observe your program.
You look at the inputs and how this inputs change the outputs
and Boundary testing is a perfect technique.
For this. What this boundary mean?
So imagine you have a very simple program, you know that
has an if if x greater than 5 do something else.
So this if divides the execution of the program so if the input
is 4 4 is not greater than 5. So the program will do
something. Then next 155 is not greater
than 5, so the program will do the same thing.
Now 66 is greater than 5 another program, there's something else.
There was a small change in the input from 5 to 6, right?
So there is more changing the integer value but the program
responded to something. Completely different.
So this is a boundary and this is precisely where you can write
a test because we love to put bugs there, right?
And as a developer that makes sense because it's very easy to
confuse, you know, a greater than with a greater than or
equals to. And the ideal of boundary
testing is look at your program and look at the inputs and how
the inputs affect the outputs. And look for those moments where
a small change in the input changes the output.
There's a paper from 1994, the reciting my book that explains
this in a more mathematical way. So this was really written.
Buy a computer scientist and that shows that if you write
tests like this, you are more likely to reveal a bug.
And I think this is something very easy to change in the
behavior of the developer, right?
And it's like a very quick win. That will give you better test,
because as a developer, you look in, if you write a test for the
true branch and for the false Branch.
But then we usually pick random numbers that exercise.
The true branch in that exercise, the false Branch
instead of picking random numbers, pick the numbers that
are close to the boundary. These two tests will be way
stronger than the other two tests.
That's Of exercise the programming the same way but
very far from the boundary. It's a very easy change and a
funny enough. I get emails from people reading
my book and this part of the book is the ones that people
like the most. And people say, oh my God, this
is so simple and that really made my test so much better
because it's also easy. So think of boundary testing for
sure. And of course in this example is
very easy to find the boundaries but more complex programs you're
going to have to spend some time looking for the boundaries,
identifying the boundaries, but that's just part of the exercise
that we have to do. Now thanks for sharing this.
This technique this tip. So I really loved the way you
phrase it right box, love the boundaries.
So before people here, maybe that's also a lesson for all of
us, right? So when you think of test cases,
now start to think in terms of boundary first then you can pick
maybe some random inputs later on after you have covered the
boundaries. So let's move on to the next
technique, which you categorize, as structural testing.
So maybe tell us more. What is structural, testing, how
can we use it to become an effective software tester?
So structural Things sort of the chemical name for using Code
coverage in practice. So the idea of structural
testing is let's use the structure of the program as a
source of inspiration to write tests.
So before we were looking at the requirements with specification
based testing, right? Maybe texts.
Now we're looking at the implementation and trying to
come up with ideas and how do you do this while we look at
them, if statements and then you think, well, I want to write a
test that actually exercises that if statement, and maybe you
can do this line by line. A line, right?
And then once you cover all the lines, you're done with your
structural testing because you tested the structure in the way
you want and your coverage tools, give you this
information, right? You can see line coverage for
example, or you can be a little bit more thorough and say, well
I want to cover all the branches and whenever there's an if I
want to make sure that there's a test for the true branch in for
the false Branch, or you can go deeper an if statement can have
multiple conditions and I want to exercise every condition as
true and as false, right? So you can keep going Crazy
about this, but this is sort of structural testing and I think
the main point of my book is that there's this hate in
Industry about called coverage, right there are many reasons for
this, you know, one, you can cheat the coverage number,
right? It's very easy to write a test
that covers a lot of stuff, but the test is not so good if
you're using coverage as a target number.
So let's say your company says 90%.
Otherwise, we don't get from merge request.
Maybe that forces you to write useless test, just so you get to
90%. Also, if you get to a hundred
percent coverage, Each that also doesn't mean your code is
perfect, right? And it's quite expensive to get
to 100% College. This is why people usually hate
it. But I think people usually hate
cold coverage because they are focusing more on the number
rather than on the output the information that coverage brings
to you, right? And in my book, I showed it
coverage should compliment specification testing because,
you know, you write the test based on the specification and
then you do, maybe boundary testing.
And then the question is, am I done?
Well, you can triangulate this and you can look at coverage,
right? And then you see, Did I cover
everything and maybe you forgot to cover something.
And then the question is, why did I forget was it?
Because I forgot was it, because there was a mismatch between the
requirements and the implementation you reflect about
it. You either write a test or you
don't try to test and it's also fine, right?
But it helped you to reflect a multi, my done writing tests.
And if you use coverage with this purpose, notice that we are
not talking about numbers anymore.
Furthermore, talking like the test that I wrote, are they good
or not? And coverage is giving me
insights about it. And I think that story You
should use coverage but then the 1 million dollar question then
is also, okay. Okay, coverage is cool.
I use structural coverage that will maybe help me increase my
coverage because I'm going to look at an obstruction that I
didn't test. I'm going to test it now, is
there a correlation between High coverage numbers and the
effectiveness of the test Suite? So that's sweet.
That has a very high coverage. Is this one more likely to find
bugs? If I have a bug in my code?
And this is a perfect question for academics to answer, right?
And then you see lots of Papers written in my Besides papers
from the 1990s to 2010 or something.
And a lot of those papers show correlation between coverage and
effectiveness of the test suite and that makes sense.
Right? Because the more code your test
Suite covers, the more likely it will find a bug.
If you introduce a bug, of course, different paper, show
different levels of correlation, some of them are strong,
correlations some others anymore, weak correlation, but
the correlation does exist. And I think the lesson that I
get from these papers are. 100 coverage, may not mean a lot.
Because once you're there, you're desert very good.
But that doesn't mean it's perfect.
And then when you're at 100% coverage coverage doesn't give
you more useful information because it's all covered.
So if you're there that stream, then coverage is not so good
anymore for you. And doesn't tell you much with
each other on the other side. That is you have very low
coverage. That's a 10% then that means a
lot, right? That means your test Suite is
still, may be poor. Maybe there's something that you
can improve their so, to summarize it in coverage can be
used as a way to complement your tests to help.
You see if your test Suite is good enough for now.
In coverage, helps you to identify poorly tested areas of
the code base and hey, this is not covered.
So maybe I should write a bunch of tests for.
They don't have to go to a hundred percent coverage, but I
have to add some text for it. Well, I really love another
pragmatic advice on this, right? Because people always debate
about code coverage. I know some companies also who
put code coverage as the so-called the build gate, right?
So if you don't pass a certain threshold, will fail your build.
So I think you remind us that code coverage should be a
complimentary thing. It should not be the main.
Main thing, right? And I really loved the way you
mentioned that 100% doesn't mean that we will not find any bugs,
right? Although there's a correlation
of course, but a low coverage actually means that there are
room for improvement and I also like one particular statement
that you mentioned, how should we use a criteria in terms of
covering more line coverage you say in your book that all code
should be covered until proven otherwise, right?
So I think that's also another good guideline I find sometimes,
we can leave some coverage behind but actually you should
have a good. Listen for that, right?
So I think that's a very good thing.
Another technique that sometimes is used for, this is actually
called mutation testing. So this is also in the
structural test kind of territory.
So tell us more for people who may not have heard about this
term mutation testing what it is and how does it help to provide
effective software testing as well?
Sure. So, how do you know if your test
Suite is good or not? You can look at coverage for
sure, right? But maybe you can get to a lot
of coverage, but maybe your assertion.
As our poor, right, and then there's a bug but your
assertions are not catching that bug but you're covering the
code. So your coverage is very high.
Mutation testing helps you to identify this Gap and what is
the idea, the idealist I'm going to create mutants of my codes.
So imagine like, I just get your production code and I own
purpose insert, the bug. So I change a greater than to a
smaller them, right? So let's say I do this, if I run
the tests, your tests, you must have a test.
That is failing because it just introduced a bug.
If that happens, that means Okay, your test can actually q
that mutants. So your tests are doing some
good stuff but if I can mutate your code so they introduced a
bug on purpose in your tests are still green.
Hey, I just found the case a possible bug that someone can
introduce that, you'll never know with your test Suite.
So that is the idea of mutation testing.
And what mutation testing tools do for you is to automate this
process. So they change your code, they
run your tests and they see for test.
Catch the bug and they repeat this in a structured way and
they give a beautiful report in the the end solely for in the
Java world. For example, like I am, you have
Pi tests which is an open source tool that does that for you.
And I love the idea of mutation testing.
It just makes a lot of sense to understand if your tests are
good or not, I think in practice a lot of companies are not there
yet to benefit from mutation testing.
You start to benefit from mutation testing when you have a
lot of tests and have very good test.
Suites if your test Suite is very poor.
Well it doesn't get it doesn't kill any Mutant.
So why would you run mutants in first place?
Right. Comfort is good enough for you
at that moment. So I should like a lot of
companies are not there yet in terms of materiality to use
mutant but if you are so if you have a beautiful day sweetened
put mutants in your pipeline tools are evolving.
So by test is a very nice tool that are ways for you to reduce
the time it takes because it's an expensive process, right?
We have to mutate your code and run.
All your test suite and we do this a million times or the tool
does this a million times. So those tools are getting
better and better you know. So that you just run the tests
that are really relevant for the mutant and blah blah blah.
There's a beautiful paper that I cite in my book called, I think
it's mutation testing at Google. It was published in an academic
conference but in the industry track and that paper sort of
describes the experience of Google in trying to put mutation
testing on scale mutation testing.
By the way, is also it dates from the 70s, right?
So the ideal is super old but I think now that we have very good
Hardware to make it work and the tools are getting better and
better. I think now that industry is
adopting it more, which is really cool.
Loved your analogy using mutants to explain how this mutation
testing works. So, I think it's true that.
I think if you have a very low coverage or kind of like the
poor test Suite, you will not be able to kill the mutant so to
speak, right? Because your test will always
pass because you don't cover enough.
Another technique that you mention in your book is called
property testing. How does this differ in what is
property testing? Yeah, so property based testing
is the way of writing a test that is different from the way
we write normal tests, right? So usually when we write, At a
normal test, which I call example based test in the book,
you know, in our minds, what sort of Branch you an exercise
in the cold or test case when I create based on requirements and
what you do is you think of a concrete inputs that room run
the program will exercise the program in their way you want.
So in the example that I gave X greater than 5, it can be Any
number greater than 5 5 6 7 8, right?
And you pick just one because you're pregnant again because
you cannot do more than that, right?
So maybe we can do two or three or four then you cannot do 50
by. Right hand, it's just too
expensive. So this is example based testing
because you're testing, by example, you pick one example in
property, based testing what you do.
Is you try to describe a property of the program and you
let the to come up with the inputs for you.
So, let's say in this program, if for any number that is
greater than 5, you know, that the output of the program,
should always be a positive number.
So, inputs, X greater than 5, then the program always prints
positive numbers. You can write the property and
then you say, create any number that is greater than 5.
And just assert, it's that the number that comes out of it is
positive. And then the two creates a lot
of inputs for you. So, for example, in the Java
world, you have J quick, when you run the test, Jake will
think of 1000 inputs for your function and it will send 1,000
input for your function. So in a way, you're exploring
way more, the domain of that input right?
Because you're not trying one example, you're trying multiple.
So the ideal is very cool. It is, of course, much harder to
write a test because you have to stop thinking of specific
functionality when a test but more like what are the
properties of my program? That I want to exercise.
But once you can do this, then it's very powerful because
you're just testing a lot. So maybe another example so that
you can see the usefulness of it imaginary just implementing some
data structure. So let's say you're implementing
a set yourself, right? So you cannot have repeated
elements on the set and you can write a property based testing
that just transfer to insert random elements in the set and
you make sure that you're having repeated elements and it doesn't
matter. The order that you insert the
elements in the end, the property is The set must only
have unique elements in there and erectus like this.
And then suddenly you're testing dozens and hundreds of
combinations of inserts of repeated stuff, new stuff
repeated stuff. Something that you would never
be able to do by hens. So that's how powerful those
things can be. Now we need Distributing
Information Systems, which is what a lot of us do property.
Based testing is I think you have less opportunities to write
this to use property based testing.
So I still feel like example based testing is something you
should do as the to Approach but considered property based
testing especially for situations like this where
sometimes you just feel unsure. This one example is enough and
it more and then property based testing can help.
Thanks for that. Toro, explanation about this
property, based testing. I personally haven't done,
mutation testing and property testing myself.
So I think it will be cool to introduce that.
Sometimes in my test Suites I guess.
So for people who also have an experienced it, I think you can
check some of the papers that Mauricio has mentioned.
Also, some more materials from his Book.
Make sure to check out as well talking about testing.
I think it will be a Miss if we don't discuss about test-driven
development or tdd. What's your view about this
workflow? Are you a proponent or do you
always do tdd? So maybe tell us more about
that. That's a tough question hearing
because usually you know you have some very passionate people
about disturbing developments, right?
And sometimes I make fun with my friends that on my Twitter
bubble, sometimes I just post something that says that he's
not that perfect. And then yeah a lot of backlash
in, right? And I'm a big fan of tdd.
I even wrote a book about TD until 2012, it's in Brazilian
Portuguese. So maybe not accessible for
everyone that's listening here and that was because I was doing
a lot of DVD myself and I really felt I was just being a better
developer doing tdd and this was 2012 and when I wrote the book
so I was doing tdd for a couple of years now, 2023.
I think I do 3D wave less and I feel that just because I found a
way to get the benefits of tdd without having to do tdd and
what are these Benefits. Right.
What am I talking about? If you like the big benefit for
me, about TD is the ability is that we work on very small
steps, right? And we make very small steady
progress towards the bigger feature before.
Knowing TD, someone would give me a feature to implement and I
would be all over the place, trying to write the full
algorithm in one goal. And that would just complicate
my life. I would code for an entire day.
A lot of frustration because you know you bump into barriers, you
break stuff that was already working.
And so on and so forth. And then after, you know, a few
days of work or whatever, then writing test is, you know, I
just want to get this feature done.
I don't want to do this anymore, right?
And we tdd it was like, oh, and I cannot do tdd if I work on
such a big chunk of code. So it forced me to see
programming as an act of writing small things, and combining
these small things. One by one to give bigger
Behavior. I think that's sort of the big
difference between TV and known tdd.
And if you can incorporate these three development practice I
think it's okay if you don't start with the test.
So today, for example, in a lot of cases, I actually start with
the production code but my coding sessions are very small.
I write a little bit of production code and then a
little bit of testing the experiments sometimes I delete
the code and I start again and do it small tests.
I think if you find your way to work on small and steady steps,
I think you got the benefit from DVD and this is actually what's
more recent research shows about test-driven development.
So if you look at the whole body of knowledge on DVD, you see
papers and What have you built of these papers in a nutshell
qualitatively. If they s developers that are
doing tdd, do you like to delete?
The answer will be. Yes, I am.
Actually one of the authors of these papers, I wrote a paper
that did qualitative research on this and that was sort of the
feedback we like tdd quantitatively.
If you compare the quality of the test Suites in comparison to
people, not doing tdd. The differences are negligible
very small. So 3D by itself doesn't do
magic. So more recent research actually
shows that the benefits are on those small steps.
And not because you're writing the test before that means side,
do I recommend you to do tdd definitely?
Yes. Especially if you've never done
it because if you never done it, odds are you're not used to work
on doing small things. We're just used to work on big
things. So TD is the best teacher, you
can have when it comes to explain to you how to program in
small bits. So, do tdd, do a lot of it.
Once you internalize the ideas, then it's okay.
Then you don't have to do it anymore.
Wow. It's like a safety will write so
too. Figure out you start using it
and once you kind of like Master it and like a bicycle, right?
You can take it off and you can choose when to use it and when
not to use it, and I like that, you mention about this Empirical
research, when I read it in your book, I was intrigued.
This research doesn't actually show that TD produces much way
better test versus the non TDS. So I think that's really cool to
know about this and I think I'll make sure to put it in the show
notes as well, for people who are interested and also have
passion about Didi. It's not saying that tdd is not.
Good, right? But I think it's the mindset of
working with small things. That is very important.
So we have a few minutes left before we go to the technical
leadership. Wisdom, I have one question,
maybe tips from you for people to be able to become a better
tester in terms of their quality or maintainability.
Is there any few tips that you can just give us here so that we
can improve? That's a very good question
because, you know, if you really write lots of tests you're going
to have lots of source code in, they're going to have to
maintain this as well. Not only Production codes,
right? So I think all the love that you
put in your production code. You also have to putting a
passcode and to me in the test code if you have to focus on one
thing because in my book I mentioned lots of ideas you know
good practices that you can have to make a cool beautiful and
blah blah blah. But if I can just talk about one
that is you have to make sure that the part of your test where
you create the data, right? That you create the inputs that
you're going to pass it to the method, or to the class, you
want to test. That part has to be crystal
clear. Because if you're working in a
very complex information, Mmm, odds are that the complexity of
the test will be in that part. So that part needs to be very
easy to read. It has to be very easy to evolve
because your entities in your domain are evolving all the
time. So it has to be easy to make an
evolution in the production code, without breaking the test
code, and it has to be easy for you to come up with complex data
inputs without having to spend 50 100 lines of codes.
So you have to have lots of utility methods or whatever you
want to call it that help you build data, right?
Test data, Builders the famous. For patterns like this, you need
to invest on them. So, that is my tip number one,
and that's what you need to focus.
Input data has to be Crystal, Clear?
Yeah, I also mentioned reading about this in the I think the
growing's object-oriented software Guided by test.
Right, I think this is also one key technique that is mentioned
in the book. So I really love it because I
have seen in my experience before seeing, you know, lines
of test code which are probably too complicated to understand
and especially the part where you produce this test data,
right? Initially, We will see a lot of
gibberish, probably like, why are these logic here?
But it's actually to produce test data and I think it's a
really, really important for us here to improve.
And if I can tell a personal story Henry, if I can tell a
personal story, you mentioned this growing object-oriented
Guided by test, right? Altered by Steve Freeman,
Annette pranks and I have a very funny story about it because in
2010 there was this Workshop in Terris about test-driven
development, the one and only on the series and I had a paper
there and Treatment. The author of the book was
giving a keynote and so he gave the keynote and it was then
giving a presenting my paper and you, my paper will siphon him a
lot, so that was like, Freeman and price.
Say this in the yearbook at some point, Steve looked at me, you
know, for like, because it was a tenth time.
I said his name on my presentation and then I said,
I'm sorry. I just like her book, very much
and we became friends, right? And when I moved to Europe, then
we were closer to each other and then Steve started to teach one
guest lecture every year in my course in delft.
So he always comes to death every year from my I course and
that book influenced me a lot and he's, in fact, one of the
forward altars of my book, right?
So one of the forward there was written by Steve, so he
influenced me a lot in this book is an amazing book if you have
the chance to read it as well. Well thanks for sharing about
this personal story. I find it really, really
interesting. So I think that's also One Good
Reason. How do you get connected to a
more famous gas of famous authors.
So I think thanks for sharing that.
So, as I mentioned, I have one last question that normally, I
ask for all. Guess which is called the tree
technical leadership, is them. You can think of it just like
advice that you want to give to the listeners here, maybe from
your experience, from the expertise.
What would be some of your wisdom that you can share?
Here, Maurice, you three. Let's do it.
So number one focused on these episodes, right?
So you should Master testing and measuring testing means not only
mastering the tools like ju and it and mockito and whatever to
you use. But also mastering creating good
test cases. And once It's you really become
proficient with it. It just becomes way easier and
way cheaper. So you start having less reasons
not to do it because it just more natural.
How do you get there by practicing?
Right, it's going to hurt at the beginning but the more you do
it, the easier it will get. So practice, testing advice
number two, for software engineers in general is you
should never stop learning, right?
I think it's very easy for us, you know, a lot of us, go
through education and then we just join a company and you
start working as a developer and then send it.
We forget that we need to keep updating ourselves.
I once heard from a developer, you know, hey, I'm here working
for two years in a row and I started to read this book.
I recommend the book to this person, and he said, I started
to read this book in a few. So nice to learn something new,
and I don't know why I wasn't reading books before.
I guess I was just too busy working, and that's life.
That's life, we're all adults, we have responsibilities, it's
very hard. So, I think you have to find
time during her work hours. To be honest, your employee
needs to be on this to upskill yourself.
So read books. Are you interested in?
I don't know, microservices with the book so just keep studying
because there's so much new stuff going on and so many best
practices and something that I feel among developers is that a
lot of them when they go, you know, in a big company and
you're going to have this big discussions to decide what
architecture to take, should we use practice a versus B?
Whatever people have intuition, but they have very little
evidence to explain to people and if a reading books you have
clear ways to explain why you want to go for this or not.
That's so there are so many benefits in studying and putting
studying as part of your daily job.
And so work on this, make sure you study a lot in advice.
Number three will be. You just learned a lot from my
advice number two, because you're studying.
Now, it's time to share your knowledge.
So make sure you also write about what you learn or give
talks at conferences, right? And this is very good for you
because it exposes you to other people, it forces you to
formalize what's in your head. And you have to put in words and
I find that putting in words is the best way for you to really
understand how much understand about something.
And maybe that's because, you know, I have this academic
background. So writing really helps me to
see how much I understand about something.
It's good for you, right? Because you're just getting
better as a person as a developer.
And it's also good for others because I'm pretty sure you have
cool stuff to share and that will be others that are willing
to listen to you. So please share knowledge.
It's as important as learning new knowledge.
So there you go, the tree. Is Henry.
Well, really beautiful. Thanks for sharing that after I
hear about our discussion about testing.
Now, I think sharing is like a form of testing itself, right?
That you actually learn from what you study, right from
reading the books and things like that, may be the best test
case that you write for those learning is actually to share it
either, you know, writing a blog or whatever YouTube video and
things like that. So I'm having this testing
mindset these days after hearing what you explained earlier.
So thank you so much for covering this testing.
I think it's really part of Out of the skill set that developers
should Master more, right? Like, what you mentioned in your
first wisdom, developers need to practice more and be good at
testing because I think that really is the key to produce a
quality software. So for people who love to
connect with you, maybe to talk more about testing, is there a
place where they can reach you on mine?
I'm always on Twitter. So it's at now, this your
nation. So my name and my surname, I
also have a free newsletter related to my book so if you go
to the website effective Dash software Dash testing, You can
subscribe to the newsletter. Those are the best ways to get
in contact with me for sure. LinkedIn, you can also add me on
LinkedIn now. Recent issue find me, feel free
to drop me a message there as well.
Cool, so I'll make sure to put the newsletter as well, so that
people can learn maybe every week, right?
How do you improve your testing experience.
So, thank you so much for your time and ratio.
I hope people enjoy learning about testing and I wish that
your book is also going to be teaching people a lot more about
how to produce the best quality code.
Thank you so much, Henry Ford invite and I hope you like it
everyone. Bye bye.
Thank you for listening to this episode and for staying, right
until the end if you highly enjoyed it.
I would appreciate if you share it with your friends and
colleagues who you think would also benefit from listening to
this episode. And if you are new to the
podcast, make sure to subscribe and leave me your valuable
review and feedback. It helps me a lot in order to
grow this podcast better. You can also find the full show
notes of this conversation on the episode page, at Tech Legion
o.f website, including the full transcript interesting.
Quotes and links to the resources mention from the
conversation. And lastly, make sure to
subscribe to the shows mailing list on package.
You know, dot f to get notified for any future episodes.
Stay tuned for the next technology.
No episode. And until then goodbye.