In the fast-paced world of software development, how do we
how do we really know if the code we're building is reliable?
How do we get that confidence? Today on the Deep Dive, we're
cutting through some of the noise to unpack a really
fundamental concept. Test coverage.
We're looking at insights from software Engineering, a modern
approach by Marco Tulio Valente. Mainly, our mission here is
simple. Get a real grip on what test
coverage means, why it matters for soccer quality, and you know
how to think about using it practically.
Exactly. And understanding metrics like
test coverage, it's not just academic, it's fundamental for
building software that actually works reliably.
It's how you avoid those, well, those common pitfalls and bugs
that can really sink a project. It's about building that
assurance. OK, let's unpack this then.
When we talk about test coverage, it sounds fairly
straightforward, but what exactly are we measuring?
Is it just like a count of lines?
You're on the right track. Think of it maybe as a check up
for your tests. Test coverage is a metric.
It tells us the percentage of your program's code, the
executable statements that are actually run when your tests
execute. So simpler terms, number of
lines your tests hit divided by the total number of executable
lines. Each statement is basically an
instruction. Right, an instruction the
computer runs. Precisely, the more statements
your test touch, the higher your coverage number goes.
That makes sense. So how do teams actually, you
know, see this? Do they calculate it by hand or
are there? Tools.
Oh definitely. Tools wouldn't do this manually.
Not efficiently anyway. Our source mentions tools like
Jacoco, Java Code coverage. Very common and these tools give
you great visual feedback, like a traffic light system almost
showing what's covered. A traffic light.
OK, tell me more. What do those colors mean?
Green. Yellow.
Red. Yeah, pretty much.
It's quite intuitive. Green means good.
Those lines are covered by your test yellow background that
flags a branch statement. Think of an if statement.
Yellow means maybe only one path got tested.
Like only the true path maybe? Exactly or only the false?
It's partial coverage, a warning sign, and then red.
That's the critical one. Red means that code wasn't
touched by any tests. That's a definite red flag.
That visual feedback sounds incredibly useful for spotting
gaps quickly. The source had the stack test
example right where they got 100% coverage because the tests
hit everything executable. But what happens when tests are
missing? What's the immediate impact?
That's exactly the point. The metric shows a gap
immediately. The source shows if you leave
out a key test like checking what happens when you pop an
empty stack, maybe called test empty stack exception.
Well, your coverage drops instantly and their example it
went from 100% down to was it 92.9% it's.
From one missing test. Because suddenly a few
instructions weren't executed by any test.
It went from 56 total instructions down to only 52
being covered. Shows a direct link.
OK, so we know what it is, how it's measured, how tools show
it. But this leads to the big
question, right? For you, the listener, building
software or just curious, is there a magic number, a perfect
percentage we should aim for like 100% all the time?
The $1,000,000 question and the short answer, no, there's no
universal target. It really varies hugely project
to project. I mean think about it, testing
software for a pacemaker versus testing a simple website.
Completely different needs. Different stakes.
Exactly. Complexity, how critical the
project is, even the team's attitude towards risk.
It all factors in and going for 100% it isn't always the best
goal. Sometimes it's not even
devisable. You often have really simple
methods like basic getters and setters just setting or getting
a value. Right, boilerplate code almost.
Pretty much testing those exhaustively.
It can be overkill, a waste of effort, plus some things are
just inherently harder to test. Well, user interfaces, complex
interactions, anything asynchronous.
Where timing messes everything up.
Yeah, forcing 100% coverage there can lead to really brittle
tests or just massive effort for tiny gains, diminishing returns.
OK, that makes perfect sense. So chasing a rigid number is
out. What should teams do instead?
How do you approach this strategically?
The better approach is usually to watch the trend.
Monitor how your coverage changes over time.
Is it going up, staying stable or oh is it slowly dropping?
That tells you about the teams commitment to testing O it's.
More about the direction than the destination.
In a way, yes, But critically, you also need to look closely at
what isn't covered. Those red lines we talked about,
you need to assess them. Are they non critical, genuinely
hard to test? Or is it just an oversight?
It's about making conscious decisions, not just hitting a
number. That's a much more nuanced way
to look at it, but are there like general rules of thumb,
industry benchmarks maybe? There are some general
observations, yeah. Teams that really focus on
testing often land somewhere around 70% coverage.
OK, 70% and. If you see coverage dip below
50%, well, they usually raises eyebrows.
Suggest large parts of the system aren't tested.
That seems low. It is concerning, yeah.
And even with really rigorous methods like Test Driven
Development, TDD coverage often gets above 90%, but it rarely
actually hits 100%. There's usually that last bit
where the cost outweighs the benefit.
Right, that point of diminishing returns again.
Now this is interesting. What about the big players like
Google? Do they aim for 100% with all
their resources? That's a great question.
We actually have some data on that.
Google shared stats back in 2014 at a developer conference.
Their median statement coverage across systems it was 78%. 78,
not 100. Nope, and the recommended target
for most systems was 85%, but even that wasn't a hard rule.
Wow OK. They also mentioned it varied by
language. C++ projects average a bit under
60%, Python a bit over 80%. So even Google doesn't mandate a
single number. Exactly.
The take away seems to be monitor consistently.
Focus on critical areas, but don't obsess over hitting an
arbitrary percentage everywhere. That's really insightful.
OK. So we've focused heavily on
statement coverage, counting lines, C0, you called it.
Are there other ways to measure coverage?
Other angles? Absolutely.
Statement coverage where C0 is the most common, Definitely, but
there are other useful definitions.
There's function coverage. Simpler idea, what percentage of
your actual functions or methods got called by any test?
Just did we run this function at all?
Pretty much the basic check. Then there's a function call
coverage that's a bit more specific.
It looks at the places in your code where one function calls
another. It measures what percentage of
those specific calls were exercised by tests.
So not just if the function ran, but if we tested the part where
it gets called from somewhere else.
Exactly. Testing the connections if you
like. And another really important
one, branch coverage, also called C1 coverage.
This looks at decision points like an if statement creates 2
branches, right? One for true, one for false.
Branch coverage measures the percentage of all possible
branches or paths through your code that the tests actually
took. Is generally seen as stricter
than just statement coverage. Stricter because you might hit
all the lines but miss a specific decision path.
Recisely you could execute every line in an if else block but
only ever test the if part, never the else statement.
Coverage might be 100%, but branch coverage would show the
gap. Can you give a quick example
just to make that distinction clearer?
Sure, Think about a simple function to get the absolute
value of a number like absecs inside it probably has something
like if f0 then it changes the sign.
Makes sense? Now, if your only test is ABS
one, that test runs through all the lines, right?
It checks the condition, finds it true, it changes the sign.
So 100% statement coverage. Correct, 100% statement
coverage, but branch coverage only 50% because you only tested
the true path of that. If you never tested what happens
when X is not less than 0? I see, so you'd need another
test. Right.
No, you need to test like ABS one to exercise the false path.
Then you'd have 100% branch coverage too.
It forces you to test both sides of the decision.
That clarifies it perfectly. So understanding test coverage,
it's really not just about the number itself, it's about making
smart choices. Exactly.
It's about knowing what your tests are actually doing, what
paths they're taking, and using that information to build more
robust, more reliable software. It's about informed confidence.
Well, thank you for joining us on this deep dive.