When a codebase shows 100% coverage, every line has been run by a test, so it seems safe to assume the code is properly checked. But code coverage only measures whether a line was executed during a test run, not whether that test would identify if the logic behind that line changed.
Mutation testing addresses this gap directly, by deliberately introducing small bugs into a codebase and checking whether the existing tests catch them. The following blog sets out why this distinction matters, how mutation testing works in practice and what it revealed when run against both a small example and a much larger, real production codebase.
Why code coverage isn't enough
Code coverage is the percentage of code touched by a unit test. Tooling tracks which lines are hit while tests run and tallies this up. The idea is to use this to target areas that remain uncovered, particularly high-risk areas or ones that are hard to reach through automated testing.
The problem is that a high coverage percentage can create a false sense of security. We conducted a demonstration to test this: starting from an example project with 100% test coverage, a single line of code was deliberately changed to introduce a bug, in this case swapping which item in a set of products was treated as the cheapest for a discount calculation. Despite this change, every existing unit test still passed. In other words, a project with 100% code coverage did not have 100% protection against this kind of fault. At that point, only code review or manual and automation testing downstream would have any chance of catching it.
What mutation testing actually measures
Mutation testing works by creating many altered versions of a codebase, known as mutants, each with a small change introduced. A mutant is considered "killed" if any unit test fails as a result of that change, and "survived" if all tests continue to pass despite the change.
Rather than asking how much code has been executed by tests, mutation testing asks how much of that code, if changed incorrectly, would be caught by a failing test. A mutation score close to 100% means that changing almost any line of code causes a test to fail, which is a more meaningful measure of test quality than coverage alone.
Running mutation testing in .NET
Mutation testing has existed for some time, though it has been available in .NET for a shorter period. The tool used here was Stryker.NET, installed and run as a .NET tool. Where unit tests span multiple projects, the tool needs to be pointed at the specific project reference containing the code under test, in this case a domain layer project.
Stryker.NET works by scanning code for specific patterns and substituting them with alternatives: an addition might be swapped for a subtraction, a greater-than comparison for a less-than, and, relevant to the example below, a LINQ OrderBy call for OrderByDescending, or vice versa. Mutants that fail to compile are discarded.
The tool produces an HTML report by default, showing which mutants were created, which were killed and which survived, alongside the resulting mutation score. It also supports JSON and Markdown output, which could be used for build pipeline reporting or posting a summary to a pull request.
A worked example
Running the tool against the small example project generated seven mutants. Six were killed and one survived, giving a mutation score of 6/7, or 85.71%. The surviving mutant pinpointed the deliberately introduced bug: the swap between OrderBy and OrderByDescending in the discount calculation, which none of the existing tests were sensitive enough to detect.
To fix this, an additional test case was added, using products with three distinct prices rather than identical ones, so that the ordering of items by price would affect the result. After removing the deliberately introduced bug and adding this test, the mutation score on a second run reached 100%, with all seven mutants killed, including the one that had previously survived.
A 100% mutation score is highly unlikely to be realistic on a larger, real-world codebase. It was only achievable here because the example was narrow and self-contained, testing a single, well-defined piece of logic.
Testing at scale
We also ran the tool against one of our larger internal production codebases, which has around 5,000 unit tests. The initial full run returned a mutation score of around 40%. In other words, the existing tests caught around four in ten of the deliberate changes made to the code.
This score isn't the same as code coverage. A low score doesn't automatically mean more tests are needed. It can just as easily mean some existing tests aren't really checking anything, and those are better improved or removed than simply added to.
So rather than treating the score as a target to push towards 100%, we recommend using it as a map: it shows where tests are doing real work, where they add runtime, and where extra effort would pay off.
What makes a unit test valuable
A recurring theme in these tests was that a unit test which never fails, regardless of what changes are made to the underlying code, is not adding much value. The test may simply assert that an operation was successful in a general sense, without checking the specific outcome the code is meant to produce. This kind of test can catch runtime errors, but it won't catch a business logic mistake.
Conclusion
Mutation testing offers a more representative view of test suite quality than code coverage alone, because it tests whether unit tests would catch a real fault, rather than whether a line of code was executed.
The trade-off is runtime: at the scale of a codebase with tens of thousands of mutants, a full mutation testing run can take hours.
Since this test was carried out, Stryker.NET has introduced a preview test runner built on the Microsoft Testing Platform, designed to remove the repeated start-up overhead that slows down large mutation testing runs. This may close the gap for large codebases, and it's a sign that runtime constraint is likely to ease over time rather than being a fixed limitation of the tool. The new runner is still maturing, though, so it's worth checking its current status before relying on it.
