Many metrics have been proposed to evaluate the quality of a piece of code: nesting depth, cyclomatic complexity, relational cohesion… although my favorite remains WTFs per Minute.
Testing also plays a big role in software quality and is therefore also being measured. One of the most popular unit testing metrics is code coverage, which evaluates the fraction of lines of code that have been executed at least once during testing.
Code coverage is not a bad metric per se, but reaching 100% is not a guarantee that your code is bug-free. Far from it. So, how could we do better?
I’m sure countless Ph.D. theses have been written on this question, but I’d like to propose another idea in the vein of WTFs per Minute. Something not totally serious but not totally ridiculous either.
I call this approach Recreational Bug Seeding and it goes as such…
On a regular basis, you invite your software engineers to do a bug seeding session. Something like a hackathon, but where the goal is to introduce bugs.
During a session, software engineers are encouraged to go through the code — not the tests! — and to modify it in any way they want. Anything goes. They can add a character to a regular expression, change the start index of a loop, return early from a function, invert the clauses in an if-else. As long as the change should break things, it’s valid.
Once a change has been made, the bug seeder runs the suite of unit tests. If they all pass then the dev scores a point and brags about it on slack.
The total number of seeded bugs gives you an interesting indication of the ingenuity of your software engineers, but — assuming this ingenuity is constant over time — it also gives you a pretty good indication of how thorough your tests are.
If devs don’t put too much effort into thinking of weird things that could go wrong, then it’s going to be fairly easy for bug seeders to score points.
One obvious drawback of that method is that it costs engineering time. However, if done well, it might be fun engineering time — with cakes and all — which could have a positive impact on your company culture. As an engineer, I know I would have liked a bug seeding session every now and then. 😀
Waverly is still too small to do this, but I’d be really curious to know if anyone has tried something similar in a larger company. Please reach out if you have!
Update: Many of you pointed me to an automated version of this idea, mutation testing. Thanks!
Phil Beaudoin 