The Test Waste Ratio

Remember the days when bugs were just part of life? I mean a big part.

If you recall those dark ages, we’ve spent most of our time fixing bugs and introducing others. In our spare time, we got to work on new stuff.

Today is different, there are a lot less bugs. So we get to work more on the new stuff, right?

Or have we found a new replacement for them?

Test Maintenance Is the New Bug

You wake up one morning, and you discover that you spend 25% on maintaining tests. True story.

This doesn’t happen in one day, you feel more work on tests creeping up, and you feel it’s just natural. Part of the work.

Then one day you actually look at the numbers. And someone needs to pick you off the floor.

Nobody Writes Bad Tests Intentionally

So, where do these tests come from?

These tests are the creaky windows in the attic. One gust of wind, and you hear a loud bang. You need to go up there and fix them. Again.

Much like those windows, they don’t give you much protection. Since they break once in a while, you get accustomed to them creaking and banging, you don’t notice them anymore. Which is bad, since sometimes they let some real menaces into the house. Like real bugs. But you won’t know because you’re sure it’s the ol’ window moving in its place.

So until you face reality, it’s not clear how much they cost us. It could amount to a lot.

Back to Test World

Let’s call these bad tests the “wasteful” tests.

Which begs the question: If those cause waste, when are tests valuable? What exactly is this value?

This is hard to quantify. A test is valuable if it catches a bug. It gets points for that. So, ridiculous as it may sound, the time spent on fixing bugs the tests found is an indicator to the value of the test. We can call those tests the “valuable” tests.

Now, here’s the Test Waste ratio:


The denominator is the total time spent due to tests breaking, the valuable and wasteful. The numerator is the wasteful work.

If your wasted time is large, the ratio gets closer to 1. If it’s small, it gets closer to zero.

You want to get a lower ratio as possible. It’s not really up to you, but it’s an indicator. And those numbers can actually be measured.

So Much For Theory

What do you think? Does it make sense? Is it helpful in any way? Do I stand a chance for a Nobel prize?

Let me know.

PS. Next week, I’m going to be at Belgium Testing Days. Apart from the Spaceship I’m smuggling through customs, I’m going to be part of an interactive session, talking about the ROI of test automation.

Share Button

3 comments on “The Test Waste Ratio”

  1. Anonymous

    Not sure if ‘a test is valuable if it finds bugs’ is the correct statement. A test is valuable if it gives you increased confidence your code is working. e.g. you wrote a test to cover the case of a bug once found. this test will probably not fail again, but it gives confidence that you are not going to fall for the same errors you did

  2. Slavoj Písek

    Gil, according to my own experience any test discovers the most defects at the
    time of its writing. Once finished and deployed, the test can find no new bug.
    But it still can discover many regressions. That makes the test valuable even
    if it hasn’t found a bug for a long time because you never know what is
    waiting behind next corner. Besides your definition of “waste tests” sounds
    awkward to me. What do you mean by “break once in a while”? Do you talk about
    the tests that fail randomly even if nothing has changed in neither
    application under the test nor the test itself? Such tests are not waste at
    all, they are much much worse. They are dangerous because they harm trust to
    the whole set of your tests and its results. If it is not possible to fix such
    a test quickly, it is necessary to replace it with something more reliable.

    On the other hand, as your application evolves and changes, your tests can be
    affected by these changes very often. It is quite common that a test breaks
    because the tested application has changed and the test must be adjusted to
    the new situation. Yes, it is annoyng, boring and very very expensive work,
    but it is not waste! It is not waste because you should always keep your lamps
    burning. And referring to your last question, I am afraid that this year’s
    Nobel Prize will claim someone else.

  3. Gil Zilberfeld

    Guys, thanks for the comments!

    Tests give us confidence. True, but not measurable, and very subjective.
    Unless we do TDD, tests are the most valuable when they find regressions. Those are either bugs, which we need to fix, or true requirements that require new tests. I suggest this is time spent that benefits the project.

    The wasteful tests can undermine the trust. But it’s not like we know that would happen when we wrote them, right? But we need to take care of them, at the cost of moving forward with the project. Thus, waste.

    If someone else gets the Nobel prize, they have my blessing.

Comments are closed.