ROI Is Dead!

Last month I was in Huib Schoots’ testing workshop. He talked about being asked “What’s the ROI of testing?”. To which he replied “I don’t know, what’s the ROI of management?”. We are really obsessed with ROI, aren’t we? We use it a lot as a tool for decision making, but are we using it wisely?

Economics. Take a deep breath.

ROI (Return On Investment) is comprised of costs and value. Cost is easy to measure: salaries, time, tools, licenses, computers, material. But can we put a number of value?

Let’s take a developer as an example. What’s their value?

Is it lines of code? If so, how do different languages compare? Delivered lines of code are hardly a measure.

What about productivity? For example, lines of code per minute? Should we subtract bugs from that? Even if everything works perfectly, are all the features count as value, or just the ones actually used?

Let’s try something else…

Maybe we’ll have better luck with testers. Is it the number of bugs they found? How do we count the ones they didn’t find? What if they documented everything they found, and then we decided to ignore all the non-major bugs, and fix only the major ones. Is all the work around minor bugs waste (non-value)?

You see the pattern – while cost is easy to measure individually, value isn’t. The reason is that as long it’s work in progress, there isn’t any really value yet, regardless on who’s doing the work. We can measure whatever we like, but we’ll be just lying to ourselves. When the product is released, the investment is the sum of the work put into it. It doesn’t matter who put the work in, the tester, the developer, the product manager, the team leader. Only then can we start talking about value.

Now we’re talking. Let’s just calculate the value of the product, at least we’ll have an ROI for that. And to make it easy, we’ll calculate income as value, because it’s easy to measure.

Wait a minute. WHEN you measure actually impacts the income going into the calculation. Is it on release day? Unless you’re Apple, or making a block-buster movie, there’s not much sense in checking the first day. (By the way: Neither Apple or any major studio stops counting on the first day).

Right, so when do you stop? 1 month after release? 6 months? 2 years? 5 years?

Just for the sake of argument, let’s say it’s 2 years. On the 2nd anniversary we evaluate the ROI to be 1.5. We’re happy because the ratio is bigger than one. Now what do I do with this number?

Do you remember what we needed the ROI for? Making decisions before we actually did the work. If we have to wait 2 years for making a decision now, let alone do the project to get the number, it’s kind of missing the point.

Oh wait, there’s more. It seems that the product was successful enough to carry the next product (released 3 years later) forward. Or it was so crappy, that the next product, although it was much better, suffered from its sibling’s reputation. It seems the value of product B is impacted by product A!

Our calculations are ruined!

Damn economics.

ROI is dead as a decision mechanism, because complexity killed it. There are too many things, external or internal, that go into its calculation, and we don’t even know most of them when do the calculation.

There’s a reason we’re talking about safe-to-fail experiments in agile. We acknowledge we don’t know enough, but we’re willing to invest, and sometimes lose,  small amounts of money. We acknowledge complexity and the only way not to lose big, is to lower the cost of failure. That means learning quickly, getting feedback fast.

The primary measure of progress is working software, says the Agile Manifesto. We’re willing to continue investing in things that continue show promise. It’s the only evidence we have, and we should trust it.

Not a made up (sorry, well calculated) number that was applicable to the last product we release.

Either that, or you can wait a few years, and then decide.

Your choice.

3 comments on “ROI Is Dead!”

  1. Ard Kramer Reply

    In addition to your blog I read some interesting new insights on ROI.

    I totaly agree that ROI is dead. there a better ways :
    For example (and it is a follow up on your blog):
    Value of Investment (VOI)
    And VOI means looking at every aspect of value: for yourself because it gives you a good feeling doing your job (e.g using automation so you don’t have to do boring things) or value for the environment, value for the organization in term of reputation, a better network or gaining knowledge
    All kinds of thing bigger then ROI
    And above VOI stand Return on environment in terms of prosperity, planet and people.

    Does it help you looking futher than ROI?

    • Gil Zilberfeld Reply

      Thanks Ard!

      Everything you say is correct, and because we’re living in a complex world, it’s hard to arrive at a number, and we can’t know when to measure it anyway. Problem is – we like simple numbers. We continue to look for simple solutions, instead of acknowledging the complex world around us.

      • Ard Kramer Reply

        Hear, Hear Gil, If there were simple solutions, they would already been implemented. But I see as our task be make people aware that value has a much broader meaning as just money as I stated above. Let we be the advocates of this opinion!

Leave A Reply

Your email address will not be published. Required fields are marked *