Last month I was in Huib Schoots’ testing workshop. He talked about being asked “What’s the ROI of testing?”. To which he replied “I don’t know, what’s the ROI of management?”. We are really obsessed with ROI, aren’t we? We use it a lot as a tool for decision making, but are we using it wisely?

Economics. Take a deep breath.

ROI (Return On Investment) is comprised of costs and value. Cost is easy to measure: salaries, time, tools, licenses, computers, material. But can we put a number of value?

Let’s take a developer as an example. What’s their value?

Is it lines of code? If so, how do different languages compare? Delivered lines of code are hardly a measure.

What about productivity? For example, lines of code per minute? Should we subtract bugs from that? Even if everything works perfectly, are all the features count as value, or just the ones actually used?

Let’s try something else…

Maybe we’ll have better luck with testers. Is it the number of bugs they found? How do we count the ones they didn’t find? What if they documented everything they found, and then we decided to ignore all the non-major bugs, and fix only the major ones. Is all the work around minor bugs waste (non-value)?

You see the pattern – while cost is easy to measure individually, value isn’t. The reason is that as long it’s work in progress, there isn’t any really value yet, regardless on who’s doing the work. We can measure whatever we like, but we’ll be just lying to ourselves. When the product is released, the investment is the sum of the work put into it. It doesn’t matter who put the work in, the tester, the developer, the product manager, the team leader. Only then can we start talking about value.

Now we’re talking. Let’s just calculate the value of the product, at least we’ll have an ROI for that. And to make it easy, we’ll calculate income as value, because it’s easy to measure.

Wait a minute. WHEN you measure actually impacts the income going into the calculation. Is it on release day? Unless you’re Apple, or making a block-buster movie, there’s not much sense in checking the first day. (By the way: Neither Apple or any major studio stops counting on the first day).

Right, so when do you stop? 1 month after release? 6 months? 2 years? 5 years?

Just for the sake of argument, let’s say it’s 2 years. On the 2nd anniversary we evaluate the ROI to be 1.5. We’re happy because the ratio is bigger than one. Now what do I do with this number?

Do you remember what we needed the ROI for? Making decisions before we actually did the work. If we have to wait 2 years for making a decision now, let alone do the project to get the number, it’s kind of missing the point.

Oh wait, there’s more. It seems that the product was successful enough to carry the next product (released 3 years later) forward. Or it was so crappy, that the next product, although it was much better, suffered from its sibling’s reputation. It seems the value of product B is impacted by product A!

Our calculations are ruined!

Damn economics.

ROI is dead as a decision mechanism, because complexity killed it. There are too many things, external or internal, that go into its calculation, and we don’t even know most of them when do the calculation.

There’s a reason we’re talking about safe-to-fail experiments in agile. We acknowledge we don’t know enough, but we’re willing to invest, and sometimes lose,  small amounts of money. We acknowledge complexity and the only way not to lose big, is to lower the cost of failure. That means learning quickly, getting feedback fast.

The primary measure of progress is working software, says the Agile Manifesto. We’re willing to continue investing in things that continue show promise. It’s the only evidence we have, and we should trust it.

Not a made up (sorry, well calculated) number that was applicable to the last product we release.

Either that, or you can wait a few years, and then decide.

Your choice.