ROI Is Dead!

Last month I was in Huib Schoots’ workshop. He talked about being asked “What’s the ROI of testing?”. To which he replied “I don’t know, what’s the ROI of management?”. We are really obsessed with ROI, aren’t we? We use it a lot as a tool for decision making, but are we using it wisely?

ROI (Return On Investment) is comprised of costs and value. Cost is easy to measure: salaries, time, tools, licenses, computers, material. But can we put a number of value?

Let’s take a developer as an example. What’s their value?

Is it lines of code? If so, how do different languages compare? Delivered lines of code are hardly a measure.

What about productivity? For example, lines of code per minute? Should we subtract bugs from that? Even if everything works perfectly, are all the features count as value, or just the ones actually used?

Let’s try something else…

Maybe we’ll have better luck with testers. Is it the number of bugs they found? How do we count the ones they didn’t find? What if they documented everything they found, and then we decided to ignore all the non-major bugs, and fix only the major ones. Is all the work around minor bugs waste (non-value)?

You see the pattern – while cost is easy to measure individually, value isn’t. The reason is that as long it’s work in progress, there isn’t any really value yet, regardless on who’s doing the work. We can measure whatever we like, but we’ll be just lying to ourselves. When the product is released, the investment is the sum of the work put into it. It doesn’t matter who put the work in, the tester, the developer, the product manager, the team leader. Only then can we start talking about value.

Now we’re talking. Let’s just calculate the value of the product, at least we’ll have an ROI for that. And to make it easy, we’ll calculate income as value, because it’s easy to measure.

Wait a minute. WHEN you measure actually impacts the income going into the calculation. Is it on release day? Unless you’re Apple, or making a block-buster movie, there’s not much sense in checking the first day. (By the way: Neither Apple or any major studio stops counting on the first day).

Right, so when do you stop? 1 month after release? 6 months? 2 years? 5 years?

Just for the sake of argument, let’s say it’s 2 years. On the 2nd anniversary we evaluate the ROI to be 1.5. We’re happy because the ratio is bigger than one. Now what do I do with this number?

Do you remember what we needed the ROI for? Making decisions before we actually did the work. If we have to wait 2 years for making a decision now, let alone do the project to get the number, it’s kind of missing the point.

Oh wait, there’s more. It seems that the product was successful enough to carry the next product (released 3 years later) forward. Or it was so crappy, that the next product, although it was much better, suffered from its sibling’s reputation. It seems the value of product B is impacted by product A!

Our calculations are ruined!

ROI is dead as a decision mechanism, because complexity killed it. There are too many things, external or internal, that go into its calculation, and we don’t even know most of them when do the calculation.

There’s a reason we’re talking about safe-to-fail experiments in agile. We acknowledge we don’t know enough, but we’re willing to invest, and sometimes lose,  small amounts of money. We acknowledge complexity and the only way not to lose big, is to lower the cost of failure. That means learning quickly, getting feedback fast.

The primary measure of progress is working software, says the Agile Manifesto. We’re willing to continue investing in things that continue show promise. It’s the only evidence we have, and we should trust it.

Not a made up (sorry, well calculated) number that was applicable to the last product we release.

Either that, or you can wait a few years, and then decide.

Your choice.

The New Agile- Certifications

Last time we looked at how things came to be. How things converged around a group of software developers in a ski resort: There were actual experiments in the field with reported successes. There was a funnel for communication to spread those ideas. And now there was a joint vision and a name.

The agile manifesto would still remain a nice website with a very modest cult, unless for Ken Schwaber’s business savvy. He decide that he’s not just going to teach scrum. There were going to be certificates involved. This is how the pyramid scheme of the CST (Certified Scrum Trainer) and the CSM (Certified Scrum Master) and derivatives started. Many people in the agile world (that do not belong to any of the training organizations, or make money of them) usually look with disdain at those certificates. “How can one become a Scrum Master in 3 days?” We ask. The simple answer is, of course, no one can become MASTER of any kind in 3 days.

We can say what we want about certificates, it is because of them that we practice agile today in many organizations. Because much like agile inspects and adapts, so did scrum. It adapted to how business hired and trained people – through certifications. This is not new – that’s how guilds were created long ago, and scrum is a sort of guild.

If Lean started it all (even before having that very cool, marketable name), and scrum led the way, the next evolution was David Anderson’s kanban. It too grew to be  a training operation – the Lean-Kanban University. While scrum and kanban are not dealing with the exact same thing, both fall under the same “process improvement” blanket. There’s no escaping the question “should we do scrum or kanban”, although both sides will say they solve a completely different set of problems.

Both scrum descendants (after the divorce between Ken Schwaber and Jeff Sutherland) continue on their own, the Scrum Guide (the ultimate body of scrum knowledge) incorporates both Scrum Inc and inputs. There’s also the Scrum Alliance, not affiliated with either father, but a place for all the children to come and play.

As scrum took over the world, managers took notice outside the development team. If development team can move more quickly, why not apply the same principles to bigger product development organizations? In fact, kanban was there first, because that’s what it did – kanban processes came from the Toyota Production System, which after all was a PRODUCTION system, not a software facility.

Scrum at that point did not want to answer that. But where there’s money to be made, certification will follow, and so we got Scott Ambler’s Disciplined Agile Delivery (DAD) and others, but these are small fries compared to Dean Leffingwell’s 800 pound gorilla – Scaled Agile Framework (SAFe).

We’ll talk about SAFe later, but so far it looks as the answer to enterprise agile: the scaling, the scrum and XP basis, along with an assuring cool name. It too offers certificates, and it looks like the current winner in the area.

Certifications will continue to crown the kings of framework, unless there’s going to be a major shift in how organizations hire and train people, which won’t happen anywhere soon. Just as today, organizations “want scrum” because it’s the “best in breed” and everyone doing it, SAFe is probably going to be in the next few years. And where there’s a want, somebody will answer it.

Agile Newspeak

Every time and again, I go back to the “4 ways that agile is declining post” to see if it is still relevant. I don’t get disappointed.

Now that I am a full time agile coach, I’m in a kind of a fix. Agile has crossed the chasm, and scrum won as we all know. So people expect and want Scrum. I usually don’t deliver on that, because scrum is not a solution, it’s a starting point. So what I do is start with some combination of scrum and kanban and try to fit a method to the organization I’m working with. This is of course a long process, with ups and downs, and leads to good results many times.

Then I come to a place that already learned scrum. Ah, you say, if they already do scrum why do they need you?  Good question.

Remember I was talking about agile as snake oil? Ok, it doesn’t really matter if they were actually sold on that oil, or that they felt that they didn’t need coaching any more, and once the coach was not there, they implemented it themselves. Or that the coach was trying to fit it to the current culture or organization. The fact is that they don’t do scrum by the book.

Which is fine, if it works. I’m all for good results, no matter how you get them. Then why would they need a coach?

There used to be a “scrum-but” idea. “We’re doing scrum, but we don’t do stand ups”, and you can replace the different “buts”.  That was a smell – it usually pointed to a conflict between scrum values and the organizational values, where the organization won. Then again, nobody who’s doing scrum successfully is doing vanilla scrum, so a smell can be just that.

Sometimes, however, that’s not a scrum-but. They are doing scrum, there’s no but. In their eyes. Dig in and you’ll see:

  • Product owners are not part of the team. In fact, in a sprint, each team works with multiple product owners.
  • Product owners commit to the customer how many story points the team (who doesn’t know the requirements yet) can deliver
  • The team can estimate all they want, but still need to meet what the POs promised. (If you think about it, estimation is really waste in that case).
  • The team leader is the scrum master.  He needs to manage conflicting requirements and assign them to the different team members.
  • So there’s not a real team, just a bunch a people who happen to have the same manager.
  • Testers are assigned and rotated between teams. But since they are remote, and have their own managers funny things can happen, I’m sure you can imagine.
  • Requirements are not all ready in the beginning of the sprint, so over time things get delayed to the end, and guess what kind of march the team is doing in the late stage of the sprint? It’s even planned ahead.
  • Daily meetings are mostly done, but POs mostly skip them. Which makes sense, because they are busy filling their requirement quota.
  • Finally, there’s also the small fact that people aren’t happy.

They are doing scrum, they’ll tell you. The top management who brought trainers and coaches before to introduce scrum, don’t understand why scrum doesn’t work.

This is not a scrum-but. This is 1984’s Newspeak. This is taking scrum vocabulary and assigning new meaning to it.

It can only end in two ways: Giving another chance to scrum, with coaches, and trainers, and management backing.

Or they can just proclaim that “scrum is snake oil” and wait for the new hot thing.

Testing Economics 101: The Video

This is a must see for everyone on the planet who wasn’t in my session at NDC London this week. The others are invited to watch again.

Watch it on the NDC Videos site. You can also get the slides here.

Testing Economics 101–The Slides

Had a great time today at NDC London. Now to enjoy the rest of it. Here are  the slides.

The New Agile–The beginning

Frequent readers of this blog know I’m a big proponent of agile, with a small ”a”. I tend to scoff at new big ideas that seem to jump out of a marketing committee.

Truth is, sometimes these are good ideas. Sometimes they are first versions of trials and errors. When we look at them at the current stage they may seem ridiculous, but if someone get exposed to them, and continues the push, we may get something wonderful. Much like quality, we get the real judgment after the fact, sometimes years later.

Therefore, I’ve decided to put some of my cynicism aside, and try to look at things in the agile world that have happened since the agile manifesto - critically. The collection of the posts called “The New Agile” stood behind the making of my “The New Agile” presentation. I presented different topics in the agile world. Some of the information will be accompanied with my observations, and some will be just an encouragement to you to study more if they interest you. There will be recommendation, but especially around what interests me. It may not be the same for everyone.

Before we start though, we need to look back at how we got here. And like many examples,  I begin by going back to the agile manifesto.

“We are uncovering better ways of developing software, by doing it and helping others do it”.

17 smart people who went skiing in Snowbird, Utah in 2001 applied science to software development in one sentence. The idea is that we don’t know everything, but we continue to try, learn, fail and try again. The idea is that if more people do it, we’ll increase our learning. And the idea that software development is not a single process, but different paths, some we didn’t even discover yet.

Great ideas.

Those are still true today, and therefore what we’ve already learned may not be the best way to develop software. It may seem, for example, that after almost 20 years of TDD, we couldn’t find anything better, and therefore it is a best practice. But we may find something better tomorrow.

Those things didn’t start at 2001. This was when they just go a name.

If we go back in history, we can see different people learning and applying lean principles, before they got their name. Take W. Edwards Deming, for example. After WWII, he went to Japan and formalized the basis of agile: The  PDCA cycle: Plan, do, check and adjust. It’s the basis of continuous learning and improvement. Deming put in place the lean foundation that Toyota would base their manufacturing processes on. It may come as a shock to you after years of saying “developing software is not like building bridges” that it really is, if you’re building the bridges correctly.

Ideas like a line-worker that can stop the line, is the origin of continuous integration. Don’t believe me? In a working CI, what happens when the build breaks? Fixing it is the biggest priority. And it’s not a managerial decree. It’s a culture where the developers stop what they are doing and getting the build on track again, and their decisions is supported by management. Just like when a line worker stops the production line. Quality is first and the culture supports it.

So we had those ideas floating around, but because until the 90s we didn’t have an actual internet, ideas were not exchanged in the rate they do today. Communication was still limited. So when people like Ken Schwaber and Jeff Sutherland started working on scrum, Kent Beck and Ron Jeffries on eXtreme Programming, Alistair Cockborn on Crystal, they were “just” working with teams. The different names came to support the consultants marketing efforts, but at its base - it was “just” working with teams.

The way they exchanged ideas was the old way: papers, books and conferences. You should remember that there weren’t many conferences about those things then. There were mostly programmer conferences  (nobody dared thinking about conferences for testers then). There were business management conferences, but they weren’t really interested with what the developers were playing with.

And then the internet happened. Much like with literacy and then print, the internet gave knowledge ways to explode and reach exponentially larger audience. Finally ideas “scaled’. They were compared to each other, discussed, confronted, and applied. Some successfully, some poorly, some deviously.

A few years after the agile manifesto, the business world was ready to stop and hear about some development methodology with a name that came from rugby.

In most cases, we’d say the rest is history. But this was only the beginning.

Upcoming Events

I’m a busy guy. That’s why I’m posting “Upcoming Events” and not real content. Don’t worry because that’s coming too.

In the meantime, after Agile Testing Days, here’s what’s coming (that I know of):

image_thumb6 Next week, Dec 3rd, I’ll be at NDC London, talking about “Testing Economics 101”. Very exciting conference, and lots of insights that you may get a glimpse from my book “Everyday Unit Testing”.

Technology, Business and Product Management Israel

Dec 14th, I’m joining a panel on the Agile Product Management conference. This is a half-day meet-up of product people, and the main topic is Product Owner vs Product Manager – what are the differences, the commonalities, and is it possible to be that super person.  Register here(free).
  Dec 15th, I’m doing a workshop on Introduction on Unit Testing and TDD. This is a full day learning the basics of unit testing, with a glimpse towards new and exciting stuff. Register here (Hebrew).
On Jan 27th, as part of the Agile Practitioners 2015 conference (that I also help organize) Lior Friedman and I will do a whole day workshop on “Advanced Agile Programming” topics. This workshop is mostly coding, and will go through a list of programming methods, refactoring techniques, introducing code smells, and basically make you better programmers. Register here.
The big event, Agile Practitioners 2015, is the on Jan 28th. Apart of caring to attendees the only way I know how,  I’ll do the “To Estimate or #NoEstimate” talk. After that I’m going to let loose, and enjoy the rest of the conference. Register here.

Yes, there are more things brewing up like coffee in a brewing thing.

And, some more content, and that book, and work, and more stuff.


Related Posts Plugin for WordPress, Blogger...
Twitter Delicious Facebook Digg Stumbleupon More