What Is A Unit Test?

What make unit tests different than other tests? They are full of FAIL.

Going to the Wikipedia  definition, you’ll get a vague and unhelpful description, and to summarize: It tests a small piece of code.

In  what language? What is small? And why does that matter?

I feel that many times in software, we’d rather concentrate on the mechanics, rather than on the goal. For example, we talk about mocking (how we do it) while we actually want isolation (which is what we need for our tested code).

First, let’s take a look at our goals for a good unit test:

  1. Tell you quickly when there’s a real problem
  2. Get you as quickly as possible from “Found a problem” to “Fixed a problem”

Let’s take a closer look, using FAIL.

Functionality: A unit test is a sensor, telling us if a former working functionality no longer works. While feedback is the requirement from every kind of test, the key thing is functionality, and in code terms - logic: if-thens, try-catches and workflows inside the code.

Accuracy: A unit test should fail for only two reasons: We broke something and should fix the code (A bug), or we broke something and should fix the test (A changed requirement). In both cases, we’re going to do valuable work. When is it not valuable? Example: If the test checked internal implementation, and we changed the implementation but not the functionality, this does not count as a real problem. The code still does what it was meant to, functionality didn’t change. But now we need to fix the test, which is waste. We don’t like waste.

Instant: A unit test runs quickly. A suite of hundreds and thousands unit tests runs in a few seconds or minutes. The success of applying a fine-grain sensor array relies on quickness in scale. This usually translates in the tested code to be short and isolated.

Locator: When there’s a problem, we need to fix it quickly. Part of it is testing a small amount of code. Then, there’s more we can do in the test to help us solve the problem. Yet, we need to think outside the context of writing the test, though. Someone else may break it, in a year or more, after we’ve moved companies twice. In other words, we’re leaving a paper trail to locate the specific problem for someone else. To do that we use accurate naming, readable test code, testing small portion of the code in a specific scenario, isolation from any undetermined or non-specific dependency, and lots of other tricks that will help our unfortunate developer achieve a quick fix.

Notice that none of these attributes are about the experience of writing a test. It’s about getting the most value out of it after it’s there.

The only value you get while writing a test, is when the code is not there yet. That’s right, in TDD. In that case, you get all of the above, plus insight about the design and safe incremental progress.

All other kinds of tests, which are also valuable, don’t have all these traits: Integration tests don’t locate the problematic code. UI tests are brittle. Full system tests are slow. Non-functional tests just give you feedback, and exploratory testing is not about functional correctness, but rather on business value (at least should be).

If your test passes the FAIL test, then it is a unit test.

image source

The Real Story of Arnie and Sly

You think that Arnold and Sylvester movie careers were head to head?

Let me tell you the story of what happened before that.

<Dissolve Effect>

Arnold and Sylvester had a struggling software development business.

Arnold wanted better code design. Because he was after better design, he chose TDD, with all the design goodness it brings. TDD usually works better on new code, rather than on legacy code. On his journey, Arnold got better at TDD and improved both his tests and production code design, although he left the legacy code untested. The new code was a shining example of SOLID code.

Sylvester wanted better quality. He understood that he needed to fix the current code base, and that meant writing tests for his legacy code, as well as the new code. This was hard, as is the way with testing legacy code (although the pain was alleviated by some cool tools). Progress was slower, since the legacy code fought back. But Sylvester persevered, and got some quality in, and kept the bugs out.

Both Arnie and Sly’s journeys were tough. They got better at what they did. Testing is a set of skills, and practice made perfect. Not perfect, really, but improved.

As they made progress, the two journeys have started to overlap.

As Arnold TDDed his new code, he put enough safeguards in, and had enough confidence to start operating on the old legacy code. As design got better, it was easier to put tests around the legacy code, and less bugs reared their ugly heads.

Sylvester was no slouch either. With much less bugs to worry about, he had enough time to improve the design on of the code, making it legacy no more.

In the end, both have arrived at better design and better quality.

As a bonus they also got better development speed, because high quality meant less rework, and good design meant quick modification for new features.

When they retired their successful software business, Arnie and Sly started shooting movies and people for a living.

And the moral is…

Different people expect different things from the process of implementing automated testing. Sometimes, it feels that we’re sacrificing something by concentrating on another. We have that feeling, since that’s how life is, a balance between goals. The cool thing about software development, usually when you achieve one of these goals, you get the others for free. After mastering them for years, true.

Arnie and Sly were had successful movies, because they gave their customers what they wanted, with good design (read: special effects), good quality (read: blockbusters one after another) and high development speed (read: movies year after year).

And they lived happily ever after.

Good Vs Evil

Posts about agile dying or becoming irrelevant come and go. God knows I’ve contributed to that movement.

The latest wave started with Agile Manifesto signatory Dave Thomas, with “Agile is dead, long live Agility”. The discussion later continued on Twitter and the XP mailing list under the title “Taking agile back”.

Taking it from whom?

From the evil consultants who are selling “agile” to unsuspecting business managers. Agile was a developer thing, and now it’s out there in the hands of the uninitiated.

However, if the consultants are not so evil (there’s this kind too), It’s a different story. When the word Agile became mainstream, it helped promote the ideas of better development practices.

Both sides are right. I’ve written about it multiple times – Consultants that sell the methodology du jour, and move on to the new hotness, leaving the organization to deal with the aftermath. They do exist, and the current ambassador is SAFe.

But the good consultants exist as well. They make their living by helping organizations. They do that because one VP just heard about agile, and although he heard “faster development”, it was enough to open the door.

Who is right?

Unfortunately, we don’t live in a galaxy far far away, where good and evil roles are pre-defined and there’s a clear-cut difference. Both exist, and the “agile” (much like the Force) is strong in both.

But there’s no taking “agile” back, unfortunately.

Yes, it’s not an English noun. And yes, it is used as one, inappropriately.

We can call it whatever we want, and describe it in different ways, but agile is not a word anymore.

Agile is a brand

The discussion therefore is moot. Once “Agile” has crossed the chasm, replacing it with another word that has another meaning, and pushing it as the “right way” will not work the way you think it is. In fact, what SAFe is doing is just that. If you’re selling it you’re doing a marvelous job. If you want to dissuade people from doing it, you already lost, since you’re using the brand SAFe in that sentence. And it goes back to what people hear, rather than what we want them to understand.

This is the hard part, for us, who want the agile way of old (XP?). Or rather, we want people to understand it the same way we do.

Some people will understand, but they are the minority.

If this new word is to succeeds at all, it needs to follow scrum and SAFe in what they do best: speak the business language, not developish.

If by some miracle it does succeed, we won’t be happy again, because life will repeat itself.

It’s really hard to please us, we agility folk.

It does sound weird, doesn’t it? Let’s stick with agile and try to make the best of it.

 

Image source: http://southernstingray.deviantart.com/art/Good-Vs-Evil-Stencils-158439358

The Test Waste Ratio

Remember the days when bugs were just part of life? I mean a big part.

If you recall those dark ages, we’ve spent most of our time fixing bugs and introducing others. In our spare time, we got to work on new stuff.

Today is different, there are a lot less bugs. So we get to work more on the new stuff, right?

Or have we found a new replacement for them?

Test Maintenance Is the New Bug

You wake up one morning, and you discover that you spend 25% on maintaining tests. True story.

This doesn’t happen in one day, you feel more work on tests creeping up, and you feel it’s just natural. Part of the work.

Then one day you actually look at the numbers. And someone needs to pick you off the floor.

Nobody Writes Bad Tests Intentionally

So, where do these tests come from?

These tests are the creaky windows in the attic. One gust of wind, and you hear a loud bang. You need to go up there and fix them. Again.

Much like those windows, they don’t give you much protection. Since they break once in a while, you get accustomed to them creaking and banging, you don’t notice them anymore. Which is bad, since sometimes they let some real menaces into the house. Like real bugs. But you won’t know because you’re sure it’s the ol’ window moving in its place.

So until you face reality, it’s not clear how much they cost us. It could amount to a lot.

Back to Test World

Let’s call these bad tests the “wasteful” tests.

Which begs the question: If those cause waste, when are tests valuable? What exactly is this value?

This is hard to quantify. A test is valuable if it catches a bug. It gets points for that. So, ridiculous as it may sound, the time spent on fixing bugs the tests found is an indicator to the value of the test. We can call those tests the “valuable” tests.

Now, here’s the Test Waste ratio:

clip_image002[7]

The denominator is the total time spent due to tests breaking, the valuable and wasteful. The numerator is the wasteful work.

If your wasted time is large, the ratio gets closer to 1. If it’s small, it gets closer to zero.

You want to get a lower ratio as possible. It’s not really up to you, but it’s an indicator. And those numbers can actually be measured.

So Much For Theory

What do you think? Does it make sense? Is it helpful in any way? Do I stand a chance for a Nobel prize?

Let me know.

PS. Next week, I’m going to be at Belgium Testing Days. Apart from the Spaceship I’m smuggling through customs, I’m going to be part of an interactive session, talking about the ROI of test automation.

Product Roadmaps Are Anti-Agile

I was listening recently to the “Global Product Management Talk” live podcast (which I recommend, by the way). The speaker talked about creating roadmaps for product lines. It’s an interesting topic for me, as I’m juggling between products everyday.

As the the interview sped along, I asked on Twitter: How are roadmaps related to agile?

The answer I got was a bit bland, I thought. It seemed along the agile lines of collaboration with marketing and sales, and indeed everybody. It wasn’t what I intended, and I explained what I was really asking: Roadmaps are big planning efforts. Agile is about adapting to change. How can they live together?

Roadmaps resist change

Ever since donning my product manager cap, I’ve created roadmaps, mainly because I was asked for them. Some people, including me, regarded them as a snapshot of our current plans. Things change, plans change and therefore “roadmaps” change. That is the agile view, isn’t it?

However, some people show roadmaps much more reverence. What if you’re not that agile in spirit, but rather spend your time skipping under waterfalls? Roadmaps incur not just the planning time, but also commit resources towards that plan.  Then the plan can move forward, it makes perfect waterfall sense.

The problem is that by early commitment, we close down options that can help us change when troubles or opportunities arise. Roadmaps are the epitome of early commitment. They resist change

I haven’t managed to get my point across through twitting. I hope things are clearer now.

However, my story doesn’t end here.

Collaborating on the roadmap is  a whole team job, which is a great agile value. But there was a nuance I caught in the talk and the tweets: “[roadmapping] requires alignment w/Agile developers included in the roadmapping process” .

Agile is how developers manage their work

Once the roadmap is in place, the developers can develop the products anyway they want. Agile if they choose too. They can and should have continuous integration, automatic tests, the whole lot.

Because they are the agile team.

Note that “they” are not “us” product managers, project managers, decision makers. We decide, they work.

Command-and-control goes hand in hand with waterfall. We can try wrapping the process in all kinds of collaboration efforts, better communication, joint vision and alignment. But when you hear the “us” and “them” you understand there’s no real agility there.

The development team doing stand-ups does not make you agile. Doing a retrospective does not make you agile. Even delivering in a constant pace (gasp!) does not make you agile.

Agility is NOT a developer thing. It’s a business thing. If you discover your roadmap is ruined because a competitor is getting traction in your area, and you cannot compete, it may not be your developers are not agile enough. It‘s probably you overcommitted resources, and now cannot turn the ship around.

At that point, it doesn’t matter who was wrong, “us” or “them”. It’s all of “you” at the bottom of the ocean.

So what about those roadmaps?

Roadmaps are tools. They visualize one way to go. But just one way.

Keep your eyes open, since you might need to capitalize on another option soon, unexpectedly.

When that happens, you’ll be thankful for being part of an agile team.

We Few, We Unhappy Few

Uncle Bob Martin recently blogged about the hordes of untrained developers coming into the industry, which eventually leads to some huge software disasters. James Bach wrote about the failures of HealthCare.gov that stemmed from incompetence in software management. These are very good articles, that make you think: Should software professionals (developers, testers, managers) be regulated?

Let’s start with the why: Obviously, most of the (first) world runs on software. We either produce it as a product, or run our company on it. In any case, software is a critical component in business. It makes sense that we’d want to to keep out the riff-raff. Software is too important to leave to the unprofessional.

Would you want an untrained person, posing as a plumber to handle your pipes? (Yes, pun intended). Of course not. And that goes triple for your business.

Why regulate?

We usually try to regulate when we have suffered a massive abuse of our trust. We start with our lives, then our livelihood. We trust doctors (some, including me, say too much) because they went through required tests. We regulate banks  (badly) because they’ve taken away our money and spent it on bonuses for incompetent executives. In normal countries, we try to avoid regulation as much as we can, because it invites gaming the system, which plays badly in a free market. Not that the free market is free of gaming. Regulation, in theory, is there to create trust.

So if we want to trust software, where do we start?

How does regulated software world look like?

Of course, the simplest thing is to regulate developers. For example, you’d need to show you have decent skills in software before you can write commercial software. We won’t discuss what’s decent here (I guess we can come up with a common ground. Ok, I’m pretty sure we can’t, but let’s continue either way). This will probably be something more than what students learn today at universities, since when they come into the real world, we and them find what they’ve learned there is 90% useless. On a good day.

Once we’ve accepted this level of competence, we’ll immediately create another industry: the software licensing administration. Like any administration it sucks money from where it really ought to go. And with clear cut defined competency levels, there’ll be training and licensing organizations ready to take a young developer’s money in order to qualify. Much like a university or college degree, it gets devalued pretty quickly. But on the upside, since there’s a money invested, it does keep away some people. The poor ones.

Like any administration, it’s here to last, and so are its competency definitions. That’s right, in 50 years time, there are still going to be questions about C++ standard strings. These things do not get updated frequently.

This doesn’t look too good. The system keeps away talent, while letting in the people who bought “Programming license for dummies”. That won’t raise the quality of software.

The current situation is not that different, by the way: we’re using the universities and colleges as a gateway. We can however bypass these rules when we identify someone with actual experience. With the new legislations, the bar just moves (it doesn’t necessarily move higher).

We’re not managing risk, we’re just adding complexity to the process, and bleeding money along the way.

The big question

So what can really make sure software quality is raised to a professional level?

The craftsmanship movement says it’s the software developers who are responsible to raise their level.  While I agree, it will work only for those who accept the challenge. That leaves, what 80% of developers out of the loop? And I’m generous. An honorable demand, but conflicts with the reality.

In order to raise the level of quality, we need a demand of professionalism from the business side. That requires understanding of what building software is from the business side, which is no small fit. The business people need to fear for their wallet for funding reckless software projects. The business needs to pay for putting out risky code, because only then it will require professionalism from their workers. when this happens, weeding out the unprofessional will become part of the business, from hiring to respecting the professional developer. Uncle Bob mentions a few companies got the gist and are taking responsibility.

Chances of that happening in the rest of the software world? The same as with bankers, insurance providers and doctors. 

If you’re reading this, you’re probably on the craftsmanship side of ship. If it starts sinking, it’s because most of the horde is tipping the other side.

I hope you weren’t expecting a clear upbeat answer. Pack a life jacket.

Fail SAFe

Last week I went to a presentation about Scaled Agile Framework - SAFe. I’ve read a bit about it before, but this was a more broad introduction to the topic.

It’s going to be a success.

When I talk about why scrum succeeded in crossing the chasms from developer world to business world, the main reason I see is that it dropped the developer jargon and talked in business language. SAFe goes the extra mile, and totally talking the business language.

But it’s done what scrum never done: it offer ALL the answers.

SAFe is detailed. Very detailed. It got details on all the needed roles, all processes, how to role it out, all specified and quantified. It got all the information up front, before you even ask. You can get all the information in the site.

Answers are good, seems everyone’s looking for them these days. And SAFe not only has them, it makes sense too. All the pieces fit together, it’s a combination of tried and true rules and processes.

You know why? Because we live in a complex world, where we don’t know all the answers. And we’re ready to pay anyone who can help us.

SAFe is the first methodological framework to tie all team, project and program information into a whole organization solution. And being first is going to probably why it will stick around for the next few years.

There’s just one thing.

Snake Oil Alert

The presentation was in the context of the Lean-Kanban group, you’d expect the audience to be a pretty agile group. Most of the questions were “How do we do X in SAFe”. While these were honest questions, I couldn’t help notice this weird thing: The agile crowd focused on “processes and tools”. Isn’t the whole point of agile dealing with an ever changing reality, where prescribed recipes don’t work? 13 years after the agile manifesto, 20 years of practice, you’d guess people actually get the picture, at least this bunch.

And nobody wonders how in a this very short time of actual agile practice we now have all the answers. And that the complexity of life can now be reduced to a couple of ceremonies.

Reality is winning. We’re confounded and continue to look for answers.

The more desperate we are for answers, we’ll gladly believe the truth is out there. And that a few consultants have that knowledge and are happy to download it to us, and give us certificates that prove that we now know.

Most people still believe in the silver bullet. It’s a safe bet (pun intended) that SAFe looks the part.

In a few years, in the post-SAFe era, when people start questioning the holes left in the process, there’ll be another solution. Which we’ll gladly by into.

Agile is not declining. We’re simply riding a sine wave.

Related Posts Plugin for WordPress, Blogger...
Twitter Delicious Facebook Digg Stumbleupon More