Working with Racer, Part II

Note: Cross posted from The Typemock Insider Blog.

Permalink

This week I played with Racer and Gallio, the current incarnation of mbUnit. Did you know that Gallio comes with Typemock Isolator extensions?

Anyway, Gallio has some thread manipulation classes, like TaskContainer and ThreadTask, as well as others. I tried to find deadlocks around those classes. No dice. I’m not discouraged though. I’m just going to need a more thread-complicated project where a deadlock is waiting to be found.

But until then, here are things I find during my exercises, useful ones, I think.

First, when using MSTest, which I do, you need to turn off deployment in the solution’s test configuration file. There’s a small bug that causes a weird exception when you run with the default deployment on.

Next, running with Isolator. Yes, Typemock tools do work together. When you have both installed, and you want to run Racer, you should disable Isolator and enable Racer. This is the default mode, since Racer runs the regular code paths.

However, there come times where you need dependency isolation. In my tests, I found that the code I want to Racer to test calls system files dependencies. In another case, the function got an abstract class as an argument. In both, those didn’t have any impact on what I was looking for, namely thread manipulation methods. So I faked them with Isolator.

In order to make that work, you need to disable from the Typemock menu and enable Isolator. Then you need to link Isolator with Racer, using Isolator’s Configuration program. And that’s it, you can now use Isolator’s APIs to your heart’s content. This can make the process of running much shorter, because as we know dependencies can be heavy on the processing side, while not important for the purpose of the test.

So far I haven’t found a worthy opponent. And by worthy I mean a complex enough multi-threaded project that might actually cause deadlocks. What I’ve encountered until now are simple locks. What do you suggest I try next?

Avoiding Fragile Tests

This post came from reading a question on our forum, as well as reading one on TDD experience. It’s about fragile tests, and the cost of maintenance.

We’re talking about refactoring. And not just for making the code prettier and maintainable. We refactor and then our tests break. Well, sometimes they do, depending on how we wrote them. But if we change functionality, yes they will have to change.

It becomes worse when our tests know too much about the inner logic of our components. If component A uses B in some way, and you change that, it maybe that integration tests for A are still passing, but the ones where you fake B break.

What should we do? Are we doomed? Is this our fate? Is there no way out?

Alas, there is no simple answer. First, remember that the cost of maintaining your tests is largely outweighed by having these tests in place at all. Having no tests at all costs a lot more than having to maintain a few tests. So if someone tries this on you, show them the door.

The second thing is to come to terms with the fact that test code is just that - code, and therefore is subject to the lifetime changes such as production code is. If you are using Isolator or any mocking framework out there, you’re bound to hit this sometime.

But you can minimize the risk of creating fragile tests. Here are a few suggestions:

  • Verify less. The Verify APIs (or Mocks in general) tie your test code to internal functionality. These have much more impact on breaking tests on changes than WhenCalled APIs (or Stubs). Use them only when you need to.
  • Write focused tests. The more focused you test, meaning testing a small portion of the code, you’ll be less affected when you change other code.
  • Minimize usage of  NonPublic. This makes a lot of sense, although obviously it’s there because you need it. Still, try. 
  • Use less WhenCalled statements. Less statements means less knowledge of tests of the tested code. Here’s Isolator biggest advantage when you use recursive fakes. Since you’re changing the behavior of an entire tree of objects (without specifying which objects) in a single line, it minimizes the risk of a broken tests.
  • Use proper tools, especially refactoring tools. Resharper, CodeRush and others make changes much safe then by doing them manually. Like I always say: use the proper tools for the right job.

What’s you learned-the-harsh-way lessons for avoiding fragile tests?

The “I don’t have time to write a test” Excuse

Note: Cross posted from The Typemock Insider Blog.

Permalink

I like this one. I usually hear it in a much larger organizational context, but hey one developer is a micro-cosmos of the entire org, right?

Alk talks about the cost saving of writing a test in terms of just running and debugging an app for a bug. We always talk about how writing tests impacts development now, but saves so much during the entire lifecycle of the application. But, writing a unit test to fix a bug is the one case where the developer sees in his own eyes how much time gets wasted when he’s debugging. It’s time saved for him, not for “everyone”.

When developers see they don’t need the entire app just to tests a single method, it’s one of those “Aha” moments we all love. If you want to convince a developer that unit testing can help him, not only the entire project, show him how a writing a simple test can save him a lot of quality time with the debugger. And then say: “Oh this little tool? It’s called Typemock Isolator. Saves me hours, and it can help you do that too”. 

Unit Testing 101 Presentation

Note: Cross posted from The Typemock Insider Blog.

Permalink

This is a nice one by Yaron Naveh. Sure, it uses Moq and not Isolator, but what the hell.

The funny thing is that currently I’m showing demos of faking DateTime.Now for testability, and suddenly wrapping it up just for testability (like in this presentation) seems weird.

Problem Solving with Isolator

Note: Cross posted from The Typemock Insider Blog.

Permalink

I’ve written before how Richard Fennell is using CThru to fake a SharePoint workflow. But he’s not the only one using Isolator to solve problems.

Take for example, Apostolis Bekiaros. He’s writing an MVC app, and wants to write more readable tests. Sure, he can do this the hard way, since MVC allows it. But his code with Isolator is much more elegant. By the way, he has a new open-source project called eXpand, which is tested with Isolator.

On the other end of the globe, Francis Cheung, uses Isolator in integration tests, by redirecting calls from the SharePoint’s server to object you construct locally. I tend to say it a lot lately: The same tools today are available both unit testing and integration testing (MSTest or NUnit, Isolator) and you can use both for different purposes in different ways. And what Francis did is really cool.

Well done guys!

How Much Does Pair Programming Save?

The InfoQ article shows how much, and how Dave Nicolette calculated it. It’s interesting, since these “you only do it once and can’t compare it to the other way” things are a real pain. How much does TDD, unit testing, finding a bug earlier saves you?

Maybe now we can tell.

TestDriven.Net for … C++

Yes, hard to believe, but some people want to test in C++. Even more so, they like to do it in style, like, you know, regular devs. So here’s an add-in that does what TD.Net did to NUnit – integrated it into VS.

Way to go!

James Whittaker on The Plague of Aimlessness

What are they teaching in testing school? The Google testing blog post talks about no past leading to no future. It’s not that bleak, though. Is it because our industry is so young, and testing inside it even younger? Are we slow learners? Or slow teachers?

I really like the analogy. And yes elves, so it really speaks to me.

Via Dror.

Richard Fennell Writes a CThru Logging Aspect

Note: Cross posted from The Typemock Insider Blog.

Permalink

CThru is something like an extension method. It allows you to add functionality to an already working system, especially when you don’t access to the code inside.

Richard Fennell from Black Marble is looking for a way to test SharePoint workflows. When we talked about it, it looked to me like the effort Roy put into isolating Silverlight in SilveUnit. In SilverUnit, the main issue was abstracting the asynchronous event messaging that Silverlight relies on, to substitute with a more robust and easy way to set properties. It allowed to create more readable tests, as well as run actual unit tests. In Richard’s case, it’s about identifying the dependency on other systems, that do things in the background.

The work here is not done, since Richard did not complete his quest yet. However, on the way, Richard created a logging aspect, that using CThru, starts narrowing down the search. With logging, you can create a log of all the calls under the hood of your system.

You can see the code here. CThru is coming up as a platform for solving things we couldn’t do before. How can it help you?

First review of Typemock Racer

Note: Cross posted from The Typemock Insider Blog.

Permalink

And here it is. Go check it out now.

Thanks Dr. Random (Casey Kramer)!

Working with Racer

Note: Cross posted from The Typemock Insider Blog.

Permalink

I’m the first one who gets to break new stuff, and so, here are my impressions from Racer: It’s the greatest invention ever!

What’s that? You say I’m a bit biased because I work at Typemock? You might have a point. So how’s my experience so far?

I decided to try Racer on open source projects. See if I can find problems there. I started with NUnit, and then tried NHibernate. I’m trying to see if I can find deadlocks in there.

Here’s how I do it:

Step 1: Open the Racer docs. Yeah, I know, RTFM. The docs specify which APIs are supported, meaning, which APIs Racer intercepts to identify deadlocks.

Step 2: Search for usage of these APIs in the code. Obviously, this is redundant if you know your code. But I’ve taken the opportunity to check out OPC (other people’s code). I learned, for example, that NHibernate (which I have to admit, is my first time looking at it) implements its own lock.

Step 3: Now that I found API usage, I climbed up the code tree to see which objects and methods get called. At this point I also looked for shortcuts – are there any tests that already call these methods?

Step 4: Finally, I wrote some tests for Racer to find deadlocks. Depending on the code I tried both cross-function calls, or running the same method on a couple of threads.

And the results are in. So far I’ve found… nothing. Actually, in NUnit, I didn’t find problems. With NHibernate, I had trouble setting it up, so I decided to let it. I do want to give it another go, though.

I’m going to continue my quest, to find deadlocks in open source projects, and then expose them to the public. (I know they are already exposed because they’re open source, humor me here). So I’m going to pick one project each week, and try it. I’ll post the results I get.

Which project should I pick next? Would you volunteer to run it on your project? You can download a 21-day evaluation now. So race ahead and take Racer for a ride now!

James Carr on TDD Anti Patterns

A long catalog of anti patterns. While interesting and lots of “I’ve done this!” moments, remember: When doing unit tests, you will make mistakes. Ok, when you program, you’ll make mistakes. You can’t say now: I got to watch out for these, you won’t write unit tests at all.

Don’t let lists of patterns or anti patterns lead you.  Patterns are communication tools. They make conversations shorter, and allow better idea transfer from one mind to another. If you know them, they are easy to spot. But that’s it. There’s no substitution for your brain. You actually need to put it to work now and again, even if to decide whether to use the pattern, or, heaven forbid, leave your code as it is, even if it looks like the anti-patterns.

Exploratory Testing by James Bach

A quick link from James Bach on exploratory testing. The edited video is funny, and it also makes you think. QA testing is different than developer testing. QA should figure out how to break the system.  Developer tests need to prove the system does what it should do.

Together – you got a mighty fine quality tool.

CRAP and Other Metrics

Uncle Bob on metrics for clean code.

Despite what he says, and I have so much respect for the guy, don’t rush out and do it now. If you start running it on your codebase, expect a harsh reality shock. Even if you expect it, it’s still a shock.

I recall doing something similar years ago with running FXCop on our codebase. The 1st thought was: this can’t be. The 2nd: screw this, there is no way I’m going to sink so much time into this to fix it.

Like all things, start gradually, and work from there. It would be interesting to see if this metric catches.

When to TDD an Interface

Note: Cross posted from The Typemock Insider Blog.

Permalink

Brett Schuchert, on the Object Mentor blog, writes about TDDing an interface.  One of the main questions raised is when to decide you need an interface rather than a concrete class.

The more I think about it, it comes back to your knowledge of the system. If the interfaces already exist, there’s no problem, you go with what you have. If not, then my experience tells me to with a concrete class. I leave extraction of interfaces to the refactoring phase.

Now, what does that mean?  There’s no refactoring phase in software development (not the one you tell yourself you’ll get to sometime). The refactoring phase in TDD, is when you leave time for thinking about the code. This is where you put 2 and 2 together, and say: Hey, doesn’t this behave like component X? This is the time to extract the interface. Or move up, any kind of refactoring.

And if you want to get there even earlier, you should do it in pairs. When you put two minds together, one of the advantages is that together, they know the system better. So chances are you’ll identify commonality earlier.

When Should I Write A Test?

Note: Cross posted from The Typemock Insider Blog.

Permalink

Sam Newman asks three questions that can help you decide whether to write tests for a feature.

  1. How easy is the feature to test?
  2. What is the likelihood of the feature breaking?
  3. What is the impact of the break?

Let’s start with the 2nd one. That’s the most important. Because if you don’t have any logic in your code, like wrapper properties, there’s not much point in putting a test around it. Also, if all code A does is call another piece of code B that is surrounded by tests, the likelihood of the coda A breaking is low, and therefore you might decide you don’t need a unit test for it. (An integration test that covers both is another story). In all other cases it makes sense to write a test.

A feature is sometimes hard to test if it requires a whole setup of servers and databases. However, with Isolator, you can, and should, write tests for the different components. If you decide it needs a test (according to the former paragraph), it is easy to unit test with Isolator, and therefore you should write the test. For integration tests it’s just as important: If a system is too complex to test, what do you think the probability is for having bugs there? Exactly. Q1 is irrelevant.

Q3 starts off with good intent. It analyzes the severity of a possible bug, and based on it directs you to either write the test or not. However, it does not take into account that a future code change might affect the current behavior of the feature. We write tests not just to make sure our code works now, but also as a safety net that accompanies the code through its lifetime. If we don’t have a test in place when we are changing our code, we don’t know we’ve broken something.

To counter that let’s take a look a different example: Let’s say I have a web page, with a yellow background. Now, I’m sure it never happened to you, but with every visit from a marketing person, the color changes. Should I have a test in place that checks that the background color is the correct one, then maintain the test with every change? Obviously not. And this is where Q3 comes from. If the impact is just changing the color identifier, I will not write a test.

So when do I really write a test? Where there is logic. It is logic that causes the state to change, and it’s logic that interacts with other components in your system. Where there’s logic there’s a chance for a bug (Q2). And bugs in logic have the biggest cost associated with them (Q3). That’s why there’s no need to test properties, if they have no logic. And where there’s a bug, write a test for it as well.

Integration tests are not as different, but they do cost you more to write, because you can write a whole lot of them, based on the system complexity. So what should you do? We write tests to lower the maintenance cost of our code. If you have a very good suite of unit tests, you can minimize the number of integration tests around the well tested code. This is an analysis driven decision, unlike the unit test question.

What’s your guidelines for writing tests?

Reuse As The Holy Grail

Udi Dahan writes about the fallacy of reuse. I couldn’t agree more.

In my previous job, reuse was a selling point to our clients. We promised that building the current system will produce components that will help build the next system at a lower cost. A noble goal.

Here’s what happened: we spent a lot of time designing for all the future needs of the component. We then implemented and had some integration issues, because, generic as it was, it didn’t fit the current system. So we fitted it to work. Later, when a new system came, we had long discussions on why the previously though generic component is no longer so. Then we branched it. And so on…

Until Microsoft came out with a new .Net framework.

Then of course there was no use for reuse. We just wrote a better generic component in the new framework. Rinse, repeat. And needless to say, the first components were still maintained in their respective systems.

The goal of lower cost was not achieved. Yet the mad race continued. You know why? “Because next time we’ll get it right”.

No you won’t. Stop trying to catch all the balls in the air, including the ones that are not there yet. Concentrate on the problems you have now, and solve them. It will cost you less. And your client will not think you’re playing them.

100% Code Coverage – Achievement Unlocked

Patrick Smacchia, of NDepend fame, brings arguments for pushing for maximum code coverage. We all know that it’s a problem reaching  over the top to cover all the code. Patrick says 100% coverage will motivate you to keep it that way.

There is some truth in that. The broken window theory states that as well.

Properties are the best candidate for raising the coverage, although they don’t do anything. You get the coverage, but you spend the time.  By the way, some may argue that the tests are in place for when we’ll add logic to our tests. Isn’t that YAGNI?

So it comes down to the question – is it worth it? Well, you might give yourself a nice pat on the back. But in most cases, it’s time and effort that are better spent on adding value to the product. It’s a waste of company money if you don’t need it.

Quality is not objective. (By the way, so is good design). Good development is not about adding quality to your code. It’s about adding value to your product. Code coverage does not add quality. It lowers the cost of maintenance. And like everything, the law of diminishing returns works here as well. The last 15% will cost more than what you receive at the end.

Use you judgment, and write tests when it makes sense. Do it for lowering costs, not for an achievement.

Unit Testing is Not a “Figure It Out Later” By

Run, don’t walk, and read and the entire post (and a very emotional one at that) about why unit testing mattes.

To quote the bottom line:

If you care about code quality, business process, the business you work for, and respect for yourself as a quality developer, less bugs for you and your end users, and ultimately getting shit done and going home sooner because you had less bugs today to work on, therefore I can now spend more time with the family, then you’ll promise yourself the very same thing starting today.

Amen!

The Dark Side of Faking DateTime.Now

Note: Cross posted from The Typemock Insider Blog.

Permalink

Add a feature and see the support calls comes in.

Our new feature of faking DateTime.Now is causing some issues. You can see Dennis’ post and Soon Hui’s post which are basically point to the same problem. What happened is that we caused a breaking change without realizing it.

Let’s look at what happened before. I’ll use Dennis’ example here:

Isolate.Verify.WasCalledWithAnyArguments(() => someObject.ProcessDate = DateTime.Now);

What we have inside the delegate looks like that:
someObject.set_ProcessDate(DateTime.Now)

When we say Isolator doesn’t support mscorlib, here’s what we mean: All mscorlib types are not intercepted by Isolator, and run outside its scope. Because we can’t intercept the calls, we can’t change their behavior.

So before 5.3.1, what would happen is that Isolator would intercept the method call to the ProcessDate setter, but not for DateTime.Now. For Isolator, it would look like a concrete value was set.

But this has now changed. Now Isolator intercepts two methods, which are not in a chain. Isolator’s AAA API has a limitation, that they don’t handle well more than a single chain. So the test that has previously passed, because there was just a single call inside, now breaks, because there are two now. And of course, that’s why Dennis’ solution works. He got the DateTime.Now outside the Verify clause, and everything returned to normal.

We admit: we break stuff. We don’t mean to. Not on purpose, anyway. But the more we delve into mscorlib land, these things may happen. It’s part of the product’s evolution. However your feedback is valuable – we learn how you write tests, then we make great products helping you write them better.

People vs Programs

SecretGeek writes about examples on why people are not like computers. He gives 4, but I’m sure there are at least 7 differences.

The first example DRY (Don’t repeat yourself) is the opposite of Horstman’s second law of management (from the excellent Manager Tools): “There’s no such thing as over-communication”.

By the way the first law is:”It’s all about people”. And the way I see this blog taking shape in its second coming is about merging the two worlds

… And we’re back!

After doing some thinking, I decided to revive the blog. I’m still at Typemock, and you can find most of what I’ve blogged in the last year and a half there. I am going to cross-post from the Typemock blog, but also add some more “philosophical” stuff here.

Stay tuned…

Related Posts Plugin for WordPress, Blogger...
Twitter Delicious Facebook Digg Stumbleupon More