During the management review, we discussed the fact that although there’s extensive testing, critical bugs slip through. One of the PMs, who is a big proponent of exploratory testing, said we’re concentrating too much on writing test procedures, and less on testing.
I did actually produce the numbers, thanks to the newly implemented MS Project server. Although there’s no pattern, since it’s very new, during the last months, we see there’s some basis to his argument.
So what happens here? are we writing too much? Currently, I think we’ve bumped into a scaling issue. The systems are complex, and in order to cover many (but not even most) test cases, we have to invest a lot of time in writing the procedures. This doesn’t scale well, as the complexity grows exponentially. Even with enough time, there’s no way to cover all possibilities, and this is where bugs slip by.
Is it possible to do some planning ahead to catch those bugs? The scaling issue rears up its ugly head again: Even with enough time, how much stuff that can go wrong is it possible to foresee? Not a lot, so the procedures do not find the bugs.
We are currently doing only manual testing, that takes a lot of time on one hand, and is not repeated, due to lack of time. I believe, automation of tests can improve quality by freeing the time for the manual exploratory testing. While it does take an effort to place the automation, it’s worth it in terms of quality.