Tuesday, October 13, 2009

How do you get better test coverage?

There are two things we use to find out whether our test design has coverage. The first is ensuring the the specification is fully covered and the second is code coverage. Even though all our testing is not done using the test design and our coverage is much higher that these numbers in reality, these numbers are still important for us because these represent what are the repetitive tests, documented, handed down to serviciability and in use for years after the product is released.
While the spec coverage is easy, as long long as you can trace each use case to a test case, you  know you are good. But often you would realise that this spec coverage does not necessarily lead to a good code coverage. For code coverage, we often work in a reactive mode. If our code coverage is not as per our acceptance criteria, we look at the code and see what we've missed, come up with tests for what we have missed. While this does give us adequate coverage in the end, there still seems to be something missing in our test design.
A better approach would be a hibrid one to our test design. We should not look only at the requirement specification to come up with our initial test design but at the dev design too. Our testing should be phased/scoped. While unit tests remain primarily a dev responsibility, our test design should not be completely agnostic to the dev design or code. While the dev design would help us come up with some responsibility based test cases at class/subsystem/integration level. The dev code would help up come up with implementation based test cases, which are necessary for certain type of bugs, example exceptions. We cannot really provide coverage for the errors our code will produce unless we know how it is implemented. The dev design on the other hand will help us come with testcases which will provide better coverage. Its important for us as testers to understand how the various components of the product integrate, what is the responsibility for each of them. If we include this kind of testing early one in our test cycle, we would have a better grip on coverage and not only that, this would gaurantee that we find some bugs earlier and closer to their introduction. This would reduce the costs of fixing these bugs too. While application level testing is important to ensure that our users are getting what has been promise, the component level testing would mean less surprises in the end.
One more side effect of this exercise would be more meaningful reviews of the dev design by the test team. Do we as testers care only about the implemented feature or should we care about what is the design in implementing the feature. The design does affect the quality of the shipped product, especially when the requirements start changing. A good design ensures we can fit the changing requirements better.
Next time we start out to design tests for a new product, lets keep these things in our minds. As testers one of our jobs is to collect information about the quality of the product, lets own this responsibility more completely by getting involved at all aspects of the project better than before.