a little madness

We have many components in our product that rely on external packages being available before tests will pass. For example, our make integration requires that make is installed, and our cvs integration tests require the cvs test server to be reachable.

The problem is that some of these resources will not always available on the box running the tests. This is often the case when we are running a build on a new box / environment for the first time. What would be very nice is if the test framework could warn me if a resource is not available, and only run the tests that have a chance of succeeding, ie, those where the required resources were available.

JUnit

Since we use JUnit for testing, I had a look to see what I could do to solve this issue.

The simplest solution is to Assert.fail during the setup/before method if the resource is not available:

This results in a:

for each test that requires the unavailable resource. This is certainly effective, but a little crude, resulting in a lot of meaningless strack traces when all I need to know is that X is not available.

An alternative, in JUnit3 at least, would be to implement a custom TestDecorator that would only execute the tests it wraps if the required resources are available.

This has the advantage of avoiding the stack traces, and allows us to control the number of error messages printed. This is certainly not as noisy as the first approach, but now we need to wrap all of our tests in this decorator, and does not seem to work well with the JUnit 4 suites.

TestNG

I have heard some good things about TestNG lately, so how does it handle this problem?

TestNG supports the concept of test dependencies. That is, if testB depends on testA, then testB will not run until testA has run successfully.

So, what we can do is define a method to check for a resource, and make tests that require that resource dependent on that method.

The result is that if the resource is not available, you get a single failure with details “resource X not found”, with tests that depend on that resource will be marked as skipped. That is pretty clean outcome. We get no duplicate failures and a short error message:

Short and definately Sweet.

Conclusion

Given these results, I certainly prefer the TestNG solution. The output is clear and the association of dependencies to tests is defined with each test case and on a per test basis, not on the per class basis as with JUnit. Now if only I was using TestNG :). Maybe it is time to run the JUnitToTestNGConverter?

I am curious to know if other people run into this issue? And if so, how do you solve it? Maybe you do one of the above, or have written custom annotations or extended JUnit. Please let me know, I would love to hear about it.

Liked this post? Share it!

This entry was posted on Wednesday, August 30th, 2006 at 2:57 pm and is filed under Java, Technology, Testing. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.