Archive for the ‘Testing’ Category

Testing Is … Freedom To Change

Wednesday, February 21st, 2007

For this second entry in the “Testing Is …” series, I decided to mention an advantage of testing that is widely known. The idea is not original, but its importance to you as a developer cannot be overstated. Perhaps the single greatest advantage of writing tests is the freedom you feel to change the code under test. All software grows cruft over time, and left unchecked the code will become an unmaintainable mess. Most developers would agree that maintaining code is not the greatest part of the job, and maintaining crufty code is about as bad as it gets. The answer is to refactor to remove the cruft over time. However, we also know from bitter experience that changing working code, no matter how crufty, will introduce new bugs1.

That is where the tests come in. If you have maintained a comprehensive test suite over time2, you need not fear nasty new surprises. You can refactor as required, and keep your code base much healthier. Now, test suites are not perfect and new bugs are still bound to happen. However, this is mitigated by two factors:

  • The difficulty in maintenance of the crufty code itself will lead to bugs.
  • A well-written test suite will cover those dark corners where bugs have been found to lurk. In particular, regression tests written to reproduce bugs as they are found are priceless at refactor time.

The logic is quite simple. Beyond that, there is also a powerful emotional difference. When the tests aren’t in place, we fear change. When we have the testing safety net, the fear is replaced with a sense of freedom. A freedom to change, and improve over time. A freedom which itself can be motivating.


1 If the cruft has grown due to the underlying complexity of the problem domain it is certain there are some nasty new bugs waiting for you!
2 Maintaining the tests as you write the code is important in itself. It is much more difficult to add tests later, not least because the problem is not fresh in your mind.

Testing Is … Instant Gratification

Wednesday, February 14th, 2007

Somewhat inspired by Alberto Savoia’s article Testivus – Test For The Rest Of Us, I have been thinking about ways to motivate more developers to write more tests. Holier than thou style arguments (”your code is useless if it isn’t tested…”) are no motivation. But there is a much simpler way to approach the topic: testing has many tangible benefits for the developer. Perhaps looking at the ways in which testing can directly benefit us will motivate us all to test more. So I’ll be writing a few posts with perspectives of testing that highlight the benefits I get from my own tests. Here it goes:

Testing Is … Instant Gratification

One of the most frustrating parts of the development cycle is the start of a new major release. This is an exciting time, where big new things are planned and frontiers are explored. However, given the size of the problems that are tackled in this phase, roadblocks are always found along the way. Even when things are sailing along smoothly, the finish line is a long way in the distance. Thus it is easy to get the feeling that you are going nowhere: as you have no concrete features to show for your efforts. That’s where testing can help. By isolating and exercising a smaller piece of the puzzle, you can demonstrate progress. And I don’t mean that you can prove to your manager that you haven’t just been surfing the web all day. Rather, you can demonstrate to yourself that things are heading the right direction. I find that even this small sense of achievement can be really motivating. Once the most basic things are working, you can’t help but add that next incremental step. Before you know it, you’re in the zone and things are really beginning to gel.

So, next time you feel stuck: write some tests!

UnitTest++: Reports

Wednesday, January 10th, 2007

In my previous post, UnitTest++: The New Choice for C++ Unit Testing?, I gave a basic introduction to UnitTest++. Moving beyond the basics, you will often want to integrate UnitTest++ into your development process. A key to this is being able to generate test reports in a format amenable to integration. The default output of UnitTest++ is a simple, human-readable summary of the test results:

jsankey@shiny:~/repo/utpp$ ./utpp
utpp.cpp(9): error: Failure in MyTest: false
FAILURE: 1 out of 1 tests failed (1 failures).
Test time: 0.00 seconds

This is fine for the developer, but not so easy to process in code for integration purposes. Luckily, the reporting mechanism is not fixed. The test runner accepts an instance of type TestReporter to use for reporting results. The default reporter, TestReporterStdout, produces the developer-friendly output shown above. A second reporter is also included in the UnitTest++ distribution: XmlTestReporter. As its name suggests, this reporter outputs results in XML format, which is much easier to digest in code.

Using the XmlTestReporter is easy. Just construct one with an output stream and pass it to the test runner when executing the tests:

#include <fstream>
#include "UnitTest++.h"
#include "XmlTestReporter.h"

int main(int, char const *[])
{
    std::ofstream f(“tests.xml”);
    UnitTest::XmlTestReporter reporter(f);
    return UnitTest::RunAllTests(reporter,
                                 UnitTest::Test::GetTestList(),
                                 NULL,
                                 0);
}

The results in this case are saved to a file named “tests.xml”. An example XML report is shown below:

<?xml version=“1.0″?>
<unittest-results tests=“3″ failedtests=“1″
                  failures=“1″ time=“0″>

    <test suite=“SuiteOne” name=“TestOne” time=“0″/>
    <test suite=“SuiteOne” name=“TestTwo” time=“0″>
        <failure message=“utpp.cpp(14) : false”/>
    </test>
    <test suite=“SuiteTwo” name=“TestOne” time=“0″/>
</unittest-results>

The report is easily interpreted, and can be easily parsed for integration in to other systems. A prime use case would be integration with a continuous integration server, such as Pulse, which is why I am looking into the reports myself :) .

Finally, if the XML format is not suitable for your purposes, you can also create your own test reporters. The interface is compact and easily implemented.

UnitTest++: The New Choice for C++ Unit Testing?

Monday, December 18th, 2006

In an earlier post on C++ Unit Testing Frameworks, I came across a relatively new framework by the name of UnitTest++. At first glance, this framework appealed to me for a couple of reasons:

  • Unlike most of the other frameworks, it is relatively recent, and in development
  • One of the originators of the project is the author of the best comparison of C++ unit testing libraries online. The experience of reviewing several other frameworks should inform the design of a new framework.

So, I’ve decided to take a closer look. I’ll start in this post with the basics: how do we write tests, fixtures and suites in UnitTest++? These are the fundamentals of a unit testing library, and should be very simple to use.

First, we need the UnitTest++ distribution. It is available as a simple tarball from SourceForge. Exploding the tarball gives a basic structure with build files at the top level, and child docs and src directories. To build the library itself, on Linux at least, requires a simple make:

jsankey@shiny:~/tools/UnitTest++$ make
src/AssertException.cpp
src/Test.cpp

Creating libUnitTest++.a library…
src/tests/Main.cpp
src/tests/TestAssertHandler.cpp

Linking TestUnitTest++…
Running unit tests…
Success: 162 tests passed.
Test time: 0.31 seconds.

The primary output is libUnitTest++.a at the top level. This, along with the header files under src (excluding src/test), forms the redistributables needed to build against UnitTest++ in your own project. It is a little awkward that no binary distributions, nor a “dist” or similar Makefile target are available. However, the source tree is so simple that it is not hard to extract what you need.

Armed with the library, the next step is to create out first test case, and run it. UnitTest++ makes use of macros to simplify creating a new test case. It could hardly get an easier:

#include "UnitTest++.h"

TEST(MyTest)
{
    CHECK(true);
}

int main(int, char const *[])
{
    return UnitTest::RunAllTests();
}

A test case is created using the TEST macro, which takes the case name as an argument. The macro adds the test case to a global list of cases automatically. The body of the test utilises the CHECK macro to assert conditions under test. Various CHECK* macros are available for common cases. Finally, to actually run the test, we call UnitTest::RunAllTests(). This runs all cases using a default reporter that prints a result summary to standard output:

jsankey@shiny:~/repo/utpp$ ./utpp
Success: 1 tests passed.
Test time: 0.00 seconds.

RunAllTests returns the number of failed cases, so using this as the program exit code works well. If we change the check to CHECK(false), we get a failure report:

jsankey@shiny:~/repo/utpp$ ./utpp
utpp.cpp(9): error: Failure in MyTest: false
FAILURE: 1 out of 1 tests failed (1 failures).
Test time: 0.00 seconds.

The next step is to create a test fixture, which allows us to surround our test cases with shared setup/teardown code. This is achieved in UnitTest++ by building upon standard C++ construction/destruction semantics. To create a fixture, you just create a standard C++ struct. The setup and teardown code go in the struct constructor and destructor respectively. Let’s illustrate how this works:

#include <iostream>
#include <string>
#include "UnitTest++.h"

struct MyFixture
{
    std::string testData;

    MyFixture() :
        testData(“my test data”)
    {
        std::cout << “my setup” << std::endl;
    }

    ~MyFixture()
    {
        std::cout << “my teardown” << std::endl;
    }
};

TEST_FIXTURE(MyFixture, MyTestCase)
{
    std::cout << testData << std::endl;
}

int main(int, char const *[])
{
    return UnitTest::RunAllTests();
}

Instead of the TEST macro, we use TEXT_FIXTURE to create a test case that uses the fixture struct. The example is artificial, but serves to illustrate the order in which the functions are called. Also of interest is how members of the fixture struct are referenced directly by name within the test case. Under the covers, the TEST_FIXTURE macro derives a type from MyTestFixture, making this possible. Running this new program gives the following:

jsankey@shiny:~/repo/utpp$ ./utpp
my setup
my test data
my teardown
Success: 1 tests passed.
Test time: 0.01 seconds.

The setup and teardown wrap execution of the test case, which has simple access to the data in the fixture. By leveraging construction/destruction, the fixture code is both familiar and concise.

The final step is to organise test cases into suites. UnitTest++ again uses macros to simplify the creation of suites. You simply wrap the tests in the SUITE macro:

#include <iostream>
#include "UnitTest++.h"

SUITE(SuiteOne)
{
    TEST(TestOne)
    {
        std::cout << “SuiteOne::TestOne” << std::endl;
    }

    TEST(TestTwo)
    {
        std::cout << “SuiteOne::TestTwo” << std::endl;
    }
}

SUITE(SuiteTwo)
{
    TEST(TestOne)
    {
        std::cout << “SuiteTwo:TestOne” << std::endl;
    }
}

int main(int, char const *[])
{
    return UnitTest::RunAllTests();
}

As shown above, it is possible to have two tests of the same name in different suites. This illustrates the first function of suites: namespacing. Running the above gives:

jsankey@shiny:~/repo/utpp$ ./utpp
SuiteOne::TestOne
SuiteOne::TestTwo
SuiteTwo:TestOne
Success: 3 tests passed.
Test time: 0.01 seconds.

Suites also have another function: they allow you to easily run a group of related tests. We can change our main function to only run SuiteOne (note we also need to include TestReporterStdout.h):

int main(int, char const *[])
{
    UnitTest::TestReporterStdout reporter;
    return UnitTest::RunAllTests(reporter,
                                 UnitTest::Test::GetTestList(),
                                 “SuiteOne”,
                                 0);
}

Running this new main will only execute SuiteOne:

jsankey@shiny:~/repo/utpp$ ./utpp
SuiteOne::TestOne
SuiteOne::TestTwo
Success: 2 tests passed.
Test time: 0.00 seconds.

So there you have it, a taste of the basics in UnitTest++. The most appealing thing about this library is simplicity: you can tell that the authors have made an effort to keep construction of cases, fixtures and suites as easy as possible. This lets you get on with writing the actual test code. In this overview I have not explored all of the details, most notably the various CHECK macros that test for equality, exceptions and so on. However, as it stands UnitTest++ is quite a simple framework, and there is not a whole lot more to it. Although you may need more features than you currently get out of the box, UnitTest++ is young and thus I expect still growing. The simplicity also makes it an easy target for customisation, which is important given the diversity of C++ environments. I’ll be keeping an eye on UnitTest++ as it evolves, and recommend you take a look yourself.

Article: Reducing the Impact of Broken Builds

Tuesday, December 5th, 2006

The “build” is the current status of any software project, and as such reflects the health, vitality and progress of the project. In this article we first review some of the impacts of a broken build. Those already familiar with the negative impacts of broken builds may wish to skip this lead in. There follows an analysis of various techniques to reduce the frequency and impact of broken builds. These techniques vary from the optimistic to the preventative, and from lightweight to quite intensive.

Read the full article at zutubi.com.

Pulse Continuous Integration Server 1.2 M3

Tuesday, December 5th, 2006

We just keep on punching out the milestones on 1.2 ;) . New features in this puppy include:

  • Customisable columns: customise build tables with drag and drop.
  • Manual release support: configure Pulse to prompt for release build properties
  • Post stage actions: hook in after each agent build to cleanup/reboot agents.
  • P4Web support: built in linking to P4Web.
  • LDAP group integration: manage permissions through LDAP
  • Broken since support for tests: differentiate new and existing test failures.
  • Executable projects: run custom build scripts without dropping down to XML.

See the early access page for M3 packages and full details.

Pulse 1.2 M1: Test Before You Commit

Tuesday, October 31st, 2006

Phew, it’s been a busy time, but finally we have the first milestone build of Pulse 1.2 ready to go! The headline feature for this release is the ability to run personal builds. A personal build takes your local changes and applies them to a pulse™ build without them being submitted to your SCM first. This allows you to test your changes before submitting them to version control.

Other major features in this release include:

  • Reports: each pulse™ project now has its own “reports” page, which displays build data for the project visually. Currently, the reports show trends over time for:
    • Build results
    • Tests run per build
    • Build time
    • Stage execution time
  • Windows System Tray Notification: a new Pulse client, Stethoscope, sits in your system tray allowing you to see your project health at a glance. You can configure Stethoscope to monitor both personal builds and project builds for your selected projects. If you like, Stethoscope will pop up a message whenever a build completes.
  • Customisable Notifications: don’t like the format of your notification emails or instant messages? In pulse™ 1.2, the notification templates can be customised using FreeMarker.
  • Automatic Agent Upgrades: we go to great effort to make pulse™ easy to install, upgrade and maintain. That is why in pulse™ 1.2 we have made the upgrade process even simpler by adding automatic upgrades for agent machines. Now, after you upgrade your main pulse™ server, your agents will be automatically upgraded for you!
  • Resource Configuration Wizard: on the same theme of keeping things simple, we have also added a new resource configuration wizard. This wizard makes it easy for you to configure common build dependencies, such as Java Development Kits and build tools (ant, make, etc). We have also improved the resource auto-discovery code to detect resource versions for you: in many cases you won’t even need the wizard!
  • Anonymous Signup: you can now optionally allow users to sign up to pulse™ themselves. This lessens the burden on the pulse™ administrator by removing the need for them to create accounts. It is also perfect for public-facing servers (e.g. open source projects) where interested parties can sign up for read-only access but with their own dashboard and preferences.

Grab a milestone build now from our Early Access Program page and try it out!

Automate Your Acceptance Tests With Python

Thursday, September 28th, 2006

Recently I’ve been cooking up some new acceptance-level tests for Pulse. The sort of tests that need to take a Pulse release package, unpack it, start the server and poke it with a stick. This is the sort of automation that I would typically use bash for: since a lot of the work involves executing and managing subprocess. However, I had learned from earlier testing that process control from bash is not very portable. More specifically, the behaviour of Cygwin bash on Windows differs considerably from *nix bash. At the time, I worked around the problem by doing some process control using Ant, but there had to be a better way.

Enter Python. Or, more specifically, the new(ish) subprocess module (introduced in Python 2.4). With this module Python finally has a concise but powerful way to launch and control subprocesses. It also includes convenient ways to handle subprocess I/O, including setting up pipelines. What’s more, I have so far found it to be quite portable between Linux and Windows.

Starting with something simple, let’s say you just want to launch a command synchronously (much as you would in a shell script):

exitcode = subprocess.call([“cat”, “foo.txt”])

Simple. Want to set up a pipeline like `dmesg | grep hda` (capturing the final output)? Here’s an example straight from the doco:

from subprocess import Popen

p1 = Popen([“dmesg”], stdout=PIPE)
p2 = Popen([“grep”, “hda”], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]

For my specific case, I used the ability to launch a Pulse server processes in the background and capture the output extensively. When done, I shut down the server using the standard shutdown script (testing of said script being a nice side-effect). An asynchronous child is started simply by creating a Popen object:

from subprocess import *

# Launch process, capturing output
out = open(“stdout.txt”, “w”)
err = open(“stderr.txt”, “w”)
server = Popen([“startup.sh”], stdout=out, stderr=err)

# Do some stuff with the process…

# Trigger a shutdown and wait for the process to exit
call([“shutdown.sh”])
exitcode = server.wait()

I have simplified this example a bit, but the real work is all there, and it’s pleasantly straightforward. This combined well with Python’s simple XML-RPC API, which I used to manipulate the running server.

So far the switch to from bash/Ant to Python has been a great success. If you need a cross-platform way to automate your acceptance testing, give it a go.

C++ Unit Testing Frameworks

Tuesday, September 26th, 2006

I like to keep an eye on various build and testing tools, for potential integration with Pulse. As such, I’ve started to amass some links to unit testing tools/resources for C++, where there are many competing options:

  • Exploring the C++ Unit Testing Framework Jungle: the most comprehensive article I’ve found on the topic, even though it is getting old.
  • CppUnit: probably the best know xUnit port, supported in Pulse already. Does a decent job.
  • Boost.Test: part of the well known set of Boost libraries. Naturally has dependencies on Boost.
  • CppUnitLite: a minimalistic rewrite of CppUnit, intended to be simpler and more portable. May be appropriate if you don’t mind getting your hands dirty to add the features you need.
  • Nano Cpp Unit: more an exercise in illustrating the barest testing framework than a usable framework itself.
  • Unit++: pitched as a more “C++ like” xUnit port. Documentation is thin on the ground, and I don’t have any practical experience with it.
  • CxxTest: takes the novel approach of using Perl to generate a test runner, simplifying the code. Also relatively portable, although of course you will need Perl.
  • TUT: a simple, template-based framework distributed as a single header file. Reasonable portablility (given a modern compiler). Lacking some features of other frameworks, but without any dependencies.
  • cutee: another framework aimed at simplicity. Looks to have test case creation down to its simplest form. Documentation is thin.
  • CppTest: you guessed it: another framework aimed to be simple and portable. Supports a few output formats out of the box, including HTML reports.
  • UnitTest++: co-authored by the author of the article above, this framework hopes to combine the best ideas from others. Documentation is non-existant, but on the plus side it is one of the more modern frameworks (developed this year, not 2004!).
  • QtUnit: probably a good option if you were using Qt, but is officially unmaintained. Mind you, most of the other frameworks are also dormant.

That’s all I have gathered so far. I have to say, there are a lot of players out there, but little action. A few interesting ideas, but no framework seems to be a clear leader. I hope to get some time to play in depth a bit more, in which case I will flesh out more details.

JUnit V TestNG: Managing external dependencies

Wednesday, August 30th, 2006

We have many components in our product that rely on external packages being available before tests will pass. For example, our make integration requires that make is installed, and our cvs integration tests require the cvs test server to be reachable.

The problem is that some of these resources will not always available on the box running the tests. This is often the case when we are running a build on a new box / environment for the first time. What would be very nice is if the test framework could warn me if a resource is not available, and only run the tests that have a chance of succeeding, ie, those where the required resources were available.

JUnit

Since we use JUnit for testing, I had a look to see what I could do to solve this issue.

The simplest solution is to Assert.fail during the setup/before method if the resource is not available:

@Before public void checkResource()
{
  if (isResourceNotAvailable())
  {
    Assert.fail(“resource not available.”);
  }
}

This results in a:

java.lang.AssertionError: resource not available.
        at org.junit.Assert.fail(Assert.java:69)
        at junit4.DependencyTest.checkResource(
        at sun.reflect.NativeMethodAccessorImpl.i
        at sun.reflect.NativeMethodAccessorImpl.i
        at sun.reflect.DelegatingMethodAccessorI
        at org.junit.internal.runners.BeforeAndAfte
        at org.junit.internal.runners.BeforeAndAfte
        at org.junit.internal.runners.BeforeAndAfte
        at org.junit.internal.runners.TestMethodRu
        at org.junit.internal.runners.TestMethodRu
        at org.junit.internal.runners.TestClassMeth
        at org.junit.internal.runners.TestClassMeth
        at org.junit.internal.runners.TestClassRunn
        at org.junit.internal.runners.BeforeAndAfte
        at org.junit.internal.runners.TestClassRunn
        at junit.framework.JUnit4TestAdapter.run(

for each test that requires the unavailable resource. This is certainly effective, but a little crude, resulting in a lot of meaningless strack traces when all I need to know is that X is not available.

An alternative, in JUnit3 at least, would be to implement a custom TestDecorator that would only execute the tests it wraps if the required resources are available.

pulbic class CheckResourceDecorator extends TestDecorator
{
  public CheckResourceDecorator(ResourceAwareTest test)
  {
    super(test);
  }

  public void run(final TestResult result) {
    Protectable p= new Protectable() {
      public void protect() throws Exception {
        if (isResourceAvailable())
        {
          basicRun(result);
        }
        else
        {
          System.err.println(“resource not available.”);
        }
      }
    };
    result.runProtected(this, p);
  }

  private boolean isResourceAvailable()
  {
    return ((ResourceAwareTest)getTest()).isResourceAvailable();
  }
}

public interface ResourceAwareTest extends Test
{
  public boolean isResourceAvailable();
}

This has the advantage of avoiding the stack traces, and allows us to control the number of error messages printed. This is certainly not as noisy as the first approach, but now we need to wrap all of our tests in this decorator, and does not seem to work well with the JUnit 4 suites.

TestNG

I have heard some good things about TestNG lately, so how does it handle this problem?

TestNG supports the concept of test dependencies. That is, if testB depends on testA, then testB will not run until testA has run successfully.

So, what we can do is define a method to check for a resource, and make tests that require that resource dependent on that method.

@Test public void checkResourceX()
{
  // if resource x is not available, then fail.
  Assert.fail(“resource X not found.”);
}

@Test(dependsOnMethods = {“checkResourceX”})
public void testThatRequiresResourceX()
{
    // run the tests.
}

The result is that if the resource is not available, you get a single failure with details “resource X not found”, with tests that depend on that resource will be marked as skipped. That is pretty clean outcome. We get no duplicate failures and a short error message:

java.lang.AssertionError: resource X not found.
        at org.testng.Assert.fail(Assert.java:76)
        at testng.dependency.DependencyTest

Short and definately Sweet.

Conclusion

Given these results, I certainly prefer the TestNG solution. The output is clear and the association of dependencies to tests is defined with each test case and on a per test basis, not on the per class basis as with JUnit. Now if only I was using TestNG :) . Maybe it is time to run the JUnitToTestNGConverter?

I am curious to know if other people run into this issue? And if so, how do you solve it? Maybe you do one of the above, or have written custom annotations or extended JUnit. Please let me know, I would love to hear about it.