a little madness

A man needs a little madness, or else he never dares cut the rope and be free -Nikos Kazantzakis

Zutubi

Archive for March, 2009

C++ Unit Testing With Boost.Test

Recently I implemented a Pulse plugin to read Boost.Test reports and integrate the tests into the build results. As usual, the first step in implementing the plugin was the creation of a small project that uses Boost.Test to produce some real reports to work from. I found the Boost.Test documentation to be detailed, but not so easy to follow when just getting started — so I thought I’d give an overview here.

Step 1: Installation

First you will need to install Boost.Test, or possibly all of Boost if you plan to use more of it. You can download Boost in its entirety from the Boost download page. Then you just need to unpack the archive somewhere appropriate, so you can include the headers and link against built libraries (see the next step).

An even easier option if you are on Linux is to install a package. On Ubuntu (and, I expect, other Debian variants), the desired package is libboost-test-dev:

$ sudo apt-get install libboost-test-dev

The downside of this is the packages are somewhat out of date, the default being built from Boost 1.34.1 (there is also a 1.35 variant available). I have not seen much impact of this when using Boost.Test, but if you need newer Boost libraries then it may be better to compile your own.

Step 2: Choose Your Compilation Model

Unlike many Boost libraries (which are implemented completely as headers), Boost.Test includes a runtime component which you need to link against: the “Program Execution Monitor”. This component includes the main entry point for running your tests, among other things. If you installed Boost from source, you will need to build the library yourself using bjam — the instructions are quite toolchain specific so I won’t go into them here. You can link statically or dynamically, but will need to configure your includes and build appropriately.

The key thing from the point of view of writing and building your tests is to include the right definitions in your source and add the right flags when linking. I opted for dynamic linking against the prebuilt library installed by my Ubuntu package. To achieve this, I needed two things:

  1. To define BOOST_TEST_DYN_LINK before including the Boost.Test headers in my source file.
  2. The addition of: -lboost_unit_test_framework to my linker flags.

With that plumbing out of the way, we can get down to testing something.

Step 3: A First Test Case

For a start, I cooked up an exceptionally useful function to add two ints, and a test case “universeInOrder” to check that it works:

#define BOOST_TEST_DYN_LINK
#define BOOST_TEST_MODULE Hello
#include <boost/test/unit_test.hpp>

int add(int i, int j)
{
    return i + j;
}

BOOST_AUTO_TEST_CASE(universeInOrder)
{
    BOOST_CHECK(add(2, 2) == 4);
}

Notice that apart from the BOOST_TEST_DYN_LINK definition, I also had to define a name for my test module via BOOST_TEST_MODULE. The case itself is defined using the BOOST_AUTO_TEST_CASE macro, giving the case name as an argument1. Finally, within the test assertions can be made the BOOST_CHECK macro. Compiling and running the test gives the following:

$ g++ -ohello -lboost_unit_test_framework hello.cpp
$ ./hello
Running 1 test case...

*** No errors detected

Simple enough, my test passes. If I deliberately make it fail, by changing the 4 to a 5, I get:

 $ ./hello
Running 1 test case...
hello.cpp(12): error in "universeInOrder": check add(2, 2) == 5 failed

*** 1 failure detected in test suite "Hello"

Here we start to see the benefits of the library: I get a nice failure message, complete with line number and the expression that failed.

Step 4: More Assertions

Unlike the assertions in many testing libraries, a failed BOOST_CHECK will not exit the test case immediately — the problem is recorded and the case continues. To immediately fail a test, you can use BOOST_REQUIRE instead:

BOOST_AUTO_TEST_CASE(universeInOrder)
{
    BOOST_REQUIRE(add(2, 2) == 4);
}

To just output a warning instead of failing the test, you can use BOOST_WARN. In fact many Boost.Test assertions come in these three variants: CHECK, REQUIRE and WARN.

Richer assertions are also possible, including these notable examples:

  • BOOST_CHECK_MESSAGE: allows you specify a custom failure message as a second argument. You can pass a string, or any type supporting the << operator.
  • BOOST_CHECK_EQUAL: checks two arguments for equality using ==. Improves upon the normal check in the above examples by showing the actual values when the assertion fails.
  • BOOST_CHECK_THROW: checks that an expression causes a specified type of exception to be thrown.

The full list of available assertions for the the version of Boost.Test I am using (1.34.1) can be found here.

Step 5: Suites

Once you have a non-trivial number of test cases, you need to organise them into suites. Note that each module (defined with BOOST_TEST_MODULE) already has a top-level suite named after that module. Further suites can be nested within a module to categorise as necessary. The easiest way to do this is to continue with the auto-registration model, and simply wrap the test cases with new macros to start and end a suite:

#define BOOST_TEST_DYN_LINK
#define BOOST_TEST_MODULE Suites
#include <boost/test/unit_test.hpp>

int add(int i, int j)
{
    return i + j;
}

BOOST_AUTO_TEST_SUITE(Maths)

BOOST_AUTO_TEST_CASE(universeInOrder)
{
    BOOST_CHECK(add(2, 2) == 4);
}

BOOST_AUTO_TEST_SUITE_END()

BOOST_AUTO_TEST_SUITE(Physics)

BOOST_AUTO_TEST_CASE(specialTheory)
{
    int e = 32;
    int m = 2;
    int c = 4;

    BOOST_CHECK(e == m * c * c);
}

BOOST_AUTO_TEST_SUITE_END()

In a normal run, you won’t see that the tests have been categorised. To show the suites in the output, you can set the log level to test_suite:

$ ./suites --log_level=test_suite
Running 2 test cases...
Entering test suite "Suites"
Entering test suite "Maths"
Entering test case "universeInOrder"
Leaving test case "universeInOrder"
Leaving test suite "Maths"
Entering test suite "Physics"
Entering test case "specialTheory"
Leaving test case "specialTheory"
Leaving test suite "Physics"
Leaving test suite "Suites"

*** No errors detected

Step 6: Fixtures

To add common setup and teardown code around your cases, Boost.Test supports fixtures. These take advantage of C++’s own mechanism for setup and teardown – construction and destruction. Indeed, you can easily add a “fixture” to a test case be just defining a type with the appropriate constructor and destructor and allocating one on the stack at the start of the case. This is repetitious, however, and not terribly explicit. From my experience the nicest way is to organise your tests into suites so that you can use one fixture per suite, and then just use BOOST_FIXTURE_TEST_SUITE in place of BOOST_AUTO_TEST_SUITE:

#define BOOST_TEST_DYN_LINK
#define BOOST_TEST_MODULE Fixtures
#include <boost/test/unit_test.hpp>

struct Massive
{
    int m;

    Massive() : m(2)
    {
        BOOST_TEST_MESSAGE("setup mass");
    }

    ~Massive()
    {
        BOOST_TEST_MESSAGE("teardown mass");
    }
};

BOOST_FIXTURE_TEST_SUITE(Physics, Massive)

BOOST_AUTO_TEST_CASE(specialTheory)
{
    int e = 32;
    int c = 4;

    BOOST_CHECK(e == m * c * c);
}

BOOST_AUTO_TEST_CASE(newton2)
{
    int f = 10;
    int a = 5;

    BOOST_CHECK(f == m * a);
}

BOOST_AUTO_TEST_SUITE_END()

Note that the test cases can refer directly to the public “m” member of the fixture type — in the background inheritance is at work, so protected members are also directly accessible. If you run this with logging, you can see that the fixture runs for each case:

$ ./fixtures --log_level=test_suite
Running 2 test cases...
Entering test suite "Fixtures"
Entering test suite "Physics"
Entering test case "specialTheory"
setup mass
teardown mass
Leaving test case "specialTheory"
Entering test case "newton2"
setup mass
teardown mass
Leaving test case "newton2"
Leaving test suite "Physics"
Leaving test suite "Fixtures"

*** No errors detected

Conclusion

I hope that gives you a decent starting point for using Boost.Test. As I mentioned in the beginning, there is plenty more documentation available at the source — it’s just daunting due to its size. Happy testing!

Update

Sample code is now available at GitHub.


1 – It is also possible to define free functions containing your tests and register them manually, although I find the automatic method simpler.

Pulse 2.1 Early Access Program

Having just made the third milestone build of the Pulse 2.1 Early Access Program, it occurred to me that it’s about time we started telling people about it! We’re baking two major new features into 2.1:

  • Project Dependency Support: until now Pulse has had only limited support for dependencies, and we’ve recommended the use of tools like Apach Ivy for the delivery of artifacts between dependent builds. In Pulse 2.1, however, we’ve embedded Ivy in Pulse — both an artifact repository and the configuration of dependencies. Now you can simply declare what projects you depend on and Pulse will deliver artifacts between your builds. We’re also adding support for smart dependency triggering.
  • Flexible Projects Without the XML: the build engine of Pulse has always been flexible, allowing you to define multiple build recipes each with an arbitrary number of commands, captured outputs and post-processing. However, to access the full flexibility you needed to write an XML Pulse file by hand. Well, no longer. In Pulse 2.1, the full flexibility of the engine is configurable via the Pulse web UI! These changes also allow support for new build tools to be plugged in more easily.

We’ve still got plenty of work to do, such as figuring out the trickier questions around dependencies and improving the usability of the new features. On top of this we also plan plenty of minor improvements based on feedback from our users. If you want to be part of the feedback process, just head over to the EAP page, download a build and tell us what you think!

Rich Hickey on Clojure at LJC

Last Thursday I rolled up to a London Java Community talk by Rich Hickey, the creator of Clojure. Rich was in town for QCon, but was good enough to do an extra talk for those of us not attending the conference — thanks Rich.

The talk, entitled “Clojure for Java Developers” was polished and interesting. Rich is a good speaker and clearly articulates the motivations behind Clojure, as well as giving a basic feature overview. In a nutshell, Clojure is a Lisp for the JVM with tight Java integration and a strong functional bias. I won’t go over all the basics here: for that you’re better of heading to the Clojure website. I did, however, take a few notes on what I found to be the more interesting aspects of the talk:

  • Rich made a distinction between languages ported to the JVM (e.g. Jython, JRuby) and those designed for the JVM like Clojure. The idea is that Clojure is designed to work in the hosted environment, and has native support for, e.g. calling Java.
  • A key reason for building a lisp-like language is the fact that the language itself can all be built on a very small number of primitive constructs (I believe it could be as few as 7).
  • Clojure has native syntax for lists, maps and vectors (unlike many lisps in which only lists are distinguished). These are each built on a simple sequence abstraction with only 3 operations: seq, first, and rest. By it’s nature this abstraction allows sequences to be lazy.
  • The built in data structures are all immutable, to enable a functional style of dealing with concurrency. To allow for efficient manipulation, they are all backed be a tree representation, which cuts down on the amount of data that must be copied to create a modified version of a structure. I presume the trees are all balanced to keep depth at O(log n), meaning only O(log n) nodes need to be copied on a change.
  • Rich emphasised the point of treating aggregate structures as unchanging values so that they may be shared among threads without locking. Building efficient support for immutable structures is key to enabling this style.
  • On top of the immutable data structures, Clojure also natively supports Software Transactional Memory (STM). Instead of changing data structures directly, threads share indirect references that point to immutable structures. Within a (language-supported) transaction you can produce a new version of a structure and on commit the shared reference is updated to point to your new version.

With the interest in dynamic languages continuing, people would do well to consider Clojure. After all, once you’ve gone dynamic, why not go all the way to Lisp — the most famously-extensible language family?

Caging Selenium With Xnest

Our automated acceptance test suite for Pulse makes heavy use of Selenium to test the web UI. The ability to automate real browsers is a godsend for testing DOM/JavaScript differences, but causes a few practical issues. Probably the most irritating of these is that running the test suite continually pops up browser windows which steal focus. Any sort of focus stealing is intolerable, but having focus stolen every 20 seconds for half an hour is downright maddening.

Luckily, in the wonderful world of X, there are a multitude of ways to solve this problem. One that immediately came to mind was to use Xvfb to run the tests headless on my development box, just as I had them running on our headless CI agents. The big drawback with this method, however, is the inability to peek at what the browser is doing. It is possible to take a screenshot and open that, but it feels a little clunky.

Enter Xnest. This is an X server that can run inside a window — effectively hosted by another X server. By pointing Selenium’s DISPLAY at such a nested server, you can cage it inside a window where it is easy to observe but can’t mess with your focus:

Caged Selenium

If you’d like to tame Selenium yourself, just follow the steps below. Installation commands are Ubuntu/Debian style, but should be easy to translate to other distributions:

  1. Install Xnest: the X server that runs inside a window:
    $ sudo apt-get install xnest
  2. Install a simple window manager: any lightweight window manager will do. I chose fvwm as it has defaults that work fine for this case, and it reminds me of days past:
    $ sudo apt-get install fvwm
  3. Start Xnest: choose an unused display number (most standard setups will already be using 0) — I chose 1. Note the -ac flag turns off access control, which you might want to be more careful about. The -display argument is the display the Xnest window will open in — you can skip this flag first time round but will need it if you start in a shell where DISPLAY is already set to something else:
    $ Xnest :1 -display :0 -ac &
  4. Set DISPLAY: to get further X programs to connect to Xnest rather than your primary X server, you need to set the environment variable DISPLAY to :1 (or whatever you passed as the first argument to Xnest above):
    $ export DISPLAY=:1
  5. Start your window manager: launch the lightweight manager you installed above to control windows in your nested X instance:
    $ fvwm &
  6. Run your tests: however you normally would:
    $ ant accept.master

Viola! You should now be able to watch your tests as they run, but contained completely within the Xnest window where they won’t interrupt your normal flow. Now if only my IDE would stop stealing my focus too…

Four Simple Rules For Mocking

Lately I’ve been thinking about how I approach mocking in my tests, and why it is that I like Mockito so much. I find that I follow a few simple rules:

  1. Don’t Mock: where possible, I prefer to use real implementations rather than mocks. Using real implementations is usually easier, is closer to reality (which is the ultimate test). So I only mock when real implementations are too hard to control, or too slow.
  2. Don’t Mock, Fake: if speed is the problem, instead of immediately reaching for a mock, I first consider if there is an alternative implementation. The classic example is an in-memory implementation of what would usually be a slow external resource (like a file system or database). This is a middle ground — close to reality, but fast enough to keep the test suite snappy.
  3. Don’t Mock, Stub: I prefer not to verify directly using mocks, as this risks testing the implementation of the unit rather than its interface. So even when using a mocking library, I prefer to stub behaviour and assert based on the output and state of the unit.
  4. OK, Mock: if I strike out on rules 1, 2 and 3, I know I’ve hit the sweet spot for full-blown mocking.

My conclusion is that although mocks are a powerful and necessary tool, my personal tendency is to avoid them until absolutely necessary. Even when I use mocks, I avoid verification where possible. This is why Mockito works so well — because it favours stubbing with selective verification over the classical expect-run-verify model.