a little madness

Any excuse is good enough to get me to Paris, especially while it is only a train ride away. Daniel has actually been tempted all the way from Sydney!¹ So you’ll find us both at CITCON Europe 2009 tomorrow night and Saturday. We’re both looking forward to a great weekend, after nothing but positive experiences at previous events. Hopefully we’ll even get a few questions about the new Pulse 2.1 Beta while we’re there!

–
¹ Although combining it with a well-deserved holiday may have been a factor…

Exciting news: today we’ve pushed the latest version of Pulse, namely 2.1, to beta! This is the culmination of months of hard work on a ton of new features and improvements, including:

• Project dependency support.
• Easier multi-command projects.
• Personal build improvements.
• Fine-grained cleanup rules.
• Built-in reference documentation.
• Pluggable commands (build tool support).
• A simpler, faster configuration UI.

The new features are described in more detail on the 2.1 beta page. The largest are the first two: dependencies and multi-command projects.

Project Dependencies

The ability to deliver artifacts from one build to another is a long-standing feature request. Pulse 2.1 supports this as part of a larger dependencies feature. Essentially you can declare one project to be dependent on another, allowing the downstream project to use artifacts built by the upstream one. Artifacts are delivered through an internal artifact repository.

The dependencies feature goes beyond artifact delivery. It also includes smarter triggering for dependent projects, the ability to rebuild a full dependency tree and a new “dependencies” tab which allows you to visualise the dependency graph.

Dependency support is built on top of Apache Ivy. Our aim is for interoperability with existing tools like Ivy and Maven, but without being Java-specific.

Multi-Command Projects

We’ve always had support for multi-command projects in the Pulse build core. However, to access this full flexibility you previously had to write an XML “pulse file” by hand. As of 2.1, the configuration GUI exposes the full flexibility of the underlying build core. This allows you to define multiple recipes per-project, each of which can have multiple commands. All of the advanced command options once restricted to XML files are now also accessible in the GUI.

A key feature related to this is the ability to plug in new commands (e.g. to support a new build tool), and have the plugin seamlessly integrated into the add project wizard. If you plug in support for a command, you get simplified wizard creation of single-command projects using your plugin for free.

Give It A Go!

You can download Pulse today to try it out. Free licenses are available for evaluation, open source projects and small teams.

I’m no Maven fanboy, but for a new, small Java project the ultra-fast setup time is compelling. So, for my latest little application, I decided to give it a go. Sadly, the default Maven archetype lumped my project with JUnit 3.8.1. Boo. And although the TestNG website mentions an alternative archetype, it appears to have disappeared off the face of the internet.

Luckily, dragging my project into the present wasn’t difficult. Along the way I also added my essential testing libraries: Hamcrest for matchers and Mockito for mocking (well, stubbing, but that’s another story). For posterity’s sake, and for others that share my testing tastes, here’s how it’s done.

Requirements

I’m assuming that you have Maven 2 installed already. If not, it’s trivial to:

1. Download the latest (2.1.0 at time of writing); and
2. Install it according to the instructions provided.

You can check if you have Maven ready to go by running:

Bootstrap

With the lack of an available alternative, I found it easiest to start with the default archetype. To create a new project, run something like:

Remember that if you’ve just installed Maven it will take this opportunity to download the internet. Be patient. If you’re new to Maven, you might also want to check out the 5 minute guide which walks through this in more detail.

Check that your bare application has been created:

Add the Dependencies

Next you need to update the project POM to add the testing libraries as dependencies to your build. This involves three changes:

1. Removing the default dependency on JUnit.
2. Adding new dependencies for TestNG, Hamcrest and Mockito to the “test” scope.
3. Configuring the compiler to accept Java 5 source.

The last step is necessary as the Maven default setting assumes Java 1.3 source, which apart from being ancient doesn’t support goodies such as annotations that are required for the new testing libraries. Your updated pom.xml file should look something like:

I’ve used the latest available versions of each of the libraries in this example — tweak them to suit your current reality.

Update the Sample Test

Now you’re ready to try updating the sample test case to use the trinity of TestNG, Hamcrest and Mockito. The easiest way to do this is to get Maven to generate a project for your IDE, e.g.

or:

Fire up your chosen IDE, open the AppTest class, and edit it to exercise all of the dependencies:

What are you waiting for? Try it out:

If you got this far, then everything you need is in place. Now you just have to … implement your project!

Extra Credit

If you poke about a bit, you will also find that the maven surefire plugin, which manages the tests, generates some reports by default. Along with HTML output, it also produces a JUnit-like XML report at:

This report is ideal for integration with a continuous integration server (in my case Pulse, naturally, but many will support it).

Happy testing!

I’ve just released another Pulse 2.0 build, and the new features are still coming . In this case, I have added the long-anticipated ability to capture custom metrics with your build results, then report trends for these metrics over time. You can capture any benchmark of your build that you can imagine, then have Pulse chart it for you. All sorts of charts are supported, with flexible configuration.

As an example, imagine capturing some simple performance measures as numbers. You can then configure Pulse to show them in a line chart:

Pulse stores the metric values with each build, so you can easily see how performance is changing over time:

Most importantly, if performance drops, it is easily visible exactly when it happened, making it much easier to figure out why. You can learn more about how to configure custom reports in the Cookbook.

As part of this change, I also tweaked the default reports that Pulse generates for you. Now all of these reports are configured in the same way as the custom ones above, allowing you to change them to suit your preference. Not only that, but more reports come “out of the box”:

Sweet! Interested in continuous monitoring of your project’s performance? Try Pulse 2.0 today with a free small team, open source or evaluation license.

The Pulse 2.0 feature train keeps on rolling. Latest stop: using Pulse to communicate responsibility for a build.

Breaking the build is something to avoid, but even more important is the reaction when it inevitably does break. The key is fixing it fast, before it starts breaking the team’s flow. The worst possible scenarios are:

• Everyone assumes somebody else is responsible, so nobody fixes it. The build stays broken.
• Multiple people assume responsibility without talking to each other. Effort is wasted as they all work on fixing the same problem.

Both of these scenarios can be fixed by the same solution: communication. Responsibility for fixing the build needs to be taken by one person, quickly, and communicated to the rest of the team. If nobody has taken responsibility, everyone needs to be aware.

Pulse now supports this directly via the take responsibility feature. When you see the build is broken, you can click a link to take responsibility, optionally adding a comment:

Everybody can see who is responsible, both on the project home page, and the summary pages for all builds for the project:

Only one person can be responsible at a time, so there’s no confusion. It’s up to the person responsible to decide when their job is done — although you can optionally have responsibility automatically cleared when a build completes successfully (a pretty good indicator that it’s fixed!).

So, start communicating today, and stop wasting time! You can download Pulse and try a free evaluation today. Happy building!

Since we initially announced Pulse 2.0 four odd months ago, most of our new feature work has naturally been poured in to Pulse 2.1 (currently in early access). This doesn’t mean, however, that Pulse 2.0 has been standing still. On the contrary, as feedback comes in from customers we continue to tweak — mostly by the addition of minor fixes and features that each make life with Pulse a little better.

Enough improvements have piled up that I thought it would be worthwhile to round up the more important ones, especially for the sake of existing customers wondering what the latest versions can offer. Even the smallest features (implementation-wise) can have a big impact, depending on your usage pattern. Here it goes…

• Reworked dashboard and browse views: both of these views have been rewritten to be more dynamic, and they now allow you to save the collapsed state of groups using a simple toolbar.
• Greater dashboard configurability: more control over which groups and projects are shown in which order on your dashboard.
• Links from configuration pages to reports: you could always jump quickly to a project or agent’s configuration, now you can jump back easily too.
• Better log time-stamping: a stamp on every line of output for easier diagnosis of build issues.
• Free-form property descriptions: document your build properties for the benefit of your users.
• Option to run hooks for personal builds: you can now choose on a per-hook basis whether the hook should be run for personal builds.
• Import support for pulse files: manage complex pulse files by breaking them down into import-able pieces.
• Trigger-specific properties: you can pass values through to your build depending on how it was triggered, including remote API triggers.
• Support for new Trac versions: updates to Trac integration for compatibility with the latest Trac release.
• Support for Boost.Test: a new plugin to integrate test results from the popular Boost C++ library.
• Support for skipped test cases: test reporting now understands that tests may be skipped and displays them accordingly.
• Improved Perforce integration: including better client management, building at a label, and support for ticket authentication.

We hope these features make Pulse just that little bit nicer to work with, and look forward to delivering some even bigger features in 2.1. If you want a slice of the action, download the latest Pulse release now!

Following on from my Boost.Test primer, the key goal for me was test result reporting in a continuous integration server¹. To support this, I needed to produce output from Boost.Test which I could easily consume in a plugin. As it happens, Boost.Test has built in support for producing XML reports, which are easy to parse and therefore integrate with other tools.

What was not obvious, though, was exactly how to produce output with the right level of detail. The most promising parameters in the documentation were report_format and report_level, and indeed these can be used to produce XML — but even the detailed version does not output the reason when a test case fails. It turns out that assertion failures are reported in log output, and the log format can be tweaked to produce XML (reformatted for readability):

$ ./main --log_format=XML
<TestLog>
    <Error file="main.cpp" line="20">check add(2, 2) == 5 failed</Error>
    <Error file="main.cpp" line="25">check add(2, 2) == 1 failed</Error>
</TestLog>

To get output for passing tests, and for test suites, I also had to adjust the log_level to test_suite

$ ./main --log_format=XML --log_level=test_suite
<TestLog>
  <TestSuite name="PulseTest">
    <TestSuite name="VariantsSuite">
      <TestCase name="simplePass">
        <TestingTime>0</TestingTime>
      </TestCase>
      <TestCase name="checkFailure">
        <Error file="main.cpp" line="20">check add(2, 2) == 5 failed</Error>
        <TestingTime>0</TestingTime>
      </TestCase>
...
    </TestSuite>
  </TestSuite>
</TestLog>

Bingo! Now the XML output has all the details required to render the test results nicely in a continuous integration server, provided the server has a plugin to read the XML.

–
¹ In my case obviously Pulse, but XML reports are likely to be the easiest way to integrate with other CI servers too.

Recently I implemented a Pulse plugin to read Boost.Test reports and integrate the tests into the build results. As usual, the first step in implementing the plugin was the creation of a small project that uses Boost.Test to produce some real reports to work from. I found the Boost.Test documentation to be detailed, but not so easy to follow when just getting started — so I thought I’d give an overview here.

Step 1: Installation

First you will need to install Boost.Test, or possibly all of Boost if you plan to use more of it. You can download Boost in its entirety from the Boost download page. Then you just need to unpack the archive somewhere appropriate, so you can include the headers and link against built libraries (see the next step).

An even easier option if you are on Linux is to install a package. On Ubuntu (and, I expect, other Debian variants), the desired package is libboost-test-dev:

$ sudo apt-get install libboost-test-dev

The downside of this is the packages are somewhat out of date, the default being built from Boost 1.34.1 (there is also a 1.35 variant available). I have not seen much impact of this when using Boost.Test, but if you need newer Boost libraries then it may be better to compile your own.

Step 2: Choose Your Compilation Model

Unlike many Boost libraries (which are implemented completely as headers), Boost.Test includes a runtime component which you need to link against: the “Program Execution Monitor”. This component includes the main entry point for running your tests, among other things. If you installed Boost from source, you will need to build the library yourself using bjam — the instructions are quite toolchain specific so I won’t go into them here. You can link statically or dynamically, but will need to configure your includes and build appropriately.

The key thing from the point of view of writing and building your tests is to include the right definitions in your source and add the right flags when linking. I opted for dynamic linking against the prebuilt library installed by my Ubuntu package. To achieve this, I needed two things:

1. To define BOOST_TEST_DYN_LINK before including the Boost.Test headers in my source file.
2. The addition of: -lboost_unit_test_framework to my linker flags.

With that plumbing out of the way, we can get down to testing something.

Step 3: A First Test Case

For a start, I cooked up an exceptionally useful function to add two ints, and a test case “universeInOrder” to check that it works:

#define BOOST_TEST_DYN_LINK
#define BOOST_TEST_MODULE Hello
#include <boost/test/unit_test.hpp>

int add(int i, int j)
{
    return i + j;
}

BOOST_AUTO_TEST_CASE(universeInOrder)
{
    BOOST_CHECK(add(2, 2) == 4);
}

Notice that apart from the BOOST_TEST_DYN_LINK definition, I also had to define a name for my test module via BOOST_TEST_MODULE. The case itself is defined using the BOOST_AUTO_TEST_CASE macro, giving the case name as an argument¹. Finally, within the test assertions can be made the BOOST_CHECK macro. Compiling and running the test gives the following:

$ g++ -ohello -lboost_unit_test_framework hello.cpp
$ ./hello
Running 1 test case...

*** No errors detected

Simple enough, my test passes. If I deliberately make it fail, by changing the 4 to a 5, I get:

 $ ./hello
Running 1 test case...
hello.cpp(12): error in "universeInOrder": check add(2, 2) == 5 failed

*** 1 failure detected in test suite "Hello"

Here we start to see the benefits of the library: I get a nice failure message, complete with line number and the expression that failed.

Step 4: More Assertions

Unlike the assertions in many testing libraries, a failed BOOST_CHECK will not exit the test case immediately — the problem is recorded and the case continues. To immediately fail a test, you can use BOOST_REQUIRE instead:

BOOST_AUTO_TEST_CASE(universeInOrder)
{
    BOOST_REQUIRE(add(2, 2) == 4);
}

To just output a warning instead of failing the test, you can use BOOST_WARN. In fact many Boost.Test assertions come in these three variants: CHECK, REQUIRE and WARN.

Richer assertions are also possible, including these notable examples:

• BOOST_CHECK_MESSAGE: allows you specify a custom failure message as a second argument. You can pass a string, or any type supporting the << operator.
• BOOST_CHECK_EQUAL: checks two arguments for equality using ==. Improves upon the normal check in the above examples by showing the actual values when the assertion fails.
• BOOST_CHECK_THROW: checks that an expression causes a specified type of exception to be thrown.

The full list of available assertions for the the version of Boost.Test I am using (1.34.1) can be found here.

Step 5: Suites

Once you have a non-trivial number of test cases, you need to organise them into suites. Note that each module (defined with BOOST_TEST_MODULE) already has a top-level suite named after that module. Further suites can be nested within a module to categorise as necessary. The easiest way to do this is to continue with the auto-registration model, and simply wrap the test cases with new macros to start and end a suite:

#define BOOST_TEST_DYN_LINK
#define BOOST_TEST_MODULE Suites
#include <boost/test/unit_test.hpp>

int add(int i, int j)
{
    return i + j;
}

BOOST_AUTO_TEST_SUITE(Maths)

BOOST_AUTO_TEST_CASE(universeInOrder)
{
    BOOST_CHECK(add(2, 2) == 4);
}

BOOST_AUTO_TEST_SUITE_END()

BOOST_AUTO_TEST_SUITE(Physics)

BOOST_AUTO_TEST_CASE(specialTheory)
{
    int e = 32;
    int m = 2;
    int c = 4;

    BOOST_CHECK(e == m * c * c);
}

BOOST_AUTO_TEST_SUITE_END()

In a normal run, you won’t see that the tests have been categorised. To show the suites in the output, you can set the log level to test_suite:

$ ./suites --log_level=test_suite
Running 2 test cases...
Entering test suite "Suites"
Entering test suite "Maths"
Entering test case "universeInOrder"
Leaving test case "universeInOrder"
Leaving test suite "Maths"
Entering test suite "Physics"
Entering test case "specialTheory"
Leaving test case "specialTheory"
Leaving test suite "Physics"
Leaving test suite "Suites"

*** No errors detected

Step 6: Fixtures

To add common setup and teardown code around your cases, Boost.Test supports fixtures. These take advantage of C++’s own mechanism for setup and teardown – construction and destruction. Indeed, you can easily add a “fixture” to a test case be just defining a type with the appropriate constructor and destructor and allocating one on the stack at the start of the case. This is repetitious, however, and not terribly explicit. From my experience the nicest way is to organise your tests into suites so that you can use one fixture per suite, and then just use BOOST_FIXTURE_TEST_SUITE in place of BOOST_AUTO_TEST_SUITE:

#define BOOST_TEST_DYN_LINK
#define BOOST_TEST_MODULE Fixtures
#include <boost/test/unit_test.hpp>

struct Massive
{
    int m;

    Massive() : m(2)
    {
        BOOST_TEST_MESSAGE("setup mass");
    }

    ~Massive()
    {
        BOOST_TEST_MESSAGE("teardown mass");
    }
};

BOOST_FIXTURE_TEST_SUITE(Physics, Massive)

BOOST_AUTO_TEST_CASE(specialTheory)
{
    int e = 32;
    int c = 4;

    BOOST_CHECK(e == m * c * c);
}

BOOST_AUTO_TEST_CASE(newton2)
{
    int f = 10;
    int a = 5;

    BOOST_CHECK(f == m * a);
}

BOOST_AUTO_TEST_SUITE_END()

Note that the test cases can refer directly to the public “m” member of the fixture type — in the background inheritance is at work, so protected members are also directly accessible. If you run this with logging, you can see that the fixture runs for each case:

$ ./fixtures --log_level=test_suite
Running 2 test cases...
Entering test suite "Fixtures"
Entering test suite "Physics"
Entering test case "specialTheory"
setup mass
teardown mass
Leaving test case "specialTheory"
Entering test case "newton2"
setup mass
teardown mass
Leaving test case "newton2"
Leaving test suite "Physics"
Leaving test suite "Fixtures"

*** No errors detected

Conclusion

I hope that gives you a decent starting point for using Boost.Test. As I mentioned in the beginning, there is plenty more documentation available at the source — it’s just daunting due to its size. Happy testing!

Update

Sample code is now available at GitHub.

–
1 – It is also possible to define free functions containing your tests and register them manually, although I find the automatic method simpler.

Having just made the third milestone build of the Pulse 2.1 Early Access Program, it occurred to me that it’s about time we started telling people about it! We’re baking two major new features into 2.1:

• Project Dependency Support: until now Pulse has had only limited support for dependencies, and we’ve recommended the use of tools like Apach Ivy for the delivery of artifacts between dependent builds. In Pulse 2.1, however, we’ve embedded Ivy in Pulse — both an artifact repository and the configuration of dependencies. Now you can simply declare what projects you depend on and Pulse will deliver artifacts between your builds. We’re also adding support for smart dependency triggering.
• Flexible Projects Without the XML: the build engine of Pulse has always been flexible, allowing you to define multiple build recipes each with an arbitrary number of commands, captured outputs and post-processing. However, to access the full flexibility you needed to write an XML Pulse file by hand. Well, no longer. In Pulse 2.1, the full flexibility of the engine is configurable via the Pulse web UI! These changes also allow support for new build tools to be plugged in more easily.

We’ve still got plenty of work to do, such as figuring out the trickier questions around dependencies and improving the usability of the new features. On top of this we also plan plenty of minor improvements based on feedback from our users. If you want to be part of the feedback process, just head over to the EAP page, download a build and tell us what you think!

This particular myth was one I hadn’t anticipated, but I’ve seen variations repeated often enough to convince me that there is some confusion about the relationship between continuous integration and nightly (or indeed daily) builds.

The first point to clarify is that having a build that runs daily is not the same as practicing continuous integration. Daily is just not frequent enough — indeed the goal should be a new build for every change, or continual builds if the former is impossible. Add on top of that the fact that having a build server is not the same as CI, and it’s clear that we are talking about different beasts.

Where things get more interesting is when people start to ask questions like “Is there any value in a nightly build when you are already practicing CI?”. It may seem, if you’re building twenty times a day, that there’s little point building a twenty-first time overnight. But far from making a nightly build redundant, a CI setup can include nightly builds as a complement to the normal continuous builds. Let’s look at some common examples:

• Fast Build/Slow Build: larger projects can often develop a test suite that takes hours to run. Although the goal should be to cut this time down mercilessly, the reality is some forms of testing are slow. A common workaround is to split the slower tests out of the continuous build (to keep it fast), and run a nightly build with the full test suite.
• Nightly Releases: a build can be used to release a full package nightly based on the last known good revision. This is a good example of how the CI build complements the nightly — by identifying the most recent revision that passed all tests, making the nightly releases more reliable.
• Resource Intensive Builds: the idle hours in the night are a great time to run tests that require a lot of resources, as the build can make use of hardware that would normally be tied up during the day.

So, nightly builds are not CI, but at the same time a CI setup should not neglect nightlies altogether.