a little madness

Just a quick note to mention that Daniel will be carrying the torch for Zutubi at CITCON Asia-Pacific 2008 in Melbourne this weekend. After enjoying the event last year we signed up to sponsor this time around. Catch up with Daniel and ask him about Pulse 2.0 beta!

Sadly I won’t make it, on account of being on the other side of the planet…

Our dog food Pulse server, which spends all day building itself, is a headless box. This presented no challenge for the Pulse 1.x series of releases, as our builds are all scripted and don’t require any GUI tools. With Pulse 2.0, however, things changed. As mentioned in my previous post, this new release has a whole new acceptance test suite based on Selenium. The problem is: Selenium runs in a real browser, and that browser requires a display.

Fortunately, the dog food server is also a Linux box. Thus there is no need to add in a full X setup with requisite hardware just to have Selenium running. This is thanks to the magic of Xvfb - the X virtual framebuffer. Basically, this is a stripped back X server that maintains a virtual display in memory. Hence no actual video hardware or driver is needed, and we can keep things simple.

Setting things up is straightforward. First, install Xvfb:

on Debian/Ubuntu; or

on Fedora/RedHat. Then, choose a display number that is unlikely to ever clash (even if you add a real display later) - something high like 99 should do. Run Xvfb on this display, with access control off¹:

Now you need to ensure that your display is set to 99 before running the Selenium server (which itself launches the browser). The easiest way to do this is to export DISPLAY=:99 into the environment for Selenium. First, make sure things are working from the command line like so:

Firefox should launch without error, and stay running (until you kill it with Control-C or similar). You won’t see anything of course! If things go well, then you need to modify the way you launch the Selenium server to ensure the DISPLAY is set. There are many ways to do this, in our Pulse setup we actually use a resource as it is a convenient way to modify the build environment.

An alternative which may suit some setups is to use the xvfb-run wrapper to launch your Selenium server. I opted against this as Pulse has resources to modify the environment and this way I do not need to change the actual build scripts at all. However if you are not using Pulse (shame! ) then you may want to look into it.

–
¹ If you are worried about access on your network, then using -ac long-term is not a good idea. Once you have things working, I would suggest tightening things up by turning access control on.

Wondering why everything has gone quiet over here? Well, all should become clear now: I have just completed a move from rainy Sydney to sunny London¹. Combine the Christmas break with an overseas move and you have a recipe for zero blog posts!

The good news is that I am back in action in a new home in central London (Baker Street area). So, for now at least, Zutubi is operating in both Sydney (Daniel - GMT+11) and London (me - GMT) - the company never sleeps!

I intend on travelling to quite a bit of the UK and continental Europe while we are living here. This will hopefully give me the opportunity to meet some of our European customers. Let me know if you are in the area, and perhaps we can arrange to hook up over the coming months at some point.

–
¹ Yes, it really has been quite sunny and mild since we got here, quite to our surprise! Reports from back home tell of plenty more rain down that way.

OK, so it’s not quite the same as Quake Done Quick, but Chris has put together a few demo videos that show just how easy it is to set up Pulse and get building. In less than 5 minutes a server is setup with a first project and build. Another 5 minutes and you have integrated tests, SCM-triggered builds and RSS notifications. Sweet.

CITCON Sydney (the Continuous Integration and Testing Conference) is coming up tomorrow. Daniel and myself will both be attending, and look forward to the experience both as vendors of a CI tool and as of developers who are always looking to improve our own build process. This is my first open spaces conference, so it will also be interesting to see how the format works in practice. I find the theory a lot more appealing than being talked at by vendors that have paid their way into speaking slots!

Anyhow, if you’re lucky enough to be in Sydney, we’ll see you there!

Coincidentally soon after my post on Keeping It Green, Scott Hanselman posts his colleague’s First Rule of Software Development:

Never commit code just before you leave for the day.

This is identical the third item on my list for keeping the build healthy. Interestingly the developer had obeyed the first item on the list, i.e. testing locally before checkin:

The CCNET (Continuous Integration) build subsequently broke – despite the fact that he ran a local build first – and team members who were still in the office had difficulty progressing with their work for several hours.

But I’m obliged to repeat that an even better way to meet the first item is with a personal build. If the developer had been using Pulse, then they could have submitted a personal build to Pulse and found out that the build would break before they checked the code in, and all in the same or less time than it takes to run the tests locally. A small investment in better tools could have avoided the costly loss of team productivity ;).

Continuous integration is all about feedback, but feedback is no good when the signal-to-noise ratio is too low. That’s when people start to ignore the feedback altogether. This starts to happen when your CI server sends out too many “broken build” emails. Eventually the team will find it is not worth their while to pay the emails any attention. Once you lose their attention, the feedback is worthless and you may as well not be doing CI at all.

How do you solve this? The absolute key is to make sure that the normal state of your build is green. Broken builds should be the minority, so when they happen people start asking why. How do you keep the build green? There are multiple ways, such as:

• Have developers test before checking in. Even better, if your CI server supports personal builds let the server test the changes before they are checked in. If the full suite is too long, at least have a fast subset that can be run pre-checkin.
• You break it, you bought it. When the build breaks, the first priority should be to fix it. The CI server should identify the suspect changes, and their authors should immediately investigate.
• No check-in-and-bust-outs. Don’t check in then leave the scene of the crime before the light goes green. If you want to run a build before taking off somewhere, make it a personal build!
• Categorise known failures separately. Just removing these failures runs the risk of the test cases being forgotten altogether, so find a way to leave them in without them breaking the build. Even a low-tech solution that reports a warning for each known failure without running the test case would suffice.
• Clean up your mess. More involved test suites often play with lots of external resources. When a build stops up short, these resources can be left hanging around. Make sure your build cleans up these resources, lest the following build (and beyond) be affected. I have found it helpful to have a cleanup step just before the tests are started, as depending on how the previous build was killed it may not have been given the opportunity to do the cleanup itself.
• Track down intermittent failures. These problems can be difficult to debug, and if they happen rarely enough it is tempting to ignore them. Don’t. Imagine how hard this problem would be to debug for a remote customer!
• Use dedicated CI machines. Don’t let other activities interfere with your builds. Machines are cheap, your time is not.

Encompassing all of these points is the creation a culture where green is The Way. Sure, failures will happen, but the team needs to know that it is not cool to be a regular build breaker.

Looking over the companies that have purchased Pulse licenses, it occurs to me that a fairly large percentage of them are in the games development industry. In fact, it appears to me that game developers are over-represented. This could mean any one of a few things:

1. For whatever reason, Pulse is particularly appealing to game developers, relative to developers in general. This could be the case, but given we have not made any special effort to target game developers it would be coincidental.
2. I have underestimated the percentage of the industry that is involved in game development. Possible, as I have no hard data to back up my intuition.
3. Game developers are more likely to adopt continuous integration than the average development team.

I could be wrong, but I think the largest factor is 3. To me, this means that developers in the games industry are more savvy than the average, at least in terms of applying helpful practices to their development process¹. Looking one step deeper, I suppose the next question is why would this be the case? Are game developers smarter than the average bear? I don’t think I would make that sort of generalisation, but I would say I think it helps that they are in a “pure” development industry. That is, their core business is to create software, so naturally as a company they are focused on doing so as efficiently as possible. Development teams in non-software industries surely strive for efficiency too, but do not always have the full backing they require.

–
¹ I guess I am biased, but to me continuous integration is one of those things every team should be doing.

Since I have previously stated my admiration for UnitTest++ as a unit testing framework for C++, I guess it was high time I added direct support for it in Pulse. Well, as of Pulse 1.2.24, there is now a UnitTest++ post-processor that will slurp your UnitTest++ test results directly into Pulse. If you’re looking to do continuous integration for C++, then Pulse + UnitTest++ is a killer combo ;).

Paul Duvall’s post The Future of Continuous Integration points out a weakness in how continuous integration is typically implemented: code must be checked in before it is tested by the continuous integration server. Although problems are picked up fast, there is still a time window where developers may be affected by broken code. This is exactly why we added personal builds to Pulse in version 1.2. Personal builds allow individual developers to submit their changes to Pulse for testing before committing to version control. Pulse does a build and test of the latest version with the developer’s changes applied, allowing to developer to check the full impact of their changes prior to affecting the shared code base. Apart from closing the time window, personal builds also allow other powerful usage patterns, such as using the distributed building features of Pulse to test across multiple platforms before checking in.

Paul also touches on a couple of different solutions to the weakness. Naturally, before implementing personal builds as we did, we considered a few options. One option that Paul mentions that we had considered was leveraging the branching functionality of the SCM. Developers (perhaps with the assistance of the CI server) could check their code into private or temporary branches for testing, and the changes could then be merged down to the shared branch if the build passed. The main attraction of this method is that it reuses functionality available already in the SCM. However, we decided not to go down this route as it can have an impact on the way developers need to work and/or the structure and history in the SCM. Our method of taking a patch from the developer’s working copy and applying it directly with Pulse is about as unobtrusive as you can get. This is important for us, as the ability to adapt to existing environments has always been a key design goal for Pulse.

For more about these and related issues, you might want to also check out an article I posted a while ago to zutubi.com. Reducing the Impact of Broken Builds talks about the affects of a broken code base and how best to avoid or alleviate them.