a little madness

Continuous integration is all about feedback, but feedback is no good when the signal-to-noise ratio is too low. That’s when people start to ignore the feedback altogether. This starts to happen when your CI server sends out too many “broken build” emails. Eventually the team will find it is not worth their while to pay the emails any attention. Once you lose their attention, the feedback is worthless and you may as well not be doing CI at all.

How do you solve this? The absolute key is to make sure that the normal state of your build is green. Broken builds should be the minority, so when they happen people start asking why. How do you keep the build green? There are multiple ways, such as:

• Have developers test before checking in. Even better, if your CI server supports personal builds let the server test the changes before they are checked in. If the full suite is too long, at least have a fast subset that can be run pre-checkin.
• You break it, you bought it. When the build breaks, the first priority should be to fix it. The CI server should identify the suspect changes, and their authors should immediately investigate.
• No check-in-and-bust-outs. Don’t check in then leave the scene of the crime before the light goes green. If you want to run a build before taking off somewhere, make it a personal build!
• Categorise known failures separately. Just removing these failures runs the risk of the test cases being forgotten altogether, so find a way to leave them in without them breaking the build. Even a low-tech solution that reports a warning for each known failure without running the test case would suffice.
• Clean up your mess. More involved test suites often play with lots of external resources. When a build stops up short, these resources can be left hanging around. Make sure your build cleans up these resources, lest the following build (and beyond) be affected. I have found it helpful to have a cleanup step just before the tests are started, as depending on how the previous build was killed it may not have been given the opportunity to do the cleanup itself.
• Track down intermittent failures. These problems can be difficult to debug, and if they happen rarely enough it is tempting to ignore them. Don’t. Imagine how hard this problem would be to debug for a remote customer!
• Use dedicated CI machines. Don’t let other activities interfere with your builds. Machines are cheap, your time is not.

Encompassing all of these points is the creation a culture where green is The Way. Sure, failures will happen, but the team needs to know that it is not cool to be a regular build breaker.

Liked this post? Share it!

This entry was posted on Friday, July 20th, 2007 at 4:05 am and is filed under Agile, Continuous Integration, Technology, Testing, Zutubi. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.