a little madness

Do The Simplest Thing That Could Possibly Work, then Refactor Mercilessly: two well known Extreme Rules. It’s hard to argue with the virtues of refactoring, but if you’ve ever had to manage parallel codelines then you might have a different perspective on “mercilessly”. Many types of refactoring are at odds with parallel codelines because they make merging - an already difficult task - more difficult. In particular, changes that physically move content around amongst files tend to confuse SCMs which take a file-based view of the world (i.e. the majority of them). How can we alleviate this tension? A few possibilities come to mind:

1. Don’t do parallel development: if you don’t have to merge, you don’t have a problem. Since the act of merging itself is pure overhead to the development process, this is a tempting idea. However, reality (typically of the business kind), dictates that parallel development is necessary to some degree. How do you support customers on older versions? How do you take on large and risky subprojects without derailing the development trunk? These are scenarios where branching is valuable enough to justify the overhead of merging. It is reasonable therefore to try and minimise parallel development, but rarely possible to avoid it completely.
2. Only refactor on the trunk (or whereever the majority of development is done). This helps to alleviate the problem to some degree, by containing the main divergence to a single codeline. It is also usually the codeline where you would naturally do most of your refactoring, as it is the bleeding edge. However, even merging a small bug fix from a maintenance branch may prove difficult due to code movements on the trunk. And this solution is not much help for active development branches that run parallel to the trunk for some time.
3. Another Extreme Rule: Integrate Often. It is no coincidence that Perforce-speak for “merge” is “integrate”. Merging is a type of integration, and like any it is less painful if done regularly, before codelines have diverged too far. This is, however, a way to mitigate problems rather than solve them.
4. Avoid problematic refactoring, such as moving code between files. In some cases it is clear that a certain refactoring is best avoided due to anticipated merging. However, avoiding all problematic refactoring is not a workable long-term strategy. At some point, the maintenance benefits of cleaning up the code will outweigh the merging cost.

In reality we use a combination of these ideas and a dose of common sense to reduce merging problems. However, the tension still exists and can be painful. A technology-base solution to this would of course be ideal, i.e. an SCM that understood refactoring and could take account of it when merging. Unfortunately I know of no such existing SCM, and there are significant barriers to creating one:

1. Many popular SCMs of today struggle enough with the file-based model that it is hard to see them moving on to such a high level.
2. File-based, text-diff change management applies genericly to a wide range of tasks. Any tool with deeper knowledge of actual file contents would likely trade off some of this genericity.
3. Any “smart” merging algorithm will still have many corner cases where the right choice is ambiguous, and making these comprehensible to the poor human that needs to resolve them is more difficult when the algorithm is more complicated.

On a more positive note, there has been some welcome innovation in the SCM space recently. Perhaps I have overlooked solutions that exist now, or perhaps the growing competition will spur innovation in this area. Either way, if you have any ideas, let me know!

OK, so it’s not quite the same as Quake Done Quick, but Chris has put together a few demo videos that show just how easy it is to set up Pulse and get building. In less than 5 minutes a server is setup with a first project and build. Another 5 minutes and you have integrated tests, SCM-triggered builds and RSS notifications. Sweet.

Server applications can often benefit from exposing a remote API. Such an API enables users to both automate tasks and integrate with other systems. In the case of Pulse, we expose an XML-RPC remote API that allows remote control and monitoring of the Pulse server.

Apart from being a great feature, a remote API has another significant benefit: it enables us to control Pulse through a (relatively) fast interface during acceptance testing. The primary Pulse UI is accessed via a web browser. Although it is possible to automate web UI testing using tools such as JWebUnit and Selenium RC, the resulting tests are slow. Where we are not testing the web UI, or are testing only an isolated part of it, the overhead of driving the UI causes a huge blowout in testing time. This leads to slow builds: the enemy of continuous integration.

Thus, in many of our acceptance tests, a lot of the peripheral work such as setting up suitable data is done using the remote API. We also use the reporting functionality of the remote API to assert the current state of Pulse where possible. Tests only drive the web UI when they are testing the operation of the UI itself. The resulting tests are a lot quicker, enabling us to run our acceptance test suite more frequently. In Pulse 2.0, where the remote API has been extended to support full configuration, we are beginning to enjoy even faster acceptance tests.

So, if you are struggling with slow acceptance tests, consider adding a remote API to your application. As a bonus, your users get a great new feature!

Today I ran across this post which quotes a testing-related analogy:

Checking code into source-control without unit-tests is like betting on a poker hand without looking at the cards. — Anon.

A nice theme, which brought to mind another gambling analogy:

Developing software without unit tests is like playing the pokies¹.

How so? In a few ways:

• Poker machines are a mug’s game: in the long term, you always lose. Just so with a lack of unit testing: it will come back to haunt you as you struggle to fix bugs discovered late and lose the confidence to refactor.
• The really insidious thing with poker machines is that they are designed to keep you playing by paying out small wins reasonably frequently. Likewise, developers get addicted to the short-term feeling of productivity when they cut reams of code without tests.
• Gambling on pokies can become a downward spiral: “just one more spin and I’m done … ok, another one, I’m due for a win …”. Sounds scarily like the development team that will put better testing in place “just after we hit this important deadline”. There’s always another spin, and there’s always another deadline.

The key thing is to think long term. Don’t play a game you are set up to lose.

–
¹ That’s poker machines/slot machines/fruit machines depending on where in the world you are throwing away your money.

Well, CITCON Asia/Pacific 2007 is done. On the whole, the event was a great success. There was a great turnout, lots of interesting discussion, and all the organisation went smoothly. Thanks goes to both organisers, Jeffrey Fredrick and Paul Julius, for pulling it off.

Looking back on the conference, both Daniel and myself gained a lot:

• A first experience of an OpenSpace conference. Going in, I liked the theory of OpenSpace a lot, and in practice I have to say that despite some challenges it is a great format. Sessions are a lot more interactive, and you can get more out of a session by putting more in yourself. Of course, not all sessions work out as you may have hoped. From my observation, smaller sessions on more focused topics are more likely to succeed. A larger session on a more general topic needs an experienced facilitator. The one topic I proposed was more of an all-in discussion, so I didn’t really feel the need to do much “facilitation”.
• Of course, we got plenty out of the sessions themselves. My personal favourite was a small session about continuous integration with distributed SCMs. This is an interest of mine that I hope to follow up soon. We also got great insights into the problems that our customers (and potential customers ) face. Daniel made the observation that people reported many problems just managing their builds over time, even before getting to automated testing and continuous integration. This is an area that is of obvious importance to us as you need a good build to apply Pulse most effectively.
• We got to meet a bunch of people from various software backgrounds all interested in continuous integration. It was great to see such a turn out and enthusiasm for this area, and really reinforces that this is a boom time for improving software build and test practices. As noted by many attendees, having food put on right at the venue really helped keep discussions going. The times between sessions were just as valuable as the sessions themselves.
• We got to meet some more of our competitors, to whom we can naturally relate. There is a great spirit between the competitors I have met in our field. It’s great to be able to have a chat about the common problems we all face, and the opportunities in the future.
• Beer. And curry. ‘Nuff said.

And all this was free. If only there were more conferences organised in this spirit.

CITCON Sydney (the Continuous Integration and Testing Conference) is coming up tomorrow. Daniel and myself will both be attending, and look forward to the experience both as vendors of a CI tool and as of developers who are always looking to improve our own build process. This is my first open spaces conference, so it will also be interesting to see how the format works in practice. I find the theory a lot more appealing than being talked at by vendors that have paid their way into speaking slots!

Anyhow, if you’re lucky enough to be in Sydney, we’ll see you there!

Coincidentally soon after my post on Keeping It Green, Scott Hanselman posts his colleague’s First Rule of Software Development:

Never commit code just before you leave for the day.

This is identical the third item on my list for keeping the build healthy. Interestingly the developer had obeyed the first item on the list, i.e. testing locally before checkin:

The CCNET (Continuous Integration) build subsequently broke – despite the fact that he ran a local build first – and team members who were still in the office had difficulty progressing with their work for several hours.

But I’m obliged to repeat that an even better way to meet the first item is with a personal build. If the developer had been using Pulse, then they could have submitted a personal build to Pulse and found out that the build would break before they checked the code in, and all in the same or less time than it takes to run the tests locally. A small investment in better tools could have avoided the costly loss of team productivity ;).

Continuous integration is all about feedback, but feedback is no good when the signal-to-noise ratio is too low. That’s when people start to ignore the feedback altogether. This starts to happen when your CI server sends out too many “broken build” emails. Eventually the team will find it is not worth their while to pay the emails any attention. Once you lose their attention, the feedback is worthless and you may as well not be doing CI at all.

How do you solve this? The absolute key is to make sure that the normal state of your build is green. Broken builds should be the minority, so when they happen people start asking why. How do you keep the build green? There are multiple ways, such as:

• Have developers test before checking in. Even better, if your CI server supports personal builds let the server test the changes before they are checked in. If the full suite is too long, at least have a fast subset that can be run pre-checkin.
• You break it, you bought it. When the build breaks, the first priority should be to fix it. The CI server should identify the suspect changes, and their authors should immediately investigate.
• No check-in-and-bust-outs. Don’t check in then leave the scene of the crime before the light goes green. If you want to run a build before taking off somewhere, make it a personal build!
• Categorise known failures separately. Just removing these failures runs the risk of the test cases being forgotten altogether, so find a way to leave them in without them breaking the build. Even a low-tech solution that reports a warning for each known failure without running the test case would suffice.
• Clean up your mess. More involved test suites often play with lots of external resources. When a build stops up short, these resources can be left hanging around. Make sure your build cleans up these resources, lest the following build (and beyond) be affected. I have found it helpful to have a cleanup step just before the tests are started, as depending on how the previous build was killed it may not have been given the opportunity to do the cleanup itself.
• Track down intermittent failures. These problems can be difficult to debug, and if they happen rarely enough it is tempting to ignore them. Don’t. Imagine how hard this problem would be to debug for a remote customer!
• Use dedicated CI machines. Don’t let other activities interfere with your builds. Machines are cheap, your time is not.

Encompassing all of these points is the creation a culture where green is The Way. Sure, failures will happen, but the team needs to know that it is not cool to be a regular build breaker.

Since I have previously stated my admiration for UnitTest++ as a unit testing framework for C++, I guess it was high time I added direct support for it in Pulse. Well, as of Pulse 1.2.24, there is now a UnitTest++ post-processor that will slurp your UnitTest++ test results directly into Pulse. If you’re looking to do continuous integration for C++, then Pulse + UnitTest++ is a killer combo ;).

Paul Duvall’s post The Future of Continuous Integration points out a weakness in how continuous integration is typically implemented: code must be checked in before it is tested by the continuous integration server. Although problems are picked up fast, there is still a time window where developers may be affected by broken code. This is exactly why we added personal builds to Pulse in version 1.2. Personal builds allow individual developers to submit their changes to Pulse for testing before committing to version control. Pulse does a build and test of the latest version with the developer’s changes applied, allowing to developer to check the full impact of their changes prior to affecting the shared code base. Apart from closing the time window, personal builds also allow other powerful usage patterns, such as using the distributed building features of Pulse to test across multiple platforms before checking in.

Paul also touches on a couple of different solutions to the weakness. Naturally, before implementing personal builds as we did, we considered a few options. One option that Paul mentions that we had considered was leveraging the branching functionality of the SCM. Developers (perhaps with the assistance of the CI server) could check their code into private or temporary branches for testing, and the changes could then be merged down to the shared branch if the build passed. The main attraction of this method is that it reuses functionality available already in the SCM. However, we decided not to go down this route as it can have an impact on the way developers need to work and/or the structure and history in the SCM. Our method of taking a patch from the developer’s working copy and applying it directly with Pulse is about as unobtrusive as you can get. This is important for us, as the ability to adapt to existing environments has always been a key design goal for Pulse.

For more about these and related issues, you might want to also check out an article I posted a while ago to zutubi.com. Reducing the Impact of Broken Builds talks about the affects of a broken code base and how best to avoid or alleviate them.