a little madness

A man needs a little madness, or else he never dares cut the rope and be free -Nikos Kazantzakis

Zutubi

Archive for January, 2009

Continuous Integration Myth: CI vs Nightly Builds

This particular myth was one I hadn’t anticipated, but I’ve seen variations repeated often enough to convince me that there is some confusion about the relationship between continuous integration and nightly (or indeed daily) builds.

The first point to clarify is that having a build that runs daily is not the same as practicing continuous integration. Daily is just not frequent enough — indeed the goal should be a new build for every change, or continual builds if the former is impossible. Add on top of that the fact that having a build server is not the same as CI, and it’s clear that we are talking about different beasts.

Where things get more interesting is when people start to ask questions like “Is there any value in a nightly build when you are already practicing CI?”. It may seem, if you’re building twenty times a day, that there’s little point building a twenty-first time overnight. But far from making a nightly build redundant, a CI setup can include nightly builds as a complement to the normal continuous builds. Let’s look at some common examples:

  • Fast Build/Slow Build: larger projects can often develop a test suite that takes hours to run. Although the goal should be to cut this time down mercilessly, the reality is some forms of testing are slow. A common workaround is to split the slower tests out of the continuous build (to keep it fast), and run a nightly build with the full test suite.
  • Nightly Releases: a build can be used to release a full package nightly based on the last known good revision. This is a good example of how the CI build complements the nightly — by identifying the most recent revision that passed all tests, making the nightly releases more reliable.
  • Resource Intensive Builds: the idle hours in the night are a great time to run tests that require a lot of resources, as the build can make use of hardware that would normally be tied up during the day.

So, nightly builds are not CI, but at the same time a CI setup should not neglect nightlies altogether.

Continuous Integration Myth: A New Practice

With the recent increase in awareness of continuous integration, not to mention the ever-expanding number of tools in this space, it would be easy to think that it is a new practice. Actually, the practice has quite a long history with roots that go very deep considering the youth of software development as an industry.

The name “Continuous Integration” itself has been kicking around since the year 2000, when Fowler’s original article was published. This article certainly popularised the term, but also hints at the deeper history:

We are using the term Continuous Integration, a term used as one of the practices of XP (Extreme Programming). However we recognize that the practice has been around for a long time and is used by plenty of folks that would never consider XP for their work.

The updated version of Fowler’s article makes the somewhat cheeky claim:

The open source CruiseControl, the first continuous integration server, was originally created at ThoughtWorks.

There is no doubt that CruiseControl is a pioneering project in this space, but to claim it as the “first continuous integration server” is playing a little with semantics. It may have been the first project to use that label, but by the time CruiseControl was created there were already in-house tools that included all of the critical pieces. In my own personal experience I worked on a project which in 1999 had a server that built on every checkin, with a web UI and email notifications — not to mention that the tool behind it had been originated by my colleague at a previous job.

No doubt there are many more examples, likely much older, of projects using the core practices of continuous integration before we had a common term to describe it. Tracing their history is difficult — we’re talking pre-internet projects at closed shops here!

What is easier to find in earlier papers are references to daily builds. Take the McConnell classic Daily Build and Smoke Test, from 1996, which describes how daily builds were used at Microsoft. There is a key difference here: daily builds are not exactly continuous, and increasing the frequency is a core idea in the practice as we view it today. McConnell does, however, illustrate that frequency was seen as an important factor:

Some organizations build every week, rather than every day. The problem with this is that if the build is broken one week, you might go for several weeks before the next good build. When that happens, you lose virtually all of the benefit of frequent builds.

In truth, for a Microsoft project in those days building daily may have been as frequent as was practical. Consider McConnell’s reference to an even earlier Microsoft paper:

By the time it was released, Microsoft Windows NT 3.0 consisted of 5.6 million lines of code spread across 40,000 source files. A complete build took as many as 19 hours on several machines, but the NT development team still managed to build every day (Zachary, 1994). Far from being a nuisance, the NT team attributed much of its success on that huge project to their daily builds.

It’s likely that if building on every change was seen as possible, these projects would have evolved to do so quite quickly. I would even guess smaller projects at Microsoft and elsewhere had discovered this and were building as frequently as practical.

The most tantalising reference I found comes from our competitor Slava Imeshev, in a ViewTier presentation. Rumour has it that the real roots of the practice could date back much further:

Integration builds … supposedly introduced by IBM software team in early 60-s when developing OS/360

It’s unsubstantiated, but there are suggestions the build was run multiple times a day. Those familiar with Mythical Man Month can make what they will of the fact that the OS/360 project was Brook’s “inspiration”.

So despite the recent growth in this area, it’s fair to say that the roots of continuous integration lie decades behind us. In an industry with a history itself measured in decades, this makes continuous integration anything but a new practice!


Interested in continuous integration and want to try it today? Download Pulse and get a free license for your small team, open source project or evaluation!

Continuous Integration Myth: Large Teams Only

It’s been a little while, but my theme hasn’t finished yet. In this case I am revisiting an idea I posted about some time ago. Due to the clear benefits of continuous integration for large teams, sometimes people forget the ways in which it can be useful for small teams. Even teams as small as one.

For small teams, the benefits are less about communication (although that is still an important factor as soon as you have more than one developer), and more about automation. Calling out the points made in my last post, these benefits include:

  • Build and release automation: nobody should waste time on manual build and release processes, least of all teams already short on development resources. Setting up a continuous integration process enforces a fully automated build, which naturally extends to automated releases.
  • Building across multiple environments: as a small team with a correspondingly small number of development environments, it’s unlikely your code is naturally exercised on all the platforms your support. A continuous integration server that supports distributed builds can make it easy for you to test across platforms.
  • Testing frequency: a small team of developers will naturally have a lower overall frequency of testing than a large team, simply because there are less people exercising the code as they work. A continuous build can help close the gap, helping to uncover Heisenbugs that might otherwise escape into the wild.

Perhaps the largest benefit is a shift to the continuous integration mindset of working — coding in short iterations and checking in at least daily. The solo developer, with nobody else waiting for their changes, can easily bash away at the keyboard for weeks between checkins. Sometimes this works fine, but other times you end up losing perspective and sometimes motivation as you keep digging and digging away. Nothing keeps development on track better than taking baby steps and constant assessment of progress, and the very feeling of progress feeds your motivation.

This is what continuous integration is all about — baby steps, constant feedback.


Small team looking for a CI solution? I’ve got good news: we offer small team licenses for free, with no obligation whatsoever!