Continuous Integration Myth: The Build Must Never Break
Continuing the theme, another misguided idea is that your CI build must never break. I think the real issue here is that this idea focuses on the wrong thing: it’s not really broken builds that matter, it’s why they break and what you do about it.
Broken Builds Are Information
If your build never breaks, then it doesn’t necessarily tell you much. Saying an always-green build implies your product works is analogous to saying that high test coverage implies your code is fully tested. Green builds are good, but how informative they are is highly dependent on their quality. I particularly like Paul Duvall’s characterisation of always-green builds as a possible case of Continuous Ignorance.
When a build breaks, however, you know something is wrong1. This is a concrete indicator that your build has some value: it has found a problem.
Broken Builds != Broken Windows
So a broken build gives you information; the key thing now is how you react to it. Teams in tune with CI react to broken builds by:
- Fixing them fast: to reduce the impact on the team, and in the knowledge that the cheapest time to make the fix is now.
- Learning from this failure: and finding ways to prevent it in the future.
Teams that don’t deal with broken builds suffer “Broken Windows” syndrome. This is not because the build broke, however, but because the team didn’t respond as they should have.
What About the Cost?
I’m not saying that a broken build is all good — after all productivity is easily lost while the build is red. In Pulse we have even added many features like personal builds specifically to help reduce the frequency of build failures. However, aiming for completely unbreakable builds also has a cost. All solutions must in the end rely on complete serialisation of checkins and builds. I see this as analogous to pessimistic locking: you take a producitivity hit on every commit just in case something breaks. Not to mention the fact that:
- For many larger projects this is simply impractical due to the combination of commit frequency and build time.
- There is still no guarantee of a green build thanks to the existence of non-deterministic bugs.
We advocate an optimistic approach: by using local smoke testing and optional personal builds on each checkin, the vast majority of problems are found. This leaves only the much less frequent chance of a failure due to a logical merge conflict2 or non-determinism. The great thing about this approach is it doesn’t enforce any heavyweight overhead such as extra branching or serialisation on the process, you can pretty much work as you always have.
Conclusion
So, put your effort into a high quality build and listen to what it is telling you. Don’t get hung up on absolute guarantees of unbreakability: with the right response to broken builds you’re better off with an optimistic approach.
—
1 OK, I am glossing over spurious failures here, but I would like to deal with that in another post.
2 As opposed to a textual merge conflict which must be fixed prior to checking in.
This entry was posted on Friday, November 21st, 2008 at 1:13 am and is filed under Agile, Continuous Integration, Technology. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.
November 21st, 2008 at 4:02 am
I totally agree on this one, and I spend loads of time telling clients about this. CI is all about feedback, and acting on that feedback. Broken builds are information, use it as such…
Chris
November 23rd, 2008 at 9:25 am
ya i agree.. i am working in a project where breaking build is one of the biggest crime, so no one wants to refactor the code even if it’s getting ugly and worse.
November 24th, 2008 at 1:59 am
Well put! Over in the urbancode office, we use the same language about pessimistic and optimistic locking. Happy to see that you guys do as well. Occasional build breakages should be ok, and if they are a major productivity killer for some team, that’s a bad smell the team needs to address.
I’m also glad to see some someone else discussing personal builds (we call them preflight) in a sane way rather than as the solution to “unbreakable builds” which would require stupid levels of serialization.
November 25th, 2008 at 9:40 am
Hi Chris, narup, Eric,
Thanks for the positive comments.
@Eric: Yep, personal/preflight builds are a brilliant tool, but no magical solution. They suit some projects and situations better than others.
January 6th, 2009 at 8:41 am
a little madness » Continuous Integration Myth: Large Teams Only says:[…] been a little while, but my theme hasn’t finished yet. In this case I am revisiting an idea I posted about some […]
September 4th, 2009 at 1:47 am
Nels Olsen says:What is an acceptable level for broken builds? I am not asking our team for 100% green, but our build failure rate is between 10 and 25%, and 85% of days have broken builds. A good 25% of build failures are “hard failures” (doesn’t compile), but about 75% of those are because the build server is buggy and the failure is not really related to the checked in code. Still, that waste’s other team members time waiting for the build server to report as clean — others don’t care why the build is failing, only that it is and that they will have to hassle with ignoring external failures if they try to continue with their own work.
The time to get a break fixed is on average over an hour. Many on the team think that our build failure frequency, percentage and time-to-fix is acceptable and that waiting 10 minutes at most 4 times a day to run a smoke test suite before checking in is a “nightmare”. They argue that the time is utterly wasted waiting, since we don’t do things like review requirements with the customer / product owner, do TDD, pair programming, etc. which allow you to start some work on the next task.