a little madness

Well, CITCON Asia/Pacific 2007 is done. On the whole, the event was a great success. There was a great turnout, lots of interesting discussion, and all the organisation went smoothly. Thanks goes to both organisers, Jeffrey Fredrick and Paul Julius, for pulling it off.

Looking back on the conference, both Daniel and myself gained a lot:

• A first experience of an OpenSpace conference. Going in, I liked the theory of OpenSpace a lot, and in practice I have to say that despite some challenges it is a great format. Sessions are a lot more interactive, and you can get more out of a session by putting more in yourself. Of course, not all sessions work out as you may have hoped. From my observation, smaller sessions on more focused topics are more likely to succeed. A larger session on a more general topic needs an experienced facilitator. The one topic I proposed was more of an all-in discussion, so I didn’t really feel the need to do much “facilitation”.
• Of course, we got plenty out of the sessions themselves. My personal favourite was a small session about continuous integration with distributed SCMs. This is an interest of mine that I hope to follow up soon. We also got great insights into the problems that our customers (and potential customers ) face. Daniel made the observation that people reported many problems just managing their builds over time, even before getting to automated testing and continuous integration. This is an area that is of obvious importance to us as you need a good build to apply Pulse most effectively.
• We got to meet a bunch of people from various software backgrounds all interested in continuous integration. It was great to see such a turn out and enthusiasm for this area, and really reinforces that this is a boom time for improving software build and test practices. As noted by many attendees, having food put on right at the venue really helped keep discussions going. The times between sessions were just as valuable as the sessions themselves.
• We got to meet some more of our competitors, to whom we can naturally relate. There is a great spirit between the competitors I have met in our field. It’s great to be able to have a chat about the common problems we all face, and the opportunities in the future.
• Beer. And curry. ‘Nuff said.

And all this was free. If only there were more conferences organised in this spirit.

CITCON Sydney (the Continuous Integration and Testing Conference) is coming up tomorrow. Daniel and myself will both be attending, and look forward to the experience both as vendors of a CI tool and as of developers who are always looking to improve our own build process. This is my first open spaces conference, so it will also be interesting to see how the format works in practice. I find the theory a lot more appealing than being talked at by vendors that have paid their way into speaking slots!

Anyhow, if you’re lucky enough to be in Sydney, we’ll see you there!

Coincidentally soon after my post on Keeping It Green, Scott Hanselman posts his colleague’s First Rule of Software Development:

Never commit code just before you leave for the day.

This is identical the third item on my list for keeping the build healthy. Interestingly the developer had obeyed the first item on the list, i.e. testing locally before checkin:

The CCNET (Continuous Integration) build subsequently broke – despite the fact that he ran a local build first – and team members who were still in the office had difficulty progressing with their work for several hours.

But I’m obliged to repeat that an even better way to meet the first item is with a personal build. If the developer had been using Pulse, then they could have submitted a personal build to Pulse and found out that the build would break before they checked the code in, and all in the same or less time than it takes to run the tests locally. A small investment in better tools could have avoided the costly loss of team productivity ;).

Continuous integration is all about feedback, but feedback is no good when the signal-to-noise ratio is too low. That’s when people start to ignore the feedback altogether. This starts to happen when your CI server sends out too many “broken build” emails. Eventually the team will find it is not worth their while to pay the emails any attention. Once you lose their attention, the feedback is worthless and you may as well not be doing CI at all.

How do you solve this? The absolute key is to make sure that the normal state of your build is green. Broken builds should be the minority, so when they happen people start asking why. How do you keep the build green? There are multiple ways, such as:

• Have developers test before checking in. Even better, if your CI server supports personal builds let the server test the changes before they are checked in. If the full suite is too long, at least have a fast subset that can be run pre-checkin.
• You break it, you bought it. When the build breaks, the first priority should be to fix it. The CI server should identify the suspect changes, and their authors should immediately investigate.
• No check-in-and-bust-outs. Don’t check in then leave the scene of the crime before the light goes green. If you want to run a build before taking off somewhere, make it a personal build!
• Categorise known failures separately. Just removing these failures runs the risk of the test cases being forgotten altogether, so find a way to leave them in without them breaking the build. Even a low-tech solution that reports a warning for each known failure without running the test case would suffice.
• Clean up your mess. More involved test suites often play with lots of external resources. When a build stops up short, these resources can be left hanging around. Make sure your build cleans up these resources, lest the following build (and beyond) be affected. I have found it helpful to have a cleanup step just before the tests are started, as depending on how the previous build was killed it may not have been given the opportunity to do the cleanup itself.
• Track down intermittent failures. These problems can be difficult to debug, and if they happen rarely enough it is tempting to ignore them. Don’t. Imagine how hard this problem would be to debug for a remote customer!
• Use dedicated CI machines. Don’t let other activities interfere with your builds. Machines are cheap, your time is not.

Encompassing all of these points is the creation a culture where green is The Way. Sure, failures will happen, but the team needs to know that it is not cool to be a regular build breaker.

There has been plenty of buzz lately about so-called “dynamic” languages. I guess we have the hype around Rails and similar frameworks to thank (blame?) for this most recent manifestation of an ages-old debate. Unfortunately, the excitement around the productivity increase (real or perceived) afforded by these languages has led to some common fallacies.

Fallacy 1: Misclassification of Type Systems

I’ve rolled a few related fallacies into one here. The most common mistakes I see are classifying static as explicit typing, and dynamic as weak typing. I won’t go into too much detail, as this one is well covered by the fallacies in What To Know Before Debating Type Systems. In any case, you can usually tell what people are getting at, and concentrate on more interesting arguments.

Fallacy 2: Less “Finger” Typing == Drastically More Productivity

This one really gets to me. When was the last time you told your customer “Sorry, I would have made that deadline: I just couldn’t type all the code fast enough!”? The number of characters I need to type has very little to do with how fast I code. This is for a few reasons:

1. For anything but short bursts, I type a lot faster than I think. Most programming time goes into thinking how to solve the non-trivial problems that come up every day. This reason alone is enough to dispel the myth.
2. Every second thing I type is spelled the same: Control-Space. Honestly, good IDEs do most of the typing for you, especially when they have plenty of type information to work off.
3. Sometimes more verbosity is better anyway, otherwise we would be naming our variables a, b, c…

The only ways in which I believe less finger typing give a significant increase in productivity are when:

• The chance to make a trivial error is removed; or
• The reduced noise in the code makes it more readable.

Personally, I do not find static typing introduces noise, even when full explicit type declarations are required. On the contrary, I find code with type information is usually easier to read without the need to refer to external documentation (see my earlier post on Duck Typing vs Readability)¹.

Fallacy 3: Dynamic Typing Is The Big Factor In Increasing Productivity

This is the end conclusion many people draw to the whole issue. I think this derives from the earlier fallacies, and misses the important points. It is true that people are finding many significant productivity improvements in dynamic languages. However, I think very few of these have anything to do with the static vs dynamic typing debate. So, where are the productivity increases? Here are a few:

1. Frameworks: Rails is creating the latest hype around dynamic languages, because a framework of its style is a big productivity boost for a certain class of applications. However, Rails-like frameworks are possible in other languages, as demonstrated by the many clones. Not everything will be cloneable in some languages, but the important things can be and those that can’t are unlikely going to be because of static typing.
2. Syntactic sugar: there is a correlation between languages that are dynamically typed and languages that have convenient syntax for common data structures and other operations (e.g. regular expressions). Note that I do not believe the value of syntactic sugar is in the reduced finger typing (see fallacy 2). Rather, this syntax can reduce the chance of silly errors (e.g. foreach constructs vs a classic for-loop) and can make for significantly more readable code (compare the verbosity of simple list operations in Java vs Python). There is nothing that prevents something like a literal syntax for lists from being supported in a statically-typed language, it is just not common in popular static languages.
3. Higher-level programming constructs: this is the big one. The biggest productivity gains given by a programming language come from the abstractions that it offers. Similar to the previous point, there is a correlation between languages that use dynamic typing and those with higher-level constructs like closures, meta-programming, (proper) macros etc. However, statically-typed languages can and in some cases do support these constructs.

Let’s also not forget that dynamic typing can harm productivity in certain ways: such as less powerful tooling.

Conclusion

My point is simple: get your argument straight before you write off static typing. Even though dynamic typing is the most obviously “different” thing when you first try a language like Python or Ruby, that does not mean it is the reason things seem much more productive. In the end, allowing higher levels of abstraction and maintaining readability are far more important factors, and these are largely independent of the static vs dynamic religious war.

–
¹ Actually, the ideal readability-wise would be to have explicit type information where the type is not immediately obvious. For example, API parameters need type information (when it is not in the code it ends up in the docs). On the other hand, most local variables do not, for example loop counters.

Looking over the companies that have purchased Pulse licenses, it occurs to me that a fairly large percentage of them are in the games development industry. In fact, it appears to me that game developers are over-represented. This could mean any one of a few things:

1. For whatever reason, Pulse is particularly appealing to game developers, relative to developers in general. This could be the case, but given we have not made any special effort to target game developers it would be coincidental.
2. I have underestimated the percentage of the industry that is involved in game development. Possible, as I have no hard data to back up my intuition.
3. Game developers are more likely to adopt continuous integration than the average development team.

I could be wrong, but I think the largest factor is 3. To me, this means that developers in the games industry are more savvy than the average, at least in terms of applying helpful practices to their development process¹. Looking one step deeper, I suppose the next question is why would this be the case? Are game developers smarter than the average bear? I don’t think I would make that sort of generalisation, but I would say I think it helps that they are in a “pure” development industry. That is, their core business is to create software, so naturally as a company they are focused on doing so as efficiently as possible. Development teams in non-software industries surely strive for efficiency too, but do not always have the full backing they require.

–
¹ I guess I am biased, but to me continuous integration is one of those things every team should be doing.

If you’re going to build a “compelling replacement for CVS”, then you need to do two key things:

1. Make commits atomic, with proper changesets
2. Handle merging properly (e.g. support repeated, bi-directional and cherry-picked merges)

Sure, there are other things to improve on, and there is further room for innovation, but these are the two most fundamental features that CVS lacks. This is why I mentioned in my last post that I have been disappointed with the progress of Subversion. I am a Subversion user, and do prefer it to CVS. However, consider that the Subversion project was started in June 2000, and version 1.0 was released in February 2004. Proper merging support is long, long overdue. Thankfully, though, it is finally coming:

http://merge-tracking.open.collab.net/

Current estimates target a 1.5 release with merge tracking in “late summer”, which I guess means August (people need to remember that there are two hemispheres…). I do hope that the new merging support is worth the wait.

Recently, Linus gave a Google tech talk on git, the SCM that he wrote to manage Linux kernel development after the falling out with BitKeeper. In the talk, Linus lambasts both CVS and Subversion, for a variety of reasons, and argues why he believes that git is far superior. In true Linus fashion, he frequently uses hyperbole and false(?) egotism as he lays out his arguments. This in itself could all be in good fun, but Linus takes it a too far. The even worse part, in my opinion, is that Linus confidently pedals arguments that are either specious or poorly focused. In the end, he just loses a lot of respect.

Now both CVS and Subversion are far from perfect. In fact, I personally dislike CVS and am disappointed with the progress of Subversion. Git clearly includes some advanced features that empower the user when compared with either of these tools. However, the way Linus presents the superiority of git is very misleading. Firstly, he throws out several insults towards CVS/Subversion that are plain wrong, such as:

1. Tarballs/patches (the way kernel development was managed way, way back) is superior to using CVS. This almost has to be a joke, but sadly I do not think it was meant that way. CVS may be lacking many powerful features, but this statement ignores the many extremely useful features that CVS does have.
2. There is no way to do CVS “right”. Only if you buy the argument that decentralised is the only way, which Linus does not convince me of one bit.
3. It is not possible to do repeated merges in CVS/Subversion. It may suck that the tools do not support this directly, but everyone knows how to do repeated merges with a small amount of extra effort on the user’s part. I dislike the extra effort, but this is very different from it being impossible.
4. To merge in CVS you end up preparing for a week and setting aside a whole day. Only a stupid process would ever lead to such a ridiculous situation; one that is not adapted to the tool being used. A more accurate criticism would be to say that CVS limits your ability to parallelise development due to weak merging capabilities.

Ignoring the insults, the very core of Linus’ arguments does not make sense. He claims the absolute superiority of the decentralised model, but the key advantages that he uses to back this up are not unique to decentralised systems. Almost all of the key advantages are much more about the ability to handle complex branching and merging than they are about decentralisation. The live audience was not fooled, and one member even questioned Linus on pretty much this exact point a bit over half way through the talk. In response, Linus starts off on a complete tangent about geographically distributed teams and network latency, leaving the question totally unanswered. The closest he gets is making some very weak points about namespace issues with branches in centralised systems (like somehow that would be difficult to solve).

The inability to answer (or perhaps the ability to avoid) important questions is in fact a recurring theme throughout the talk. Another prime example is when an audience member asks an insightful question about whose responsibility it is to merge changes. In a centralised system, the author is naturally tasked with merging their own new work into the shared code base. The advantage here is they know their own changes and are thus well equipped to merge them. In a decentralised system, people pull changes from others and thus end up merging the new work of others into their own repository. Linus is so happy with this question he quips that he had payed to be asked it. Such a shame, then, that he fails to answer it in any meaningful way. Instead, he waffles about the powerful merging capabilities of git (again, not unique to distributed systems). He then hypothesises a case where he pulls a conflict and instead of merging the change himself he requests the author of the change to do the merging for him. He concludes proudly that this is so “natural” in the distributed model. But hang on a second, the original point was that this would have actually been more natural in the centralised model: the author would have merged their own change and Linus need never have been bothered. Honestly, this is as close as you can get to stand up comedy when giving a talk about an SCM!

All in all, a very disappointing talk. Distributed SCMs are a great thing, and bring many advantages. The powerful merging that the distributed model has brought (through necessity) is something that other SCM vendors should be taking a good look at. This is no way to promote the distributed model, however. A more thoughtful view on the pros and cons of distributed versus centralised models, including what each can take from the other, would have been much more informative. My feel is that centralisation has many benefits for certain projects, and like most other things, the best solution would adapt the good parts of each model to the project at hand.

Since I have previously stated my admiration for UnitTest++ as a unit testing framework for C++, I guess it was high time I added direct support for it in Pulse. Well, as of Pulse 1.2.24, there is now a UnitTest++ post-processor that will slurp your UnitTest++ test results directly into Pulse. If you’re looking to do continuous integration for C++, then Pulse + UnitTest++ is a killer combo ;).

Paul Duvall’s post The Future of Continuous Integration points out a weakness in how continuous integration is typically implemented: code must be checked in before it is tested by the continuous integration server. Although problems are picked up fast, there is still a time window where developers may be affected by broken code. This is exactly why we added personal builds to Pulse in version 1.2. Personal builds allow individual developers to submit their changes to Pulse for testing before committing to version control. Pulse does a build and test of the latest version with the developer’s changes applied, allowing to developer to check the full impact of their changes prior to affecting the shared code base. Apart from closing the time window, personal builds also allow other powerful usage patterns, such as using the distributed building features of Pulse to test across multiple platforms before checking in.

Paul also touches on a couple of different solutions to the weakness. Naturally, before implementing personal builds as we did, we considered a few options. One option that Paul mentions that we had considered was leveraging the branching functionality of the SCM. Developers (perhaps with the assistance of the CI server) could check their code into private or temporary branches for testing, and the changes could then be merged down to the shared branch if the build passed. The main attraction of this method is that it reuses functionality available already in the SCM. However, we decided not to go down this route as it can have an impact on the way developers need to work and/or the structure and history in the SCM. Our method of taking a patch from the developer’s working copy and applying it directly with Pulse is about as unobtrusive as you can get. This is important for us, as the ability to adapt to existing environments has always been a key design goal for Pulse.

For more about these and related issues, you might want to also check out an article I posted a while ago to zutubi.com. Reducing the Impact of Broken Builds talks about the affects of a broken code base and how best to avoid or alleviate them.