a little madness

Well, if they try to do anything useful anyway. Last week I was working on improvements to the Windows service installation for Pulse. The existing service installation was handled by a pretty simple batch file wrapping the service executable. However, this method had some major limitations, mostly related to location of a compatible JVM. I needed to expand the logic in the batch script considerably as part of the solution to these limitations.

That’s when I realised: logic and batch scripts don’t really go together. In fact, scripting and batch scripts don’t really go together. Even something as simple as reading the output of an executed command is bizarre in batch (there is an obscure for loop syntax to do it). Fed up with the archaic nature and limitations of batch files, I went looking for an alternative. I had heard of powershell (aka Monad), but of course it is not yet widely installed. So I turned to Windows Script Host, which has been around since Windows 98. I hadn’t used it before, but I discovered you could write scripts in JScript (Javascript-like) and easily access the filesystem and registry, so it seemed like a good fit.

In fact, apart from the pain of navigating MSDN documentation online, my foray into WSH started quite promisingly. Then, just as I was really starting to get somewhere, Norton Antivirus chimed in. Apparently, this script that I was running was trying to do something “malicious”. Whenever I tried to do something moderately useful in the script, like access file or registry information, Norton would warn me in no uncertain terms. Brilliant. No matter that I was running the script directly, just as I could run any executable that could do equally “malicious” things. I suppose Symantec doesn’t care about false positives; that might take effort. Instead, WSH has been rendered useless to software distributors, except for those willing to have their customers warned that their software is “malicious” by a virus scanner.

In the end I implemented the solution as a standalone C++ program, because at least that way I knew I could get it to work. It’s a sad state of affairs when I am forced to write a few hundred lines of C++ when a 50 line script should do. That’s what happens when a platform is completely devoid of useful scripting options.

Well, it’s finally here! The first public beta of pulse 1.1. This is a huge milestone for us, just take a look at what’s new to see what I mean. The biggest single feature is powerful distributed building, but as you’ll see the list is long!

Thanks so much to our private pre-release testers, your feedback has been invaluable. Now for the rest of you: bring it on!

Pulse version 1.0.6 has been released. This release fixes some minor issues in 1.0.5, see the release notes for details.

Zutubi is proud to announce the availability of the Pulse automated build (or continuous integration) server for sale from today. This is the culmination of many months of development and beta testing. We would like to thank all of our beta testers for their feedback during this period.

If you haven’t tried it yet, download pulse today and let us know what you think!

When your bash script is going haywire, and you’re having trouble sorting out why, one of the simplest ways to see what is going on is to use:

set -x

This turns on the xtrace shell option, which prints the commands the shell is executing with expanded arguments, as they are executed. From the man page:

This shows you exactly what is going on as your script executes. Take the following, contrived shell script:

When executed, the script will print the following:

Note that the trace lines are prefixed with ‘+’ characters (in fact, whatever the value of $PS4 is), and regular output appears as normal. You can also see that the for loop and if conditions are displayed. This is all very handy for tracking down obscure problems.

One issue with tracing is the output can be very verbose, making the useful bits hard to find. If you have a rough idea of where the problem is, you can alleviate this problem by turning tracing on and off selectively around the interesting parts of the script (set +x turns it off). Happy scripting!

—
Automation crazy? Need a serious automated build server? Try pulse.

Every good continuous integration server provides some form of notification of build results. The most basic (and one of the best) forms of notification is via email. Email is ubiquitous, and thus there is nothing extra to set up or teach before you can use email notifications. The better continuous integration servers also allow filtering of notifications based on the build result. This is simply about increasing the signal/noise ratio: if you get emails after every build, and a large percentage of builds are successful, it is easy to lose the useful notifications in the deluge.

Given this flexibility to choose when to receive notifications, it is tempting to start filtering out aggresively. It is a matter of personal preference, but there are many people who want to keep their email traffic to an absolute minimum. This is fair enough, but in my opinion there is a minimum level of notification that every developer should subscribe to: all failed builds. I don’t care who caused it, if a build of a project you are working on fails, you should know about it. I can imagine some complaints already:

• But that’s too many emails, it’s annoying!: Well, then your project spends too much of its time broken, and you deserve to be annoyed . Failed builds should be the exception, not the rule. Receiving the deluge of emails will hopefully motivate your team to keep things working.
• But it’s not my fault, somebody else broke the build!: You are all one big happy team, remember? In this case, it is still useful for you (and other team members) to know something is broken. You may help fix the build, or hold off on other changes. Further, the threat of public shame will make your teammates more careful .
• But once it fails, it keeps failing, and the emails are piling up!: Then fix it faster! Seriously! OK, in some cases where you know the fix will take a while, you may want to pause the server. But often the fix is quick, and should be applied ASAP.

Of course, this is all just one developer’s opinion, and no continuous integration server should enforce this. However, I believe that if all developers receive notifications of all failures (at least), failures will become less frequent as a matter of course. Otherwise that pile of emails will just get too annoying!

—
Into continuous integration? Want to be? Try pulse. You can choose when you want to be notified using arbitrary boolean expressions.

Local build is a unique feature of pulse that allows you to execute the pulse build engine in a local checkout of your project’s source code. Why would you want to do this? First of all, let me be clear that this is not a replacement for your regular build script (e.g. build.xml, Makefile): pulse is designed to be layered on top of an existing build system. There are three main use cases for local builds:

1. Debugging your pulse file: the pulse file describes to pulse how to build your project. You can choose to either configure the build via the pulse web interface (great for simple projects where you don’t want to mess with text configuration) or by writing the file by hand. In the latter case you can also store the configuration in your SCM where it is versioned with your project source code. This is a powerful way to manage the configuration, but it suffers from a problem: when you need to modify the pulse file, you have a lengthy debug cycle. You need to make the edits, check the file in, trigger a build and wait for the pulse server to get back to you. This is painful, especially for debugging simple typos. Local build allows you to avoid the problem completely by enabling you to run the build in your local checkout of the project’s source code. Simple errors are caught immediately, and you can fine tune your changes to your heart’s content before committing them.
2. Taking advantage of pulse’s post-processing model to make sense of your build output. Pulse uses post-processing as a simple but powerful way to extract useful information from a build. All build artifacts, including the output of build commands, may be processed using one of several built-in processors to find features (errors, warnings, test failures). You can even add your own post-processors based on regular expressions. By executing your build scripts via pulse local build, you get simple summaries of each command executed, with any useful features listed in the build result. There is no need to scan through line after line of build output: let pulse local build do it for you!
3. Reproducing the build: imagine a continuous integration build fails in a mysterious way on your pulse server and you are having a hard time reproducing the failure. A key to reproduction is to execute everything exactly the same way as the pulse server. Although you should be able to reproduce almost exactly the same build as the pulse server using just your regular build scripts, local build allows you to go one better.

You can use local build even if you configure your project via the web interface (rather than writing the pulse file by hand): the pulse server generates a pulse file for every build and makes them available for download.

Pulse local build is available as a separate download to the pulse server package. Try it today: local build is free!

Back in my post Your Next Programming Language I mentioned I would post occassional tips about bash scripting. As soon as I started writing my next script, it occured to me: the first thing I always do when writing a new bash script is set the errexit option:

This option makes your script bail out when it detects an error (a command exiting with a non-zero exit code). Without this option the script will plough on, and mayhem often ensues. In all the noise generated it can be a pain to found the root cause of the problem. So I make it a rule to set this option and fail as early as possible.

—
Into continuous integration? Want to be? Try pulse.

A typical continuous integration server will be set up to build each time a change is detected in source control. This achieves fast feedback for changes by testing them as soon as they are available. However, I believe many projects should be built even more frequently than this: they should be built AFAP (As Frequently As Possible). Why build when there are no new changes to the code? Simple: more testing. Not all bugs show up every time you test. In fact, the nastiest bugs are those that only show up every once in a while. These are the bugs you really want to catch before you ship. If you think it’s nasty debugging race conditions in your cozy little cube, wait until your customers start reporting intermittent problems .

Since building AFAP can have very little cost (you have a dedicated build machine, right?), many teams have no reason not to do it. The only significant cost that comes to mind is delaying feedback when a change does occur, as the continuous integration server may be busy performing an AFAP build. On average a build with a change is delayed by half the time for a full build cycle. For projects with long build times, this may be a real issue (indeed, one of many issues: get those build times down!). For those projects, it may be better to stick to building on change throughout the day. Even so, there is no need to waste those overnight and weekend hours! Schedule your AFAP builds to run throughout the night, every night and all day over the weekend. These are also good opportunities to run longer build/test cycles with your most thorough test suites.

——-
Into continuous integration? Want to be? Try pulse.

Recently, a beta tester of pulse asked a fair question:

I can see the benefits of automated build software for large software teams. Would pulse considerably benefit a development team of 1-5?

Now, I am surely biased, but my answer is most certainly yes. Firstly, there are many benefits that apply to both, such as frequent testing, testing outside of the developers’ environments, early detection of integration problems, isolating changes that introduce problems, and so on.

It is also fair to say that some of the benefits for large teams do not apply to the same degree for a team of 1-5. For example, as the team grows the chance of an integration problem increases even more rapidly due to potentail semantic conflicts between changes committed by various developers. Even when the developers are all disciplined enough to update and run tests locally (which we all are, right? ), there is still a greater chance of a submit race. For large teams the continuous integration server can also be a great way to communicate recent project activity, as developers are notified of builds, and the web UI may allow you to see recent project changes.

Smaller teams do not have such problems with submit races and typically find communication much easier. However, some of the benefits of continuous integration are more important to small teams than to large ones. A prime example is the increased frequency of testing. Without continuous integration, tests are only run as frequently as the developers run them. The fewer developers, the less frequent the testing. Problems take longer to show up, especially intermittent bugs (those nasty blighters that only show up once every 364 test runs). If your tests run every fifteen minutes on your continuous integration server, even those intermittent bugs can’t hide for long. Further, small teams necessarily have a smaller number of environments that they are naturally testing on during development. Adding another one, even a single continuous integration server, is significant. Lastly, a smaller team, with naturally limited resources, needs to apply automation even more aggresively than a large team. Dedicating one or more engineers to release and integration management may not seem like a big deal to a large team, but for a small team that may be a third of the manpower!

Although they benefit in different ways, I wouldn’t say a large team needs CI more than a small one, or vice-versa. Experience has taught me that once a team employs continuous integration, whether it be 2 or 200 strong, the engineers will never want to do without it again.

——-
Into continuous integration? Want to be? Try pulse.