a little madness

A man needs a little madness, or else he never dares cut the rope and be free -Nikos Kazantzakis

Zutubi

Open Source at Zutubi

At Zutubi we’ve reaped massive benefits over the years from open source. We’ve contributed back bits and pieces in that time — feedback and patches — but always felt we could do more. You may have noticed recent posts here about some of our projects, but we thought they deserved a bit more attention. So we’ve recently launch an new open source section on our website. This is a home for a few small projects of our own that we’ve opened up to the community. We hope you find them useful!

So far we’ve listed three projects:

  1. android junit report: a custom test runner for Android projects that produces XML test reports in a format similar to the Ant JUnit task.
  2. zutubi android ant: a collection of Ant tasks that supplement the standard Android build tooling.
  3. zutubi diff: a Java library for reading, inspecting and applying patch files in unified diff and other formats.

Over time we hope to publish more of our work in this way. You can keep an eye out for new projects on our website and/or follow our organisation on GitHub (or indeed my own GitHub account for Android-specific projects).

And as always: feedback is welcome, so fork away!

Android JUnit Report Updates

Recently I’ve made a few updates to my custom Android test runner, android-junit-report. This runner makes it easy to integrate your Android test results into a continuous integration process by producing de-facto standard JUnit-style XML reports. The latest changes bring the runner up to speed with the latest version of Android and include better documentation. A summary of the updates follows:

  • A new home page for the project.
  • Within the home page, full documentation.
  • Updates to where the reports are written by default. The runner no longer attempts to use the internal storage of the test application, instead it always defaults to the internal storage of the main application (i.e. the one under test).
  • Support for a new __external__ token that allows you to place the reports in the external storage area of the application under test (given availability and permission to do so).
  • Changes to the token syntax to avoid the need to escape when calling the runner via a shell.

I’d like to acknowledge the help of Sebastian Schuberth and Christopher Orr in fine-tuning some aspects of the runner for this release. The latest runner version, 1.5.8, is available for download now.

Android: Easier Source Linking in Eclipse

In my prior post Android: Attaching Sources to Libraries in Eclipse I noted that ADT version r20 added the ability to link sources to libraries in the Android Dependencies classpath container in Eclipse. Although this was a welcome addition, the method used to link sources is a bit cumbersome: you must create a .properties file alongside each library, and add a property that points to the source jar.

I felt this could be a lot simpler if you are willing to adopt a convention for the location of source jars. Once the convention is established, adding linked sources is as simple as placing the source jar in the expected location. This makes the process as seamless as the Android Dependency container is for adding libraries in the first place.

So I’ve made this possible via a new libproperties task in my zutubi-android-ant project. This task inspects the jars in your libs/ directory, looks for corresponding source jars by naming convention, and adds appropriate .properties files when sources are found. If you’re willing to accept the default convention, where source a source jar for library libs/foo.jar is placed at libs/src/foo-source.jar, then this simple Ant target can be used to generate the required .properties files for you:

<target name="link-sources">
    <zaa:libproperties/>
</target>

Of course you’ll need to import the zutubi-android-ant tasks into your build file first. If you want to ensure sources are present for all libraries, set the attribute failOnMissingSrc to true and your build will fail if any source jar is missing. You can even establish your own conventions, using a regular expression to match the libraries and a replacement string to generate paths to source jars:

<target name="link-sources">
    <-- libs/alib-1.3.jar maps to libs/src/alib-src-1.3.jar -->
    <zaa:libproperties failOnMissingSrc="true"
                         jarPattern="(.+)-([0-9.]+)\.jar"
                         srcReplacement="src/$1-src-$2.jar"/>
</target>

Head over to the new zutubi-android-ant home page for more details, including quick-start instructions to get the tasks installed in your build in minutes.

New Android Ant Tasks: lint and ndkbuild

I’m back into Android development these days, and expect plenty more to come (especially with a Nexus 7 due to arrive today!). So my Android-related projects on GitHub should be getting some much-needed attention. In fact I’ve already added a couple of new tasks to zutubi-android-ant (my extensions to the Android SDK Ant tooling): zaa:lint and zaa:ndkbuild.

The names of the tasks are pretty self-explanatory. The zaa:lint task runs Android Lint, which checks your source for common bugs and problems. If you don’t use lint, do yourself a favour and add it to your builds, it will save you a lot of grief! The zaa:ndkbuild task runs the Android NDK build command, which builds your native code.

Both tasks are simple extensions of the venerable exec task which make it simpler to run the underlying tools regardless of their install location and host operating system. This makes it easier to write machine-independent targets in your build files, which is great for teams working in diverse environments (and, of course, your continuous integration server). The tasks rely on the standard Android Ant properties (i.e. sdk.dir and ndk.dir) to locate the tools.

Because the tasks extend the built-in exec, they support all of the standard configuration. Adding arguments, tweaking the environment, and so on should all be familiar to Ant users. In future I may expand the tasks further to support some more tailored configuration, specific to the tools, but they are already fully-capable and very useful.

You’ll find more documentation in the README over at the zutubi-android-ant repository. Expect more tasks to follow, too!

Android: Attaching Sources to Libraries in Eclipse

Since ADT r17, Android projects in Eclipse have had a nice property: all jars in your libs directory have been automatically picked up by Eclipse projects under a magic “Android Dependencies” entry in the build path. Ant builds use the same conventions, so adding a library is as simple as dropping it into the right directory. Unfortunately, though, this magic came with a major limitation: there was no way to attach sources to these libraries (see issue 27940). This led me to duplicate build path entries manually just to get source attachments.

Good news: in the just-released r20 version of ADT, Google have provided a solution to this problem. It’s not well advertised, but if you check out comment 21 on the aforementioned issue, you see that by creating a properties file for each library, you can tell ADT where to find the sources. So, for example, if you have a jar named:

some-library-1.0.jar

you can create a properties file alongside it named:

some-library-1.0.jar.properties

Within the properties file you can add a src property set to the relative (or absolute, but that’s version-control-unfriendly) path of the source jar, zip or directory. Say, like me, you put it in the libs/src subdirectory, your properties file might look like:

src=src/some-library-1.0-sources.jar

Once you’ve added the properties file, refresh the project in Eclipse and voilà: the sources will be attached! A similar property named doc is supported for javadoc attachments.

Although this new feature is very welcome, I feel like it could use a bit more fine tuning. The main issue is the need to create a separate properties file for each jar. I keep my source jars in predictable locations, with conventional names, so I should be able to configure the convention once and have everything Just Work from then on. Heck, I’m not sure why there isn’t a default convention, which I could just follow with no further configuration!

Pulse Continuous Integration Server 2.5 Released!

After a smooth beta period, we’re excited to announce the release of Pulse 2.5! Massive credit for the new feature and improvement ideas in this release go to the Pulse community, whose feedback was invaluable. Here’s the brief list of what’s new:

  • Browse and dashboard view filtering (by project health).
  • A new MSTest post-processor plugin.
  • Increased visibility of build (and other) comments.
  • Comments on agents, to communicate with other users.
  • Properties on agents, for simpler parameterisation.
  • The ability to reference properties in SCM configuration.
  • The ability to configure SCMs using resources.
  • A resource repository on the Pulse master.
  • A new option to attach build logs to notification emails.
  • Upstream change reporting (changes via dependencies).
  • Downstream build notifications (again via dependencies).
  • Simpler delivery of directory trees via dependencies.
  • Pre-stage hooks, which run just before the stage is dispatched to the agent.
  • Stage terminate hooks, allowing you to run custom cleanup.
  • SCM inclusion filters, complementing existing exclusion filters.
  • Inverse resource requirements (i.e. requiring the absence of a resource).
  • Smart label renaming.
  • A distinct warnings status for UI and API consistency.
  • A separate permission for the ability to clean up build directories.
  • Support for git submodules.
  • Support for Subversion 1.7 in personal builds.
  • Support for Perforce streams.

For more details, see the new in 2.5 page on our website. Or, why not just download Pulse and try it free today?

Bash Tip: Reliable Clean Up With Trap

Wow, it has been over 6 years since my last bash tip. Bash scripts are still my go-to for automating the mundane and repetitive — it might not be beautiful but bash is feature rich and installed everywhere. Although a lot of my scripts are throw-away, others make it into the permanent tool kit. For scripts in the latter category, it’s worth the extra bit of effort to make them more robust.

A key part of a robust script is detecting and sensibly reporting errors. Related to this is proper clean up of work in progress when the script encounters a fatal error. Such clean up is often overlooked, as it is difficult to get right in a naive fashion. Thankfully, bash has a built-in mechanism that makes it simple: trap.

Setting a trap instructs bash to run a command when the shell process receives a specified signal. Further, bash defines pseudo-signals ERR and EXIT that can be used to trap any error or exit of the shell. I usually pair an exit trap with a clean up function to ensure my scripts don’t leave a mess. If the script runs successfully, I reset the trap (by specifying – as the command to run) and call the clean up function directly (if required). For example:

#! /usr/bin/env bash

set -e

# Defines a working area on the file system.
SCRATCH=/tmp/$$.scratch

function cleanUp() {
    if [[ -d "$SCRATCH" ]]
    then
        rm -r "$SCRATCH"
    fi
}

trap cleanUp EXIT
mkdir "$SCRATCH"

# Actual work here, all temp files created under $SCRATCH.

# We succeeded, reset trap and clean up normally.
trap - EXIT
cleanUp
exit 0

Note that the EXIT signal pairs well with set -e: if the script exits on error your EXIT trap is executed. You may also consider applying the trap to interrupts (signal INT) and kills (signal TERM) of your script. Multiple signals can be specified in one trap statement:

trap cleanUp EXIT INT TERM

If you’re used to programming with exceptions, you can think of a trap as a “finally” clause for your whole script. It’s certainly a whole lot easier and less error-prone that trying to figure out all the ways the script might exit and adding calls to clean up manually.

—-
Want Continuous Integration without the Needless Duplication? Try pulse.

Pulse Continuous Integration Server 2.5 Beta!

I’m happy to report that the Pulse 2.5 release stream has been upgraded to beta! This marks completion of the 2.5 feature set. Since the alpha release, we’ve powered through more features and improvements; examples include:

  • A new warning status, making it easier to work with builds with warnings.
  • The ability to reference properties in SCM configuration.
  • Another new stage hook points on stage termination.
  • Upstream change reporting to show how dependencies influenced a build.
  • Dependency notifications to show how your changes affected downstream builds.
  • Artifact archives, allowing easy delivery of directory trees between projects.
  • Flexible bootstrapping allows control over how your builds are prepared.
  • Subversion 1.7 working copy format support.
  • Git submodule support.

Our plan now is to focus on performance and stability for the next few builds before pushing 2.5 as the stable release. In the mean time you can find out more, and even try beta builds, by visiting our beta program page. Happy building!

Pulse Continuous Integration Server 2.5 Alpha!

Cobwebs may be creeping over the blog, but not over Pulse: we’ve just published the first Pulse 2.5 alpha build! The Pulse 2.5 series is focused (even more so than usual) on customer feedback. We’ve got plenty of great ideas from the Pulse community, and we’re folding in some of the best ones in this release series. Updates so far include:

  • Browse and dashboard view filtering (by project health).
  • A new MSTest post-processor plugin.
  • Increased visibility of build (and other) comments.
  • Comments on agents, to communicate with other users.
  • Properties on agents, for simpler parameterisation.
  • The ability to configure SCMs using resources.
  • A resource repository on the Pulse master.
  • A new option to attach build logs to notification emails.
  • Pre-stage hooks, which run just before the stage is dispatched to the agent.
  • SCM inclusion filters, complementing existing exclusion filters.
  • Inverse resource requirements (i.e. requiring the absence of a resource).
  • Smart label renaming.
  • A separate permission for the ability to clean up build directories.
  • Support for Perforce streams.

There’s plenty more to come in subsequent alpha builds, too. In fact, we’ve started the groundwork for some other larger changes already in 2.5.0.

To grab yourself a fresh-from-the-oven 2.5 alpha build, or just to find out more, head over to our alpha program page.

Pulse Sample and Community Plugins

Since version 2.0, Pulse has supported plugins to extend integration with external tools. Whenever customers have asked for help with plugin implementation, we’ve always provided samples as the easiest starting point. Seeing a complete plugin project, with build and packaging support, is the easiest way to get a new plugin up and running.

Based on this we’ve decided to start maintaining a repository of open source Pulse plugins. These will act both as samples and in some cases as early versions of plugins that may migrate into the core Pulse distribution. The sample plugins are hosted in a Mercurial repository which has an hgweb interface:

http://hg.zutubi.com/

So far we have provided a sample JUnit test post-processor (simplified from the version shipped with Pulse), and the beginnings of a Subversion SCM plugin (wrapping the svn command line). Provided you have Mercurial installed, you can easily clone the samples over HTTP, e.g.:

$ hg clone http://hg.zutubi.com/com.zutubi.pulse.core.postprocessors.junits

The samples have README files to get you started.

You may notice there is a third sample: a plugin that provides basic SCM integration with Bazaar by wrapping the bzr command-line. I’m happy to report that this sample was generously donated by a member of the community, Michiel van Slobbe. Michiel is the first to try to get Pulse and Bazaar working together and we appreciate the effort! Hopefully it will provide inspiration for other community members.

Although we are happy to provide hosting for such community plugins, you may also choose your own path. Another member of the community, Orvid, is using Pulse for C#/Mono projects. Orvid has taken the initiative of writing a command plugin to integrate XBuild, the Mono equivalent of Microsoft’s MSBuild, with Pulse. You can find the XBuild plugin on GitHub:

https://github.com/Orvid/XBuildIntegration

You might also notice that Orvid has been working on a .Net wrapper for the Pulse remote API:

https://github.com/Orvid/Zutubi.Pulse.Api

These efforts are a great contribution that we are most thankful for!