a little madness

Recently I’ve made a few updates to my custom Android test runner, android-junit-report. This runner makes it easy to integrate your Android test results into a continuous integration process by producing de-facto standard JUnit-style XML reports. The latest changes bring the runner up to speed with the latest version of Android and include better documentation. A summary of the updates follows:

• A new home page for the project.
• Within the home page, full documentation.
• Updates to where the reports are written by default. The runner no longer attempts to use the internal storage of the test application, instead it always defaults to the internal storage of the main application (i.e. the one under test).
• Support for a new __external__ token that allows you to place the reports in the external storage area of the application under test (given availability and permission to do so).
• Changes to the token syntax to avoid the need to escape when calling the runner via a shell.

I’d like to acknowledge the help of Sebastian Schuberth and Christopher Orr in fine-tuning some aspects of the runner for this release. The latest runner version, 1.5.8, is available for download now.

In my prior post Android: Attaching Sources to Libraries in Eclipse I noted that ADT version r20 added the ability to link sources to libraries in the Android Dependencies classpath container in Eclipse. Although this was a welcome addition, the method used to link sources is a bit cumbersome: you must create a .properties file alongside each library, and add a property that points to the source jar.

I felt this could be a lot simpler if you are willing to adopt a convention for the location of source jars. Once the convention is established, adding linked sources is as simple as placing the source jar in the expected location. This makes the process as seamless as the Android Dependency container is for adding libraries in the first place.

So I’ve made this possible via a new libproperties task in my zutubi-android-ant project. This task inspects the jars in your libs/ directory, looks for corresponding source jars by naming convention, and adds appropriate .properties files when sources are found. If you’re willing to accept the default convention, where source a source jar for library libs/foo.jar is placed at libs/src/foo-source.jar, then this simple Ant target can be used to generate the required .properties files for you:

<target name="link-sources">
    <zaa:libproperties/>
</target>

Of course you’ll need to import the zutubi-android-ant tasks into your build file first. If you want to ensure sources are present for all libraries, set the attribute failOnMissingSrc to true and your build will fail if any source jar is missing. You can even establish your own conventions, using a regular expression to match the libraries and a replacement string to generate paths to source jars:

<target name="link-sources">
    <-- libs/alib-1.3.jar maps to libs/src/alib-src-1.3.jar -->
    <zaa:libproperties failOnMissingSrc="true"
                         jarPattern="(.+)-([0-9.]+)\.jar"
                         srcReplacement="src/$1-src-$2.jar"/>
</target>

Head over to the new zutubi-android-ant home page for more details, including quick-start instructions to get the tasks installed in your build in minutes.

I’m back into Android development these days, and expect plenty more to come (especially with a Nexus 7 due to arrive today!). So my Android-related projects on GitHub should be getting some much-needed attention. In fact I’ve already added a couple of new tasks to zutubi-android-ant (my extensions to the Android SDK Ant tooling): zaa:lint and zaa:ndkbuild.

The names of the tasks are pretty self-explanatory. The zaa:lint task runs Android Lint, which checks your source for common bugs and problems. If you don’t use lint, do yourself a favour and add it to your builds, it will save you a lot of grief! The zaa:ndkbuild task runs the Android NDK build command, which builds your native code.

Both tasks are simple extensions of the venerable exec task which make it simpler to run the underlying tools regardless of their install location and host operating system. This makes it easier to write machine-independent targets in your build files, which is great for teams working in diverse environments (and, of course, your continuous integration server). The tasks rely on the standard Android Ant properties (i.e. sdk.dir and ndk.dir) to locate the tools.

Because the tasks extend the built-in exec, they support all of the standard configuration. Adding arguments, tweaking the environment, and so on should all be familiar to Ant users. In future I may expand the tasks further to support some more tailored configuration, specific to the tools, but they are already fully-capable and very useful.

You’ll find more documentation in the README over at the zutubi-android-ant repository. Expect more tasks to follow, too!

After a smooth beta period, we’re excited to announce the release of Pulse 2.5! Massive credit for the new feature and improvement ideas in this release go to the Pulse community, whose feedback was invaluable. Here’s the brief list of what’s new:

• Browse and dashboard view filtering (by project health).
• A new MSTest post-processor plugin.
• Increased visibility of build (and other) comments.
• Comments on agents, to communicate with other users.
• Properties on agents, for simpler parameterisation.
• The ability to reference properties in SCM configuration.
• The ability to configure SCMs using resources.
• A resource repository on the Pulse master.
• A new option to attach build logs to notification emails.
• Upstream change reporting (changes via dependencies).
• Downstream build notifications (again via dependencies).
• Simpler delivery of directory trees via dependencies.
• Pre-stage hooks, which run just before the stage is dispatched to the agent.
• Stage terminate hooks, allowing you to run custom cleanup.
• SCM inclusion filters, complementing existing exclusion filters.
• Inverse resource requirements (i.e. requiring the absence of a resource).
• Smart label renaming.
• A distinct warnings status for UI and API consistency.
• A separate permission for the ability to clean up build directories.
• Support for git submodules.
• Support for Subversion 1.7 in personal builds.
• Support for Perforce streams.

For more details, see the new in 2.5 page on our website. Or, why not just download Pulse and try it free today?

Wow, it has been over 6 years since my last bash tip. Bash scripts are still my go-to for automating the mundane and repetitive — it might not be beautiful but bash is feature rich and installed everywhere. Although a lot of my scripts are throw-away, others make it into the permanent tool kit. For scripts in the latter category, it’s worth the extra bit of effort to make them more robust.

A key part of a robust script is detecting and sensibly reporting errors. Related to this is proper clean up of work in progress when the script encounters a fatal error. Such clean up is often overlooked, as it is difficult to get right in a naive fashion. Thankfully, bash has a built-in mechanism that makes it simple: trap.

Setting a trap instructs bash to run a command when the shell process receives a specified signal. Further, bash defines pseudo-signals ERR and EXIT that can be used to trap any error or exit of the shell. I usually pair an exit trap with a clean up function to ensure my scripts don’t leave a mess. If the script runs successfully, I reset the trap (by specifying – as the command to run) and call the clean up function directly (if required). For example:

#! /usr/bin/env bash

set -e

# Defines a working area on the file system.
SCRATCH=/tmp/$$.scratch

function cleanUp() {
    if [[ -d "$SCRATCH" ]]
    then
        rm -r "$SCRATCH"
    fi
}

trap cleanUp EXIT
mkdir "$SCRATCH"

# Actual work here, all temp files created under $SCRATCH.

# We succeeded, reset trap and clean up normally.
trap - EXIT
cleanUp
exit 0

Note that the EXIT signal pairs well with set -e: if the script exits on error your EXIT trap is executed. You may also consider applying the trap to interrupts (signal INT) and kills (signal TERM) of your script. Multiple signals can be specified in one trap statement:

trap cleanUp EXIT INT TERM

If you’re used to programming with exceptions, you can think of a trap as a “finally” clause for your whole script. It’s certainly a whole lot easier and less error-prone that trying to figure out all the ways the script might exit and adding calls to clean up manually.

—-
Want Continuous Integration without the Needless Duplication? Try pulse.

Cobwebs may be creeping over the blog, but not over Pulse: we’ve just published the first Pulse 2.5 alpha build! The Pulse 2.5 series is focused (even more so than usual) on customer feedback. We’ve got plenty of great ideas from the Pulse community, and we’re folding in some of the best ones in this release series. Updates so far include:

• Browse and dashboard view filtering (by project health).
• A new MSTest post-processor plugin.
• Increased visibility of build (and other) comments.
• Comments on agents, to communicate with other users.
• Properties on agents, for simpler parameterisation.
• The ability to configure SCMs using resources.
• A resource repository on the Pulse master.
• A new option to attach build logs to notification emails.
• Pre-stage hooks, which run just before the stage is dispatched to the agent.
• SCM inclusion filters, complementing existing exclusion filters.
• Inverse resource requirements (i.e. requiring the absence of a resource).
• Smart label renaming.
• A separate permission for the ability to clean up build directories.
• Support for Perforce streams.

There’s plenty more to come in subsequent alpha builds, too. In fact, we’ve started the groundwork for some other larger changes already in 2.5.0.

To grab yourself a fresh-from-the-oven 2.5 alpha build, or just to find out more, head over to our alpha program page.

The Android SDK documentation offers a guide for Versioning Your Applications. In short, applications have two version fields:

1. versionCode: a simple integer that you should increase with each new release. Being an integer, the code can be processed in software (e.g. other applications) if required.
2. versionName: a free-form string you can use as the customer-facing version for your application. A typical format would be dotted-decimal (e.g. 1.2.0), although any form is allowed.

This version scheme is simple and allows you the flexibility to choose your own naming scheme. One annoyance, however, is that the version is specified in the AndroidManifest.xml file:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
        package="com.zutubi.android.example"
        android:versionCode="1"
        android:versionName="1.0">
    <application android:name=".ExampleApplication">
        ...
    </application>
</manifest>

The problem here is that the manifest is a version controlled file that also contains other vital information about the application. This makes it inconvenient to automate stamping your application with a new version for each release.

This motivated me to create a couple of Ant tasks which make it easy to set the version of your Android applications from your Ant build. I’ve released them as part of an initial version of the zutubi-android-ant project. So far there are two tasks: setversion and bumpversion.

setversion

This task allows you to explicitly set the version code and/or name for your application to specified values. Typically these values would be specified indirectly via Ant properties. The properties would be set by some other process (e.g. automatic generation based on the date, an input prompt, a property from your build server, or an read from an unversioned properties file).

For example, to set the versions based on Ant properties, prompting if those properties are not yet defined:

<target name="set-version">
    <input addproperty="version.code" message="Version Code:"/>
    <input addproperty="version.name" message="Version Name:"/>
    <zaa:setversion code="${version.code}" name="${version.name}"/>
</target>

bumpversion

Perhaps you simply want to increment the version code and point number for your application each time you make a build. (You may also change the major.minor version at times, but that is a manual choice.) You can do this simply with the bumpversion task. By default, this task will just increment the existing version code. If you set the optional attribute bumpname to true, the last element of the version name will also be incremented. For example:

<target name="bump-version">
    <zaa:bumpversion bumpname="true"/>
</target>

If your existing manifest has a version code of 3 and a version name of 1.0.3, then after bumping you will have a code of 4 and a name of 1.0.4. Simple!

Try It Out

The zutubi-android-ant GitHub page has a full README and simple examples showing in more detail how you can use these tasks. Happy versioning!

Introduction

Ant may be unfashionable these days, but it still has its advantages. Key among these are familiarity and simplicity: most Java developers have worked with Ant, and with an Ant build what you get is what you see. A major disadvantage, though, is that Ant provides very little out-of-the-box. When you start a new project, you’ve got a lot of grunt work to endure just to get your code compiled, packaged, and tested. An all-too-common solution, in the grand tradition of make, is to copy a build file from an existing project as an easy starting point.

Over the years, though, Ant has gradually expanded support for creating reusable build file snippets. On top of this a few projects have emerged which aim to simplify and standardise your Ant builds, including:

• EasyAnt
• Ant Master Build Scripts
• Ant Script Library

Today I’ve taken my first proper look at the latter, and so far I like what I see.

The Ant Script Library

In the author Joe Schmetzer’s own words:

The Ant Script Library (ASL) is a collection of re-usable Ant scripts that can be imported into your own projects. The ASL provides a number of pre-defined targets that simplify setting up build scripts for a new project, bringing re-use and consistency to your own Ant scripts.

ASL consists of several Ant XML files, each of which provides a group of related functionality via predefined targets. For example, the asl-java-build.xml file defines targets for compiling and packaging Java code. The asl-java-test.xml file extends this with the ability to run JUnit tests, and so on. Essentially, ASL packages up all the grunt work, allowing you to concentrate on the small tweaks and extra targets unique to your project. The modular structure of ASL, combined with the fact that it is just Ant properties and targets, makes it easy to take what you like and leave the rest.

An Example

Allow me to illustrate with a simple project I have been playing with. This project has a straightforward directory structure:

• <project root>
  • asl/ – the Ant Script Library
  • build.xml – Ant build file
  • lib/ – Jar file depedencies
  • src/ – Java source files
  • test/ – JUnit-based test source files

To add ASL to my project, I simply downloaded it from the project download page and unpacked it in the asl/ subdirectory of my project¹. Then I can start with a very simple build file that supports building my code and running the tests:

<?xml version="1.0" encoding="utf-8"?>
<project name="zutubi-android-ant" default="dist">
    <property name="java-build.src-dir" location="src"/>
    <property name="java-test.src-dir" location="test"/>
    <property name="java-build.lib-dir" location="libs"/>
	
    <property name="asl.dir" value="asl"/>

    <import file="${asl.dir}/asl-java-build.xml"/>
    <import file="${asl.dir}/asl-java-test.xml"/>
</project>

Notice that I am using non-standard source locations, but that is easily tweaked using properties which are fully documented. With this tiny build file, let’s see what targets ASL provides for me:

$ ant -p
Buildfile: build.xml

Main targets:

 clean                 Deletes files generated by the build
 compile               Compiles the java source
 copy-resources        Copies resources in preparation to be packaged in jar
 dist                  Create a distributable for this java project
 generate              Generates source code
 jar                   Create a jar for this java project
 test-all              Runs all tests
 test-integration      Runs integration tests
 test-run-integration  Runs the integration tests
 test-run-unit         Runs the unit tests
 test-unit             Runs unit tests
Default target: dist

It’s delightfully simple!

Adding Reports

It gets better: ASL also provides reporting with tools like Cobertura for coverage, FindBugs for static analysis and so on via its asl-java-report.xml module. The full range of supported reports can be seen in the report-all target:

<target name="report-all"
        depends="report-javadoc, report-tests, report-cobertura, report-jdepend, report-pmd, report-cpd, report-checkstyle, report-findbugs" 
        description="Runs all reports"/>

Having support for several tools out-of-the-box is great. For my project, however, I’d like to keep my dependencies down and I don’t feel that I need all of the reporting. Although the choice of reports is not something that is parameterised by a property, it is still trivial to override by providing your own report-all target. This shows the advantage of everything being plain Ant targets:

<?xml version="1.0" encoding="utf-8"?>
<project name="zutubi-android-ant" default="dist">
    <property name="java-build.src-dir" location="src"/>
    <property name="java-test.src-dir" location="test"/>
    <property name="java-build.lib-dir" location="libs"/>
	
    <property name="asl.dir" value="asl"/>

    <import file="${asl.dir}/asl-java-build.xml"/>
    <import file="${asl.dir}/asl-java-test.xml"/>
    <import file="${asl.dir}/asl-java-report.xml"/>
    
    <target name="report-all"
            depends="report-javadoc, report-tests, report-cobertura, report-pmd, report-checkstyle" 
            description="Runs all reports"/>
</project>

Here I’ve included the java-report module, but defined my own report-all target that depends on just the reports I want. This keeps things simple, and allows me to trim out a bunch of ASL dependencies I don’t need.

Conclusion

I’ve known of ASL and such projects for a while, but this is the first time I’ve actually given one a go. Getting started was pleasantly simple, as was applying the small tweaks I needed. So next time you’re tempted to copy an Ant build file, give ASL a shot: you won’t regret it!

–
¹ In this case I downloaded the full tarball including dependencies, which seemed on the large side (21MB!) but in fact can be easily trimmed by removing the pieces you don’t need. Alternatively, you can start with the basic ASL install (sans dependencies) and it can pull down libraries for you. Sweet .

Earlier in the year I put Selenium in a cage using Xnest. This allows me to run browser-popping tests in the background without disturbing my desktop or (crucially) stealing my focus.

On that post Rohan stopped by to mention a nice alternative to Xnest: Xephyr. As the Xephyr homepage will tell you:

Xephyr is a kdrive based X Server which targets a window on a host X Server as its framebuffer. Unlike Xnest it supports modern X extensions ( even if host server doesn’t ) such as Composite, Damage, randr etc (no GLX support now). It uses SHM Images and shadow framebuffer updates to provide good performance. It also has a visual debugging mode for observing screen updates.

It sounded sweet, but I hadn’t tried it out until recently, on a newer box where I didn’t already have Xnest setup. The good news is the setup is as simple as with Xnest in my prior post:

1. Install Xephyr: which runs an X server inside a window:
   $ sudo apt-get install xserver-xephyr
2. Install a simple window manager: again, for old times’ sake, I’ve gone for fvwm:
   $ sudo apt-get install fvwm
3. Start Xephyr: choose an unused display number (most standard setups will already be using 0) — I chose 1. As with Xnest, the -ac flag turns off access control, which you might want to be more careful about. My choice of window size is largely arbitrary:
   $ Xephyr :1 -screen 1024×768 -ac &
4. Set DISPLAY: so that subsequent X programs connect to Xephyr, you need to set the environment variable DISPLAY to whatever you passed as the first argument to Xephyr above:
   $ export DISPLAY=:1
5. Start your window manager: to manage windows in your nested X instance:
   $ fvwm &
6. Run your tests: however you normally would:
   $ ant accept.master

Then just sit back and watch the browsers launched by Selenium trapped in the Xephyr window. Let’s see them take your focus now!

Lately one of our Pulse agents has been bogged down, to the extent that some of our heavier acceptance tests started to genuinely time out. Tests failing due to environmental factors can lead to homicidal mania, so I’ve been trying to diagnose what is going on before someone gets hurt!

The box in question runs Windows Vista, and I noticed while poking around that some disk operations were very slow. In fact, deleting even a handful of files via Explorer took so long that I gave up (we’re talking hours here). About this time I fired up the Reliability and Performance Manager that comes with Vista (Control Panel > Administrative Tools). I noticed that there was constant disk activity, and a lot of it centered around C:\$MFT — the NTFS Master File Table.

I had already pared back the background tasks on this machine: the Recycle Bin was disabled, Search Indexing was turned off and Defrag ran on a regular schedule. So why was my file system so dog slow? The answer came when I looked into the AppData\Local\Temp directory for the user running the Pulse agent. The directory was filled with tens of thousands of entries, many of which were directories that themselves contained many files.

The junk that had built up in this directory was quite astounding. Although some of it can be explained by tests that don’t clean up after themselves, I believe a lot of the junk came from tests that had to be killed forcefully without time to clean up. It was also evident that every second component we were using was part of the problem – Selenium, JFreechart, JNA, Ant and Ivy all joined the party.

So, how to resolve this? Of course any tests that don’t clean up after themselves should be fixed. But in reality this won’t always work — especially given the fact that Windows will not allow open files to be deleted. So the practical solution is to regularly clean out the temporary directory. In fact, it’s quite easy to set up a little Pulse project that will do just that, and let Pulse do the work of scheduling it via a cron trigger. With Pulse in control of the scheduling there is no risk the cleanup will overlap with another build.

A more general solution is to start with a guaranteed-clean environment in the first place. After all, acceptance tests have a habit of messing with a machine in other ways too. Re-imaging the machine after each build, or using a virtual machine that can be restored to a clean state, is a more reliable way to avoid the junk. Pulse is actually designed to allow reimaging/rebooting of agents to be done in a post-stage hook — the agent management code on the master allows for agents to go offline at this point, and not try to reuse them until their status can be confirmed by a later ping.