a little madness

A man needs a little madness, or else he never dares cut the rope and be free -Nikos Kazantzakis

Zutubi

Archive for the ‘Continuous Integration’ Category

Pulse Continuous Integration Server 2.5 Alpha!

Cobwebs may be creeping over the blog, but not over Pulse: we’ve just published the first Pulse 2.5 alpha build! The Pulse 2.5 series is focused (even more so than usual) on customer feedback. We’ve got plenty of great ideas from the Pulse community, and we’re folding in some of the best ones in this release series. Updates so far include:

  • Browse and dashboard view filtering (by project health).
  • A new MSTest post-processor plugin.
  • Increased visibility of build (and other) comments.
  • Comments on agents, to communicate with other users.
  • Properties on agents, for simpler parameterisation.
  • The ability to configure SCMs using resources.
  • A resource repository on the Pulse master.
  • A new option to attach build logs to notification emails.
  • Pre-stage hooks, which run just before the stage is dispatched to the agent.
  • SCM inclusion filters, complementing existing exclusion filters.
  • Inverse resource requirements (i.e. requiring the absence of a resource).
  • Smart label renaming.
  • A separate permission for the ability to clean up build directories.
  • Support for Perforce streams.

There’s plenty more to come in subsequent alpha builds, too. In fact, we’ve started the groundwork for some other larger changes already in 2.5.0.

To grab yourself a fresh-from-the-oven 2.5 alpha build, or just to find out more, head over to our alpha program page.

Pulse Sample and Community Plugins

Since version 2.0, Pulse has supported plugins to extend integration with external tools. Whenever customers have asked for help with plugin implementation, we’ve always provided samples as the easiest starting point. Seeing a complete plugin project, with build and packaging support, is the easiest way to get a new plugin up and running.

Based on this we’ve decided to start maintaining a repository of open source Pulse plugins. These will act both as samples and in some cases as early versions of plugins that may migrate into the core Pulse distribution. The sample plugins are hosted in a Mercurial repository which has an hgweb interface:

http://hg.zutubi.com/

So far we have provided a sample JUnit test post-processor (simplified from the version shipped with Pulse), and the beginnings of a Subversion SCM plugin (wrapping the svn command line). Provided you have Mercurial installed, you can easily clone the samples over HTTP, e.g.:

$ hg clone http://hg.zutubi.com/com.zutubi.pulse.core.postprocessors.junits

The samples have README files to get you started.

You may notice there is a third sample: a plugin that provides basic SCM integration with Bazaar by wrapping the bzr command-line. I’m happy to report that this sample was generously donated by a member of the community, Michiel van Slobbe. Michiel is the first to try to get Pulse and Bazaar working together and we appreciate the effort! Hopefully it will provide inspiration for other community members.

Although we are happy to provide hosting for such community plugins, you may also choose your own path. Another member of the community, Orvid, is using Pulse for C#/Mono projects. Orvid has taken the initiative of writing a command plugin to integrate XBuild, the Mono equivalent of Microsoft’s MSBuild, with Pulse. You can find the XBuild plugin on GitHub:

https://github.com/Orvid/XBuildIntegration

You might also notice that Orvid has been working on a .Net wrapper for the Pulse remote API:

https://github.com/Orvid/Zutubi.Pulse.Api

These efforts are a great contribution that we are most thankful for!

Pulse Continuous Integration Server 2.4 Released!

Happy days: we’ve now released Pulse 2.4! Thanks to all those that provided feedback during the Pulse 2.4 beta period. Here’s a recap of the major updates in this release:

  • Mercurial support: in the form of a new plugin.
  • Maven 3 support: including a command, post-processor and resource discovery.
  • Agents page updates: with graphical status and more convenient navigation.
  • Reworked agent status tab: with more build links and efficient live updates.
  • New agent history tab: quickly browse all builds that involved an agent.
  • Reworked server activity tab: showing build stages nested under active builds.
  • Pause server: admins can pause the build queue, so all triggers are ignored.
  • New server history tab: showing build history across all projects.
  • Restyled info and messages tabs: for both the agents and server sections.
  • Improved process termination: to make terminating builds more reliable.
  • Kill build action: for quicker build termination forgoing full clean up.
  • Improved changelist views: these views have been reworked in the new style.
  • Pinned builds: mark builds that should never be deleted or cleaned.
  • Templated field actions: easily find or revert to an inherited value.
  • Introduce parent refactoring: adjust your template hierarchy over time.
  • Pluggable resource discovery: automatically locate build tools and libraries.
  • Subversion changelist support: easily submit a changelist as a personal build.
  • … and more: extra UI touches, improved performance, more plugin support implementations and more.

The new in 2.4 page on our website has more details and a few screenshots. Or you can simply download and try Pulse 2.4 for free.

Pulse 2.4 Release Candidate

After a few iterations of Pulse 2.4 beta builds, we’ve finally reached a stable enough state to declare our first release candidate. Since the original 2.4 beta post, we haven’t just been squishing bugs, but have come up with several more improvements:

  • New agent history tab: quickly browse all builds that involved an agent.
  • Pause server: admins can pause the build queue, so all triggers are ignored.
  • Improved changelist views: these views have been reworked in the new style.
  • Pinned builds: mark builds that should never be deleted or cleaned.
  • Subversion changelist support: easily submit a changelist as a personal build.
  • Faster browse view: most of these optimisations also appear in Pulse 2.3.
  • Improved process termination: to make terminating builds more reliable.

We know from feedback that these changes will be popular! Check them out yourself: release candidate builds are available from the Beta Program page.

Android JUnit XML Reports: Multiple File Support

Due to popular demand, I’ve added support for multiple output files (one per test suite) to android-junit-report.

For simplicity and efficiency, android-junit-report does not produce files in the exact format used by the Ant JUnit task. In the 1.1 release there are two main differences:

  1. A single report file is produced containing all test suites.
  2. Redundant information, such as the number of cases, failures etc is not added using attributes on the testsuite tag.

It turns out the first of these restrictions caused multiple users issues with tools accustomed to handling a single report file per suite. So in the latest 1.2 release I have added a new multiFile option. When this option is enabled, android-junit-report produces a separate output file for each test suite. This does mean that to retrieve the results from the device you will need to pull a whole directory.

To enable this option from an Ant build, you can override the default run-tests target as follows:


Cleaning up previous test reports...







Running tests...














Downloading XML test reports...








You can learn more about android-junit-report and/or download the new release using the links below:

Android JUnit XML Reports: Now With Test Durations

I’ve been planning to add test case durations to the XML reports generated by android-junit-report for some time. This morning, however, the magic of open source caught up with me. I received a pull request from another GitHub user who had implemented durations already!

So, with thanks to Tim from todoroo, there is a new release of android-junit-report which outputs the duration (in seconds) for each test case. This matches the output produced by the regular Ant JUnit report task, and thus should be compatible with all tools that read the format. Durations are not added to test suites, for the same reason the test case counts are not: it would require buffering. For my own use this is no big deal, because Pulse will sum the times of all cases to give an idea of the total suite time when it is not provided directly.

You can grab this new version of (1.1, build 4) from the GitHub downloads page. Or, as always, you can access the latest release build directly from the build server (click on “jar” in the “featured artifacts” table on the right of the page).

Pulse Continuous Integration Server 2.2!

Big news today: Pulse 2.2 has graduated to stable! This release includes a stack of new features and improvements, including:

  • Build UI overhaul: all tabs improved and restyled.
  • New logs tab: making it easier to access stage logs.
  • Featured artifacts: choose which artifacts should appear prominently.
  • Build navigator: easily move forward and backward through history.
  • Working copy browser: view working copies for in progress builds.
  • Move refactoring: move projects and agents in the template hierarchy.
  • Template navigation: navigate directly up and down a hierarchy.
  • Subscription by label: subscribe to notifications by project groups.
  • Agent executing stages: see what all agents are building at a glance.
  • Subversion exports: for faster and smaller builds.
  • Performance improvements: key for larger installations.

See the new in 2.2 page for full details.

In conjunction with this release, we’ve also given our website a complete overhaul. The new site has a fresher look, and communicates the key features of Pulse more directly. The updates also include some new features:

  • RSS feeds for news items.
  • Links to our latest blog posts on the front page.
  • An improved buying process, allowing multiple licenses to be purchased in one transaction.
  • Self-service renewal payments – just enter your current license key and go!
  • A more user-friendly downloads page.

We hope you enjoy the new release, and the new site. And please, let us know what you think!

Android Testing: XML Reports for Continuous Integration

Summary

This post introduces the Android JUnit Report Test Runner, a custom instrumentation test runner for Android that produces XML test reports. Using this runner you can integrate your Android test results with tools that understand the Ant JUnit task XML format, e.g. the Pulse Continuous Integration Server.

The motivation and details of the runner are discussed below. For the impatient: simply head on over to the project home page on GitHub and check out the README.

Introduction

If you’ve been following my recent posts you’ll know that I’ve been figuring out the practical aspects of testing Android applications. And if you’ve been following for longer, you might know that my day job is development of the Pulse Continuous Integration Server. So it should come as no surprise that in my latest foray into the world of Android testing I sought to bring the two together :).

Status Quo

Out of the box, the Android SDK supports running functional tests on a device or emulator via instrumentation. Running within Eclipse, you get nice integrated feedback. Unfortunately, though, there are no real options for integrating with other tools such as continuous integration servers. Test output from the standard Ant builds is designed for human consumption, and lacks the level of detail I’d like to see in my build reports.

The Solution

On the upside, having access to the Android source makes it possible to examine how the current instrumentation works, and therefore how it can be customised. I found that the default InstrumentationTestRunner may be fairly easily extended to hook in extra test listeners. So I’ve implemented a custom JUnitReportTestRunner that does just that, with a listener that generates a test report in XML format. The format is designed to be largely compatible with the output of the Ant JUnit task’s XML formatter — the most widely supported format in the Java world. Tools like Pulse can read in this format to give rich test reporting.

How It Works

As mentioned, the JUnitReportTestRunner extends the default InstrumentationTestRunner, so it can act as a drop-in replacement. The custom runner acts identically to the default, with the added side-effect of producing an XML report.

For consistency with the SDK’s support for generating coverage reports, the XML report is generated in the file storage area of the target application. The default report location is something like:

/data/data/<tested application package>/files/junit-report.xml

on the device. To retrieve the report, you can use adb pull, typically as part of your scripted build.

Using the Runner

Full details on using the runner are provided in the README on the project home page. Briefly:

  • Add the android-junit-report-<version>.jar to the libraries for your test application.
  • Replace all occurrences of android.test.InstrumentationTestRunner with com.zutubi.android.junitreport.JUnitReportTestRunner:
    • In the android:name attribute of the instrumentation tag in you test application’s AndroidManifest.xml.
    • In the test.runner property in the Ant build for your test application (before calling the Android setup task).
    • In the Instrumentation runner field of all Android JUnit Run Configurations in your Eclipse project.
  • Add logic to your Ant build to run adb pull to retrieve the report after the tests are run.

As an example for retrieving the report in your Ant build:

<target name="fetch-test-report">
    <echo>Downloading XML test report...</echo>
    <mkdir dir="${reports.dir}"/>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}"/>
        <arg value="pull" />
        <arg value="/data/data/${tested.manifest.package}/files/junit-report.xml" />
        <arg value="${reports.dir}/junit-report.xml" />
    </exec>
</target>

In the Wild

You can see a complete example of this in action in my simple DroidScope Android application. The custom runner is applied in the droidscope-test application in the test/ subdirectory. You can even see the test results being picked up by Pulse on our demo server. Note that some of the tests are pure unit tests, which are run on a regular JVM, whereas others are run with the custom runner on an emulator. It’s nice for all the results to be collected together!

Android Testing: Using Pure Unit Tests

Introduction

The Android SDK comes with support for testing, allowing tests to be run on an Android device (or emulator) via instrumentation. This is useful for functional tests that require a realistic environment, but for the majority of tests it is overkill. The instrumentation and emulation layers add complexity to the process, making tests much slower to run and harder to debug.

The good news is that there is no need to run most of your tests via instrumentation. Because Android applications consist of regular Java code, it is possible to isolate much of the implementation from the Android environment. In fact, if you’ve separated concerns in your application already, it’s likely that large parts of it are already independent of the Android APIs. Those sections of your code can be tested on a regular JVM, using the rich ecosystem of tools available for unit testing.

Unit Testing Requirements

To put this idea into practice, I set out the following requirements for unit testing my Android application:

  1. The unit tests should run on a regular JVM, with no dependency on the Android APIs or tools.
  2. It should be possible to run the tests within Eclipse.
  3. It should be possible to run tests using Ant.
  4. Running tests via Ant should produce reports suitable for use with a Continuous Integration server.

These requirements allow the tests to be run quickly within the development environment, and on every commit on a build server.

Adding a Unit Testing Project

In keeping with my existing Android project setup, I decided to use an additional project specifically for unit testing. To recap, in the original setup I had two projects:

  1. The main project: containing the application itself.
  2. The test project: containing an Android test project for instrumentation testing, in a test/ subdirectory of the root.

Both projects had Ant build files and Eclipse projects. Similar to the use of a test/ subdirectory for instrumentation tests, I added my new unit test project in a unit/ subdirectory of the root. As with the other projects, the source code for the unit tests lives in a src/ subdirectory, giving the following overall layout:

my-app/
    src/        - main application source
    test/
        src/    - functional tests
    unit/
        src/    - unit tests

Creating the Eclipse project for unit testing was trivial: I just added a new Java Project named my-app-unit. I then edited the build path of this project to depend on my main my-app project, so that I could build against the code under test.

Testing Libraries

The main tool required for this setup is a unit testing framework. I decided to go with JUnit 4 as it is well supported in Eclipse, Ant and CI servers. (JUnit is also used by the instrumentation testing support in the Android SDK.) In addition, for mocking I am a fan of Mockito. Note, though, that the beauty of using pure Java tests is you can use any of the myriad of mocking (and other) libraries out there.

For consistency with the existing projects, I added the JUnit and Mockito jars to a libs/ subdirectory of the unit project. I then added those jars to the build path of my Eclipse project, and I was ready to implement some tests!

A Trivial Test

To make sure the setup works, you can try adding a trivial JUnit 4 test case:

package com.zutubi.android.myapp;

import static org.junit.Assert.*;

import org.junit.Test;

public class MyAppTest
{
    @Test
    public void testWorld()
    {
        assertEquals(2, 1 + 1);
    }
}

If all is well you should be able to run this in Eclipse as a JUnit test case. Once you have this sanity test passing, you can proceed to some Real Tests.

Adding an Ant Build

Setting up an Ant build took a little more effort than for the original projects, as their build files import Android rules from the SDK. For the unit tests, I wrote a simple build file from scratch, trying to keep within the conventions established by the Android rules:

<?xml version="1.0" encoding="UTF-8"?>
<project name="my-app-unit" default="test">
    <property name="source.dir" value="src"/>
    <property name="libs.dir" value="libs"/>

    <property name="out.dir" value="build"/>
    <property name="classes.dir" value="${out.dir}/classes"/>
    <property name="reports.dir" value="${out.dir}/reports"/>
    <property name="tested.dir" value=".."/>
    <property name="tested.classes.dir" value="${tested.dir}/build/classes"/>
    <property name="tested.libs.dir" value="${tested.dir}/libs"/>
    
    <path id="compile.classpath">
        <fileset dir="${libs.dir}" includes="*.jar"/>
        <fileset dir="${tested.libs.dir}" includes="*.jar"/>
        <pathelement location="${tested.classes.dir}"/>
    </path>

    <path id="run.classpath">
        <path refid="compile.classpath"/>
        <pathelement location="${classes.dir}"/>
    </path>
    
    <target name="clean">
        <delete dir="${out.dir}"/>
    </target>
    
    <target name="-init">
    	<mkdir dir="${out.dir}"/>
    	<mkdir dir="${classes.dir}"/>
    	<mkdir dir="${reports.dir}"/>
    </target>
    
    <target name="-compile-tested">
        <subant target="compile" buildpath="${tested.dir}"/>
    </target>
    
    <target name="compile" depends="-init,-compile-tested">
        <javac target="1.5" debug="true" destdir="${classes.dir}">
            <src path="${source.dir}"/>
            <classpath refid="compile.classpath"/>
        </javac>
    </target>
    
    <target name="run-tests" depends="compile">
        <junit printsummary="yes" failureproperty="test.failure">
            <classpath refid="run.classpath"/>
            
            <formatter type="xml"/>
            
            <batchtest todir="${reports.dir}">
                <fileset dir="${source.dir}" includes="**/*Test.java"/>
            </batchtest>
        </junit>
        
        <fail message="One or more test cases failed" if="test.failure"/>
    </target>
</project>

The run-tests target in this build file compiles all of the unit test code against the libraries in the unit test project, plus the classes and libraries from the project under test. It then runs all JUnit tests in classes that have names ending with Test, printing summarised results and producing full XML reports in build/reports/. These XML reports are ideal for integrating your results with a CI server (Pulse in my case, of course!).

Wrap Up

The Android SDK support for testing is useful for functional tests, but too slow and cumbersome for rapid-feedback unit testing. However, there is nothing to stop you from isolating the pure Java parts of your application and testing them separately. In fact this is one of those rare win-wins: by clean design of your code you also get access to all the speed and tool support of testing on a regular JVM!

Pulse Continuous Integration Server 2.2 Beta!

Great news: today the latest incarnation of Pulse, version 2.2, went beta! In this release we’ve focused primarily on usability, largely in the build reporting UI. A new build navigation widget allows you to easily step forwards and backwards in your build history – while sticking to the same build tab. All of the build tabs themselves have been overhauled with new styling and layout. Here’s a sneak peak at the artifacts tab, for example:

Artifacts Tab

Artifacts Tab

It not only shows additional information, with greater clarity, but also allows you to sort and filter artifacts so you can find the file you are after. Other UI changes go beyond style too – for example the new build summary tab shows related links and featured artifacts for the build. More information, and screenshots, are available on the new in 2.2 page.

We’ve also squeezed in some less obvious updates, such as:

  • The much-requested ability to move projects and agents in the template hierarchy.
  • Convenient navigation up and down the template hierarchy.
  • The ability to subscribe to projects by label.
  • An option to use subversion exports for smaller and faster builds.
  • Improved cleanup of persistent working directories (when requesting a clean build).
  • Performance improvements for large configuration sets.

The first beta build, Pulse 2.2.0, is available for download now. We’d love you to give it a spin and let us know what you think!