a little madness

A man needs a little madness, or else he never dares cut the rope and be free -Nikos Kazantzakis

Zutubi

Android Testing: XML Reports for Continuous Integration

Summary

This post introduces the Android JUnit Report Test Runner, a custom instrumentation test runner for Android that produces XML test reports. Using this runner you can integrate your Android test results with tools that understand the Ant JUnit task XML format, e.g. the Pulse Continuous Integration Server.

The motivation and details of the runner are discussed below. For the impatient: simply head on over to the project home page on GitHub and check out the README.

Introduction

If you’ve been following my recent posts you’ll know that I’ve been figuring out the practical aspects of testing Android applications. And if you’ve been following for longer, you might know that my day job is development of the Pulse Continuous Integration Server. So it should come as no surprise that in my latest foray into the world of Android testing I sought to bring the two together :).

Status Quo

Out of the box, the Android SDK supports running functional tests on a device or emulator via instrumentation. Running within Eclipse, you get nice integrated feedback. Unfortunately, though, there are no real options for integrating with other tools such as continuous integration servers. Test output from the standard Ant builds is designed for human consumption, and lacks the level of detail I’d like to see in my build reports.

The Solution

On the upside, having access to the Android source makes it possible to examine how the current instrumentation works, and therefore how it can be customised. I found that the default InstrumentationTestRunner may be fairly easily extended to hook in extra test listeners. So I’ve implemented a custom JUnitReportTestRunner that does just that, with a listener that generates a test report in XML format. The format is designed to be largely compatible with the output of the Ant JUnit task’s XML formatter — the most widely supported format in the Java world. Tools like Pulse can read in this format to give rich test reporting.

How It Works

As mentioned, the JUnitReportTestRunner extends the default InstrumentationTestRunner, so it can act as a drop-in replacement. The custom runner acts identically to the default, with the added side-effect of producing an XML report.

For consistency with the SDK’s support for generating coverage reports, the XML report is generated in the file storage area of the target application. The default report location is something like:

/data/data/<tested application package>/files/junit-report.xml

on the device. To retrieve the report, you can use adb pull, typically as part of your scripted build.

Using the Runner

Full details on using the runner are provided in the README on the project home page. Briefly:

  • Add the android-junit-report-<version>.jar to the libraries for your test application.
  • Replace all occurrences of android.test.InstrumentationTestRunner with com.zutubi.android.junitreport.JUnitReportTestRunner:
    • In the android:name attribute of the instrumentation tag in you test application’s AndroidManifest.xml.
    • In the test.runner property in the Ant build for your test application (before calling the Android setup task).
    • In the Instrumentation runner field of all Android JUnit Run Configurations in your Eclipse project.
  • Add logic to your Ant build to run adb pull to retrieve the report after the tests are run.

As an example for retrieving the report in your Ant build:

<target name="fetch-test-report">
    <echo>Downloading XML test report...</echo>
    <mkdir dir="${reports.dir}"/>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}"/>
        <arg value="pull" />
        <arg value="/data/data/${tested.manifest.package}/files/junit-report.xml" />
        <arg value="${reports.dir}/junit-report.xml" />
    </exec>
</target>

In the Wild

You can see a complete example of this in action in my simple DroidScope Android application. The custom runner is applied in the droidscope-test application in the test/ subdirectory. You can even see the test results being picked up by Pulse on our demo server. Note that some of the tests are pure unit tests, which are run on a regular JVM, whereas others are run with the custom runner on an emulator. It’s nice for all the results to be collected together!

Liked this post? Share it!

7 Responses to “Android Testing: XML Reports for Continuous Integration”

  1. July 15th, 2010 at 3:11 am

    Manfred Moser says:

    Very nice. I just wanted to clarify one omission I noticed. There are in fact other options for running an Android build on a continuous integration server. The Maven Android Plugin supports unit as well as integration(instrumentation) testing and other things like start/stop the emulator. Since it is using Maven pretty much any CI server can use it. In fact the samples that come with the plugin are run on Hudson for a few month…

  2. July 15th, 2010 at 11:09 am

    Jason says:

    Hi Manfred,

    My post was not to suggest that there was no way to run an Android build on a CI server previously, so I think it’s a stretch to call that an “omission”. The piece that was missing, as far as I was aware, was some way to get nicely integrated test reporting. Although the tests would run and appear in the build log, many CI servers support richer integration and reporting of test results. To do this they need access to a parseable test report. Hence my new runner that produces such reports.

    My new runner itself could equally be used with a Maven build, provided your plugin allows the user to customise the test runner somehow. In fact it could be nice to figure out how to put the two together for people that prefer Maven!

  3. September 6th, 2010 at 10:56 am

    Junit Test Runner for ANT « MobileBytes says:

    […] on the SD card that than can be pushed down to a desktop folder using an ant task. The blog post is here and github project page is […]

  4. December 11th, 2010 at 9:21 am

    Testing Android Apps | Sam Snyder says:

    […] Android Testing – XML Reports for Continuous Integration: more advice from […]

  5. February 11th, 2011 at 3:05 am

    mayank says:

    Hi,

    I use the steps suggested above and I am able to get a basic XML file generated after running my test cases in Android. However when I try to use that with an industry accepted CI tool like Quickbuild, its not able to consume the XML file and throws errors.

    Wonder if the XML file generated is pure JUnit-XML or something that works with Pulse but may/maynot work with other CI tools or standard XML-HTML converters to give me a nice GUI interface to look at tests run/pass/fail etc?

    Another question I have is on the jar file that is mentioned, what licensing terms is it released under? Can I use it for commercial distribution as part of my application?

    Regards

  6. February 18th, 2011 at 4:38 pm

    Jason says:

    Mayank,

    The README for android-junit-report mentions some caveats about the report file format, the main ones being that:

    – All suites are dumped in a single file; and
    – No redundant information is given.

    Pulse is flexible enough in its handling of these types of reports to read this output, but I’m not surprised that some tools aren’t, as it differs from the junit task’s output.

    However, today I have added a new feature to android-junit-report that allows you to generate multiple output files – oe per suite. Perhaps Quickbuild will have more luck with them:

    https://github.com/jsankey/android-junit-report

    Re: the license, this is with the source code in GitHub (Apache 2). Commercial distribution is fine.

  7. February 23rd, 2011 at 4:08 pm

    a little madness » Blog Archive » Android JUnit XML Reports: Multiple File Support says:

    […] Original blog post […]

Leave a Reply