a little madness

A man needs a little madness, or else he never dares cut the rope and be free -Nikos Kazantzakis

Zutubi

Archive for the ‘Testing’ Category

Pulse 3.0 Alpha

With great pleasure I can announce the very first build of Pulse 3.0 is now available from our Alpha Program page! Although this build is very much incomplete and unstable, this is a huge milestone after months of work in both the back and front ends of Pulse.

As this is a larger release we’ve made the choice to release earlier in the cycle than we otherwise would, for a couple of reasons:

  • To show the massive progress that has already been achieved.
  • To solicit feedback about the all-new administration UI (in time to apply said feedback before feature freeze).

We know from previous feedback that the new UI will address some pain points, mainly around quickly navigating and understanding your configuration in a large setup. We’ve also taken advantage of modern browser features to fix clunky corners: e.g. proper history integration and drag-and-drop reordering (no more clickety-click!). With such major changes, though, we’re always keen to hear what you think — good or bad — so we can keep on the right track.

So please do find the time to download and play with an alpha build, then get in touch with us via support email. We’ll be iterating fast on this release stream now, so expect to see updates about regular builds with new UI features each time. Happy building!

Android: JUnit XML Reports with Gradle

The Android development tools project has seen big changes over the last year. The original Eclipse ADT development environment was superseded late last year by Android Studio — a new IDE based on Intellij. Under the hood Android Studio also uses a new command line build system based on Gradle, replacing the previous Ant-based system. I’ve been keen to find out how these changes impact the integration of Android test reports with continuous integration servers like Pulse.

Summary

  • Android JUnit Report is redundant.
  • Run on-device Android tests with:
    ./gradlew connectedAndroidTest
  • Collect reports from:
    app/build/outputs/androidTest-results/connected/*.xml

 

Details

The original Ant-based build system for Android didn’t produce XML test reports for instrumentation tests (i.e. those that run on-device), prompting me to create the Android JUnit Report project. Android JUnit Report produced XML output similar to the Ant JUnit task, making it compatible with most continuous integration servers. The good news is: Android JUnit Report is now redundant. The new Gradle-based build system produces sane XML test reports out of the box. In fact, they’re even more complete than those produced by Android JUnit Report, so should work with even more continuous integration servers.

The only downside is the documentation, which is a little confusing (while there are still documents for the old system about) and not very detailed. With a bit of experimentation and poking around I found how to run on-device (or emulator) tests and where the XML reports were stored. With a default project layout as created by Android Studio:

ASDemo.iml
app/
  app.iml
  build.gradle
  libs/
  proguard-rules.pro
  src/
    androidTest/
    main/
build.gradle
gradle
gradle.properties
gradlew
gradlew.bat
local.properties
settings.gradle

You get a built-in version of Gradle to use for building your project, launched via gradlew. To see available tasks, run:

$ ./gradlew tasks

(This will download a bunch of dependencies when first run.) Amongst plenty of output, take a look at the Verification Tasks section:

Verification tasks
------------------
check - Runs all checks.
connectedAndroidTest - Installs and runs the tests for Debug build on connected devices.
connectedCheck - Runs all device checks on currently connected devices.
deviceCheck - Runs all device checks using Device Providers and Test Servers.
lint - Runs lint on all variants.
lintDebug - Runs lint on the Debug build.
lintRelease - Runs lint on the Release build.
test - Run all unit tests.
testDebug - Run unit tests for the Debug build.
testRelease - Run unit tests for the Release build.

The main testing target test does not run on-device tests, only unit tests that run locally. For on-device tests you use the connectedAndroidTest task. Try it:

$ ./gradlew connectedAndroidTest
...
:app:compileDebugAndroidTestJava
:app:preDexDebugAndroidTest
:app:dexDebugAndroidTest
:app:processDebugAndroidTestJavaRes UP-TO-DATE
:app:packageDebugAndroidTest
:app:assembleDebugAndroidTest
:app:connectedAndroidTest
:app:connectedCheck

BUILD SUCCESSFUL

Total time: 33.372 secs

It’s not obvious, but this produces compatible XML reports under:

app/build/outputs/androidTest-results/connected

with names based on the application module and device. In your continuous integration setup you can just collect all *.xml files in this directory for reporting.

Although the new build system has killed the need for my little Android JUnit Report project, this is a welcome development. Now all Android developers get better test reporting without an external dependency. Perhaps it will even encourage a few more people to use continuous integration servers like Pulse to keep close tabs on their tests!

Pulse 2.7 Released

I’m dusting off the blog with a bang, announcing that Pulse 2.7 has gone gold! This release brings a broad range of new features and improvements, including:

  • New agent storage management options, including the ability to prevent builds when disk space is low.
  • Configuration system performance improvements.
  • Live logging performance improvements.
  • Xcode command updates, including a new clang output processor.
  • A new plugin for integration of XCTest reports.
  • More flexibility and feedback for manual triggering.
  • New service support, including integration with systemd and upstart.
  • Improved support for git 2.x, especially partial clones.
  • Support for Subversion 1.8.
  • Improved dependency revision handling across multiple SCMs.
  • More convenient actions for cancelling builds.
  • The ability to run post build hooks on agents.

As always we based these improvements on feedback from our customers, and we thank all those that took the time to let us know their priorities.

Pulse 2.7 packages can de downloaded from the downloads page. If you’re an existing customer with an active support contract then this is a free upgrade. If you’re new to Pulse, we also provide free licenses for evaluation, open source projects and small teams!

Android JUnit Report Updates

Recently I’ve made a few updates to my custom Android test runner, android-junit-report. This runner makes it easy to integrate your Android test results into a continuous integration process by producing de-facto standard JUnit-style XML reports. The latest changes bring the runner up to speed with the latest version of Android and include better documentation. A summary of the updates follows:

  • A new home page for the project.
  • Within the home page, full documentation.
  • Updates to where the reports are written by default. The runner no longer attempts to use the internal storage of the test application, instead it always defaults to the internal storage of the main application (i.e. the one under test).
  • Support for a new __external__ token that allows you to place the reports in the external storage area of the application under test (given availability and permission to do so).
  • Changes to the token syntax to avoid the need to escape when calling the runner via a shell.

I’d like to acknowledge the help of Sebastian Schuberth and Christopher Orr in fine-tuning some aspects of the runner for this release. The latest runner version, 1.5.8, is available for download now.

Pulse Continuous Integration Server 2.5 Released!

After a smooth beta period, we’re excited to announce the release of Pulse 2.5! Massive credit for the new feature and improvement ideas in this release go to the Pulse community, whose feedback was invaluable. Here’s the brief list of what’s new:

  • Browse and dashboard view filtering (by project health).
  • A new MSTest post-processor plugin.
  • Increased visibility of build (and other) comments.
  • Comments on agents, to communicate with other users.
  • Properties on agents, for simpler parameterisation.
  • The ability to reference properties in SCM configuration.
  • The ability to configure SCMs using resources.
  • A resource repository on the Pulse master.
  • A new option to attach build logs to notification emails.
  • Upstream change reporting (changes via dependencies).
  • Downstream build notifications (again via dependencies).
  • Simpler delivery of directory trees via dependencies.
  • Pre-stage hooks, which run just before the stage is dispatched to the agent.
  • Stage terminate hooks, allowing you to run custom cleanup.
  • SCM inclusion filters, complementing existing exclusion filters.
  • Inverse resource requirements (i.e. requiring the absence of a resource).
  • Smart label renaming.
  • A distinct warnings status for UI and API consistency.
  • A separate permission for the ability to clean up build directories.
  • Support for git submodules.
  • Support for Subversion 1.7 in personal builds.
  • Support for Perforce streams.

For more details, see the new in 2.5 page on our website. Or, why not just download Pulse and try it free today?

Pulse Continuous Integration Server 2.5 Alpha!

Cobwebs may be creeping over the blog, but not over Pulse: we’ve just published the first Pulse 2.5 alpha build! The Pulse 2.5 series is focused (even more so than usual) on customer feedback. We’ve got plenty of great ideas from the Pulse community, and we’re folding in some of the best ones in this release series. Updates so far include:

  • Browse and dashboard view filtering (by project health).
  • A new MSTest post-processor plugin.
  • Increased visibility of build (and other) comments.
  • Comments on agents, to communicate with other users.
  • Properties on agents, for simpler parameterisation.
  • The ability to configure SCMs using resources.
  • A resource repository on the Pulse master.
  • A new option to attach build logs to notification emails.
  • Pre-stage hooks, which run just before the stage is dispatched to the agent.
  • SCM inclusion filters, complementing existing exclusion filters.
  • Inverse resource requirements (i.e. requiring the absence of a resource).
  • Smart label renaming.
  • A separate permission for the ability to clean up build directories.
  • Support for Perforce streams.

There’s plenty more to come in subsequent alpha builds, too. In fact, we’ve started the groundwork for some other larger changes already in 2.5.0.

To grab yourself a fresh-from-the-oven 2.5 alpha build, or just to find out more, head over to our alpha program page.

Pulse Sample and Community Plugins

Since version 2.0, Pulse has supported plugins to extend integration with external tools. Whenever customers have asked for help with plugin implementation, we’ve always provided samples as the easiest starting point. Seeing a complete plugin project, with build and packaging support, is the easiest way to get a new plugin up and running.

Based on this we’ve decided to start maintaining a repository of open source Pulse plugins. These will act both as samples and in some cases as early versions of plugins that may migrate into the core Pulse distribution. The sample plugins are hosted in a Mercurial repository which has an hgweb interface:

http://hg.zutubi.com/

So far we have provided a sample JUnit test post-processor (simplified from the version shipped with Pulse), and the beginnings of a Subversion SCM plugin (wrapping the svn command line). Provided you have Mercurial installed, you can easily clone the samples over HTTP, e.g.:

$ hg clone http://hg.zutubi.com/com.zutubi.pulse.core.postprocessors.junits

The samples have README files to get you started.

You may notice there is a third sample: a plugin that provides basic SCM integration with Bazaar by wrapping the bzr command-line. I’m happy to report that this sample was generously donated by a member of the community, Michiel van Slobbe. Michiel is the first to try to get Pulse and Bazaar working together and we appreciate the effort! Hopefully it will provide inspiration for other community members.

Although we are happy to provide hosting for such community plugins, you may also choose your own path. Another member of the community, Orvid, is using Pulse for C#/Mono projects. Orvid has taken the initiative of writing a command plugin to integrate XBuild, the Mono equivalent of Microsoft’s MSBuild, with Pulse. You can find the XBuild plugin on GitHub:

https://github.com/Orvid/XBuildIntegration

You might also notice that Orvid has been working on a .Net wrapper for the Pulse remote API:

https://github.com/Orvid/Zutubi.Pulse.Api

These efforts are a great contribution that we are most thankful for!

Android JUnit XML Reports: Multiple File Support

Due to popular demand, I’ve added support for multiple output files (one per test suite) to android-junit-report.

For simplicity and efficiency, android-junit-report does not produce files in the exact format used by the Ant JUnit task. In the 1.1 release there are two main differences:

  1. A single report file is produced containing all test suites.
  2. Redundant information, such as the number of cases, failures etc is not added using attributes on the testsuite tag.

It turns out the first of these restrictions caused multiple users issues with tools accustomed to handling a single report file per suite. So in the latest 1.2 release I have added a new multiFile option. When this option is enabled, android-junit-report produces a separate output file for each test suite. This does mean that to retrieve the results from the device you will need to pull a whole directory.

To enable this option from an Ant build, you can override the default run-tests target as follows:

<target name="run-tests" depends="-install-tested-project, install"
       description="Runs tests from the package defined in test.package property">
    <property name="reports.dir" value="${out.dir}/reports"/>
    <property name="files.dir" value="/data/data/${tested.manifest.package}/files"/>
    <echo>Cleaning up previous test reports…</echo>
    <delete dir="${reports.dir}"/>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}" />
        <arg value="shell" />
        <arg value="rm" />
        <arg value="${files.dir}/*" />
    </exec>
    <echo>Running tests…</echo>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}"/>
        <arg value="shell" />
        <arg value="am" />
        <arg value="instrument" />
        <arg value="-w" />
        <arg value="-e" />
        <arg value="coverage" />
        <arg value="@{emma.enabled}" />
        <arg value="-e" />
        <arg value="multiFile" />
        <arg value="true" />
        <arg value="${manifest.package}/${test.runner}" />
    </exec>      
    <echo>Downloading XML test reports…</echo>
    <mkdir dir="${reports.dir}"/>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}"/>
        <arg value="pull" />
        <arg value="${files.dir}" />
        <arg value="${reports.dir}" />
    </exec>
</target>

You can learn more about android-junit-report and/or download the new release using the links below:

Android JUnit XML Reports: Now With Test Durations

I’ve been planning to add test case durations to the XML reports generated by android-junit-report for some time. This morning, however, the magic of open source caught up with me. I received a pull request from another GitHub user who had implemented durations already!

So, with thanks to Tim from todoroo, there is a new release of android-junit-report which outputs the duration (in seconds) for each test case. This matches the output produced by the regular Ant JUnit report task, and thus should be compatible with all tools that read the format. Durations are not added to test suites, for the same reason the test case counts are not: it would require buffering. For my own use this is no big deal, because Pulse will sum the times of all cases to give an idea of the total suite time when it is not provided directly.

You can grab this new version of (1.1, build 4) from the GitHub downloads page. Or, as always, you can access the latest release build directly from the build server (click on “jar” in the “featured artifacts” table on the right of the page).

Android Testing: XML Reports for Continuous Integration

Summary

This post introduces the Android JUnit Report Test Runner, a custom instrumentation test runner for Android that produces XML test reports. Using this runner you can integrate your Android test results with tools that understand the Ant JUnit task XML format, e.g. the Pulse Continuous Integration Server.

The motivation and details of the runner are discussed below. For the impatient: simply head on over to the project home page on GitHub and check out the README.

Introduction

If you’ve been following my recent posts you’ll know that I’ve been figuring out the practical aspects of testing Android applications. And if you’ve been following for longer, you might know that my day job is development of the Pulse Continuous Integration Server. So it should come as no surprise that in my latest foray into the world of Android testing I sought to bring the two together :).

Status Quo

Out of the box, the Android SDK supports running functional tests on a device or emulator via instrumentation. Running within Eclipse, you get nice integrated feedback. Unfortunately, though, there are no real options for integrating with other tools such as continuous integration servers. Test output from the standard Ant builds is designed for human consumption, and lacks the level of detail I’d like to see in my build reports.

The Solution

On the upside, having access to the Android source makes it possible to examine how the current instrumentation works, and therefore how it can be customised. I found that the default InstrumentationTestRunner may be fairly easily extended to hook in extra test listeners. So I’ve implemented a custom JUnitReportTestRunner that does just that, with a listener that generates a test report in XML format. The format is designed to be largely compatible with the output of the Ant JUnit task’s XML formatter — the most widely supported format in the Java world. Tools like Pulse can read in this format to give rich test reporting.

How It Works

As mentioned, the JUnitReportTestRunner extends the default InstrumentationTestRunner, so it can act as a drop-in replacement. The custom runner acts identically to the default, with the added side-effect of producing an XML report.

For consistency with the SDK’s support for generating coverage reports, the XML report is generated in the file storage area of the target application. The default report location is something like:

/data/data/<tested application package>/files/junit-report.xml

on the device. To retrieve the report, you can use adb pull, typically as part of your scripted build.

Using the Runner

Full details on using the runner are provided in the README on the project home page. Briefly:

  • Add the android-junit-report-<version>.jar to the libraries for your test application.
  • Replace all occurrences of android.test.InstrumentationTestRunner with com.zutubi.android.junitreport.JUnitReportTestRunner:
    • In the android:name attribute of the instrumentation tag in you test application’s AndroidManifest.xml.
    • In the test.runner property in the Ant build for your test application (before calling the Android setup task).
    • In the Instrumentation runner field of all Android JUnit Run Configurations in your Eclipse project.
  • Add logic to your Ant build to run adb pull to retrieve the report after the tests are run.

As an example for retrieving the report in your Ant build:

<target name="fetch-test-report">
    <echo>Downloading XML test report...</echo>
    <mkdir dir="${reports.dir}"/>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}"/>
        <arg value="pull" />
        <arg value="/data/data/${tested.manifest.package}/files/junit-report.xml" />
        <arg value="${reports.dir}/junit-report.xml" />
    </exec>
</target>

In the Wild

You can see a complete example of this in action in my simple DroidScope Android application. The custom runner is applied in the droidscope-test application in the test/ subdirectory. You can even see the test results being picked up by Pulse on our demo server. Note that some of the tests are pure unit tests, which are run on a regular JVM, whereas others are run with the custom runner on an emulator. It’s nice for all the results to be collected together!