a little madness

A man needs a little madness, or else he never dares cut the rope and be free -Nikos Kazantzakis

Zutubi

Archive for the ‘Android’ Category

Android: JUnit XML Reports with Gradle

The Android development tools project has seen big changes over the last year. The original Eclipse ADT development environment was superseded late last year by Android Studio — a new IDE based on Intellij. Under the hood Android Studio also uses a new command line build system based on Gradle, replacing the previous Ant-based system. I’ve been keen to find out how these changes impact the integration of Android test reports with continuous integration servers like Pulse.

Summary

  • Android JUnit Report is redundant.
  • Run on-device Android tests with:
    ./gradlew connectedAndroidTest
  • Collect reports from:
    app/build/outputs/androidTest-results/connected/*.xml

 

Details

The original Ant-based build system for Android didn’t produce XML test reports for instrumentation tests (i.e. those that run on-device), prompting me to create the Android JUnit Report project. Android JUnit Report produced XML output similar to the Ant JUnit task, making it compatible with most continuous integration servers. The good news is: Android JUnit Report is now redundant. The new Gradle-based build system produces sane XML test reports out of the box. In fact, they’re even more complete than those produced by Android JUnit Report, so should work with even more continuous integration servers.

The only downside is the documentation, which is a little confusing (while there are still documents for the old system about) and not very detailed. With a bit of experimentation and poking around I found how to run on-device (or emulator) tests and where the XML reports were stored. With a default project layout as created by Android Studio:

ASDemo.iml
app/
  app.iml
  build.gradle
  libs/
  proguard-rules.pro
  src/
    androidTest/
    main/
build.gradle
gradle
gradle.properties
gradlew
gradlew.bat
local.properties
settings.gradle

You get a built-in version of Gradle to use for building your project, launched via gradlew. To see available tasks, run:

$ ./gradlew tasks

(This will download a bunch of dependencies when first run.) Amongst plenty of output, take a look at the Verification Tasks section:

Verification tasks
------------------
check - Runs all checks.
connectedAndroidTest - Installs and runs the tests for Debug build on connected devices.
connectedCheck - Runs all device checks on currently connected devices.
deviceCheck - Runs all device checks using Device Providers and Test Servers.
lint - Runs lint on all variants.
lintDebug - Runs lint on the Debug build.
lintRelease - Runs lint on the Release build.
test - Run all unit tests.
testDebug - Run unit tests for the Debug build.
testRelease - Run unit tests for the Release build.

The main testing target test does not run on-device tests, only unit tests that run locally. For on-device tests you use the connectedAndroidTest task. Try it:

$ ./gradlew connectedAndroidTest
...
:app:compileDebugAndroidTestJava
:app:preDexDebugAndroidTest
:app:dexDebugAndroidTest
:app:processDebugAndroidTestJavaRes UP-TO-DATE
:app:packageDebugAndroidTest
:app:assembleDebugAndroidTest
:app:connectedAndroidTest
:app:connectedCheck

BUILD SUCCESSFUL

Total time: 33.372 secs

It’s not obvious, but this produces compatible XML reports under:

app/build/outputs/androidTest-results/connected

with names based on the application module and device. In your continuous integration setup you can just collect all *.xml files in this directory for reporting.

Although the new build system has killed the need for my little Android JUnit Report project, this is a welcome development. Now all Android developers get better test reporting without an external dependency. Perhaps it will even encourage a few more people to use continuous integration servers like Pulse to keep close tabs on their tests!

Open Source at Zutubi

At Zutubi we’ve reaped massive benefits over the years from open source. We’ve contributed back bits and pieces in that time — feedback and patches — but always felt we could do more. You may have noticed recent posts here about some of our projects, but we thought they deserved a bit more attention. So we’ve recently launch an new open source section on our website. This is a home for a few small projects of our own that we’ve opened up to the community. We hope you find them useful!

So far we’ve listed three projects:

  1. android junit report: a custom test runner for Android projects that produces XML test reports in a format similar to the Ant JUnit task.
  2. zutubi android ant: a collection of Ant tasks that supplement the standard Android build tooling.
  3. zutubi diff: a Java library for reading, inspecting and applying patch files in unified diff and other formats.

Over time we hope to publish more of our work in this way. You can keep an eye out for new projects on our website and/or follow our organisation on GitHub (or indeed my own GitHub account for Android-specific projects).

And as always: feedback is welcome, so fork away!

Android JUnit Report Updates

Recently I’ve made a few updates to my custom Android test runner, android-junit-report. This runner makes it easy to integrate your Android test results into a continuous integration process by producing de-facto standard JUnit-style XML reports. The latest changes bring the runner up to speed with the latest version of Android and include better documentation. A summary of the updates follows:

  • A new home page for the project.
  • Within the home page, full documentation.
  • Updates to where the reports are written by default. The runner no longer attempts to use the internal storage of the test application, instead it always defaults to the internal storage of the main application (i.e. the one under test).
  • Support for a new __external__ token that allows you to place the reports in the external storage area of the application under test (given availability and permission to do so).
  • Changes to the token syntax to avoid the need to escape when calling the runner via a shell.

I’d like to acknowledge the help of Sebastian Schuberth and Christopher Orr in fine-tuning some aspects of the runner for this release. The latest runner version, 1.5.8, is available for download now.

Android: Easier Source Linking in Eclipse

In my prior post Android: Attaching Sources to Libraries in Eclipse I noted that ADT version r20 added the ability to link sources to libraries in the Android Dependencies classpath container in Eclipse. Although this was a welcome addition, the method used to link sources is a bit cumbersome: you must create a .properties file alongside each library, and add a property that points to the source jar.

I felt this could be a lot simpler if you are willing to adopt a convention for the location of source jars. Once the convention is established, adding linked sources is as simple as placing the source jar in the expected location. This makes the process as seamless as the Android Dependency container is for adding libraries in the first place.

So I’ve made this possible via a new libproperties task in my zutubi-android-ant project. This task inspects the jars in your libs/ directory, looks for corresponding source jars by naming convention, and adds appropriate .properties files when sources are found. If you’re willing to accept the default convention, where source a source jar for library libs/foo.jar is placed at libs/src/foo-source.jar, then this simple Ant target can be used to generate the required .properties files for you:

<target name="link-sources">
    <zaa:libproperties/>
</target>

Of course you’ll need to import the zutubi-android-ant tasks into your build file first. If you want to ensure sources are present for all libraries, set the attribute failOnMissingSrc to true and your build will fail if any source jar is missing. You can even establish your own conventions, using a regular expression to match the libraries and a replacement string to generate paths to source jars:

<target name="link-sources">
    <-- libs/alib-1.3.jar maps to libs/src/alib-src-1.3.jar -->
    <zaa:libproperties failOnMissingSrc="true"
                         jarPattern="(.+)-([0-9.]+)\.jar"
                         srcReplacement="src/$1-src-$2.jar"/>
</target>

Head over to the new zutubi-android-ant home page for more details, including quick-start instructions to get the tasks installed in your build in minutes.

New Android Ant Tasks: lint and ndkbuild

I’m back into Android development these days, and expect plenty more to come (especially with a Nexus 7 due to arrive today!). So my Android-related projects on GitHub should be getting some much-needed attention. In fact I’ve already added a couple of new tasks to zutubi-android-ant (my extensions to the Android SDK Ant tooling): zaa:lint and zaa:ndkbuild.

The names of the tasks are pretty self-explanatory. The zaa:lint task runs Android Lint, which checks your source for common bugs and problems. If you don’t use lint, do yourself a favour and add it to your builds, it will save you a lot of grief! The zaa:ndkbuild task runs the Android NDK build command, which builds your native code.

Both tasks are simple extensions of the venerable exec task which make it simpler to run the underlying tools regardless of their install location and host operating system. This makes it easier to write machine-independent targets in your build files, which is great for teams working in diverse environments (and, of course, your continuous integration server). The tasks rely on the standard Android Ant properties (i.e. sdk.dir and ndk.dir) to locate the tools.

Because the tasks extend the built-in exec, they support all of the standard configuration. Adding arguments, tweaking the environment, and so on should all be familiar to Ant users. In future I may expand the tasks further to support some more tailored configuration, specific to the tools, but they are already fully-capable and very useful.

You’ll find more documentation in the README over at the zutubi-android-ant repository. Expect more tasks to follow, too!

Android: Attaching Sources to Libraries in Eclipse

Since ADT r17, Android projects in Eclipse have had a nice property: all jars in your libs directory have been automatically picked up by Eclipse projects under a magic “Android Dependencies” entry in the build path. Ant builds use the same conventions, so adding a library is as simple as dropping it into the right directory. Unfortunately, though, this magic came with a major limitation: there was no way to attach sources to these libraries (see issue 27940). This led me to duplicate build path entries manually just to get source attachments.

Good news: in the just-released r20 version of ADT, Google have provided a solution to this problem. It’s not well advertised, but if you check out comment 21 on the aforementioned issue, you see that by creating a properties file for each library, you can tell ADT where to find the sources. So, for example, if you have a jar named:

some-library-1.0.jar

you can create a properties file alongside it named:

some-library-1.0.jar.properties

Within the properties file you can add a src property set to the relative (or absolute, but that’s version-control-unfriendly) path of the source jar, zip or directory. Say, like me, you put it in the libs/src subdirectory, your properties file might look like:

src=src/some-library-1.0-sources.jar

Once you’ve added the properties file, refresh the project in Eclipse and voilĂ : the sources will be attached! A similar property named doc is supported for javadoc attachments.

Although this new feature is very welcome, I feel like it could use a bit more fine tuning. The main issue is the need to create a separate properties file for each jar. I keep my source jars in predictable locations, with conventional names, so I should be able to configure the convention once and have everything Just Work from then on. Heck, I’m not sure why there isn’t a default convention, which I could just follow with no further configuration!

Android JUnit XML Reports: Multiple File Support

Due to popular demand, I’ve added support for multiple output files (one per test suite) to android-junit-report.

For simplicity and efficiency, android-junit-report does not produce files in the exact format used by the Ant JUnit task. In the 1.1 release there are two main differences:

  1. A single report file is produced containing all test suites.
  2. Redundant information, such as the number of cases, failures etc is not added using attributes on the testsuite tag.

It turns out the first of these restrictions caused multiple users issues with tools accustomed to handling a single report file per suite. So in the latest 1.2 release I have added a new multiFile option. When this option is enabled, android-junit-report produces a separate output file for each test suite. This does mean that to retrieve the results from the device you will need to pull a whole directory.

To enable this option from an Ant build, you can override the default run-tests target as follows:

<target name="run-tests" depends="-install-tested-project, install"
       description="Runs tests from the package defined in test.package property">
    <property name="reports.dir" value="${out.dir}/reports"/>
    <property name="files.dir" value="/data/data/${tested.manifest.package}/files"/>
    <echo>Cleaning up previous test reports…</echo>
    <delete dir="${reports.dir}"/>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}" />
        <arg value="shell" />
        <arg value="rm" />
        <arg value="${files.dir}/*" />
    </exec>
    <echo>Running tests…</echo>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}"/>
        <arg value="shell" />
        <arg value="am" />
        <arg value="instrument" />
        <arg value="-w" />
        <arg value="-e" />
        <arg value="coverage" />
        <arg value="@{emma.enabled}" />
        <arg value="-e" />
        <arg value="multiFile" />
        <arg value="true" />
        <arg value="${manifest.package}/${test.runner}" />
    </exec>      
    <echo>Downloading XML test reports…</echo>
    <mkdir dir="${reports.dir}"/>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}"/>
        <arg value="pull" />
        <arg value="${files.dir}" />
        <arg value="${reports.dir}" />
    </exec>
</target>

You can learn more about android-junit-report and/or download the new release using the links below:

Android: Automated Build Versioning

The Android SDK documentation offers a guide for Versioning Your Applications. In short, applications have two version fields:

  1. versionCode: a simple integer that you should increase with each new release. Being an integer, the code can be processed in software (e.g. other applications) if required.
  2. versionName: a free-form string you can use as the customer-facing version for your application. A typical format would be dotted-decimal (e.g. 1.2.0), although any form is allowed.

This version scheme is simple and allows you the flexibility to choose your own naming scheme. One annoyance, however, is that the version is specified in the AndroidManifest.xml file:

<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
        package="com.zutubi.android.example"
        android:versionCode="1"
        android:versionName="1.0">
    <application android:name=".ExampleApplication">
        ...
    </application>
</manifest>

The problem here is that the manifest is a version controlled file that also contains other vital information about the application. This makes it inconvenient to automate stamping your application with a new version for each release.

This motivated me to create a couple of Ant tasks which make it easy to set the version of your Android applications from your Ant build. I’ve released them as part of an initial version of the zutubi-android-ant project. So far there are two tasks: setversion and bumpversion.

setversion

This task allows you to explicitly set the version code and/or name for your application to specified values. Typically these values would be specified indirectly via Ant properties. The properties would be set by some other process (e.g. automatic generation based on the date, an input prompt, a property from your build server, or an read from an unversioned properties file).

For example, to set the versions based on Ant properties, prompting if those properties are not yet defined:

<target name="set-version">
    <input addproperty="version.code" message="Version Code:"/>
    <input addproperty="version.name" message="Version Name:"/>
    <zaa:setversion code="${version.code}" name="${version.name}"/>
</target>

bumpversion

Perhaps you simply want to increment the version code and point number for your application each time you make a build. (You may also change the major.minor version at times, but that is a manual choice.) You can do this simply with the bumpversion task. By default, this task will just increment the existing version code. If you set the optional attribute bumpname to true, the last element of the version name will also be incremented. For example:

<target name="bump-version">
    <zaa:bumpversion bumpname="true"/>
</target>

If your existing manifest has a version code of 3 and a version name of 1.0.3, then after bumping you will have a code of 4 and a name of 1.0.4. Simple!

Try It Out

The zutubi-android-ant GitHub page has a full README and simple examples showing in more detail how you can use these tasks. Happy versioning!

Android JUnit XML Reports: Now With Test Durations

I’ve been planning to add test case durations to the XML reports generated by android-junit-report for some time. This morning, however, the magic of open source caught up with me. I received a pull request from another GitHub user who had implemented durations already!

So, with thanks to Tim from todoroo, there is a new release of android-junit-report which outputs the duration (in seconds) for each test case. This matches the output produced by the regular Ant JUnit report task, and thus should be compatible with all tools that read the format. Durations are not added to test suites, for the same reason the test case counts are not: it would require buffering. For my own use this is no big deal, because Pulse will sum the times of all cases to give an idea of the total suite time when it is not provided directly.

You can grab this new version of (1.1, build 4) from the GitHub downloads page. Or, as always, you can access the latest release build directly from the build server (click on “jar” in the “featured artifacts” table on the right of the page).

Android Testing: XML Reports for Continuous Integration

Summary

This post introduces the Android JUnit Report Test Runner, a custom instrumentation test runner for Android that produces XML test reports. Using this runner you can integrate your Android test results with tools that understand the Ant JUnit task XML format, e.g. the Pulse Continuous Integration Server.

The motivation and details of the runner are discussed below. For the impatient: simply head on over to the project home page on GitHub and check out the README.

Introduction

If you’ve been following my recent posts you’ll know that I’ve been figuring out the practical aspects of testing Android applications. And if you’ve been following for longer, you might know that my day job is development of the Pulse Continuous Integration Server. So it should come as no surprise that in my latest foray into the world of Android testing I sought to bring the two together :).

Status Quo

Out of the box, the Android SDK supports running functional tests on a device or emulator via instrumentation. Running within Eclipse, you get nice integrated feedback. Unfortunately, though, there are no real options for integrating with other tools such as continuous integration servers. Test output from the standard Ant builds is designed for human consumption, and lacks the level of detail I’d like to see in my build reports.

The Solution

On the upside, having access to the Android source makes it possible to examine how the current instrumentation works, and therefore how it can be customised. I found that the default InstrumentationTestRunner may be fairly easily extended to hook in extra test listeners. So I’ve implemented a custom JUnitReportTestRunner that does just that, with a listener that generates a test report in XML format. The format is designed to be largely compatible with the output of the Ant JUnit task’s XML formatter — the most widely supported format in the Java world. Tools like Pulse can read in this format to give rich test reporting.

How It Works

As mentioned, the JUnitReportTestRunner extends the default InstrumentationTestRunner, so it can act as a drop-in replacement. The custom runner acts identically to the default, with the added side-effect of producing an XML report.

For consistency with the SDK’s support for generating coverage reports, the XML report is generated in the file storage area of the target application. The default report location is something like:

/data/data/<tested application package>/files/junit-report.xml

on the device. To retrieve the report, you can use adb pull, typically as part of your scripted build.

Using the Runner

Full details on using the runner are provided in the README on the project home page. Briefly:

  • Add the android-junit-report-<version>.jar to the libraries for your test application.
  • Replace all occurrences of android.test.InstrumentationTestRunner with com.zutubi.android.junitreport.JUnitReportTestRunner:
    • In the android:name attribute of the instrumentation tag in you test application’s AndroidManifest.xml.
    • In the test.runner property in the Ant build for your test application (before calling the Android setup task).
    • In the Instrumentation runner field of all Android JUnit Run Configurations in your Eclipse project.
  • Add logic to your Ant build to run adb pull to retrieve the report after the tests are run.

As an example for retrieving the report in your Ant build:

<target name="fetch-test-report">
    <echo>Downloading XML test report...</echo>
    <mkdir dir="${reports.dir}"/>
    <exec executable="${adb}" failonerror="true">
        <arg line="${adb.device.arg}"/>
        <arg value="pull" />
        <arg value="/data/data/${tested.manifest.package}/files/junit-report.xml" />
        <arg value="${reports.dir}/junit-report.xml" />
    </exec>
</target>

In the Wild

You can see a complete example of this in action in my simple DroidScope Android application. The custom runner is applied in the droidscope-test application in the test/ subdirectory. You can even see the test results being picked up by Pulse on our demo server. Note that some of the tests are pure unit tests, which are run on a regular JVM, whereas others are run with the custom runner on an emulator. It’s nice for all the results to be collected together!