a little madness

A man needs a little madness, or else he never dares cut the rope and be free -Nikos Kazantzakis

Zutubi

Archive for the ‘Project Automation’ Category

Obscure Gradle Error: RSA premaster secret error

Just a quick post in case anyone else runs into the same obscure scenario. Setting up a new Gradle project on my OSX dev machine, the build could not download any files from Maven Central. When trying to establish an SSH connection I was getting:

RSA premaster secret error

A web search didn’t turn up much, making it clear this was not a common issue. The only hits I found where outdated or unusual configurations, whereas I believed I had a pretty vanilla setup.

Long story short, the problem was I had globally set the java.ext.dirs system property to the empty string to prevent another, unrelated (and equally obscure) error in the past. That was too blunt an approach — the JVM at the very least needs $JAVA_HOME/jre/lib/ext to be included in java.ext.dirs to load some core jars, including sunjce_provider.jar which includes implementations of the encryption algorithms required to establish SSL connections. User error on my part, which I paid for with wasted time — I hope this post saves someone from the same mistake!

Pulse 3.0 Alpha

With great pleasure I can announce the very first build of Pulse 3.0 is now available from our Alpha Program page! Although this build is very much incomplete and unstable, this is a huge milestone after months of work in both the back and front ends of Pulse.

As this is a larger release we’ve made the choice to release earlier in the cycle than we otherwise would, for a couple of reasons:

  • To show the massive progress that has already been achieved.
  • To solicit feedback about the all-new administration UI (in time to apply said feedback before feature freeze).

We know from previous feedback that the new UI will address some pain points, mainly around quickly navigating and understanding your configuration in a large setup. We’ve also taken advantage of modern browser features to fix clunky corners: e.g. proper history integration and drag-and-drop reordering (no more clickety-click!). With such major changes, though, we’re always keen to hear what you think — good or bad — so we can keep on the right track.

So please do find the time to download and play with an alpha build, then get in touch with us via support email. We’ll be iterating fast on this release stream now, so expect to see updates about regular builds with new UI features each time. Happy building!

Android: JUnit XML Reports with Gradle

The Android development tools project has seen big changes over the last year. The original Eclipse ADT development environment was superseded late last year by Android Studio — a new IDE based on Intellij. Under the hood Android Studio also uses a new command line build system based on Gradle, replacing the previous Ant-based system. I’ve been keen to find out how these changes impact the integration of Android test reports with continuous integration servers like Pulse.

Summary

  • Android JUnit Report is redundant.
  • Run on-device Android tests with:
    ./gradlew connectedAndroidTest
  • Collect reports from:
    app/build/outputs/androidTest-results/connected/*.xml

 

Details

The original Ant-based build system for Android didn’t produce XML test reports for instrumentation tests (i.e. those that run on-device), prompting me to create the Android JUnit Report project. Android JUnit Report produced XML output similar to the Ant JUnit task, making it compatible with most continuous integration servers. The good news is: Android JUnit Report is now redundant. The new Gradle-based build system produces sane XML test reports out of the box. In fact, they’re even more complete than those produced by Android JUnit Report, so should work with even more continuous integration servers.

The only downside is the documentation, which is a little confusing (while there are still documents for the old system about) and not very detailed. With a bit of experimentation and poking around I found how to run on-device (or emulator) tests and where the XML reports were stored. With a default project layout as created by Android Studio:

ASDemo.iml
app/
  app.iml
  build.gradle
  libs/
  proguard-rules.pro
  src/
    androidTest/
    main/
build.gradle
gradle
gradle.properties
gradlew
gradlew.bat
local.properties
settings.gradle

You get a built-in version of Gradle to use for building your project, launched via gradlew. To see available tasks, run:

$ ./gradlew tasks

(This will download a bunch of dependencies when first run.) Amongst plenty of output, take a look at the Verification Tasks section:

Verification tasks
------------------
check - Runs all checks.
connectedAndroidTest - Installs and runs the tests for Debug build on connected devices.
connectedCheck - Runs all device checks on currently connected devices.
deviceCheck - Runs all device checks using Device Providers and Test Servers.
lint - Runs lint on all variants.
lintDebug - Runs lint on the Debug build.
lintRelease - Runs lint on the Release build.
test - Run all unit tests.
testDebug - Run unit tests for the Debug build.
testRelease - Run unit tests for the Release build.

The main testing target test does not run on-device tests, only unit tests that run locally. For on-device tests you use the connectedAndroidTest task. Try it:

$ ./gradlew connectedAndroidTest
...
:app:compileDebugAndroidTestJava
:app:preDexDebugAndroidTest
:app:dexDebugAndroidTest
:app:processDebugAndroidTestJavaRes UP-TO-DATE
:app:packageDebugAndroidTest
:app:assembleDebugAndroidTest
:app:connectedAndroidTest
:app:connectedCheck

BUILD SUCCESSFUL

Total time: 33.372 secs

It’s not obvious, but this produces compatible XML reports under:

app/build/outputs/androidTest-results/connected

with names based on the application module and device. In your continuous integration setup you can just collect all *.xml files in this directory for reporting.

Although the new build system has killed the need for my little Android JUnit Report project, this is a welcome development. Now all Android developers get better test reporting without an external dependency. Perhaps it will even encourage a few more people to use continuous integration servers like Pulse to keep close tabs on their tests!

Pulse 2.7 Released

I’m dusting off the blog with a bang, announcing that Pulse 2.7 has gone gold! This release brings a broad range of new features and improvements, including:

  • New agent storage management options, including the ability to prevent builds when disk space is low.
  • Configuration system performance improvements.
  • Live logging performance improvements.
  • Xcode command updates, including a new clang output processor.
  • A new plugin for integration of XCTest reports.
  • More flexibility and feedback for manual triggering.
  • New service support, including integration with systemd and upstart.
  • Improved support for git 2.x, especially partial clones.
  • Support for Subversion 1.8.
  • Improved dependency revision handling across multiple SCMs.
  • More convenient actions for cancelling builds.
  • The ability to run post build hooks on agents.

As always we based these improvements on feedback from our customers, and we thank all those that took the time to let us know their priorities.

Pulse 2.7 packages can de downloaded from the downloads page. If you’re an existing customer with an active support contract then this is a free upgrade. If you’re new to Pulse, we also provide free licenses for evaluation, open source projects and small teams!

Android JUnit Report Updates

Recently I’ve made a few updates to my custom Android test runner, android-junit-report. This runner makes it easy to integrate your Android test results into a continuous integration process by producing de-facto standard JUnit-style XML reports. The latest changes bring the runner up to speed with the latest version of Android and include better documentation. A summary of the updates follows:

  • A new home page for the project.
  • Within the home page, full documentation.
  • Updates to where the reports are written by default. The runner no longer attempts to use the internal storage of the test application, instead it always defaults to the internal storage of the main application (i.e. the one under test).
  • Support for a new __external__ token that allows you to place the reports in the external storage area of the application under test (given availability and permission to do so).
  • Changes to the token syntax to avoid the need to escape when calling the runner via a shell.

I’d like to acknowledge the help of Sebastian Schuberth and Christopher Orr in fine-tuning some aspects of the runner for this release. The latest runner version, 1.5.8, is available for download now.

Android: Easier Source Linking in Eclipse

In my prior post Android: Attaching Sources to Libraries in Eclipse I noted that ADT version r20 added the ability to link sources to libraries in the Android Dependencies classpath container in Eclipse. Although this was a welcome addition, the method used to link sources is a bit cumbersome: you must create a .properties file alongside each library, and add a property that points to the source jar.

I felt this could be a lot simpler if you are willing to adopt a convention for the location of source jars. Once the convention is established, adding linked sources is as simple as placing the source jar in the expected location. This makes the process as seamless as the Android Dependency container is for adding libraries in the first place.

So I’ve made this possible via a new libproperties task in my zutubi-android-ant project. This task inspects the jars in your libs/ directory, looks for corresponding source jars by naming convention, and adds appropriate .properties files when sources are found. If you’re willing to accept the default convention, where source a source jar for library libs/foo.jar is placed at libs/src/foo-source.jar, then this simple Ant target can be used to generate the required .properties files for you:

<target name="link-sources">
    <zaa:libproperties/>
</target>

Of course you’ll need to import the zutubi-android-ant tasks into your build file first. If you want to ensure sources are present for all libraries, set the attribute failOnMissingSrc to true and your build will fail if any source jar is missing. You can even establish your own conventions, using a regular expression to match the libraries and a replacement string to generate paths to source jars:

<target name="link-sources">
    <-- libs/alib-1.3.jar maps to libs/src/alib-src-1.3.jar -->
    <zaa:libproperties failOnMissingSrc="true"
                         jarPattern="(.+)-([0-9.]+)\.jar"
                         srcReplacement="src/$1-src-$2.jar"/>
</target>

Head over to the new zutubi-android-ant home page for more details, including quick-start instructions to get the tasks installed in your build in minutes.

New Android Ant Tasks: lint and ndkbuild

I’m back into Android development these days, and expect plenty more to come (especially with a Nexus 7 due to arrive today!). So my Android-related projects on GitHub should be getting some much-needed attention. In fact I’ve already added a couple of new tasks to zutubi-android-ant (my extensions to the Android SDK Ant tooling): zaa:lint and zaa:ndkbuild.

The names of the tasks are pretty self-explanatory. The zaa:lint task runs Android Lint, which checks your source for common bugs and problems. If you don’t use lint, do yourself a favour and add it to your builds, it will save you a lot of grief! The zaa:ndkbuild task runs the Android NDK build command, which builds your native code.

Both tasks are simple extensions of the venerable exec task which make it simpler to run the underlying tools regardless of their install location and host operating system. This makes it easier to write machine-independent targets in your build files, which is great for teams working in diverse environments (and, of course, your continuous integration server). The tasks rely on the standard Android Ant properties (i.e. sdk.dir and ndk.dir) to locate the tools.

Because the tasks extend the built-in exec, they support all of the standard configuration. Adding arguments, tweaking the environment, and so on should all be familiar to Ant users. In future I may expand the tasks further to support some more tailored configuration, specific to the tools, but they are already fully-capable and very useful.

You’ll find more documentation in the README over at the zutubi-android-ant repository. Expect more tasks to follow, too!

Pulse Continuous Integration Server 2.5 Released!

After a smooth beta period, we’re excited to announce the release of Pulse 2.5! Massive credit for the new feature and improvement ideas in this release go to the Pulse community, whose feedback was invaluable. Here’s the brief list of what’s new:

  • Browse and dashboard view filtering (by project health).
  • A new MSTest post-processor plugin.
  • Increased visibility of build (and other) comments.
  • Comments on agents, to communicate with other users.
  • Properties on agents, for simpler parameterisation.
  • The ability to reference properties in SCM configuration.
  • The ability to configure SCMs using resources.
  • A resource repository on the Pulse master.
  • A new option to attach build logs to notification emails.
  • Upstream change reporting (changes via dependencies).
  • Downstream build notifications (again via dependencies).
  • Simpler delivery of directory trees via dependencies.
  • Pre-stage hooks, which run just before the stage is dispatched to the agent.
  • Stage terminate hooks, allowing you to run custom cleanup.
  • SCM inclusion filters, complementing existing exclusion filters.
  • Inverse resource requirements (i.e. requiring the absence of a resource).
  • Smart label renaming.
  • A distinct warnings status for UI and API consistency.
  • A separate permission for the ability to clean up build directories.
  • Support for git submodules.
  • Support for Subversion 1.7 in personal builds.
  • Support for Perforce streams.

For more details, see the new in 2.5 page on our website. Or, why not just download Pulse and try it free today?

Bash Tip: Reliable Clean Up With Trap

Wow, it has been over 6 years since my last bash tip. Bash scripts are still my go-to for automating the mundane and repetitive — it might not be beautiful but bash is feature rich and installed everywhere. Although a lot of my scripts are throw-away, others make it into the permanent tool kit. For scripts in the latter category, it’s worth the extra bit of effort to make them more robust.

A key part of a robust script is detecting and sensibly reporting errors. Related to this is proper clean up of work in progress when the script encounters a fatal error. Such clean up is often overlooked, as it is difficult to get right in a naive fashion. Thankfully, bash has a built-in mechanism that makes it simple: trap.

Setting a trap instructs bash to run a command when the shell process receives a specified signal. Further, bash defines pseudo-signals ERR and EXIT that can be used to trap any error or exit of the shell. I usually pair an exit trap with a clean up function to ensure my scripts don’t leave a mess. If the script runs successfully, I reset the trap (by specifying – as the command to run) and call the clean up function directly (if required). For example:

#! /usr/bin/env bash

set -e

# Defines a working area on the file system.
SCRATCH=/tmp/$$.scratch

function cleanUp() {
    if [[ -d "$SCRATCH" ]]
    then
        rm -r "$SCRATCH"
    fi
}

trap cleanUp EXIT
mkdir "$SCRATCH"

# Actual work here, all temp files created under $SCRATCH.

# We succeeded, reset trap and clean up normally.
trap - EXIT
cleanUp
exit 0

Note that the EXIT signal pairs well with set -e: if the script exits on error your EXIT trap is executed. You may also consider applying the trap to interrupts (signal INT) and kills (signal TERM) of your script. Multiple signals can be specified in one trap statement:

trap cleanUp EXIT INT TERM

If you’re used to programming with exceptions, you can think of a trap as a “finally” clause for your whole script. It’s certainly a whole lot easier and less error-prone that trying to figure out all the ways the script might exit and adding calls to clean up manually.

—-
Want Continuous Integration without the Needless Duplication? Try pulse.

Pulse Continuous Integration Server 2.5 Alpha!

Cobwebs may be creeping over the blog, but not over Pulse: we’ve just published the first Pulse 2.5 alpha build! The Pulse 2.5 series is focused (even more so than usual) on customer feedback. We’ve got plenty of great ideas from the Pulse community, and we’re folding in some of the best ones in this release series. Updates so far include:

  • Browse and dashboard view filtering (by project health).
  • A new MSTest post-processor plugin.
  • Increased visibility of build (and other) comments.
  • Comments on agents, to communicate with other users.
  • Properties on agents, for simpler parameterisation.
  • The ability to configure SCMs using resources.
  • A resource repository on the Pulse master.
  • A new option to attach build logs to notification emails.
  • Pre-stage hooks, which run just before the stage is dispatched to the agent.
  • SCM inclusion filters, complementing existing exclusion filters.
  • Inverse resource requirements (i.e. requiring the absence of a resource).
  • Smart label renaming.
  • A separate permission for the ability to clean up build directories.
  • Support for Perforce streams.

There’s plenty more to come in subsequent alpha builds, too. In fact, we’ve started the groundwork for some other larger changes already in 2.5.0.

To grab yourself a fresh-from-the-oven 2.5 alpha build, or just to find out more, head over to our alpha program page.