a little madness

We all know the Java promise, Write Once Run Anywhere. And we all know some things are too good to be true. On the one hand, there are those features that don’t quite work the same on different platforms. That’s just a fact of life in portable programming, which I can deal with. On the other hand, there is the lowest common denominator syndrome. Important features that are only available on some platforms are non-existent in the Java APIs. The excuse for the absence of such APIs is, of course, that they can’t be implemented on all platforms. This I can’t stand¹. I don’t expect every platform-specific API to be exposed, but some are just too important to ignore. Unfortunately for Java programmers, Sun has managed to ignore several for quite some time, for example:

• Unix style file permissions: applications that work with Files (yes, there are a few of them) often need access to the read/write/execute permissions. I have seen (and written) plenty of ugly workarounds to use the stat/chmod commands to achieve the desired result. A request for this functionality can be found in the bug database from as early as 1997. Sure, the permission systems are different on other platforms (notably Windows), but you can cover a huge percentage of use cases with a small amount of work. Finally, in Mustang, we have the ability to get and set the permissions.
• Free disk space: those applications working with the file system may also want to make sure they don’t fill it. It is a lot easier to prevent disk exhaustion than to recover from it. Looking back, poor Java developers have been hoping for this ability since 1997. Finally, Mustang delivers.
• Masked command line input: the primitive Java APIs for command line interaction make it near impossible to create a usable command line interface. An example of something simple which you cannot do without crazy workarounds is prompt for masked input, e.g. to get a password. yep, you guessed it, there is a request from way back in 1997. Finally, in Mustang, we have the new java.io.Console class that provides a readPassword method.
• System tray icons: many applications that wish to integrate with popular desktops on both Windows and Unix need the ability to use the system tray. It’s an expected part of the user’s desktop experience. The earliest request I found for this one is from 2000. The nice part is the Mustang support grew from an external project, so in this case the users were a bit more empowered.

It’s quite sad that Java developers have had to wait a decade for such commonly-required and relatively simple APIs. No wonder Java has had trouble getting a foothold on the desktop when it has been so difficult to make an application that integrates well with the underlying platform. I just hope the arrival of this new functionality in Mustang is a sign of a new attitude towards such features. No more hiding behind the platform independence argument, time to just get on and give people the tools they need to get the job done.

–
1: It’s worth noting that this sort of attitude is not restricted to the Java community. I can’t count the number of times that I have seen people in the C++ community reject ideas for additions to the standard library because “C++ runs on platforms that don’t even have a file system”. Who cares when 90% of C++ code runs on systems that could use this functionality!

–
Interested in automated builds? Pulse 1.2 is now in Early Access Release. Check it out!

Phew, it’s been a busy time, but finally we have the first milestone build of Pulse 1.2 ready to go! The headline feature for this release is the ability to run personal builds. A personal build takes your local changes and applies them to a pulse™ build without them being submitted to your SCM first. This allows you to test your changes before submitting them to version control.

Other major features in this release include:

• Reports: each pulse™ project now has its own “reports” page, which displays build data for the project visually. Currently, the reports show trends over time for:
  • Build results
  • Tests run per build
  • Build time
  • Stage execution time
• Windows System Tray Notification: a new Pulse client, Stethoscope, sits in your system tray allowing you to see your project health at a glance. You can configure Stethoscope to monitor both personal builds and project builds for your selected projects. If you like, Stethoscope will pop up a message whenever a build completes.
• Customisable Notifications: don’t like the format of your notification emails or instant messages? In pulse™ 1.2, the notification templates can be customised using FreeMarker.
• Automatic Agent Upgrades: we go to great effort to make pulse™ easy to install, upgrade and maintain. That is why in pulse™ 1.2 we have made the upgrade process even simpler by adding automatic upgrades for agent machines. Now, after you upgrade your main pulse™ server, your agents will be automatically upgraded for you!
• Resource Configuration Wizard: on the same theme of keeping things simple, we have also added a new resource configuration wizard. This wizard makes it easy for you to configure common build dependencies, such as Java Development Kits and build tools (ant, make, etc). We have also improved the resource auto-discovery code to detect resource versions for you: in many cases you won’t even need the wizard!
• Anonymous Signup: you can now optionally allow users to sign up to pulse™ themselves. This lessens the burden on the pulse™ administrator by removing the need for them to create accounts. It is also perfect for public-facing servers (e.g. open source projects) where interested parties can sign up for read-only access but with their own dashboard and preferences.

Grab a milestone build now from our Early Access Program page and try it out!

I would like to draw your attention to the recent release of the new Java Persistence API for the Berkeley DB. In short, the Berkeley DB is designed to be a very efficient embedded database. With no SQL query support, you instead talk directly to its internal BTree via a very simple API. A very good write up is available at the server side for those interested in the full details.

This style of persistence mechanism is certainly not for everyone. If you want adhoc query support for writing reports and extracting data, look elsewhere.

If you don’t, then Berkeley should be considered when defining the persistence architecture of your next project, just don’t tell your DBAs. Not only is it reportedly very fast, due largely to the lack of SQL style interface, but it also supports transactions, hot backups and hot failover, all of the things that help you sleep at night. However, what has me intrigued is idea of not having to deal with SQL schema migration.

I consider Schema migration to be one of the more tedious and yet non-trivial tasks that is required by any application that employs relational persistence. Yes, Hibernate makes this task somewhat easier to deal with. However, even with Hibernate, you will still need to roll up your sleeves and write some SQL to handle the migration of the data.

Managing schema migration within the Berkeley DB is different. Where as previously you extracted the data via SQL, converted it and then updated the DB via SQL, with Berkeley, you just convert the data in Plain Old Java. They have some examples in there javadoc that gives a reasonable idea of what is involved. Below is one of these examples, a case where the Person object’s address field is split out into a new Address object with 4 fields. The core of the work is done by the convert method:

Personally, I think this is a great improvement. Now, if only I had been aware of this at the start of this project, things might be a little different, and faster, and some other good stuff as well.

So are schema upgrades a thing of the past? Maybe not, but they don’t have to be a part of every project.

Well, it may not be a new idea, but it’s still a fun one (and still useful!). Todd reports a colleague hooking up Pulse notifications to a pair of lava lamps. Full write up is here. Maybe we should make a boxed set? :).

People regularly write about good and bad tech interview questions. Certain companies are famous for their interviews, there are books and websites full of such questions, and there are countless blogs on the topic. Still, many of the questions I have heard asked and seen quoted in blogs are just plain ordinary. Heck, even questions I have used myself look worse to me as time goes on.

Certain common question types are useful, such as:

• Simple coding tasks: as a quick screen. If the candidate can’t reverse a list, it’s an immediate no.
• Problem-solving puzzles: made famous by Microsoft. A good puzzle or two can be a fun way to find out how the candidate approaches a problem.

However, the quick screen doesn’t do the hard job of separating decent coders and great coders. And it can be hard to draw conclusions from a puzzle question: is the candidate just a fan of puzzles (i.e. clued in to the tricks)? I think part of the problem is we need to look at the issue a slightly different way. These questions are trying to emulate the challenges of the job, but they fall short in two key ways:

1. To fit the task into a 10 minute question, it is scaled down to a toy size. Nice for an exercise, but not very realistic.
2. The interviewers themselves know the answer, giving them a position of power over the candidate. They may be biased towards their known answer (despite a unique alternative from the candidate). They could either lead or intimidate the candidate from this position.

So what kind of question can be used to overcome these shortfalls? Simple: a question with no answer. Or, at least, a question that you don’t yet have an answer to. Every day on the job we run across these problems, and have to solve them. It’s the essence of the job. And we solve them together, not as individuals, and certainly not with an all-knowing interviewer smugly sitting on the perfect solution.

Sure, this sort of question is not nice and clean to ask. As the interviewer, you have to be willing to be as out of depth as the candidate. The benefit is you are now working on a level playing field together with the candidate, just as you would if they were hired. This both adds realism and allows you to assess their ability to fit in with the team. You can brainstorm solutions, debate alternatives, hit dead ends and make breakthroughs together. You may have no answer, you may come across many, it’s not really important. Tomorrow always brings a new problem.

So how do you come up with these questions? One possibility is harvesting them from real life examples. Is there a problem you are struggling with today? If it makes sense in isolation, it could be a perfect candidate. Is there a design issue you run across every now and again, and never seem to have a good answer? Note it down. If you’re really stuck, try what we did a couple of times at my last company. Pick a random question out of a programming competition archive: one that you have never seen before. Take it into the interview blind, don’t have a head start on the candidate. Some realism is lost, but many of the benefits remain.

Happy interviewing!

Over the past couple of weeks, I have been working on adding plugin support to pulse. One of the primary requirements is to provide as much functionality out of the box as possible, allowing plugin writers to focus on the features, not support code. For example, if a plugin requires configuration, then the plugin developer should define what input is required, what validation rules should be applied, but not have to worry about the details of rendering and validating a web form. But more on that later.

Whilst doing some research into what was already being done, I came across an approach to using annotations that I have since found very useful. For reference, the project was the excellent trails project from Brian Topping.

The approach centres around defining the annotation, and annotating the annotation with a reference to its handler.

In the code example above, you can see that the Required annotation has a reference to the RequiredValidator, its handler.

By adding a simple annotation processor that searches for annotated annotations and executes the handler, you have the basis of a very flexible meta data processing facility. The core of the flexibility here is that a custom annotation does not need to be registered with the processor. The information normally conveyed by the registration process is embedded in the annotation.

Lets see how this works in practice by applying it to the validation domain by implementing custom Name validation rules.

Firstly, the validator:

Then the annotation:

And finally, we apply it to our business object:

The core of the annotation processor would look like:

We are also applying this same technique to the plugin frameworks’ automatic form generation, and anywhere else where we want plugin authors to be able to use annotations to customise the default behaviour of the framework.

Recently I’ve been cooking up some new acceptance-level tests for Pulse. The sort of tests that need to take a Pulse release package, unpack it, start the server and poke it with a stick. This is the sort of automation that I would typically use bash for: since a lot of the work involves executing and managing subprocess. However, I had learned from earlier testing that process control from bash is not very portable. More specifically, the behaviour of Cygwin bash on Windows differs considerably from *nix bash. At the time, I worked around the problem by doing some process control using Ant, but there had to be a better way.

Enter Python. Or, more specifically, the new(ish) subprocess module (introduced in Python 2.4). With this module Python finally has a concise but powerful way to launch and control subprocesses. It also includes convenient ways to handle subprocess I/O, including setting up pipelines. What’s more, I have so far found it to be quite portable between Linux and Windows.

Starting with something simple, let’s say you just want to launch a command synchronously (much as you would in a shell script):

Simple. Want to set up a pipeline like `dmesg | grep hda` (capturing the final output)? Here’s an example straight from the doco:

For my specific case, I used the ability to launch a Pulse server processes in the background and capture the output extensively. When done, I shut down the server using the standard shutdown script (testing of said script being a nice side-effect). An asynchronous child is started simply by creating a Popen object:

I have simplified this example a bit, but the real work is all there, and it’s pleasantly straightforward. This combined well with Python’s simple XML-RPC API, which I used to manipulate the running server.

So far the switch to from bash/Ant to Python has been a great success. If you need a cross-platform way to automate your acceptance testing, give it a go.

I like to keep an eye on various build and testing tools, for potential integration with Pulse. As such, I’ve started to amass some links to unit testing tools/resources for C++, where there are many competing options:

• Exploring the C++ Unit Testing Framework Jungle: the most comprehensive article I’ve found on the topic, even though it is getting old.
• CppUnit: probably the best know xUnit port, supported in Pulse already. Does a decent job.
• Boost.Test: part of the well known set of Boost libraries. Naturally has dependencies on Boost.
• CppUnitLite: a minimalistic rewrite of CppUnit, intended to be simpler and more portable. May be appropriate if you don’t mind getting your hands dirty to add the features you need.
• Nano Cpp Unit: more an exercise in illustrating the barest testing framework than a usable framework itself.
• Unit++: pitched as a more “C++ like” xUnit port. Documentation is thin on the ground, and I don’t have any practical experience with it.
• CxxTest: takes the novel approach of using Perl to generate a test runner, simplifying the code. Also relatively portable, although of course you will need Perl.
• TUT: a simple, template-based framework distributed as a single header file. Reasonable portablility (given a modern compiler). Lacking some features of other frameworks, but without any dependencies.
• cutee: another framework aimed at simplicity. Looks to have test case creation down to its simplest form. Documentation is thin.
• CppTest: you guessed it: another framework aimed to be simple and portable. Supports a few output formats out of the box, including HTML reports.
• UnitTest++: co-authored by the author of the article above, this framework hopes to combine the best ideas from others. Documentation is non-existant, but on the plus side it is one of the more modern frameworks (developed this year, not 2004!).
• QtUnit: probably a good option if you were using Qt, but is officially unmaintained. Mind you, most of the other frameworks are also dormant.

That’s all I have gathered so far. I have to say, there are a lot of players out there, but little action. A few interesting ideas, but no framework seems to be a clear leader. I hope to get some time to play in depth a bit more, in which case I will flesh out more details.

We have many components in our product that rely on external packages being available before tests will pass. For example, our make integration requires that make is installed, and our cvs integration tests require the cvs test server to be reachable.

The problem is that some of these resources will not always available on the box running the tests. This is often the case when we are running a build on a new box / environment for the first time. What would be very nice is if the test framework could warn me if a resource is not available, and only run the tests that have a chance of succeeding, ie, those where the required resources were available.

JUnit

Since we use JUnit for testing, I had a look to see what I could do to solve this issue.

The simplest solution is to Assert.fail during the setup/before method if the resource is not available:

This results in a:

for each test that requires the unavailable resource. This is certainly effective, but a little crude, resulting in a lot of meaningless strack traces when all I need to know is that X is not available.

An alternative, in JUnit3 at least, would be to implement a custom TestDecorator that would only execute the tests it wraps if the required resources are available.

This has the advantage of avoiding the stack traces, and allows us to control the number of error messages printed. This is certainly not as noisy as the first approach, but now we need to wrap all of our tests in this decorator, and does not seem to work well with the JUnit 4 suites.

TestNG

I have heard some good things about TestNG lately, so how does it handle this problem?

TestNG supports the concept of test dependencies. That is, if testB depends on testA, then testB will not run until testA has run successfully.

So, what we can do is define a method to check for a resource, and make tests that require that resource dependent on that method.

The result is that if the resource is not available, you get a single failure with details “resource X not found”, with tests that depend on that resource will be marked as skipped. That is pretty clean outcome. We get no duplicate failures and a short error message:

Short and definately Sweet.

Conclusion

Given these results, I certainly prefer the TestNG solution. The output is clear and the association of dependencies to tests is defined with each test case and on a per test basis, not on the per class basis as with JUnit. Now if only I was using TestNG :). Maybe it is time to run the JUnitToTestNGConverter?

I am curious to know if other people run into this issue? And if so, how do you solve it? Maybe you do one of the above, or have written custom annotations or extended JUnit. Please let me know, I would love to hear about it.

I have been inspired by recent discussions on upgrade frameworks to show how hibernate can be used to provide simple incremental database schema maintenance. Database schema maintenance is one of the more difficult aspects of upgrading applications, particularly when the application supports multiple databases, so I am very happy that hibernate helps out during upgrades.

SchemaUpdate

Hibernate provides a class called SchemaUpdate that is able to synchronise a set of hibernate mappings with a database schema. The following code snippet shows how easy it is:

This example uses the spring ClassPathResource to load the mappings file from the classpath, and the UpgradeTaskConnectionProvider to inject a datasource into the process.

.hbm.xml fragments

This by itself is not overly interesting. What people usually do not realise is that the mappings files do not need to hold your entire schema. When making incremental changes to your schema, all you need in the mappings are those incremental changes. This comes in very handy when you have lots of mappings to manage.

For example. You have the following mapping of a user:

Some time later, you want to store a password field with this user. By passing the following mapping to the SchemaUpdate, it will add that column to your existing table, leaving the existing schema as it is.

You still need to ensure that the mapping file is valid, hence the inclusion of the ID field in the second mapping.

Versioning

So, to support incremental schema upgrades within your application, you will need to keep two sets of hibernate mapping files. The first will be the latest version of your mappings. This is what is used for new installations. The second will be a set of versioned mapping fragments as described above.

You will need to version them so that you can track which fragments you need to apply and in which order, based on the version of the schema you are upgrading from. I use directory names like build_010101 to store my schema fragments and a properties file to store the current schema version. Other people use a special table in the database to hold the current schema version. Use which ever is most appropriate to your situation.

Generating upgrade SQL

For those of you that do not want or can not allow Hibernate to run the schema update, you can use the following code to generate the SQL that Hibernate would otherwise execute:

This code would replace the last line in the first example.

Things to remember about SchemaUpdate

Okay, so just a couple of final things to be aware of with hibernates schema update.

The hibernate schema update will:

• create a new table
• add a new column

The hibernate schema update will not:

• drop a table
• drop a column
• change a constraint on a column
• add a column with a not-null constraint to an existing table

Final tip

Oh, and the class name that you provide in the update mapping can be anything you want. It is not checked, which is great, otherwise you would need to handle versioning of your class files as well.

Happy upgrading!