a little madness

Pulse 2.0 has a completely overhauled configuration UI. The current 1.2 UI uses standard web forms, whereas the new UI is ExtJS-based and makes heavy use of AJAX. This makes for a huge improvement in usability, unfortunately at the cost of making all of our existing acceptance tests for the UI redundant! So, we needed to go back to square one with our acceptance test suite. It was a painful process, but going through it again reminded me of something important:

The setup cost of your first few acceptance tests is high, but after that it is all gravy.

The process from zero to gravy is similar for most projects, so I’ve summarised the phases we experienced below.

Phase 1: Choosing the Technology

The first step was to switch out jWebUnit for Selenium. Although jWebUnit can execute much of the Javascript in our new UI, it falls down in two key ways:

1. Executing much or even most of the Javascript is not good enough – it makes writing important tests impossible.
2. It does nothing to test real-world differences in the various Javascript engines that are the source of many bugs.

As Selenium drives the actual browser, you can test anything that will run in the browser and can also test compatibility. The main (well known) drawback is that Selenium is slow. As important as speed can be, however, accuracy is far more important.

Phase 2: Scripting Setup and Teardown

This is where the slog started. The most accurate way to acceptance test is to start with your actual release artifact (installer, tarball, WAR file, whatever). In our case, this is the Pulse package, which comes in various forms. Before we can actually test the UI, we need to get the Pulse server unpacked, started and setup. When the tests are complete, we also need scripts to stop the server and clean up. This way we can integrate the tests into our build and, of course, run the acceptance tests using our own Pulse installation!

So, a day of scripting later, and we don’t have a single test. Sigh.

Phase 3: The First Test

Now things got serious: writing the first test was painfully hard. I constantly hit snags:

• How do I use this newfangled Selenium thingy?
• How do I test a UI that is asynchronous?
• How can I verify the state of ExtJS forms?

The important thing here was to keep pushing. The amount of effort to get to the first test case is not worth it for that test alone, but I knew it was an investment in knowlege that would help us develop further tests.

Phase 4: Abstraction

As I continued to write the first few test cases, repetition crept in. Verifying the state of a form is similar for all forms. Many tests navigate over the same pages, examining the state in similar ways. The code was ripe for refactoring! I started abstracting away the actual clicks, keystrokes and inspections and building up a model of the application. The classic way to do this is to represent pages and forms as individual classes that provie high-level interfaces. Over time, our tests changed from something like:

to:

and then to:

Phase 5: Gravy

With the right abstractions in place, adding acceptance tests is now much easier. The tests themselves are also easier to understand and maintain due to their declarative nature. Now we can reap the rewards of those painful early stages!

Conclusion

The process of setting up an acceptance test suite is quite daunting, and initially painful. But if you persevere and constantly look for useful abstractions, you’ll reap the rewards in the long run.

Liked this post? Share it!

This entry was posted on Friday, February 29th, 2008 at 3:42 am and is filed under Agile, Continuous Integration, Technology, Testing. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.