Pulse Continuous Integration Server 1.0.4

May 25th, 2006

We are happy to announce version 1.0.4 of pulse. Highlights in this release include:

  • Support for CppUnit
  • More flexible notification conditions: you can now write arbitrary boolean expressions to configure when you get notified!
  • The ability to clone projects: simplifying setup for multiple similar projects.
  • Improvements to the test results view for large test suites: expand/collapse suites, show/hide successful tests.

Full release notes are available. Thanks to the beta testers!

The internet, best viewed in 1280 x 1024

May 24th, 2006

Every time that I have upgraded my monitor, I have marvelled at how much more screen realestate is available. Each time I have gone through a period of adjustment where I adjusted the layouts of my applications to take maximum advantage of the extra space. My latest move from a 19inch to one of the nice Dell 24inch screens has allowed me to move to a three column layout in IDEA with the project view on the left, the structure on the right, and still plenty of room for 120 char lines in the editor. Awesome. Moving on to Thunderbird has yielded a similarly pleasing outcome with its nice three column vertical layout. Nice. Even JEdit supports a configurable layout that allows me to make good use of the screen.

Unfortunately my amusement came to an end when I opened up my browser. Either one, it matters not. Where ever I go, whether it’s my favourite news or community website, any one of the blogs I try to keep up with, I end up with one of two things. Either a fixed width page with half my screen empty on either side (but all of the content nicely squeezed into the middle), or a 100% width page where the lines are so wide that they become difficult to read and large patches of whitespace spread around the page.

Now I understand that large screens are only just becoming affordable, and so I do not expect websites to suddenly rearrange their content to suit me and my dislike of unused pixels. It does however bring to attention one aspect of building web applications that can be very frustrating. The page layout. Now certainly some people are doing nice things with the tools that are available, namely css, but that can only go so far. What I would really like to see now is a good old fashioned trustworthy layout manager.

Only just last week I was trying to achieve a relatively simple task. Opening a popup window that is an appropriate size for the content it will display. Well, I had hoped that it would be a relatively simple task. After loading the content into an iframe, doing my best to work out what its visible width and height would be and then dropping it into a popup, I am still no closer to a solution that works in both Firefox and IE. With a slightly better API for handling page layouts, this type of task would become dead simple. Speaking of layout managers, those guys over at qooxdoo are doing some pretty interesting things. Now, if only it didn’t look so much like windows.

Bash Tip: Exit on Error

May 24th, 2006

Back in my post Your Next Programming Language I mentioned I would post occassional tips about bash scripting. As soon as I started writing my next script, it occured to me: the first thing I always do when writing a new bash script is set the errexit option:

set -e

This option makes your script bail out when it detects an error (a command exiting with a non-zero exit code). Without this option the script will plough on, and mayhem often ensues. In all the noise generated it can be a pain to found the root cause of the problem. So I make it a rule to set this option and fail as early as possible.


Into continuous integration? Want to be? Try pulse.

Continuous Integration: Build AFAP

May 22nd, 2006

A typical continuous integration server will be set up to build each time a change is detected in source control. This achieves fast feedback for changes by testing them as soon as they are available. However, I believe many projects should be built even more frequently than this: they should be built AFAP (As Frequently As Possible). Why build when there are no new changes to the code? Simple: more testing. Not all bugs show up every time you test. In fact, the nastiest bugs are those that only show up every once in a while. These are the bugs you really want to catch before you ship. If you think it’s nasty debugging race conditions in your cozy little cube, wait until your customers start reporting intermittent problems :) .

Since building AFAP can have very little cost (you have a dedicated build machine, right?), many teams have no reason not to do it. The only significant cost that comes to mind is delaying feedback when a change does occur, as the continuous integration server may be busy performing an AFAP build. On average a build with a change is delayed by half the time for a full build cycle. For projects with long build times, this may be a real issue (indeed, one of many issues: get those build times down!). For those projects, it may be better to stick to building on change throughout the day. Even so, there is no need to waste those overnight and weekend hours! Schedule your AFAP builds to run throughout the night, every night and all day over the weekend. These are also good opportunities to run longer build/test cycles with your most thorough test suites.

——-
Into continuous integration? Want to be? Try pulse.

Continuous Integration: Not Just for Large Teams

May 11th, 2006

Recently, a beta tester of pulse asked a fair question:

I can see the benefits of automated build software for large software teams. Would pulse considerably benefit a development team of 1-5?

Now, I am surely biased, but my answer is most certainly yes. Firstly, there are many benefits that apply to both, such as frequent testing, testing outside of the developers’ environments, early detection of integration problems, isolating changes that introduce problems, and so on.

It is also fair to say that some of the benefits for large teams do not apply to the same degree for a team of 1-5. For example, as the team grows the chance of an integration problem increases even more rapidly due to potentail semantic conflicts between changes committed by various developers. Even when the developers are all disciplined enough to update and run tests locally (which we all are, right? ;) ), there is still a greater chance of a submit race. For large teams the continuous integration server can also be a great way to communicate recent project activity, as developers are notified of builds, and the web UI may allow you to see recent project changes.

Smaller teams do not have such problems with submit races and typically find communication much easier. However, some of the benefits of continuous integration are more important to small teams than to large ones. A prime example is the increased frequency of testing. Without continuous integration, tests are only run as frequently as the developers run them. The fewer developers, the less frequent the testing. Problems take longer to show up, especially intermittent bugs (those nasty blighters that only show up once every 364 test runs). If your tests run every fifteen minutes on your continuous integration server, even those intermittent bugs can’t hide for long. Further, small teams necessarily have a smaller number of environments that they are naturally testing on during development. Adding another one, even a single continuous integration server, is significant. Lastly, a smaller team, with naturally limited resources, needs to apply automation even more aggresively than a large team. Dedicating one or more engineers to release and integration management may not seem like a big deal to a large team, but for a small team that may be a third of the manpower!

Although they benefit in different ways, I wouldn’t say a large team needs CI more than a small one, or vice-versa. Experience has taught me that once a team employs continuous integration, whether it be 2 or 200 strong, the engineers will never want to do without it again.

——-
Into continuous integration? Want to be? Try pulse.

Your Next Programming Language

May 9th, 2006

Many people talk about how, as software developers, we should learn new programming languages frequently. I couldn’t agree more: the broader perspective improves your skills and opens your eyes to the dark corners of the language you are currently using. It strikes me, however, that many developers are missing out on a class of languages that are extremely useful every day. People learn high-level languages like Java and C++, and often a scripting language or two like Perl or Python. Maybe they will even dabble in a functional language to get a really different take on the world. But for me, the single programming language I use most frequently day-to-day, alongside my primary language, is bash scripting. Yep, plain old hackish shell scripts.

Why? Because like most programmers, I’m lazy. I don’t like to do anything I can make a computer do for me, and there are a whole raft of such things that are easily achieved via a shell script. Often it will just be a one-liner to perform a batch operation on a bunch of files. A find/exec/sed sure beats the pants off changing 200 files by hand, and is even quicker than writing a Perl script. Shell scripting is also a boon for project automation. Is packaging your project a headache? Need to pull in a bunch of resources, munge a few files, run some tests and squeeze it all together? Build tools such as Ant or make may get you part of the way, but they are not designed to write scripts. I often use a script to do all the gathering and munging, and call out to those scripts from my build file.

So, no excuses! Even those of you more inclined to the Windows way of life have easy access to bash (and other shells) via Cygwin. Get a taste and you won’t look back. There’s something quite gratifying about replacing an arduous, multi-step task with a script that you can run without breaking a sweat. You’ll never have to work again!

——-
Into continuous integration? Want to be? Try pulse.

Pulse Continuous Integration Server 1.0.2

May 9th, 2006

After a solid week of beta testing and feedback (thanks to our beta testers!) pulse version 1.0.2 is ready! You can download the new release at the pulse downloads page. Highlights in this release:

  • Much improved RSS support
  • Support for maven 2 projects
  • Improved “latest changes by me” view
  • Several other improvements and bug fixes

See the full release notes for further details. Thanks again to our beta testers, keep on stretching it!

Wallet friendly RSS feeds

May 4th, 2006

I have been subscribing to RSS feeds for some time now, but it was not until I had to implement one that I realised that there was more to it than just a structured XML response.

Below is a quick dissection of what I would consider the absolute minimum implementation for an RSS feed. I’m calling it “wallet friendly” not because it makes you money, but because it will save your users in bandwidth costs by not spitting out a full feed for every request. Whilst the example is in Java, using Rome and Webwork, the details apply equally well to other frameworks and languages.

This example is an extension of the WebWorkResultSupport class.


protected void doExecute(String format, ActionInvocation action....
{
    HttpServletResponse resp = ServletActionContext.getResponse();
    HttpServletRequest req = ServletActionContext.getRequest();
    OgnlValueStack stack = actionInvocation.getStack();  

Firstly, what do we do if there is no feed to display? The answer will depend on what that means in your system. If it’s an error, then it’s time for a 500 (Internal Server Error) response. If, on the other hand, it means that the feed request is invalid, it’s best to return a 410 (GONE) so that whoever is making the request knows that they should stop.

         
    SyndFeed feed = (SyndFeed) stack.findValue("feed");
    if (feed == null)
    {
        resp.sendError(HttpServletResponse.SC_GONE);
        return;
    }

Now set the content type and disposition. This will help the browser to deal with the response appropriately, and give it a friendly name if the user decides to ’save as’.


    resp.setContentType("application/rss+xml; charset=UTF-8");
    resp.setHeader("Content-Disposition", "filename=rss.xml");

When handling an RSS request, you should not return the feed unless it has changed since it was last requested. You can find a very good discussion on this by Charles in his fishbowl, and by Randy at the kbcafe. (For those in a hurry, the relevant details are that the “Last-Modified” and “ETag” headers are returned in the following request as “If-Modified-Since” and “If-None-Match” respectively)

There are two steps to this. The first is to always set the “Etag” and “Last-Modified” response headers. The “Last-Modified” details can be taken from the feed as so:


    // A happy default here is the If-Modified-Since header.
    // If we don't have any feed entries, then this will result
    // in a 304 Not modified
    Date lastModified =
            new Date(request.getDateHeader("If-Modified-Since"));
    List entries = feed.getEntries();
    if (entries.size() > 0)
    {
        // Get the latest feed entry - assuming the latest is
        // at the top and that you set a published/updated
        // date on the feed entries.
        SyndEntry entry = (SyndEntry) entries.get(0);
        lastModified = entry.getPublishedDate();
        Date updated = entry.getUpdatedDate();
        if (updated != null && lastModified.compareTo(updated) < 0)
        {
            lastModified = updated;
        }
    }

The Etag should uniquely identify this feed (read – the latest item in this feed). Unless you expect feed entries to be created at exactly the same time (or your database does not provide a high degree of accuracy in the time field) it’s sufficient to use the last modified timestamp for the Etag. If this is not unique enough in your case, you will need to create a unique hash from the content.


    String etag = Long.toString(lastModified.getTime());
    resp.setHeader("ETag", etag);

Before you can use the last modified date for the header, you may need to drop the milliseconds since they are not part of the date format used by HTTP.


    Calendar cal = Calendar.getInstance();
    cal.setTime(lastModified);
    cal.set(Calendar.MILLISECOND, 0);
    lastModified = cal.getTime();

Now we can set the “Last-Modified” header.


    // always set
    resp.setDateHeader("Last-Modified", lastModified.getTime());

That completes the first step (setting the “Last-Modified” and “Etag” headers on every response). The second step is to check the “If-None-Match” and “If-Modified-Since” on the request (remembering that they ’should’ contain what you sent our in the previous response). If they match the “ETag” and “Last-Modified” values we just set on the response then we do not need to return the feed. A 304 Not Modified will suffice.


    // Check the headers to determine whether or not a response
    // is required.
    if (TextUtils.stringSet(req.getHeader("If-None-Match")) ||
    TextUtils.stringSet(req.getHeader("If-Modified-Since")))
    {
        if (etag.equals(req.getHeader("If-None-Match")) &&
            lastModified.getTime() ==
                         req.getDateHeader("If-Modified-Since"))
        {
            // If response is not required, send 304 Not modified.
            resp.sendError(HttpServletResponse.SC_NOT_MODIFIED);
            return;
        }
    }

Now, let’s generate the feed data and send it on its way.


    // Render the feed in the requested format.
    WireFeed outFeed = feed.createWireFeed(format);
    outFeed.setEncoding("UTF-8");
    new WireFeedOutput().output(outFeed, response.getWriter());
    resp.flushBuffer();

Oh, and one last thing. If you want to return error details when things go wrong, do the RSS reader a favour and format the error response in valid RSS format. But be sure to set appropriate Last Modified and Etag header. For example, set an error token in the ETag header that you can check next time round. If your current error token matches the ETag in the request, respond with the 304.

——-
Into continuous integration? Want to be? Try pulse.

10 Ways To Improve Wikis; 2

May 3rd, 2006

Sorry readers, I know you have all been waiting with bated breath for the final installment in the wiki series :) . Well, here goes:

  1. Unreadable syntax for tables

    The problem is actually more generic: the simple wiki syntax works well for inline elements, but poorly for structured elements. Why? Mainly because whitespace is significant. This is convenient most of the time, but gives no way to format a structured entity such that it can be easily read. A possible solution is to allow insignificant whitespace so that structured elements can be formatted with indentation. Naturally, for regular paragraphs, significant whitespace would have to stay. I think the semantics for whitespace can still be clear if the insignificant whitespace is only allowed “outside” of the actual content, in the space between whatever syntax elements are used to delimit the table (or whatever it happens to be). Or you could just use a WYSIWYG editor, but you won’t catch me doing it :) .

  2. Editing in a browser text area sucks

    A couple of readers actually pointed out some neat solutions for this one, such as the mozex extension for Firefox, which lets you use an external program for editing instead of the browser text area. Nice idea, but still a little awkward, as you can’t avoid interacting with the web UI for other actions (this is a wiki after all!). I think the real answer is the slow, painful progression towards browsers as a richer application platform. Many have tried to create the ubiquitous thin client technology for the web, but none can compete with the inertia of the trusty old web browser. Unfortunately for us users, this means we’ll be waiting a long time for browser technology to evolve to a state where web UIs are actually enjoyable to use. Just look at how excited we all get when AJAX lets us emulate two-decade-old desktop technology in our web apps.

  3. Poor support for versioning

    Let’s start simple: a wiki that doesn’t have at least RCS-level versioning support (simple revisions, diffs, history) should not even be allowed to exist. A wiki that doesn’t have some form of merging support (with conflict resolution) isn’t worth using. Personally, my requirements are even more strict. I maintain documentation for software products, and these products have multiple release streams. I manage these streams in my code base using branches, and ideally would do the same with the documentation. I don’t know of a wiki that supports this. This seems a shame, because the version control technology has been around for a long time. Rather than reinventing a weaker versioning scheme, I don’t understand why more wikis don’t use a full-blown version control system (like Subversion) in the back end. There is a wealth of existing tools and technologies that just aren’t being leveraged here.

  4. Lost edits due to browser crashes

    First off, for those wikis with autosave, this is much less of a problem. I would like to mention one insightful comment from reddit, however. There, poster rahul turned this problem back at the browser itself. A good point: why don’t the web browsers implement their own autosave for form data? They are already part of the way there by remembering form input, it is not much of a leap to get from there to autosaving for text areas. The obvious benefit of this approach is that once it is implemented in the browser, there is no longer a need for every web app to reimplement the same functionality.

  5. Wiki discussions

    People, please just stop using flat wiki pages for discussions! It hurts us, precious! Some wikis have a more enlightened approach, and allow threaded comments on each page. This is a huge improvement, no doubt. Still, will they ever match dedicated forum solutions? Maybe wiki implementors should be looking again to adopt existing technologies. I wonder how feasible it would be to integrate an existing forum solution into the wiki. Perhaps a looser coupling would even be benefitial, allowing more diverse tools to be linked to a wiki page. The counterbalancing force is of course the benefit of tight integration.

So there you have it: some good can come of a rant after all. At the very least, I have picked up a few new ideas myself!

——-
Into continuous integration? Want to be? Try pulse.

Pulse CI Server 1.0.1 Released!

May 1st, 2006

Things have been a bit quiet around this blog the past week as we launched both our new website zutubi.com and the first public beta of pulse. Pulse is an automated build (or continuous integration) server built on the principles that are important to us:

  • Adaptability to existing environments: everything within pulse is built on a generic core engine that can run any command line build and extract useful information from the results. When we add specific tool support, we build it upon this core, ensuring the core is flexible enough to deal with many tools.
  • Developer control: pulse gives every developer their own account. Each developer has a configurable dashboard that shows the information they want to see. Developers can also control how and when they are notified of build events.
  • Simple configuration: pulse has a full web interface that allows you to configure the server in minutes. You can build a project without editing a single text file. Alternatively, you can choose to configure your project using a simple XML file and version it in your SCM with your code.

That’s enough marketing here :) . If you are interested, head over to the website for full details. Give pulse a free trial for 30 days and let us know what you think. Sign up for the beta program and get discounts on commercial licenses!