a little madness

Since I have previously stated my admiration for UnitTest++ as a unit testing framework for C++, I guess it was high time I added direct support for it in Pulse. Well, as of Pulse 1.2.24, there is now a UnitTest++ post-processor that will slurp your UnitTest++ test results directly into Pulse. If you’re looking to do continuous integration for C++, then Pulse + UnitTest++ is a killer combo ;).

Paul Duvall’s post The Future of Continuous Integration points out a weakness in how continuous integration is typically implemented: code must be checked in before it is tested by the continuous integration server. Although problems are picked up fast, there is still a time window where developers may be affected by broken code. This is exactly why we added personal builds to Pulse in version 1.2. Personal builds allow individual developers to submit their changes to Pulse for testing before committing to version control. Pulse does a build and test of the latest version with the developer’s changes applied, allowing to developer to check the full impact of their changes prior to affecting the shared code base. Apart from closing the time window, personal builds also allow other powerful usage patterns, such as using the distributed building features of Pulse to test across multiple platforms before checking in.

Paul also touches on a couple of different solutions to the weakness. Naturally, before implementing personal builds as we did, we considered a few options. One option that Paul mentions that we had considered was leveraging the branching functionality of the SCM. Developers (perhaps with the assistance of the CI server) could check their code into private or temporary branches for testing, and the changes could then be merged down to the shared branch if the build passed. The main attraction of this method is that it reuses functionality available already in the SCM. However, we decided not to go down this route as it can have an impact on the way developers need to work and/or the structure and history in the SCM. Our method of taking a patch from the developer’s working copy and applying it directly with Pulse is about as unobtrusive as you can get. This is important for us, as the ability to adapt to existing environments has always been a key design goal for Pulse.

For more about these and related issues, you might want to also check out an article I posted a while ago to zutubi.com. Reducing the Impact of Broken Builds talks about the affects of a broken code base and how best to avoid or alleviate them.

One of the reasons I love programming is that I enjoy the challenge of problem solving. On the flip side, nothing is more boring than repetitive, mind-numbing tasks. The great thing is that when something in my job becomes repetitive, that is a problem in itself that can be solved by programming. Meta.

Good developers don’t write the same five lines over and over, we abstract into a function. We abstract groups of functions (or objects) into libraries, and libraries into frameworks. Done right, we both take the tedium out of the job and enjoy the challenge of doing so. Then we get to enjoy the challenge of focusing on the real problems. If you like a challenge (and who doesn’t?), then you’re in the right place.

So why is it, then, that many developers avoid testing because they see it as a mind-numbing and repetitive activity? It is almost as if developers believe testing is an unskilled task that is doomed to be menial. In actual fact, testing takes many of the same skills as developing features. If you find it repetitive, ask yourself why and challenge yourself to do something about it. It is a perfect opportunity to put your problem solving skills to work.

The Testivus “manifestivus” says Testing Beats Debugging:

We believe that time invested writing and running tests is better than time spent debugging.

The Google Testing Blog is headlined by a graphic that states:

Debugging Sucks. Testing Rocks.

I agree wholeheartedly. Debugging is a fact of development life¹. It even has its own rewards, such as the sense of achievement when you find and squish a particularly nasty bug. Simply put, however, debugging is a massive waste of time². Any time spent debugging could otherwise have been spent actually writing code, or otherwise being productive. Ergo, anything that helps to reduce the amount of time spent debugging is a Good Thing.

Testing reduces time spent debugging in various ways:

• Well-crafted assertions in your tests will narrow bugs down right to the source. Operating at the unit test level in particular is more precise than manual testing.
• Automated tests catch problems early in the development cycle, when they are easier to debug. It is well known that the earlier a problem is found, the easier it is to fix.
• Regression tests catch problems that have been debugged before, so you don’t need to debug them again. This is particularly powerful in complex areas where bugs pop up frequently.

So, if you embrace testing, you can spend less time debugging and more time doing what you enjoy: writing great code ;).

–
¹ At least until we start writing perfect code…
² Note that is debugging, not using a debugger: they are quite different.

Fear not, despite the title I’m not going to start pontificating about design theory or TDD. Like always, the “Testing Is …” series sticks to straightforward ways in which tests make our lives easier. Being the lazy programmer that I am, I’m always looking for ways to make writing test cases easier. I suppose you might call this “designing for testability”.

Certain coding practices lead to code that is difficult to test: at best they make writing the tests painful, at worst they discourage testing and lead to maintenance pain down the track. The good news is that many of these practices are a sign of bad design in general, so by keeping testability in mind while you code you will often end up with a cleaner design. I look at it as the tests being a client of my code that help to keep the design honest.

An example is probably in order. First off, I find that it is unit tests that really drive the improved design. In unit tests we try to isolate a distinct piece (or “unit”) of code so that it can tested independently. The better we can isolate the units, the better we can cover the code as a whole without a combinatorial explosion in the possible inputs and outputs. The great thing is that well isolated units exhibit a key design practice: Separation of Concerns. To be truly isolated, a unit should be concerned with only one task. A complex system is much more powerful if it is built out of small, distinct units that can be combined in interesting ways (the “Unix way”).

A common counter-example that suggests testability and good design are actually competing goals is the apparent need to access internal unit state in tests. The problem with this of course is allowing the test to access that state is at odds with another key design practice: Encapsulation. In practice, I actually find that the need to access internal state is exceedingly rare. I would go so far as to say I consider it a design smell: if you feel the need to assert over some internal state in your tests, there is a good chance that the unit granularity is wrong (perhaps you have not truly separated concerns). There are exceptions of course, but in most cases refactoring is a better solution to subverting encapsulation. Thus, rather than countering, this example enforces that having test code as a client drives improvements to design.

The most common excuse I have heard for the failure to write automated tests is that there is no time. People have a deadline to hit, and it seems there is no time to waste writing test code when there are features that need implementing. To me, this is a terrible excuse. Not because I think that code without automated tests can’t provide your customers with valuable features. Rather, it is because you are going to have to test the features somehow, and if you can’t do it automatically you are going to have to do it by hand. As I mentioned in a previous post, I think that in the majority of cases this will be slower than just writing the automated tests.

The crux of the argument is that features never work the first time that you try them. I don’t know about you, but when my new code works the first time I try it I’m not happy, I’m suspicious. This suspicion comes from years of getting things wrong. Typically, each feature will take several testing iterations to get right. This is trivial with an automated test, and time consuming with a manual one. I have not measured the typical time for each alternative, but I would say that most of us naturally assume the manual testing treadmill is less time consuming than it turns out to be¹. So, when we have that short term deadline to hit, automating the tests can be a lot more effective at saving time than we realise.

Over the longer term, the fact that automated testing saves time is pretty much a no-brainer. In fact, I would say that this is just one case of a general principle:

Over the long term, the right way is also the fastest way.

Cutting corners is at best a near-term win, and cannot be sustained. Over the long term, automated testing continues to pay dividends, with only the (hopefully) small cost of maintaining the test suite. The most dramatic time savings come when your automated test suite finds a new bug as it is introduced by a developer. With the right process in place², this bug can be fixed before it breaks the stride of the development team, and long before it reaches a customer and leads to a costly support incident.

–
¹ Perhaps a specific case of the general optimism that makes most of us terrible at estimating software schedules.
² And tools!

For this second entry in the “Testing Is …” series, I decided to mention an advantage of testing that is widely known. The idea is not original, but its importance to you as a developer cannot be overstated. Perhaps the single greatest advantage of writing tests is the freedom you feel to change the code under test. All software grows cruft over time, and left unchecked the code will become an unmaintainable mess. Most developers would agree that maintaining code is not the greatest part of the job, and maintaining crufty code is about as bad as it gets. The answer is to refactor to remove the cruft over time. However, we also know from bitter experience that changing working code, no matter how crufty, will introduce new bugs¹.

That is where the tests come in. If you have maintained a comprehensive test suite over time², you need not fear nasty new surprises. You can refactor as required, and keep your code base much healthier. Now, test suites are not perfect and new bugs are still bound to happen. However, this is mitigated by two factors:

• The difficulty in maintenance of the crufty code itself will lead to bugs.
• A well-written test suite will cover those dark corners where bugs have been found to lurk. In particular, regression tests written to reproduce bugs as they are found are priceless at refactor time.

The logic is quite simple. Beyond that, there is also a powerful emotional difference. When the tests aren’t in place, we fear change. When we have the testing safety net, the fear is replaced with a sense of freedom. A freedom to change, and improve over time. A freedom which itself can be motivating.

–
¹ If the cruft has grown due to the underlying complexity of the problem domain it is certain there are some nasty new bugs waiting for you!
² Maintaining the tests as you write the code is important in itself. It is much more difficult to add tests later, not least because the problem is not fresh in your mind.

Somewhat inspired by Alberto Savoia’s article Testivus - Test For The Rest Of Us, I have been thinking about ways to motivate more developers to write more tests. Holier than thou style arguments (”your code is useless if it isn’t tested…”) are no motivation. But there is a much simpler way to approach the topic: testing has many tangible benefits for the developer. Perhaps looking at the ways in which testing can directly benefit us will motivate us all to test more. So I’ll be writing a few posts with perspectives of testing that highlight the benefits I get from my own tests. Here it goes:

Testing Is … Instant Gratification

One of the most frustrating parts of the development cycle is the start of a new major release. This is an exciting time, where big new things are planned and frontiers are explored. However, given the size of the problems that are tackled in this phase, roadblocks are always found along the way. Even when things are sailing along smoothly, the finish line is a long way in the distance. Thus it is easy to get the feeling that you are going nowhere: as you have no concrete features to show for your efforts. That’s where testing can help. By isolating and exercising a smaller piece of the puzzle, you can demonstrate progress. And I don’t mean that you can prove to your manager that you haven’t just been surfing the web all day. Rather, you can demonstrate to yourself that things are heading the right direction. I find that even this small sense of achievement can be really motivating. Once the most basic things are working, you can’t help but add that next incremental step. Before you know it, you’re in the zone and things are really beginning to gel.

So, next time you feel stuck: write some tests!

As programmers we tend to get attached to code we have written. After all, we spent all that effort dreaming it up, typing it in, and (sometimes) making it work. Many of us also like to look over how much we have churned out, to see that lines of code metric crack the 500,000 (or whatever) mark. These sorts of things make us loathe to delete code we have written. But we forget these important points:

1. Maintenance: someone has to maintain all of that code, and I’m sorry, but it’s you! The less code you have to maintain, the better. So next time you try and crack the million LOC, think about how fun it is going to be to maintain it all :).
2. Dead code rots: leaving unused code around “just in case” is a particularly bad idea. If it isn’t exercised, it will become broken very soon. Its very presence distracts the reader from their real task.
3. Source control: think you might need that dead code again? Think that a bad refactor may bring you unstuck? That’s what source control is for: it has got your back.
4. Efficiency: every little bit of code slows everything down just that little bit. More disk space. Longer compile times. Slower IDE performance. Larger reports. Over time, all these little things add up.
5. Elegance: verbose code is often a sign of a non-optimal solution. When refactoring, you should be able to use your superior knowledge of the problem to simplify. Cutting back code in this way removes the cruft that builds up over time.

So, stop being so happy that you’ve produced so much code. You should be much happier to achieve the same result with a smaller codebase, even if it means throwing away yesterday’s hard work!

I was reading through the Pragamatic Programmers’ List of Tips today, which is filled with fantastic, pithy advice. Undoubtably the most-quoted is the DRY principle: Don’t Repeat Yourself. More than just avoidance of code duplication, the principle states that:

Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.

It occurred to me that there is another, similar principle that could make the list. DRO: Don’t Repeat Others. These days we as developers are spoilt by the richness of open source libraries that we can choose from. Still, at times we fall prey to NIH (Not Invented Here) Syndrome, and are tempted to write our own frameworks and libraries. When there is an established and well-maintained open source alternative, it is almost certainly a mistake to write your own. The advantages of reusing open source are plentiful, with the most important being:

• Code for free. As obvious as you can get: code you reuse is code you don’t have to write. More importantly, it’s code you don’t have to debug.
• Developers for free. A well-maintained open source project brings more than just the code. The developers will keep improving the code over time, and you can take advantage of those improvements.
• Battle tested. A mature open source project will have been tested in the Real World. New code you write will take some time before it even reaches its first Real User for this acid test.
• Low Risk. For the manager wary of not having the code under their complete control: you always have access to the code if absolutely necessary.

So next time NIH hits you, fight the temptation. And next time you are struggling with the learning curve of taking on a new library, persevere. Leveraging open source pays: in the short term and more so over time.