Testing web-based applications with FIT

I have had a need to automate testing of our web-based applications because manual testing could never be completed in a reasonable timeframe.

For the last two weeks, I've been writing a fixture for FIT specifically to test web-based applications. The idea is to use very sophisticated HtmlUnit library for actual HTTP interactions and dealing with HTML (and Javascript) and to use FIT for specifing, running and reporting on test cases.

update: project is now hosted on public subversion repository. See http://opensvn.csie.org/webfixture

For those who have missed it, here is a short description of the FIT framework. FIT is a web-based tool for specifying and executing acceptance tests for all kinds of application, with a particular accent on simplicity of the specs (they could be understood and may be even written by a non-programmer, i.e. customer). The central role in the FIT framework is devoted to a fixture. A fixture implements a vocabulary, which is used by user to describe things he wants FIT to perform. For example, to press a button or to follow a link. Or to evaluate 2+2 and to check that result is 4. Thus, user writes these checks using vocabulary and programmer implements this vocabulary with a fixture in a meaningful way. If you're intrigued, Michael Feathers has written a more elaborate intro in his Pitching a Fit post.

Originally, I tried to write a custom ActionFixture for each web page I had to handle but soon discovered that it was a bad idea. Firstly, it was too much work -- to write a Java class for each page I need to test. Secondly, and more important, each fixture would use different vocabulary for each page which means that test writer had to be familiar with an underlying Java class. Therefore I decided it would be better to write a single class (a fixture) that could be used to exercise any web page. Luckily web languages' expressiveness is rather limited so the resulting vocabulary to describe test cases should be manageable.

Shortly after I started to work on my WebFixture, I discovered a (semi?)-official HtmlFixture, created for the same purpose and which even uses the same HtmlUnit library! Nevertheless, I decided to continue to develop my own fixture, for these reasons:
1. To get experience with Java platform.
One of my goals with this mini-project has been to try out a Java platform. I've been on most C++, Python, Tcl camps these years and my Java programs never went beyond the "Hello, world!" exercise. This project seems like a perfect opportunity to learn a new platform.
2. To control fixture's development.
I want a control over this fixture to add things I want in a way I want. While the HtmlFixture is mature, its source code is undocumented and poorly written (to me). This probably a funny thing to say from a novice Java developer, but still, the source code seems like a mess to me and don't want to dive into it.
3. To design it differently.
If I decided to base my work on HtmlFixture there would be still no guarantee that my changes would ever go into original codebase. Why? Because I my approach is different and therefore there is a good chance that changes I propose won't "fit" author's concepts. Don't get me wrong, HtmlFixture is a fine work, I just want to implement the same concept a bit differently.
So how my fixture does look like?

The important design goal to me is to make fixture's vocabulary as simple and as high-level as possible. It should be similar in terms to those used by a web browser metaphor. It's better to sacrifice some underlying versatility of HtmlUnit to gain in simplicity and consistency for a user. It's better to overload verbs (where appropriate) than to invent new ones.

For example, I use verb click to follow links and submit buttons and use verb enter to enter data to input boxes, text areas and select boxes. This way, the same verb can performs technically different things as long as it means one thing to the user.

There are some other features I've added so far: site configuration, Java delegates, erroneous page detection.

The site command allows you to set some site options (or preferences) which include site root, timing information, error page patterns and more. These preferences are kept in a static storage so that you could set them in SetUp page and use on test pages.

The error page patterns used to detect erroneous web pages - normal HTML pages which are returned with HTTP response code 200 but which nevertheless are a sign of some internal error in the web application. For example, some application servers (like WebKit) show crash reports in this way and fixture could be configured to detect these automatically.

Java delegates are extension point to be able to hook custom checks that are beyond normal web processing, like verifying the database state. A delegate is passed a current HtmlElement and should return a boolean result - whether a test has passed or not.

And finally, some closing words on my first impressions on Java platform.

The IDE for a statically-typed language could do amazing things and Eclipse actually does them! Moreover, it is very capable in other aspects making handling of Java verbosity almost imperceptible. With advanced browsing support and context assist I managed to grasp an essence of both FIT and HtmlUnit library very quickly. And QuickFix assist helps greatly to correct my naive syntax errors. Overall, I must say, the development process goes much more smoothly and is much more fun that I have expected, thanks to IDE.