Playing with Hudson for running UI tests

I have a whole bunch of UI automation tests that for one reason of another I never gotten around to running them automatically. I had planned to include them in our short regression tests sequence; but they are just not stable enough at the moment. The problem with testing UI in a complicated environment such as JDeveloper as it only take a minor change by someone else to derail your tests. So I wanted an environment I could run them locally in.

I decided to use Hudson to drive the tests after being impressed by it at a presentation at JavaOne last year. I don't have to worry about building the product, only testing the results of the build system. It is very easy to set up, just download the war and either deploy it to a container or just run it from the command line and its starts its own server.

Now while Hudson has lots of nice support for various source control systems it of course doesn't support Oracle's in-house ADE source control system. Also I was kinda worried that since as control our build with make that it would be hard to integrate. I shouldn't have worried as Hudson has some really nice scripting support which makes integration to any SCM system really quite easy. Perhaps I will write a proper integration one day; but not really necessary for what I need to do day to day.

Anyhow a job can be made up of multiple steps, be they script, ant or maven. Here is a fragment of a job I set up to work out what the latest labels are of each of our current development branches. This might seem like an odd thing to do; but you can trigger hudson jobs off changes in a given URL if you install the right plugin - strangely there isn't one that will trigger of a script.

There are loads of plugins and they are easily installed using the hudson plugin manager ("/pluginManager"), just make sure that you pass in the proxy options at the command line if they are not set up in your environment so it can contact the home server. The Xvnc, Emma and URL Change Trigger plugins are the most important as far I am concerned.

Hudson can run multiple jobs concurrently; but for the purposes of running tests in JDeveloper I only want one running at a time as it means I can rely on default ports on all of my tests. So I reduced the number of executors to 1 on the configuration page ("/configure"). It did have to restart the server to get this to take effect though.

As mentioned before I am using the URL trigger to cause new jobs to run, you can configure this on your job configuration page ("/job/%job_name%/configure").

At the bottom of this page you can also configure how the results of the build steps are processed. In this case capturing junit and emma results. I also want to archive the results of the automation tests, we take a screen grab at the end of each test failure and capture a log of all http traffic. This all gets collected up for each test.

For each job you get a nice overview of test results and coverage, not particular proud of the test coverage values here but it is something I am working on...

Obviously with UI tests you need a display per test run, now with only run executor this isn't so much of a problem; but if you wanted to run multiple tests on the same machine then you should make use of the Xvnc plugin and turn it on for the relevant tests. This create a new vncserver for each job and configure the DISPLAY environment variable correctly. One important thing to consider is to make sure you are running the tests in the same environment as your users. VNC default to running tvm; but for a higher level test tool such as Abbot to work properly you need to run a proper window manager. So I edit my ~/.vnc/xstartup as follows to run gnome, note the "--sm-disable" mean you can run multiple instances concurrently:

#!/bin/sh
[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &

gnome-session --choose-session=VNC  --sm-disable &

The other thing to consider is that the default display dimensions for xvnc is too small to run big complex Swing application. You can control the command line options on the hudson top level configuration page. It would be nice to be able to configure this for each job.

I guess the reason why I like this system so much is that it was easy to set up and you got a lot for the time you have invested. This are load more features, particularly in the area of notification with everything from email to rss to teddy bear lamps. The ability of Hudson to bring a machine in to the build pool by just running a WebStart application is also intriguing and something I am going to look at as I pull a windows machine into the build pool when my desktop become avaliable.