Friday, July 27, 2007

"Wait, Don't Delay" or how to test complex async Java GUIs

I have been working with abbot and costello for some time now and people tend to have one of two problems getting there scripts to run reliably: the first is not being able to find a component again; and the second is the real time nature of most complex swing UIs.

The first issue can usually be solved by ensuring the that program is accessible and that use of the "name" property where required. Things do get more complicated with nested component; but they can generally be solved with a little bit of common sense.

The second issue is the real topic of this post and it sometimes takes quite a while to change around the mindset required to create tests that will run reliably on a range of platforms and systems.

Take for example an wizard that needs to query a database between pages. This means that the required components might not be enabled instantly after the user action. Now a naive user might try to include a "delay" step because they know how long the action might take; but they will find that eventually the test will fail on one platform or another. They might also try to use the waitForIdle action which is a better guess; but unfortunately not reliable and it might never return in the case of a dialog with an ongoing animation.

In all cases where delays and waits have been used in our tests we have seen then resulting in unstable test suites. The only thing I can think that delay should be used for in day to day tests is using it to ensure a piece of functionality doesn't take too long to run. Even then it will be hard to pick a value that will work on all platforms being tested upon.

Lets look at a few ideas on how you can detect whether the task has completed.

The first and most important thing you can look at is obvious UI changes that signal that the asynchronous task as finished. This might be as simple as waiting for a component to be showing using "Insert"->"Wait For"->"Component Showing". Or in the case where the UI component is always displayed you can use an assert step to ensure that a given component is enabled. In Costello this is done by selecting the "References" tab, then go to the "Properties" tab and find the "enabled" property, hold down Control and use the button that appears to insert a "Wait For" step.

The one exception to this are progress dialogs which are generally not a reliably way of tracking a task. If a progress dialog is implemented properly they might never show if the task is really quite quick.

You can also monitor lists and trees for the addition of new nodes. In costello you only need to add a selectRow step for a tree and code being the scenes will wait until that node path becomes available. (I did some of the work for this). You do find that when you have some experience with costello that you instinctively select elements in list and tree because they provide you will a cheap way to test the current state of the model.

If you are working in a product such as an IDE then you will find that many asynchronous tasks write to a log window of some kind. The example of this in JDeveloper is when we try to start and embedded oc4j instance and we have to monitor the log window to know when it has finished starting. To deal with this we use the "Call" step to invoke a static java method that does the work for us. We might also have achieved the same results with an "Expression" step; but they can be harder debug.

If the worst comes to the worst then you do end up having to provide some kind of behind the scenes API. In the work I have been doing on JDeveloper it proved to be easier to invoke the build system programatically using an Expression step rather than the UI directly. You can also use "Call" or "Expression" steps to query the state of UI components that you don't have tester for, although in the long term you are much better or writing the testers in the first place. An example I currently have on my queue waiting is a custom tester for the "gutter" in the code editor in JDeveloper. Without that it is very hard to work out when "Audit" has finished applying its rules in order to invoke a "Quick Fix". Currently a test is this area is using a very long delay which I am less than pleased with; but I hope to remove this soon.

Finally we have implemented a kind of "internal" log window using Logger for those cases where we can't think of something more creative. This provide a way for actions to say "I am finished" and for abbot to be able to pick this up. In general this is only for use as a sticking plaster until we can put the right tester in place.

No comments: