After fighting with our test suite of roughly 40 Selenium tests I finally managed to get them all passing on the build machine. The test cases were created with Selenium IDE and saved in HTML format. The build machine has Hudson and it executes the test suite.

To have tests as stable as possible there are some points to watch out:

In case of AJAX application timing issues are a major problem. One option is to set test speed to slow enough. The obvious downside is that the test suite takes forever to pass. The other option is to use pause or waitFor commands in the test case. This requires more effort from the test case writer or recorder.

Not all Selenium components are equal. Selenium RC, IDE and Core all have a bunch of minor variations. For example clickAt command works nicely on IDE but on Core you might want to try mouseDown. And don't get me started about right clicks.

Last but not least is the external input. Another developer playing with mouse cursor while test runs on the browser might cause focus loss and test case failure. The only clue of this is randomly failing test cases.

There are some alternatives for Selenium but I have no clue how well they compare against Selenium. I am not 100% happy with Selenium so alternatives are welcome.

Edit: Hudson is an open source continuous integration tool. It supports on demand builds and scheduled builds, so you can set it up to run a full Selenium test suite during the night.