[Product-Developers] Re: Selenium tests

[Balazs Ree]

Hi Martin,

I am not following this list and Russ called my attention to your post.


On Sat, 24 May 2008 09:32:02 +0100, Martin Aspeli wrote:
 
> Is anyone successfully using Selenium testing in projects? My goal is:
> 
>   - to use existing PloneTestCase + a custom layer for fixture setup -
>   to use DemoStorage
>   - to record tests using the Selenium IDE for firefox (which supports
> Python output)
>   - to run tests as part of a buildbot
> 
> I know of Zelenium and PloneSelenium, but they both seem quite old and
> TTW driven. Perhaps not?

You may not be aware that we did some progress on this during the Sorrento 
sprint. Our goals were somewhat different from the ones you state above, 
though.

http://kssproject.org/blog/archive/2008/04/08/plone-app-kss-sprint-in-sorrento

There is also an overlapping PSPS topic:

http://dev.plone.org/plone/ticket/7871

This targets Plone to be Selenium-testable, but testing applications is 
in fact not different from testing Plone itself so this makes it
relevant to the topic of testing custom applications.

It was also a requirement that testing should be possible on all 
platforms and browsers, and that we can both support testing invoked by 
the developer on any choosen browser, and also run tests nightly in an 
automated manner on several supported browsers (this last piece is 
currently broken and will work once "kissbot" is fixed.) 

The main approach we used was extending the kss testrunner to handle 
application level tests. This means it's not only usable for kss but
for anything in general. Although, the structure to be used to set up a 
test suite is still to be discussed and finalized in kss.demo.

For the recording of tests, Selenium IDE is used. For the running of 
tests, Zelenium is used which means it runs on every browser. 
It was not a goal to be runnable from within
PloneTestCase, instead we wanted to make a group of tests 
runnable from a browser with a single click. 

It also does not use DemoStorage but operates on an existing server, but 
this could be extended if needed.

For the test runner we decided not to use setup and teardown (like 
PloneTestCase) instead we applied a simple testlayer based approach.
The main reason is that running tests with Zelenium is slow and thus 
separating all tests in the same layer with setup/teardown would be 
unbearably slow in practice. However layers work effectively, although tests 
are not allowed to leave trash behind, in order for this to work.

As far the results, the testrunner works very well in practice, I am 
using it to test plone.app.kss on every browser before releasing. 
There are not yet many tests written for Plone, but this is an issue separate 
from the testrunner itself.

You can easily check out this method by using its buildout and following
the step to step documentation.


The second approach that Sylvain was working on the sprint is funittest 
and it may also fit your needs:

This is a different method, it has a model based approach for creating 
tests. Recording tests with Selenium IDE is not possible, you write code 
only. For running the tests Selenium RC is needed which means additional 
requirements for installation on the server side. May not work on all 
platforms and browsers, but it has a working buildout available.

We are looking for a solution to allow a combination of both kinds of tests,
for the Plone testing story. There has been a long discussion about this on
the sprint, and we came to important conclusions.


You find links of all that I mentioned here from the pages I referred to 
above. 
 
Best wishes,

-- 
Balazs Ree