[Plone-testing-team] Future of robot testing in Plone & add-ons

[Godefroid Chapelle]

Le 24/10/13 21:51, Asko Soukka a écrit :
> Hi,
>
> I'm currently (slowly) working on p.a.robotframework 0.7.0. For the
> revised docs, I'd love to have your opinion on, what should be the
> recommended use cases for Robot testing with Plone and add-ons?

Short answer: cross-browser testing of JS based features.

> Originally Robot Framework and its SeleniumLibrary were introduced to us
> for testing Java Script -based features in Plone. But Plone 5 all Java
> Script is be tested already in Plone-mockup
 > and Selenium-tests would only make sense only for acceptance tests.

As far as I understand Plone-mockup is mainly unit-tested. Is that 
correct ? Are the tests run in various browsers ?

For me, the goal of robot-framework usage has always been to do 
acceptance testing of JS based features. And to build test suites that 
would be runnable against as many browsers as possible.

The reason I had that goal is the following : when we did KSS, its JS 
code was already fairly well unit-tested. However, it became quickly 
obvious that this was far from enough... and that integration/acceptance 
testing was also more than needed : the UI integration would break 
randomly in new browsers or would break because of slight changes in html.

I understand that recent browsers are much more stable than the one we 
were using back in 2006.

I still think that those acceptance tests are really needed.

Selenium tests can be run against all major browsers. Especially now 
that Saucelabs allow us to run freely open source tests.

> On the other hand, Robot is easy and fun to learn and use. Yet, Selenium
> tests are still slow to run and pain to debug.
>
> p.a.robotframework 0.7.0 (with robotframework 1.8.2) includes helpers
> (and still in-progress docs) for creating Robot testable documentation
> with Robot (Selenium) generated screenshots. Practically all the stuff,
> I was piloting in collective.usermanual:
>
> http://jenkins.plone.org/view/User%20Docs/job/collective.usermanual/
>
>> From CI point-of-view, Robot tests in documentation makes sense only
> with pure Robot Framework, without zope.testrunner (or robotsuite). They
> are not yet as optimal in layer setup/teardowns, but allow bundling
> tests so that there are less setup/teardown cycles than with
> zope.testrunner.
>
>
> What do you think, should I update docs to recommend using Robot only in
> documentation and direct to Timo's testing documentation for other tests?

As stated above, I think acceptance tests are indispensible.

> Also, if we can have acceptance testable documentation, are there need
> for acceptance outside the documentation?

Experience with doctests has shown that testable documentation is very 
different from tests. It is very useful. But not enough. And often, do 
not cover all use cases/corner cases.

> Any thoughts?

I'd be happy to be kept cced if there is feedback from the list.

> Cheers,
> Asko

PS: Asko, I was really happy that you received the Plone award. It is 
well deserved... and I am happy to have a chance to congratulate you.

-- 
Godefroid Chapelle (aka __gotcha) http://bubblenet.be