[Plone-testing-team] storing Travis-generated Selenium logs?

[Asko Soukka]

Héctor Velarde wrote:
> is there a way to store the Travis-generated Selenium logs/screenshots 
> on some place?
Yes, but it's not very convenient to set up:

http://about.travis-ci.org/blog/2012-12-18-travis-artifacts/

> we're working on a new release of collective.cover and it's a pain in 
> the but trying to fix some random failing tests.
Sound like Selenium. I guess, there's only a hard way to learn to write 
tests "defensively" for it :/

Since you have already migrated to p.a.robotframework, it would be about 
same amount of work to start using SauceLabs (which stores raw Selenium 
logs, random screenshots and complete videos of Selenium tests).

http://developer.plone.org/reference_manuals/external/plone.app.robotframework/happy.html#integrate-with-sauce-labs

It mainly comes down to this (and an updated .travis.yml):

Resource  plone/app/robotframework/saucelabs.robot

Test Setup  Open SauceLabs test browser
Test Teardown  Run keywords  Report test status  Close all browsers


(And all these have fallback to normal testing when proper variables for 
Sauce testing does not exist.)


Also, I only now noticed, that Travis-CI has implemented dedicated 
support for SauceLabs, which would make it easier to set the connection 
in travis.yml (this has not been updated in Plone docs yet):

http://about.travis-ci.org/docs/user/addons/

> we are currently using the latest versión of p.a.robotframework and I 
> have successfully implemented autologin to speed up the tests.
CMFPlone has an example of bundling custom keywords to RobotRemote-library:

https://github.com/plone/Products.CMFPlone/blob/master/Products/CMFPlone/tests/robot/robot_setup.py

https://github.com/plone/Products.CMFPlone/blob/master/Products/CMFPlone/testing.py

Brief "what is remote library and why":

For robot, remote library is an XML-RPC-service, which implements 
specific interface to work like a custom keyword library for robot. With 
p.a.robotframework it is an public object created in test layer setup 
and removed in test layer teardown. The main reasons to use it instead 
of pure Python library are: 1) provide proper request for each keyword 
call (and those request can be authorized with autologin) 2) support 
testing with pybot from separate proces: 
http://datakurre.pandala.org/2013/05/test-fixture-driven-development-for.html

-Asko