[Product-Developers] Re: Running all Plone tests?

Martin Aspeli optilude+lists at gmail.com
Sat Jan 10 17:27:38 UTC 2009

Chris Rossi wrote:
> I guess what I'm trying to glean, though, is whether it is a normal 
> thing to want to run all of the tests and whether there is an 
> expectation that they will all pass in a vanilla install.

Yes, there is. If a test is broken on a vanilla install, please consider 
helping to fix the code or the test and committing back! As far as I 
know, the release manager runs all the tests prior to making a release.

When I've run all the tests in the past, I've tended to enumerate over 
all the packages in src/ in the coredev ploneout and run them 
one-by-one, e.g.:

$ cd src ; packages=$(ls -1) ; cd ..
$ for x in $packages; do ./bin/instance test -s $x ; done

On trunk, I think you can just do:

$ ./bin/instance test -s plone

and run all the plone.* and plone.app.* tests, and probably with a few 
more commands you can run most of our tests like that.

However, most people don't run all the tests all the time whilst they 
are developing third party components, and if one of your third party 
components breaks a Plone test when it's installed, it's not 100% 
certain that this is because the component broke Plone - it may just 
have altered some assumptions that the Plone tests made that are not key 
to the test.

Conversely, your component could break Plone, but the test setup for 
Plone's test may cause your package's changes to be ignored/not loaded 
(this is kind of preferable - it means the package's tests are much more 
like unit tests).

>  If there is 
> no expectation of such, amongst the community, then I'm not inclined to 
> investigate further why individual tests are failing.  I am perfectly 
> happy to just run the tests for my packages and leave well enough 
> alone.  If I were doing coding that affected the core, though, I would 
> be more concerned, since I'd be wondering how to establish a good 
> baseline for regression testing.

All core tests should pass, of course.

> To  ask in a different way, when Mr. Aspeli, in his tutorial on testing, 
> says to "run all of the tests" what does he mean by "all"?  All of the 
> tests for your package?  Or, really, all of the tests?

Heh, Mr. Aspeli means all the ones for the code you're working on.


Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book

More information about the Product-Developers mailing list