[Framework-Team] Re: ploneenv - Or how using workingenv for a common Zope2 project might look like ;-)

[Martin Aspeli]

First of all, I think this is great. :) The important thing here is that 
people can use what they feel comfortable with - at the end of the day, 
all the actual software should become eggs that we can share.

However, I'd like to try and clarify what I see as some of the 
philosophical differences between ploneout and ploneenv. Hopefully, this 
is useful to people.

I honestly don't think it's a fair criticism of zc.buildout-based 
solutions that they are more heavyweight or complex (not that Daniel has 
said that, but it seems to be a fairly common sentiment). The only hard 
things for me when I started to work with zc.buildout to help Hanno 
complete ploneout were (a) getting over my initial fear of something new 
with a scary name and (b) understanding setuptools and eggs in general. 
The whole of zc.buildout + ploneout's specific recipes (more on that 
later) is probably a bit bigger than workingenv + ploneenv, but not by 
an order of magnitude or anything like that. Most of the code is fairly 
trivial "plain python".

Now, I suspect ploneenv and ploneout are not equivalent. ploneenv gives 
you a new Zope instance which you can use for your own development or 
deployment. ploneout is aimed squarely at core Plone developers, because 
it configures all of Plone's eggs as development eggs and does something 
similar for its products.

So, let's instead compare ploneenv with the PasteScript template I've 
been making that lets you create a new "buildout" for your own project. 
This uses the same recipes as ploneout in order to create a Zope 
instance (and more), but considers Plone as a dependency to be managed 
by buildout. Let's also assume that the Plone egg that Daniel mentions 
exists, and that it takes care of all the products. For comparison's 
sake, let's also say that you want to use an existing Zope (2.10.2) 
installation rather than let zc.buldout download, compile and configure 
it for you.


  $ paster create -t plone3_buildout myproject zope2_install=~/zope2.10

This creates buildout_test/buildout.cfg after asking a few questions 
(about Zope username/password, port, development mode, verbose 
security). Here's minimal version (using the fabled Plone egg, and you 
get a bit more documentation in comments when you do it for real):

     [buildout]
     parts = instance
     eggs = Plone

     [instance]
     recipe = z2c.recipe.zope2instance
     zope2-location = ~/zope2.10
     user = admin:admin
     eggs = ${buildout:eggs}

You then run (once):

  $ python bootstrap.py

And then (each time you change buildout.cfg):

  $ bin/buildout


zc.buildout is basically just a command runner. It makes it easy to pass 
stuff around in "options" between the different sections of 
buildout.cfg. These options are used by various "recipes" (eggs with 
commands that buildout finds using entry points). For ploneout, Hanno 
and I wrote a couple of recipes, the most important one being 
z2c.recipe.zope2instance, which can create and configure a zope 2 
instance. To do so, it needs to know where to find Zope, the root user, 
and which eggs you want Zope to know about (here referring back to the 
"global" list of eggs in the [buildout] section).

Because Zope 2 isn't terribly good with eggs, the recipe uses 
zc.buildout's fairly sophisticated egg support to tell Zope exactly 
which eggs you want it to know about (in this case, that's just the 
Plone egg, and all its dependencies, i.e. all of Plone). Hanno also made 
use of this recipe to create a wrapper around 'zopectl' that (a) works 
on Windows and (b) knows how to run tests in eggs.

The main difference, as I see it, is that zc.buildout (and thus by 
extension ploneout) takes a step back from the notion of a Zope 
instance. In the above, the Zope instance is one part. Well, the only 
part here, but the Zope 2 installation itself could be another part, as 
could 'zopepy', a python interpreter that is pre-configured to know all 
about Zope's PYTHONPATH so that you can run scripts and interactively 
execute code as if you were Zope (almost). If your environment has other 
requirements, e.g. let's say you were using WSGI to serve up parts of 
your application from another web server and you needed to build and 
configure that, you'd need an appropriate recipe and then wire it into 
your buildout.cfg.

For this reason, we also install the 'instance' script (the zopectl 
wrapper) in the top-level bin/ directory in the buildout. In fact, you 
tend not to deal with the Zope 2 instance directly at all. Instead, you 
tell buildout.cfg which eggs and development eggs you want it to 
install, you have a top-level products/ directory for Products you want 
Zope to find.

By contrast, ploneenv is very much focused around the instance itself. 
Wen you run it, it creates an instance and patches Zope in similar ways 
to the z2c.recipe.zope2instance recipe, to make it a bit more egg friendly.

Here are some things I like about the zc.buildout approach:

  - It's very expansible, meaning I'm pretty sure that if I had any 
particular thing I needed to set up as part of my development or 
deployment environment, I could write a recipe to do so. Things that are 
not solved by installing an egg are outside the scope of ploneenv from 
what I can tell. Of course, that may be YAGNI. :)

  - There is a single file that shows me what eggs and development eggs 
make up my package. This makes it easy to add new eggs, for example - 
and also easy to remove them again. With workingenv-based solutions, you 
can specify a file full of eggs to install when it's first set up, but 
from that point onwards, the environment can grow as you easy_install 
new things. It's not immediately clear to me how you reconcile all the 
eggs you've installed (not all of which may be needed when you're done 
trying things out, and some of which may just be dependencies you don't 
want to think about directly) into a list that are clearly dependencies 
of your application.

  - With workingenv, when I run easy_install SomePackage I need to worry 
about whether I'm actually in the global environment or the workingenv 
of the instance. That is, workingenv requires activation (putting your 
shell into a special state where the python environment is the one in 
your workingenv, until you deactivate it). By contrast, other people may 
not like the "add to buildout.cfg, re-run buildout dance" that buildout 
uses instead.

  - It works in Windows. :) I have no idea how hard it's to make 
ploneenv work on Windows, but I hope it's not too bad. The scripts Hanno 
wrote do give us a near-proper zopectl for Windows as well, which is 
nice. I imagine these could be adapted to be used with plain Zope 
instances, though. I assume workingenv gives us setuptools script 
support locally as well.

  - It has a convention for where development eggs go (src/) and how 
they're installed (mention them in 'develop-eggs' in buildout.cfg). It 
also explicitly prefers development eggs over regular eggs, so that if 
you're working on something, you don't have to worry about dependencies 
of other eggs taking precedence over the code you're trying to debug. Of 
course, ploneenv support development eggs as well (run python setup.py 
develop), again it's just less explicit.

  - Multiple developers can share a single buildout.cfg (e.g. in svn) 
and have a fully repeatable environment. This is essentially what 
ploneout tries to do for Plone developers.

  - It can build Zope 2 as well, if needed (this is the default, the 
example above specified an explicit path to an existing Zope installation).

Here are some things I don't always like:

  - Packages that have been easy_install'd in your global python 
environment are not available to Zope immediately. You have to tell 
buildout.cfg about them. Again, this comes back to control.

  - Some recipes can be a bit heavy-handed. For example, a first, naive 
implementation of the zope 2 install recipe would download and compile 
Zope 2 (from svn!) each time you ran buildout. We no longer do that :) 
However, recipes require proper testing to ensure that when you re-run 
buildout (which you do often) it behaves well.

  - Complex buildouts can be slow, especially if you don't run it in 
offline mode (bin/buildout -o), as it's checking for updates online.

  - People are used to Zope 2 instances being the centre of their world. 
With buildout, they are but one part.

I could imagine myself using both ploneenv and a buildout approach. For 
example, I would like to have a throwaway zope instance for testing new 
things - here, ploneenv could be great (since I can easy_install new 
things easily). If I were building something for a customer that I 
wanted to have a lot of control and visibility over, I think I would 
prefer making my own buildout. This would also reassure me that if I had 
more complex deployment scenarios I could handle them through a single 
system, in a way that would be repeatable across different environments.

In any case, I hope this has been useful to people. Please let me know 
if you think I've missed something or that I'm wrong on any point. :)

Martin