Migration profiles for add-on products?

Maurits van Rees m.van.rees at zestsoftware.nl
Sat May 12 01:51:55 UTC 2007


Martin Aspeli, on 2007-05-11:
> Maurits van Rees wrote:
>
> Now you've set my mind racing... :) I've got a revised counter-proposal 
> below that I'm starting to quite like.... very interested in your thoughts!

I am glad to keep your brain busy. ;-)

> I think the name SchemaManager is a bit misleading, though. Is it 
> managing an Archetypes schema? :)

I am not tied to the name 'SchemaManager'.  Something like
'UpgradeManager' might be more to the point.  What I am thinking about
when pondering an evolution step may be as simple as the addition of a
column or index to the portal_catalog.

As an example, when seen in the context of Plone, the upgrade from
beta 2 to beta 3 might mean going from generation 42 to 47.


As an aside, probably the UpgradeManager class that I define needs a
flag PleaseReindexTheCatalog_ThankYou that an upgrade step can set to
True when the catalog needs to be reindexed after all steps have been
taken.  If this is integrated in some control panel page then the page
template that is rendered should probably have a red div that says
that a reindexing is needed, together with a button to do just that.

> We are still relying on naming conventions. Also, the minimum_generation 
> thing is for zope.generation's auto-migration thing, which I don't think 
> can (or should) have an analogue in Plone.

I do not see much need for a minimum_generation either.

There might be made a case for one schema manager that handles
generations 0 through 12 for a product and a next for generations 13
through 19.  But I cannot think of a good use for that really.

> The other problem here is that users don't really think of 
> "generations", they think of versions, which are sequential things.

Generations are also sequential things.  I am really thinking of 1, 2,
3, 4, etcetera.  But yes: 'versions' could be a better name.  But then
as a user I would expect to see a generation/version of 1.1, 3.0, etc.
And my current rather easy idea is to simply use 0, 1, 2, etc as
profile names.  I like the simplicity of that.

In my theory these numbers are just names of GenericSetup profiles.
If after a while Plone has 531 of these profiles in one directory then
they can easily be moved to subdirectories, as long as they are
registered under their simple integer name.


BTW, I added 'collective.generations' to the collective.  (Feel free
to rename it.) But when I add this literally to my Products dir or my
lib/python dir as collective.generations I get import errors like
this:

    ImportError: No module named collective.generations

Should I rename it to a 'dot-less' name when putting it in my Products
dir or can I do this in a different way?

> We could solve this by being a bit more explicit:
>
> class IVersionManager(Interface):
>      """Keep track of the current version for a product
>      """
>
>      current_version = schema.TextLine("The current version")
>
>      product_name = schema.TextLine("Describe the product")
>
>      migration_steps = schema.Tuple("An ordered list of migration steps")

My current idea (subject to change) is that a version manager can
claim to handle evolutions to a *maximum* number of generations.  So
it would not keep a current version number itself.  The current
version/generation could be stored by a more general tool or utility
or property sheet.

At any rate /some/ code should know which version is currently
installed.  Something like this IVersionManager could do the trick.

> class IMigrationStep(Interface):
>      """Describes a migration
>      """
>
>      to_version = schema.ASCIILine("Migrates to version - displayed to 
> the user")
>
>      def __call__(portal):
>          """Execute this migration step
>          """

I think this can be handled by simply registering a GS profile with
'to_version' as the name..

> class VersionManager(object):
>      implements(IVersionManager)
>
>      def __init__(self, current, name, steps=[]):
>          self.current_version = current
>          self.product_name = name
>          self.steps = steps

So an instance of this class would need to be initialized with a list
of steps?  I think that might take too many lines of code.

But maybe we can add a method somewhere like listStepsFor(context)
where context is some package name or maybe an interface, like
IMigratingPloneSite.

BTW, can you think of a reason to have more than one VersionManager
for one Plone Product?  I do not think there are any reasons, but the
zope 3 implementation of generations does leave room for it, so I
wonder if there is a use case for that in Plone/CMF.

> class GSMigrationStep(object):
>       implements(IMigrationStep)
>
>       def __init__(self, version, profile):
>           self.to_version = version
>           self.profile = profile

This leaves room for GSMigrationSteps and something like
PlainPythonMigrationSteps.  This might be good or this might be giving
room for python migrations that are just as well rendered in the
importVarious step.

I propose supporting only GS steps.  Do you think that is too strict?

> ... now, in e.g. Poi, I'd do:
>
> version = VersionManager(
> 	name=u"Poi",
>          steps=[
> 		MigrationStep("1.1", "profile-Products.Poi:1.1"),
> 		MigrationStep("1.2", "profile-Products.Poi:1.2"),
> 		MigrationStep("1.3", "profile-Products.Poi:1.2"),
> 	]
> )

My proposal does not need this explicit registering.  Registering
profile-Products.Poi:1 implicitly registers a migration step from
generation 0 to 1.

> provideUtility(version, name=u"Products.Poi") # do this with ZCML

I would then vote for a name like
Products.Poi.generations/migration/installation to make the goal of
the utility clearer.  But that is a minor point.

> So, we have one global utility, named after the product by convention. 
> We specify the name of the product to display in the migration UI.

In my proposal we would initialize a SchemaManager and have an argument
package_name that also is the product name by convention/necessity.

> The migration steps are listed explicitly, in order. The "latest" 
> version is the last one in the list (remember, it specifies "versions 
> being migrated to"). That makes sense - it's the last thing you can 
> migrate to.

In my proposal we could probably do away with the minimum_generation
as required by the base zope 3 generations and only state the number
of the last generation.

> We reference a profile by name (we could simplify by putting the product 
> name in the VersionManager, and/or defaulting the profile name to 
> profile=${product_name}:${version}, but I think it makes sense to be 
> able to specify one explicitly.

You can already specify it explicitly by registering some directory
under a specific name. 

> When we migrate current version to 1.3, say, it goes through the list 
> from the current/pre-migration version, and does each step up until and 
> including the last one.

I am inclined to have several migration steps between version 1.2 and
1.3.  That may be more guided by a wish to make testing with a
production database during development easier though.

If you have a zope/plone site that uses the svn trunk of a
Product and the official version is 1.2 but there are some

Let's say you have a production site that runs on Plone 2.5.2, which
was out in January of this year.  Meanwhile there are some migration
steps.  Let's say there are two migration steps defined already (one
column added to the catalog and one skin directory change).  With my
strategy you could perform those steps already.  With the current
Plone migration strategy I think either you can perform those steps
now and cannot perform future steps to 2.5.3 easily or you cannot
perform the currently available steps.

That is my idea of Plone migration at least: migrating from the latest
stable version of 2.5 to the subversion branch of 2.5 is a bit iffy.

And I think your strategy is closer to the current plone strategy.
That may just be a definition of what constitutes a
generation/migration though: I do not think your strategy mandates the
evolution from one official version to the next but can also handle
intermediate changes.

> Adding a new migration step means adding another element to that list, 
> which should be pretty simple.

Yes.

> To me, this is easier to understand and manager than sequential ints 
> that have no tie to the actual version number I'm releasing, and will 
> make more sense to users.

You can register a GenericSetup profile in
migrations/v3_0/profiles/beta1_beta2 under the name 42 and still give
if a readable title and description.

Maybe we could define a SchemaManager for say profile numbers 42
through 47 that handles the upgrade between beta 1 and beta 2.

> Of course, you may then end up with "pseudo-versions" if you have a need 
> for migrations in-betwen releases, but I think this is a minor need, and 
> that people should be making more releases anyway. :) But we do need to 
> make sure it's possible to "force migrate" from a version.
>
> The missing pieces now are:
>
>   - a local utility which keeps track of the current/last migrated 
> version of a package

A property sheet could do for now, but I agree some utility is
probably better.

>   - some standard way of refreshing this if a product is re-installed 
> from scratch

When reinstalling we could look at the current generation and simply
apply all GenericSetup steps from that generation onwards.

Or we could just apply the default profile.

Uninstalling should set the generation to zero or completely erase
that value.

>   - UI for this, including the ability to force-migrate from a given version
>
> In general, I don't like encoding meaning into names of things. It's too 
> easy to get it wrong. Also, I find strings that contain version numbers 
> really hard to read and easy to get wrong: "migrate-1.1a2-2.0rc1" - yipes!

I think the following should then suit you just fine:

profile-Products.Poi:0
profile-Products.Poi:1
profile-Products.Poi:2
profile-Products.Poi:3
profile-Products.Poi:4
profile-Products.Poi:5

;-)

>>> then, you register a profile, like Products.Poi:migrate-1.0-2.0 and
>>> Products.Poi:migration-2.0-2.1
>> 
>> That would be something like this:
>> 
>>   <genericsetup:registerProfile
>>       name="3"
>>       title="Migration profile for eXtremeManagement generation 2 to 3."
>>       description="This profile contains configuration changes that
>>                    are applied during the migration from
>>                    eXtremeManagement generation 2 to 3, which is part
>>                    of the change from 1.1 to 1.5."
>>       directory="migrations/v1.5"
>>       provides="Products.GenericSetup.interfaces.MIGRATION"
>>       />
>> 
>> Note that Products.GenericSetup.interfaces.MIGRATION does not
>> currently exist.  It is just an idea.  But that is not really an issue
>> here.
>
> It can be collection.migration.interfaces.IMigrationProfiles. No need 
> for it to be in CMF Core. :)

True.

If we want some support for this in portal_setup though like the code
I wrote earlier then it should be in CMF core.  But that may not be
necessary.

>> No utilities need to be registered with the collective.generations
>> approach.  That one SchemaManager should be enough.  Maybe that can be
>> done in zcml though if wanted.
>
> I don't see any benefit of doing this with ZCML at this point, just more 
> work for you. :)

Agreed.

>>> It ought to be an explicit action in Install.py when reinstall=True, but
>>> that sounds sensible, yes.
>> 
>> Extensions/install.py could have something like this:
>> 
>>   from Products.collective.generations import UpgradeManager
>>   from Products.collective.generations import SchemaManager
>> 
>>   sm = SchemaManager(
>>       minimum_generation = 0,
>>       generation = 5,
>>       package_name = 'Products.eXtremeManagement'
>>       )
>> 
>> 
>>   def install(context):
>>       um = UpgradeManager(context) # or getUtility/getToolByName
>>       um.install(sm)
>
> It may make more sense if this was either an adapter or a utility, 
> rather than a helper class. I think a utility makes more sense, even if 
> we have to pass in the context:
>
> migrations = getUtility(IMigrationManager)
> migrations.migrate_to_current_version(portal, "Products.Poi")

Looks good.

>> And during a reinstall it is probably enough to evolve from the
>> current iteration to the maximum defined generation, using this
>> method:
>> 
>>       um.evolve_maximum(sm)
>>       
>> I do not know if this works, but I *am* beginning to like the
>> approach. :-)
>
> I'm beginning to like it less. :) The more I think about it, the more 
> wrong it seems to tie this to an arbitrary, incremented integer. Users 
> and developers think in terms of releases, which have unique and 
> sequential versions. There's lots of scope for things to go subtly 
> pear-shaped during migration, and being explicit and keeping things 
> simple seems a good general principle.

Knowing that evolving to generation 13 goes wrong may be of more help
in debugging than knowing that the switch from 1.1 to 1.2 spanning 5
generations went wrong though.

But registering some migration steps in a different way and sorting
them by name could work as well.

> Also, I'm not sure I see the need for the min/max generation stuff 
> (which is easier to do with integer generations, for sure). From what i 
> remember, zope.generation does it so that you can do auto-migration on 
> Zope startup for "Zope won't even run properly if you don't do this" 
> kind of migrations. That's fine for fundamental Zope things, but I don't 
> think auto-migration is *ever* a good idea in the context of Plone, let 
> alone technically difficult (when do you trigger it?).

Agreed, auto-migration or something else with minimum_generations seem
unworkable for Plone.

Well, let's try an example:

- Generation 7 of an application is currently installed.

- Generation 8 introduces an extra column in the catalog, which is
  needed by several templates in that version.

- Generation 9 introduces an extra workflow state for a content type.

In this case it would make sense to auto-migrate to generation 8 as
otherwise you can get errors when viewing a page.  Evolving to
generation 9 is optional.

So: there can be situations where not migrating/evolving can result in
errors and also situations where not evolving has no unwanted effects.

Usually it would be logical to go to the latest generation though and
to let the user do it explicitly.

> Besides that, the only migration that ever makes sense to invoke 
> explicitly is from "current version" to "latest version", with a 
> fallback option of explicitly choosing the "current version" if the 
> system's lost track. This is how Plone's migration has worked for years, 
> and I think this is how people expect it to be.
>
> Thoughts?

I agree.

-- 
Maurits van Rees | http://maurits.vanrees.org/ [NL]
            Work | http://zestsoftware.nl/
"Do not worry about your difficulties in computers,
 I can assure you mine are still greater."





More information about the Product-Developers mailing list