Migration profiles for add-on products?

[Martin Aspeli]

Maurits van Rees wrote:

Now you've set my mind racing... :) I've got a revised counter-proposal 
below that I'm starting to quite like.... very interested in your thoughts!

>> class Migration(object):
>>      implements(IMigration)
>>      def __init__(self, id, to_version, profile):
>>          self.id = self
>>          self.to_version = version
>>          self.profile = profile
>>
> 
> Let's compare this to my untested code in
> https://svn.plone.org/svn/collective/collective.generations/trunk
> 
>   from Products.collective.generations import SchemaManager
> 
>   sm = SchemaManager(
>       minimum_generation = 0,
>       generation = 5,
>       package_name = 'Products.eXtremeManagement'
>       )

I think the name SchemaManager is a bit misleading, though. Is it 
managing an Archetypes schema? :)

We are still relying on naming conventions. Also, the minimum_generation 
thing is for zope.generation's auto-migration thing, which I don't think 
can (or should) have an analogue in Plone.

The other problem here is that users don't really think of 
"generations", they think of versions, which are sequential things.

We could solve this by being a bit more explicit:

class IVersionManager(Interface):
     """Keep track of the current version for a product
     """

     current_version = schema.TextLine("The current version")

     product_name = schema.TextLine("Describe the product")

     migration_steps = schema.Tuple("An ordered list of migration steps")

class IMigrationStep(Interface):
     """Describes a migration
     """

     to_version = schema.ASCIILine("Migrates to version - displayed to 
the user")

     def __call__(portal):
         """Execute this migration step
         """

class VersionManager(object):
     implements(IVersionManager)

     def __init__(self, current, name, steps=[]):
         self.current_version = current
         self.product_name = name
         self.steps = steps

class GSMigrationStep(object):
      implements(IMigrationStep)

      def __init__(self, version, profile):
          self.to_version = version
          self.profile = profile



... now, in e.g. Poi, I'd do:

version = VersionManager(
	name=u"Poi",
         steps=[
		MigrationStep("1.1", "profile-Products.Poi:1.1"),
		MigrationStep("1.2", "profile-Products.Poi:1.2"),
		MigrationStep("1.3", "profile-Products.Poi:1.2"),
	]
)

provideUtility(version, name=u"Products.Poi") # do this with ZCML

So, we have one global utility, named after the product by convention. 
We specify the name of the product to display in the migration UI.

The migration steps are listed explicitly, in order. The "latest" 
version is the last one in the list (remember, it specifies "versions 
being migrated to"). That makes sense - it's the last thing you can 
migrate to.

We reference a profile by name (we could simplify by putting the product 
name in the VersionManager, and/or defaulting the profile name to 
profile=${product_name}:${version}, but I think it makes sense to be 
able to specify one explicitly.

When we migrate current version to 1.3, say, it goes through the list 
from the current/pre-migration version, and does each step up until and 
including the last one.

Adding a new migration step means adding another element to that list, 
which should be pretty simple.

To me, this is easier to understand and manager than sequential ints 
that have no tie to the actual version number I'm releasing, and will 
make more sense to users.

Of course, you may then end up with "pseudo-versions" if you have a need 
for migrations in-betwen releases, but I think this is a minor need, and 
that people should be making more releases anyway. :) But we do need to 
make sure it's possible to "force migrate" from a version.

The missing pieces now are:

  - a local utility which keeps track of the current/last migrated 
version of a package

  - some standard way of refreshing this if a product is re-installed 
from scratch

  - UI for this, including the ability to force-migrate from a given version

In general, I don't like encoding meaning into names of things. It's too 
easy to get it wrong. Also, I find strings that contain version numbers 
really hard to read and easy to get wrong: "migrate-1.1a2-2.0rc1" - yipes!

>> then, you register a profile, like Products.Poi:migrate-1.0-2.0 and
>> Products.Poi:migration-2.0-2.1
> 
> That would be something like this:
> 
>   <genericsetup:registerProfile
>       name="3"
>       title="Migration profile for eXtremeManagement generation 2 to 3."
>       description="This profile contains configuration changes that
>                    are applied during the migration from
>                    eXtremeManagement generation 2 to 3, which is part
>                    of the change from 1.1 to 1.5."
>       directory="migrations/v1.5"
>       provides="Products.GenericSetup.interfaces.MIGRATION"
>       />
> 
> Note that Products.GenericSetup.interfaces.MIGRATION does not
> currently exist.  It is just an idea.  But that is not really an issue
> here.

It can be collection.migration.interfaces.IMigrationProfiles. No need 
for it to be in CMF Core. :)

> No utilities need to be registered with the collective.generations
> approach.  That one SchemaManager should be enough.  Maybe that can be
> done in zcml though if wanted.

I don't see any benefit of doing this with ZCML at this point, just more 
work for you. :)

>> It ought to be an explicit action in Install.py when reinstall=True, but
>> that sounds sensible, yes.
> 
> Extensions/install.py could have something like this:
> 
>   from Products.collective.generations import UpgradeManager
>   from Products.collective.generations import SchemaManager
> 
>   sm = SchemaManager(
>       minimum_generation = 0,
>       generation = 5,
>       package_name = 'Products.eXtremeManagement'
>       )
> 
> 
>   def install(context):
>       um = UpgradeManager(context) # or getUtility/getToolByName
>       um.install(sm)

It may make more sense if this was either an adapter or a utility, 
rather than a helper class. I think a utility makes more sense, even if 
we have to pass in the context:

migrations = getUtility(IMigrationManager)
migrations.migrate_to_current_version(portal, "Products.Poi")

> And during a reinstall it is probably enough to evolve from the
> current iteration to the maximum defined generation, using this
> method:
> 
>       um.evolve_maximum(sm)
>       
> I do not know if this works, but I *am* beginning to like the
> approach. :-)

I'm beginning to like it less. :) The more I think about it, the more 
wrong it seems to tie this to an arbitrary, incremented integer. Users 
and developers think in terms of releases, which have unique and 
sequential versions. There's lots of scope for things to go subtly 
pear-shaped during migration, and being explicit and keeping things 
simple seems a good general principle.

Also, I'm not sure I see the need for the min/max generation stuff 
(which is easier to do with integer generations, for sure). From what i 
remember, zope.generation does it so that you can do auto-migration on 
Zope startup for "Zope won't even run properly if you don't do this" 
kind of migrations. That's fine for fundamental Zope things, but I don't 
think auto-migration is *ever* a good idea in the context of Plone, let 
alone technically difficult (when do you trigger it?).

Besides that, the only migration that ever makes sense to invoke 
explicitly is from "current version" to "latest version", with a 
fallback option of explicitly choosing the "current version" if the 
system's lost track. This is how Plone's migration has worked for years, 
and I think this is how people expect it to be.

Thoughts?

Martin