[Setup] keyError on upgrade, path/mount point issues

John Schinnerer john at eco-living.net
Fri Dec 18 21:51:19 UTC 2015


I've managed to get past this - posting my solution in case it helps 
someone else somewhere somehow sometime.

By setting the sub-transactions threshold in the portal_catalog Advanced 
tab to a relatively small number, I was able to get catalog update, and 
catalog clear/rebuild, to finish successfully.

Default appeared to be 10000; I tried 1000 but it still failed with the 
"too many files open" error in the log.
I just took it down to 100 for a next try and that worked. This was on a 
test/dev install of the site so it didn't matter much if that setting 
would have a negative effect on the production site. It is only for 
getting through the upgrade tangles and challenges.

Once the catalog update and clear/rebuild were done, I copied the 
Data.fs from the 3.3.5 site to the 4.0.8 site and the upgrade completed 
successfully. No more keyError.

cheers,
John S.


On 12/15/2015 02:35 PM, John Schinnerer wrote:
> Hi again,
>
> Here is what I was hoping to work around, possibly with cut/copy and
> paste of the site.
>
> On 12/15/2015 11:19 AM, Steve McMahon wrote:
>
>> Other strategies:
>>
>> * Set up a mirror of the old site (using backup data) and simply delete
>> everything but the target site. If you want to combine it others at the
>> end of the update process, you may use mount points.
>
> Not the problem (multiple sites in one Data.fs). It's just one site in
> one Data.fs I'm dealing with here.
>
>> * Use Zope's export/import facility, restoring to a mirror of the old
>> setup (same versions/add-ons). If you restore to a different folder, be
>> ready to rebuild the catalog.
>
> Tried this also. The export works OK; import hits the ImportError I
> mentioned in previous post - looking for something in
> collective.recaptcha, removed in upgrade because in the 3.3.5 it causes
> an error (with a huge mass of code output, not a typical orderly
> traceback) on every page render attempt.
>
> The bottom line upgrade problem, which I've failed to work around with
> export/import or cut/copy-paste, is that when I put the 3.3.5 Data.fs
> into the 4.0.8 instance and dry run the upgrade, I get a keyError on a
> traversal. Specifically:
>
> [lots of upgrade steps completing OK]
> ...
> Ran upgrade step: Upgrade DCMI metadata storage in portal_metadata
> Ran upgrade step: Add the recursive groups PAS plugin.
> Upgrade aborted. Error:
> Traceback (most recent call last):
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Plone-4.0.8-py2.6.egg/Products/CMFPlone/MigrationTool.py",
> line 175, in upgrade
> step['step'].doStep(setup)
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Products.GenericSetup-1.6.3-py2.6.egg/Products/GenericSetup/upgrade.py",
> line 142, in doStep
> self.handler(tool)
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
> line 480, in updateLargeFolderType
> update(brain)
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/plone.app.upgrade-1.0.8-py2.6.egg/plone/app/upgrade/v40/alphas.py",
> line 473, in update
> obj = brain.getObject()
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/Products/ZCatalog/CatalogBrains.py",
> line 83, in getObject
> parent = parent.unrestrictedTraverse(path[:-1])
> File
> "/opt/Plone-4.0.8-fresh/buildout-cache/eggs/Zope2-2.12.19-py2.6-linux-x86_64.egg/OFS/Traversable.py",
> line 293, in unrestrictedTraverse
> raise e
> KeyError: 'puna-cdp'
>
> The most likely explanation I have found so far is in this thread:
> http://sourceforge.net/p/plone/mailman/message/27276817/
>
> Specifically the part about paths to objects not matching:
> "The error you are getting is...happening while the migration script is
> trying to find a particular piece of content...by moving through the
> path elements leading to that object. 'KeyError'...is raised when
> dictionary lookup on an object fails to find an expected key in the
> dictionary...you used to have the site on a separate Data.fs...on a
> mountpoint...my guess is that the ZODB has persisted path information
> for that site and everything in it that includes the mountpoint...it may
> be that the ZODB and its catalog think the site is still in the old
> location and are looking for it there. Since it is not there, you get a
> KeyError."
>
> This Data.fs I am trying to upgrade has one migration/upgrade in its
> history that I did not do. It was a server OS upgrade during which my
> sysadmin did something while reinstalling/re-configuring that led to a
> path issue.
>
> Before, there was a ZODB mount point 'domain.tld' in the ZMI root folder
> (configured in buildout as usual) and in that 'domain.tld' folder was
> the plone site, like:
> /domain.tld/plone-site
>
> After, that somehow changed to have one more 'domain.tld' folder in the
> path in the ZMI, like:
> /domain.tld/domain.tld/plone-site
>
> BUT on the file system (as configured in buildout all along) the mount
> point was still just
> domain.tld/Data.fs
>
> In other words, when I copy this Data.fs into
> domain.tld/Data.fs
> on the file system, the result is
> domain.tld/domain.tld/plone-site
> in the ZMI.
>
> I've tried to fix by export/import and recatalog, but import hits the
> collective.recaptcha import error (with collective.recaptcha breaking
> the 3.3.5 site if I add it). I tried leaving collective.recaptcha in and
> dry-running upgrade, just to see what would happen. That fixed the
> collective.recaptcha import error but then gave an ImportError on
> registry.interfaces. No idea why or what to do with that.
>
> Then I tried to fix by cut/copy paste to move the site and then
> recatalog, with the copy/paste failing on import for same reasons as
> above, and the cut/paste failing for reasons yet TBD.
>
> I also tried to create the duplicate-directory mount point path in the
> 4.0.8 instance, without success.
>
> If I have this in buildout:
> ---------------
> [filestorage]
> recipe = collective.recipe.filestorage
> blob-storage = var/blobstorage-%(fs_part_name)s
> parts =
>      domain.tld
> ----------------
>
> ...it works just like in the 3.3.5 instance, namely, the filesystem
> mount point path is
> domain.tld/Data.fs
>
> I tried changing it to
>
> ---------------
> ...
> parts =
>      domain.tld/domain.tld
> ------------------
>
> ...but that gave me a triplicate path on the file system!
> In other words, when I went to add the mount point in the ZMI, it showed as
> domain.tld/domain.tld/domain.tld
>
> No idea why or if that's a bug (seems like one, but maybe it's not even
> allowed to have a multi-part path for the 'parts' declaration, or the /
> has to be escaped, or...?).
> In any case, it didn't get me a duplicate of the internal path, so I
> don't know if that would work if it did.
>
> So...what to try next?
> I am about out of ideas.
> Or, I'm thinking of something rather over my head, like a script that
> would go through the whole catalog and fix the path strangeness, remove
> the duplicate path element, something like that (if that's even possible).
>
> Any assistance appreciated.
>
> thanks,
> John S.
>
>
>> On Tue, Dec 15, 2015 at 10:59 AM, John Schinnerer <john at eco-living.net
>> <mailto:john at eco-living.net>> wrote:
>>
>>     Hi,
>>
>>     I think this is the place for install/upgrade issues. If not please
>>     direct me to the appropriate place.
>>
>>     Plone 3.3.5 is the platform here, but the reason for the question is
>>     to facilitate upgrading a 3.3.5 site to 4.x.
>>
>>     Is it possible to cut/copy and paste a "large" plone site, within a
>>     given zope instance that has one or more existing plone sites?
>>
>>     Also, is there a way to verify that "clear and rebuild" and "update"
>>     of catalog (portal_catalog -> Advanced tab) complete, or not, for a
>>     "large" site?
>>
>>     "Large" in this case means a Data.fs of ~5.7GB.
>>
>>     I am asking because I have started "clear and rebuild" and "update"
>>     operations multiple times, but after some long time of the browser
>>     "waiting for domain.tld...," the right pane of the ZMI refreshes
>>     with a "connection refused."
>>
>>     I am getting the same result when I try and copy or cut and paste
>>     the site. In that case it seems clear the operation failed at some
>>     point, since the site remains in the original location when I
>>     refresh the ZMI tab (was not moved or copied).
>>
>>     Does one of the logs track these operations? Or is there any other
>>     way to find out what is actually happening, especially with the
>>     catalog operations?
>>
>>     thanks,
>>     John S.
>>
>>     --
>>     John Schinnerer - M.A., Whole Systems Design
>>     --------------------------------------------
>>     - Eco-Living -
>>     Whole Systems Design Services
>>     People - Place - Learning - Integration
>>     john at eco-living.net <mailto:john at eco-living.net> - 510.982.1334
>>     <tel:510.982.1334>
>>     http://eco-living.net
>>     http://sociocracyconsulting.com
>>     _______________________________________________
>>     Setup mailing list
>>     Setup at lists.plone.org <mailto:Setup at lists.plone.org>
>>     https://lists.plone.org/mailman/listinfo/plone-setup
>>
>>
>>
>>
>> _______________________________________________
>> Setup mailing list
>> Setup at lists.plone.org
>> https://lists.plone.org/mailman/listinfo/plone-setup
>>
>

-- 
John Schinnerer - M.A., Whole Systems Design
--------------------------------------------
- Eco-Living -
Whole Systems Design Services
People - Place - Learning - Integration
john at eco-living.net - 510.982.1334
http://eco-living.net
http://sociocracyconsulting.com


More information about the Setup mailing list