plone/products.contentmigration

Generic content migration framework helping to write custom content migrations

Content migration utilities

by Martin Aspeli <[email protected]> and others

Released under the GNU Lesser General Public License (LGPL) version 2.0

This is a generic content migration framework, which should help you write your own content migrations. It has no UI and no value on its own, but makes it easy to write certain type of content migrations.

This replaces the ATContentTypes migration framework, and provide three useful extensions:

• A CustomQueryWalker can be used to specify a more specific catalog query for a walker to use (e.g. which content to actually migrate). This can be used with any migrator.

• A BaseInlineMigrator is similar to BaseMigrator, but does not migrate by copying the old object to a temporary location, creating a new object and applying migration methods. Instead, migration methods are applied in-place. This simplifies the code significantly, because attributes, local roles etc. does not need to be copied over.

  Note that whereas BaseMigrator works in terms of self.old and self.new as the objects being migrated, BaseInlineMigrator only has a single object, stored in self.obj. This can be used with any walker.

• An extension of this class called FieldActionMigrator uses the action-based migration framework for Archetypes fields, found in field.py. Please refer to that file for full details, but briefly, you specify a list of attributes to migrate at the storage level, instructing the migrator whether to rename, transform, unset or change the storage for an attribute.

Please see the docstrings in walker.py, migrator.py and field.py for full details. For examples, see tests/cmtc.py and tests/testATFieldMigration.py.

products.contentmigration's Issues

Ordering with migrate_children sometimes inaccurate

https://github.com/plone/Products.contentmigration/blob/master/Products/contentmigration/basemigrator/migrator.py#L598

Trying to reorder a folder's contents based on a dictionary (non-ordered) mapping can sometimes lead to unintended behavior. Consider the following example:

>>> self.orderMap
>>> {'0923.docx': 2, '1217.docx': 4, '0416.docx': 1, '0326.docx': 5, '0129.docx': 0, '1029.docx': 3}
>>> self.new.objectIds()
>>> ['0923.docx', '1217.docx', '0416.docx', '0326.docx', '0129.docx', '1029.docx']

The result is the following for self.new.objectIds():

['0129.docx', '0923.docx', '0416.docx', '1029.docx', '1217.docx', '0326.docx']

Notice that 0923.docx comes before 0416.docx instead of vice-versa. You can trace through step by step to get a better understanding of why this is if you want, but basically this is just not an accurate sort algorithm. What we can do is sort the dict items first:

>>> orderMap = sorted(self.orderMap.items(), key=lambda term: term[1])
>>> for id, pos in orderMap:
>>>     self.new.moveObjectToPosition(id, pos)

Folder migration duplicate indexing of children unindexes AT references

The folder migrator avoids redundant reindexing by removing objects with _delOb instead manage_delObject but it uses _setObject instead of _setOb which does do reindexing. Aside from the redundant indexing slowing down migrations, this also unindexes AT references which interferes with work to fix reference migration. There's a comment saying it's necessary but I'm not aware of any reason why it would be.

CatalogWalkerWithLevel incompatible with LinguaPlone

If more than one language is installed, the query for level 1 will include Language='all', the query for level 2 will include Language=None, level 3 and onwards will have Language=['en',''], or whatever you have configured as your preferred Language, thereby skipping almost all content in non-preferred languages.

(Boiled-down example to be supplied later.)

Root cause: the query dictionary in CatalogWalkerWithLevel's walk is passed "by-reference", and LinguaPlone's catalog patch will modify it in-place.

I'm filing it here because the quick fix is, AFAICT, to call catalog(**dict) or catalog(dict.copy()) instead. (Feel free to argue that LP should copy the dict itself and mangle and use the copy, I won't disagree.)

After migrating from plone.app.contenttypes 1.0 to 1.1.1, all the events are private and with mangled title

The simulation was the same as the one in plone/plone.app.contenttypes#414.

Had to call self.new.reindexObject() after https://github.com/plone/Products.contentmigration/blob/2.1.11/Products/contentmigration/basemigrator/migrator.py#L416 in a patch to avoid the error.

Flickering test on Jenkins

#### Running tests for Products.contentmigration ####
Running Products.contentmigration.tests.layer.PloneContentMigrationTestCase:Functional tests:
  Set up plone.testing.zca.LayerCleanup in 0.000 seconds.
  Set up plone.testing.z2.Startup in 0.179 seconds.
  Set up plone.app.testing.layers.PloneFixture /home/jenkins/.buildout/eggs/setuptools-5.8-py2.7.egg/pkg_resources.py:1032: UserWarning: /home/jenkins/.python-eggs is writable by group/others and vulnerable to attack when used with get_resource_filename. Consider a more secure location (set with .set_extraction_path or the PYTHON_EGG_CACHE environment variable).
in 9.999 seconds.
  Set up plone.app.testing.bbb.PloneTestCaseFixture in 2.143 seconds.
  Set up Products.contentmigration.tests.layer.PloneTestCaseFixture in 1.450 seconds.
  Set up Products.contentmigration.tests.layer.PloneContentMigrationTestCase:Functional in 0.000 seconds.


Failure in test /home/jenkins/workspace/plone-5.0-python-2.7-at/src/Products.contentmigration/Products/contentmigration/tests/../archetypes.txt
Failed doctest test for archetypes.txt
  File "/home/jenkins/workspace/plone-5.0-python-2.7-at/src/Products.contentmigration/Products/contentmigration/tests/../archetypes.txt", line 0

----------------------------------------------------------------------
File "/home/jenkins/workspace/plone-5.0-python-2.7-at/src/Products.contentmigration/Products/contentmigration/tests/../archetypes.txt", line 85, in archetypes.txt
Failed example:
    ff_migrator.migrate()
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1315, in __run
        compileflags, 1) in test.globs
      File "<doctest archetypes.txt[47]>", line 1, in <module>
        ff_migrator.migrate()
      File "/home/jenkins/workspace/plone-5.0-python-2.7-at/src/Products.contentmigration/Products/contentmigration/basemigrator/migrator.py", line 220, in migrate
        method()
       - __traceback_info__: (<Products.contentmigration.archetypes.ATFolderMigrator object at 0x7f687c8d0090>, <bound method ATFolderMigrator.migrate_extension_fields of <Products.contentmigration.archetypes.ATFolderMigrator object at 0x7f687c8d0090>>, <ATFolder at /plone/fold2_MIGRATION_>, 'fold2')
      File "/home/jenkins/workspace/plone-5.0-python-2.7-at/src/Products.contentmigration/Products/contentmigration/archetypes.py", line 203, in migrate_extension_fields
        self.new.update(**kwargs)
      File "/home/jenkins/.buildout/eggs/Products.Archetypes-1.10.4-py2.7.egg/Products/Archetypes/BaseObject.py", line 449, in update
        self.Schema().updateAll(self, **kwargs)
      File "/home/jenkins/.buildout/eggs/Products.Archetypes-1.10.4-py2.7.egg/Products/Archetypes/Schema/__init__.py", line 528, in updateAll
        method(kwargs[name])
      File "/home/jenkins/.buildout/eggs/Products.Archetypes-1.10.4-py2.7.egg/Products/Archetypes/BaseObject.py", line 216, in setId
        parent.manage_renameObject(self.id, value)
      File "/home/jenkins/workspace/plone-5.0-python-2.7-at/src/Products.CMFPlone/Products/CMFPlone/PloneFolder.py", line 84, in manage_renameObject
        result = method(self, id, new_id, REQUEST)
      File "/home/jenkins/.buildout/eggs/Zope2-2.13.22-py2.7.egg/OFS/CopySupport.py", line 348, in manage_renameObject
        action ='manage_main'))
    CopyError: 
    <HTML>
    <HEAD>
    <TITLE>Invalid Id</TITLE>
    </HEAD>
    <BODY BGCOLOR="#FFFFFF">
    <FORM ACTION="manage_main" METHOD="GET" >
    <TABLE BORDER="0" WIDTH="100%" CELLPADDING="10">
    <TR>
      <TD VALIGN="TOP">
      <BR>
      <CENTER><B><FONT SIZE="+6" COLOR="#77003B">!</FONT></B></CENTER>
      </TD>
      <TD VALIGN="TOP">
      <BR><BR>
      <CENTER>
      The id "fold2_MIGRATION_" is invalid - it is already in use.
      </CENTER>
      </TD>
    </TR>
    <TR>
      <TD VALIGN="TOP">
      </TD>
      <TD VALIGN="TOP">
      <CENTER>
      <INPUT TYPE="SUBMIT" VALUE="   Ok   ">
      </CENTER>
      </TD>
    </TR>
    </TABLE>
    </FORM>
    </BODY></HTML>

  Ran 14 tests with 1 failures, 0 errors and 0 skipped in 4.513 seconds.
Tearing down left over layers:
  Tear down Products.contentmigration.tests.layer.PloneContentMigrationTestCase:Functional in 0.000 seconds.
  Tear down Products.contentmigration.tests.layer.PloneTestCaseFixture in 0.011 seconds.
  Tear down plone.app.testing.bbb.PloneTestCaseFixture in 0.004 seconds.
  Tear down plone.app.testing.layers.PloneFixture in 0.082 seconds.
  Tear down plone.testing.z2.Startup in 0.006 seconds.
  Tear down plone.testing.zca.LayerCleanup in 0.001 seconds.
#### Finished tests for Products.contentmigration ####

P.c doesn't handle portlets

Migrated content loses all locally-assigned portlets.
Source for 2.1.9 doesn't contain the string portl nor col, so I assume there's simply no attempt to copy it. Even if the name of the manager needed to be hard-coded, having code for the standard side columns would give a person a good starting point.

additionalQuery breaks test-isolation

Using additionalQuery in a test poisons CustomQueryWalker.additionalQuery for following tests and layers. I'll remove it as a class-attribute to keep it contained. The feature added in 5b2e942 is now unused anyway.