zopefoundation/products.genericsetup

Mini-framework for expressing the configured state of a Zope Site as a set of filesystem artifacts

products.genericsetup's Issues

TypeError: '>' not supported between instances of 'NoneType' and 'NoneType'

Plone 5.2RC5, GS 2.0

    setup_layer(options, layer, setup_layers)
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.testrunner-5.0-py3.7.egg/zope/testrunner/runner.py", line 787, in setup_layer
    setup_layer(options, base, setup_layers)
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.testrunner-5.0-py3.7.egg/zope/testrunner/runner.py", line 792, in setup_layer
    layer.setUp()
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/plone.app.testing-6.1.3-py3.7.egg/plone/app/testing/helpers.py", line 374, in setUp
    self.setUpZope(portal.getPhysicalRoot(), configurationContext)
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/src/ugent.vacancy/src/ugent/vacancy/tests/base.py", line 37, in setUpZope
    xmlconfig.file("configure.zcml", mod, context=configurationContext)
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.configuration-4.3.1-py3.7.egg/zope/configuration/xmlconfig.py", line 659, in file
    context.execute_actions()
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.configuration-4.3.1-py3.7.egg/zope/configuration/config.py", line 800, in execute_actions
    None, sys.exc_info()[2])
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.configuration-4.3.1-py3.7.egg/zope/configuration/_compat.py", line 30, in reraise
    raise value.with_traceback(tb)
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.configuration-4.3.1-py3.7.egg/zope/configuration/config.py", line 791, in execute_actions
    callable(*args, **kw)
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0-py3.7.egg/Products/GenericSetup/registry.py", line 438, in registerStep
    if already and already['version'] > version:
zope.configuration.config.ConfigurationExecutionError: File "/home/ajung/sandboxes/ugent-portaal-plone-4x/src/ugent.ldap/src/ugent/ldap/configure.zcml", line 92.2-97.6
    <genericsetup:importStep
      name="ugent.ldap-setuphandlers"
      title="ugent.ldap Setuphandlers"
      description="Additional import step for package ugent.ldap"
      handler="ugent.ldap.setuphandlers.import_various"
      />

    TypeError: '>' not supported between instances of 'NoneType' and 'NoneType'

The version parameter and the already['version'] are both None.

Possibly a bad package configuration. However the error message is not directly helpful here.

(Pdb) args
self = <Products.GenericSetup.registry.ImportStepRegistry object at 0x7efc0eae2860>
id = 'ugent.ldap-setuphandlers'
version = None
handler = <function import_various at 0x7efc06d16158>
dependencies = ()
title = 'ugent.ldap Setuphandlers'
description = 'Additional import step for package ugent.ldap'
(Pdb) pp already
{'dependencies': (),
 'description': 'Additional import step for package ugent.ldap',
 'handler': 'ugent.ldap.setuphandlers.import_various',
 'id': 'ugent.ldap-setuphandlers',
 'invalid': False,
 'title': 'ugent.ldap Setuphandlers',
 'version': None}
(Pdb) 

Question: is non-utf8 supported on Python 3?

While creating PR #110, I ran into code and tests that tried to support other encodings, for example iso-8859-1, at least when importing an XML file. But some parts of the code ignore this on Python 3.

So I wonder: does GenericSetup on Python 3 only support utf-8?
If so, then some code could be simplified, for example the code in that PR.
But simplifying may need to wait until we drop support for Python 2, and/or we require Zope 5.3, where lines properties are text instead of bytes.

I wonder the same about Zope 5: does that only support bytes encoded as utf-8, at least when stored in the Data.fs?

Note: since Plone 4.3, core Plone assumes all bytes that are read from Data.fs are utf-8 encoded (or simple ascii), where previously you could set a default_charset property (preferably only set it once, never change it). See this commit from 2012.

ZMI uses old markup/classes

Fix management/ZMI markup to follow Zope 4+ /Bootstrap style:

• manage_tool
• manage_fullImport #112
• manage_exportSteps #112
• manage_upgrades #112
• manage_importSteps #112
• manage_tarballImport
• manage_snapshots
• manage_showDiff
• manage_stepRegistry

PythonScript _body attribute initialized incorrectly on import

Python 3.6, Products.PythonScripts-4.6-py3.6.egg

I get this error while importing an older workflow definition containing some PythonScripts in Plone 5.2:

  <string>(3)installProducts()
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/AccessControl-4.0b6-py3.6-linux-x86_64.egg/AccessControl/requestmethod.py(85)_curried()
-> return callable(*args, **kw)
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.CMFQuickInstallerTool-4.0.0-py3.6.egg/Products/CMFQuickInstallerTool/QuickInstallerTool.py(710)installProducts()
-> omitSnapshots=omitSnapshots
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.CMFQuickInstallerTool-4.0.0-py3.6.egg/Products/CMFQuickInstallerTool/QuickInstallerTool.py(626)installProduct()
-> blacklisted_steps=blacklistedSteps,
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/tool.py(393)runAllImportStepsFromProfile()
-> dependency_strategy=dependency_strategy)
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/tool.py(1465)_runImportStepsFromContext()
-> message = self._doRunImportStep(step, context)
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/tool.py(1277)_doRunImportStep()
-> return handler(context)
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.CMFCore-2.4.0b8-py3.6.egg/Products/CMFCore/exportimport/workflow.py(128)importWorkflowTool()
-> importObjects(tool, '', context)
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/utils.py(890)importObjects()
-> importObjects(sub, path + '/', context)
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/utils.py(886)importObjects()
-> importer.body = body
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.DCWorkflow-2.4.0b2-py3.6.egg/Products/DCWorkflow/exportimport.py(92)_importBody()
-> self.environ)
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.DCWorkflow-2.4.0b2-py3.6.egg/Products/DCWorkflow/exportimport.py(981)_initDCWorkflow()
-> _initDCWorkflowScripts(workflow, scripts, context)
> /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.DCWorkflow-2.4.0b2-py3.6.egg/Products/DCWorkflow/exportimport.py(1150)_initDCWorkflowScripts()
-> script.write(file)
  /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.PythonScripts-4.6-py3.6.egg/Products/PythonScripts/PythonScript.py(416)write()
-> m = _nonempty_line.search(text, st)
  File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.PythonScripts-4.6-py3.6.egg/Products/PythonScripts/PythonScript.py", line 416, in write
    m = _nonempty_line.search(text, st)
TypeError: cannot use a string pattern on a bytes-like object

/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.PythonScripts-4.6-py3.6.egg/Products/PythonScripts/PythonScript.py(419)write()
-> if not m:
(Pdb) args
self =
text = b'## Script (Python) "setPortalMessage"\n##bind container=container\n##bind context=context\n##bind namespace=\n##bind script=script\n##bind subpath=traverse_subpath\n##parameters=data\n##title=\n##\ncontext.plone_utils.addPortalMessage(u'Opgelet : door de status van dit item aan te passen naar concept haalt u deze pagina offline en is het niet meer toegankelijk', 'warn')\n'

Zope 4 ZMI for setup tool: Make sure the add dialog is not shown

See zopefoundation/Zope#536

Add a class variable zmi_show_add_dialog to the GS tool class so no modal add dialog is shown during creation.

Python 3: showing old upgrades throws an exception

I have a package with two upgrade steps. For each upgrade step source = 1, destination = 1, and sortkey = 1. When I try to list the old upgrades, I get the following exception:

Traceback (innermost last):
Module ZPublisher.WSGIPublisher, line 142, in transaction_pubevents
Module ZPublisher.WSGIPublisher, line 295, in publish_module
Module ZPublisher.WSGIPublisher, line 229, in publish
Module ZPublisher.mapply, line 85, in mapply
Module ZPublisher.WSGIPublisher, line 57, in call_object
Module Shared.DC.Scripts.Bindings, line 330, in call
Module Shared.DC.Scripts.Bindings, line 367, in _bindAndExec
Module Products.PageTemplates.PageTemplateFile, line 141, in _exec
Module Products.PageTemplates.PageTemplate, line 83, in pt_render
Module zope.pagetemplate.pagetemplate, line 135, in pt_render
Module Products.PageTemplates.engine, line 90, in call
Module z3c.pt.pagetemplate, line 173, in render
Module chameleon.zpt.template, line 306, in render
Module chameleon.template, line 209, in render
Module chameleon.utils, line 75, in raise_with_traceback
Module chameleon.template, line 187, in render
Module 561e1882708296964a293b8943c621a1.py, line 685, in render
Module Products.GenericSetup.tool, line 1002, in listUpgrades
Module Products.GenericSetup.upgrade, line 270, in listUpgradeSteps
TypeError: '<' not supported between instances of 'dict' and 'dict'

• Expression: " python:context.listUpgrades(profile_id, show_old=show_old"
• Filename: manage_upgrades
• Location: (line 69: col 27)
• Arguments: template: <ImplicitAcquisitionWrapper manage_upgrades at 0x7f7c13b013a8>
  here: <ImplicitAcquisitionWrapper portal_setup at 0x7f7c13b01090>
  context: <ImplicitAcquisitionWrapper portal_setup at 0x7f7c13b01090>
  container: <ImplicitAcquisitionWrapper portal_setup at 0x7f7c13b01090>
  nothing: <NoneType - at 0xa02bc0>
  options: {...} (1)
  root: <ImplicitAcquisitionWrapper at 0x7f7c12cb4948>
  request: <WSGIRequest - at 0x7f7c0e1c3780>
  modules: <_SecureModuleImporter - at 0x7f7c21f42630>
  user: <ImplicitAcquisitionWrapper - at 0x7f7c13b01a20>
  default: <object - at 0x7f7c248ecac0>
  repeat: {...} (0)
  loop: {...} (1)
  wrapped_repeat: <SafeMapping - at 0x7f7c122d7b88>
  target_language: <NoneType - at 0xa02bc0>
  translate: <function translate at 0x7f7c0dac7ea0>
  profile_id: ugent.editmode:default
  prof_with_upgrades: <list - at 0x7f7c12b67988>
  prof_with_pending_upgrades: <list - at 0x7f7c12086bc8>
  prof_uptodate: <list - at 0x7f7c133bf848>
  has_pending_upgrades: False
  show_old: 1

I get this problem with Python 3, but not with Python 2. It looks like the method listUpgradeSteps tries to sort a list of upgrade steps by comparing dicts, which works in Python 2, but not in Python 3.

Problem with dependency-profiles

There seems to be an issue with #16

After this was merged and enabled in the Plone 5 coredev the following problem was observed the workflow 'comment_review_workflow' from plone.app.discussion is not installed upon creating a new site.

Steps to reproduce:

• create new site
• go to http://localhost:8080/Plone/portal_workflow/manage_main

Rolemap importer offers no way to append roles into existing permissions

In https://bugs.launchpad.net/zope-cmf/+bug/1227290, @seanupton reported:

If a GenericSetup base profile defines a set of roles for a permission in its rolemap.xml, it is impossible to supplement (append) roles to that permission from an extension profile. Rather, the extension profile rolemap.xml must include duplicative (all) roles (for each permission) listed in upstream base profile and cannot merely supplement.

I have verified the outcome of this looking at manage_access in a Plone site, and by reviewing the code.

AccessControl.rolemanager.RoleManager.manage_permission() is passed only what is explicitly in the rolemap.xml for an extension profile by Products.GenericSetup.rolemap.importRolemap(). Permission.setRoles() does a setattr() replacement of the roles tuple/list in each permission. The consequence of this is that it is impossible to merge/supplement/append roles.

The work-around is to copylift and duplicate all roles listed in upstream package profiles.

Note: ac_roles is fine, is supplemented correctly. This is a per-permission mapping issue.

Versions: Zope: 2.13.21, Products.GenericSetup 1.7.4, Products.CMFCore 2.2.7 on Python 2.7.3

TarballImportContext crashes when comparing the last char of empty strings

Products.GenericSetup, v1.7.6:

In Products/GenericSetup/context.py, there are two comparisons of path[-1] to ‘/’, which must fail for empty strings. IMO, here the endswith method must be used.

The empty string seems to be a known special case for tarballs.

See pull request #13; this ticket is identical with https://bugs.launchpad.net/zope2/+bug/1479268.

Replace zope.formlib with z3c.form

Should that be something desirable? Plone 5 already made a strong move out of zope.formlib in favor of z3c.form and one of the only few packages that do still depend on it is Products.GenericSetup: https://github.com/zopefoundation/Products.GenericSetup/search?utf8=%E2%9C%93&q=formlib

Import handler not found

BUG/PROBLEM REPORT (OR OTHER COMMON ISSUE)

What I did:

I installed a package with two namespace levels foo.bar.baz and inside i have an import step handler foo.bar.baz.exportimport.import_handler as development package or from an sdist.

What I expect to happen:

Generic Setup finds the function and uses it.

What actually happened:

It does not and log about an invalid import handler.

What version of Python and Zope/Addons I am using:

• Python 3.9.9
• Zope 5.4
• Plone 6.0.0a3
• pip 22 based installation
• pas.plugins.ldap

Additional Infos

It used to work in Plone 5.2/buildout based installation.

For more Info, the story behind this report and the (possible) solution see https://community.plone.org/t/dotted-name-resolving-fails/14810

A PR will evolve at #117

snapshot comparison broken in Plone 5.2 - Python 3

Run an instance of Plone 5.2 in a Python 3
When you run a snapshot comparison in ZMI - portal_setup you get an error
Seems related to the str - bytes change in python 3, code is expecting bytes, but getting str
example buildout.cfg:

[buildout]
parts = instance
extends = https://dist.plone.org/release/5.2-latest/versions.cfg

[instance]
recipe = plone.recipe.zope2instance
wsgi = on
eggs =
    Plone
    Pillow

traceback:

2019-09-24 10:08:13,548 INFO    [GenericSetup.workflow:98][waitress] Workflow tool exported.
/Users/KP/projects/plone/minimalplone52/eggs/Zope-4.1.1-py3.7.egg/OFS/ObjectManager.py:463: ResourceWarning: unclosed file <_io.FileIO name=19 mode='rb+' closefd=True>
  return [(id, self._getOb(id)) for id in self.objectIds(spec)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/KP/projects/plone/minimalplone52/eggs/Zope-4.1.1-py3.7.egg/OFS/ObjectManager.py:463: ResourceWarning: unclosed file <_io.FileIO name=20 mode='rb+' closefd=True>
  return [(id, self._getOb(id)) for id in self.objectIds(spec)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/KP/projects/plone/minimalplone52/eggs/Zope-4.1.1-py3.7.egg/OFS/ObjectManager.py:463: ResourceWarning: unclosed file <_io.FileIO name=21 mode='rb+' closefd=True>
  return [(id, self._getOb(id)) for id in self.objectIds(spec)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/KP/projects/plone/minimalplone52/eggs/Zope-4.1.1-py3.7.egg/OFS/ObjectManager.py:463: ResourceWarning: unclosed file <_io.FileIO name=23 mode='rb+' closefd=True>
  return [(id, self._getOb(id)) for id in self.objectIds(spec)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
2019-09-24 10:08:13,605 INFO    [GenericSetup.contenttypes:98][waitress] Content type registry exported.
2019-09-24 10:08:56,450 ERROR   [Zope.SiteErrorLog:251][waitress] 1569312536.4484330.7780034037876496 http://localhost:8080/Plone/portal_setup/manage_showDiff
Traceback (innermost last):
  Module ZPublisher.WSGIPublisher, line 155, in transaction_pubevents
  Module ZPublisher.WSGIPublisher, line 337, in publish_module
  Module ZPublisher.WSGIPublisher, line 255, in publish
  Module ZPublisher.mapply, line 85, in mapply
  Module ZPublisher.WSGIPublisher, line 61, in call_object
  Module Shared.DC.Scripts.Bindings, line 335, in __call__
  Module Shared.DC.Scripts.Bindings, line 372, in _bindAndExec
  Module Products.PageTemplates.PageTemplateFile, line 145, in _exec
  Module Products.PageTemplates.PageTemplate, line 85, in pt_render
  Module zope.pagetemplate.pagetemplate, line 135, in pt_render
  Module Products.PageTemplates.engine, line 88, in __call__
  Module z3c.pt.pagetemplate, line 173, in render
  Module chameleon.zpt.template, line 306, in render
  Module chameleon.template, line 209, in render
  Module chameleon.utils, line 75, in raise_with_traceback
  Module chameleon.template, line 187, in render
  Module 653a823b417046947bff895cf59cce43, line 714, in render
  Module Products.GenericSetup.tool, line 464, in markupComparison
TypeError: startswith first arg must be bytes or a tuple of bytes, not str

 - Expression: "python: context.markupComparison( comparison )"
 - Filename:   manage_showDiff
 - Location:   (line 129: col 29)
 - Arguments:  template: <ImplicitAcquisitionWrapper manage_showDiff at 0x11c1ea230>
               here: <ImplicitAcquisitionWrapper portal_setup at 0x11036f050>
               context: <ImplicitAcquisitionWrapper portal_setup at 0x11036f050>
               container: <ImplicitAcquisitionWrapper portal_setup at 0x11036f050>
               nothing: <NoneType - at 0x109854058>
               options: {...} (1)
               root: <ImplicitAcquisitionWrapper  at 0x1152f80f0>
               request: <WSGIRequest - at 0x11727e410>
               modules: <_SecureModuleImporter - at 0x10c0e8d10>
               user: <ImplicitAcquisitionWrapper - at 0x11c0d56e0>
               default: <object - at 0x10a9baf40>
               repeat: {...} (0)
               loop: {...} (1)
               wrapped_repeat: <SafeMapping - at 0x11c1eabe0>
               target_language: <NoneType - at 0x109854058>
               translate: <function translate at 0x11401f7a0>
               lhs: snapshot-snapshot-20190924080710
               rhs: snapshot-snapshot-20190924080812
               mae: 0
               ib: 0
               mcc: <method manage_compareConfigurations at 0x1209fe0a0>
               comparison: <bytes - at 0x7f91bf9c9600>

See changes in tool.py in macbook icts kim
/Users/u0045445/projects/plone/minimal52/eggs/Products.GenericSetup-2.0b5-py3.6.egg/Products/GenericSetup/tool.py

See also: python - startswith TypeError in function - Stack Overflow

After changing some str to bytes (b'') and line to bytes(line)

Module chameleon.template, line 183, in render
  Module 5f634d11d3bcaaa0faf3cdadf322dc59.py, line 704, in render
  Module Products.GenericSetup.tool, line 500, in markupComparison
  Module Products.GenericSetup.tool, line 500, in <listcomp>
  Module html, line 19, in escape
TypeError: a bytes-like object is required, not 'str'

 - Expression: "python: context.markupComparison( comparison )"
 - Filename:   manage_showDiff
 - Location:   (line 129: col 29)
 - Arguments:  template: <ImplicitAcquisitionWrapper manage_showDiff at 0x11225ca20>
               here: <ImplicitAcquisitionWrapper portal_setup at 0x11225c8b8>
               context: <ImplicitAcquisitionWrapper portal_setup at 0x11225c8b8>
               container: <ImplicitAcquisitionWrapper portal_setup at 0x11225c8b8>
               nothing: <NoneType - at 0x10091f810>
               options: {...} (1)
               root: <ImplicitAcquisitionWrapper  at 0x1122dda20>
               request: <WSGIRequest - at 0x119f865f8>
               modules: <_SecureModuleImporter - at 0x10eba2518>
               user: <ImplicitAcquisitionWrapper - at 0x11225c798>
               default: <object - at 0x101e95a90>
               repeat: {...} (0)
               loop: {...} (1)
               wrapped_repeat: <SafeMapping - at 0x11a1e3bc8>
               target_language: <NoneType - at 0x10091f810>
               translate: <function translate at 0x11338f8c8>
               lhs: snapshot-snapshot-20190924141937
               rhs: snapshot-snapshot-20190924142041
               mae: 0
               ib: True
               mcc: <method manage_compareConfigurations at 0x1190a7748>
               comparison: <bytes - at 0x1296e9000>
`

Non-existing dependency-profiles cause multiple errors

Refs: #64, plone/Products.CMFPlone#2228.

The multiple errors still occur, I think the culprit lies in getProfileDependenciesChain, because it is calling itself recursively:

Products.GenericSetup/Products/GenericSetup/tool.py

Line 1331 in 3ca5ee7

chain.extend(self.getProfileDependencyChain(dependency, seen))

And the try/except-block in _runImportStepsFromContext of where it's called initially, is masking this:

Products.GenericSetup/Products/GenericSetup/tool.py

Line 1386 in 3ca5ee7

try:

Probably because the first try succeeds and later an exception occurs from a recursive-call.

So, I tried raising an error explicitly in getProfileDependenciesChain, but that also doesn't stop the execution of further code.

The only solution I came up with, is to return None in getDependenciesForProfile for symbolizing dependencies are broken, like it was sugested in #64:
dd11374

That will cause the interpreter to raise a "not-iterable"-error in getProfileDependenciesChain and fail hard, as wished.

@tseaver, @icemac, @sunew, if you think this is feasible, I'd like to provide another PR.

List of available profiles in the ZMI needs fixing

In https://bugs.launchpad.net/zope-cmf/+bug/501668, @dataflake reported:

When visiting a setuptool in the ZMI it shows all registered base profiles on the Profiles tab and all registered extension profiles on the Import tab. However, all those are registered for specific interfaces, and still show up even if the tool's container does not provide them.

The code that generates the list of available profiles should check the tool container's interfaces and only list those profiles that are appropriate for the container.

TypeError: initial_value must be str or None, not bytes

Zope5.2.1 on Mac OS 10.13.5 Python3.7.7

What I did:

From the ZMI I choosed "Configured PAS" and the action generate the following error:

2021-06-09 11:27:16 ERROR [waitress:404][waitress-0] Exception while serving /oneMoreTestFolder/manage_addProduct/PluggableAuthService/addConfiguredPAS
Traceback (most recent call last):
  File "/Applications/zope521bo/eggs/waitress-2.0.0-py3.7.egg/waitress/channel.py", line 397, in service
    task.service()
  File "/Applications/zope521bo/eggs/waitress-2.0.0-py3.7.egg/waitress/task.py", line 168, in service
    self.execute()
  File "/Applications/zope521bo/eggs/waitress-2.0.0-py3.7.egg/waitress/task.py", line 434, in execute
    app_iter = self.channel.server.application(environ, start_response)
  File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/httpexceptions.py", line 30, in __call__
    return self.application(environ, start_response)
  File "/Applications/zope521bo/eggs/Paste-3.5.0-py3.7.egg/paste/translogger.py", line 69, in __call__
    return self.application(environ, replacement_start_response)
  File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/WSGIPublisher.py", line 376, in publish_module
    response = _publish(request, new_mod_info)
  File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/WSGIPublisher.py", line 279, in publish
    bind=1)
  File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/mapply.py", line 85, in mapply
    return debug(object, args, context)
  File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/WSGIPublisher.py", line 68, in call_object
    return obj(*args)
  File "/Applications/zope521bo/eggs/Products.PluggableAuthService-2.6.3-py3.7.egg/Products/PluggableAuthService/PluggableAuthService.py", line 1419, in addConfiguredPAS
    tool.runAllImportStepsFromProfile('profile-%s' % base_profile)
  File "/Applications/zope521bo/eggs/Products.GenericSetup-2.1.1-py3.7.egg/Products/GenericSetup/tool.py", line 405, in runAllImportStepsFromProfile
    dependency_strategy=dependency_strategy)
  File "/Applications/zope521bo/eggs/Products.GenericSetup-2.1.1-py3.7.egg/Products/GenericSetup/tool.py", line 1483, in _runImportStepsFromContext
    message = self._doRunImportStep(step, context)
  File "/Applications/zope521bo/eggs/Products.GenericSetup-2.1.1-py3.7.egg/Products/GenericSetup/tool.py", line 1295, in _doRunImportStep
    return handler(context)
  File "/Applications/zope521bo/eggs/Products.PluggableAuthService-2.6.3-py3.7.egg/Products/PluggableAuthService/exportimport.py", line 29, in importPAS
    IFilesystemImporter(context.getSite()).import_(context, 'PAS', True)
  File "/Applications/zope521bo/eggs/Products.GenericSetup-2.1.1-py3.7.egg/Products/GenericSetup/content.py", line 159, in import_
    stream = cStringIO(objects)
TypeError: initial_value must be str or None, not bytes

Log file objects accessible to anonyone with View permission on the tool's container

Log file objects stored in Generic Setup Tool instances (and the tool instances themselves) don't have any specific permission settings applied, they just acquire all settings from their container. If anonymous users have e.g. View permission on the enclosing container the log files are accessible to anonymous users if they can guess the file name. This is an information disclosure bug.

P.S.: The snapshots folder is affected as well, but actual snapshot data is not visible, only the page templates can be listed.

GenericSetup upgradeStep ZCML discriminator should include the profile

In https://bugs.launchpad.net/zope-cmf/+bug/924721, @rpatterson reported:

It appears the ZCML handler for the upgradeSetp directive doesn't include the profile id in the discriminator. This means that if you register the same handler for two different profiles but with the same source and destination versions, you get a ConfigurationConflictError.

Fix tests with Products.ZCatalog 4.1

See PR #61 which temporarily fixes it by pinning Products.ZCatalog back to 4.0.1.

Tests are broken

What I did:

I looked at the test runs in GHA.
e. g. https://github.com/zopefoundation/Products.GenericSetup/actions/runs/4274869009/jobs/7441795588

What I expect to happen:

Green tests.

What actually happened:

Tests are broken probably due to changes in newer setuptools versions resp. its bundled packaging module.

What version of Python and Zope/Addons I am using:

• Zope 5
• Python >= 3.7

Cc-ing some contributors:

@dataflake @mauritsvanrees

Port to Python 3

Todo:

• zopefoundation/Products.MailHost#6
• port Products.GenericSetup (this package) to Python 3 #51
• undo 98f06b9

Online docs are outdated

The GenericSetup PyPI page points to documentation at http://packages.python.org/Products.GenericSetup/ and https://pythonhosted.org/Products.GenericSetup/. The first url redirects to the second one.

What we see there, is documentation for version 1.7.0. Does anyone know how to update this?

Alternatively, we could rearrange things and move to readthedocs.

GenericSetup has problems with BrowserLayers

In https://bugs.launchpad.net/zope-cmf/+bug/530943, @do3cc reported:

When trying to modfiy the peristent registry with the manage_components tab, it usually fails if browserlayers are registered as utilities.

The code tries to generate a xml configuration for the utility and assumes type(utility) as the factory for the BrowserLayers. The BrowserLayer utility is an interface, type resorts to zope.interface.interface.InterfaceClass. This cannot be called without arguments. Thus when loading the new utilities, it fails for Browserlayers.

and attached a patch for a testcase to reproduce the issue:

https://bugs.launchpad.net/zope-cmf/+bug/530943/+attachment/1173798/+files/patch

with the comment:

Here is a traceback:

  Module plone.postpublicationhook.hook, line 74, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 42, in call_object
  Module zope.formlib.form, line 769, in __call__
  Module Products.Five.formlib.formbase, line 55, in update
  Module zope.formlib.form, line 750, in update
  Module zope.formlib.form, line 594, in success
  Module Products.Five.formlib.formbase, line 69, in handle_edit_action
  Module zope.formlib.form, line 528, in applyChanges
  Module zope.schema._bootstrapfields, line 183, in set
  Module Products.GenericSetup.browser.components, line 55, in _setBody
  Module Products.GenericSetup.utils, line 544, in _importBody
  Module Products.GenericSetup.components, line 76, in _importNode
  Module Products.GenericSetup.components, line 173, in _initUtilities
TypeError: __init__() takes at least 2 arguments (1 given)

This code is version 1.4.5

I have written a test for it.

I am not sure what a clean way of fixing would be. It seems to me that the code tries to create its own xml serialization and fails. I am not sure whether having your own serialisation here makes sense.

Later @do3cc wrote:

I talked with hannosch about the issue. In the end, the view never really worked in all cases, but some objected to the removal.

Since it is easily feasable to get a list of all utilities and adapters, but not so easy to serialize and deserialize them, a potential fix would be to create a new simple ui that lists local adapters and utilities and offers a delete action for each.

Yvo Schubbe responded:

Thanks for the report! AFAICS this issue is not specific to the manage_components tab, your testcase is for the ComponentRegistryXMLAdapter that is primarily used for importing and exporting GenericSetup profiles.

ComponentRegistryXMLAdapter has its limitations and a real UI for five.localsitemanager site managers would be better than the current view, but that's a completely different issue.

Your testcase just shows that imports using InterfaceClass as factory don't work, but that's not really the problem. The real issue seems to be the export/serialization. And if you report issues with the ComponentRegistryXMLAdapter please report them against version 1.5+ because it has many components handler fixes 1.4.5 does not contain.

@flipmcf followed up:

sorry to raise the dead here, but I'm seeing this behavior also with an 3rd party add-on which will go unnamed.

I may have a workaround, but it's unconfirmed:

I think a browserlayer.xml like this will create the utility which exports in componentregistry with the bad factory:

 <?xml version="1.0"?>
 <layers>
    <layer name="MyLayer"
      interface="IMyLayer"/>
 </layers>

But a browserlayer.xml like this:

 <?xml version="1.0"?>
 <layers name="MyLayers" meta_type="ComponentRegistry">
    <layer name="MyLayer"
      interface="IMyLayer"/>
 </layers>

Seems to avoid the issue and doesn't muck about with componentregistry's export process.

Would love to get a confirmation of this from someone, and an official "Nod of the head" that the 2nd example of a browserlayer.xml generic setup profile is the "Correct" way to do it.

@flipmcf later wrote:

Please disregard previous message. I experimented and saw no difference.

and then again:

I would like to raise the severity of this issue as it seems related to causing zope to spin in some situations. Our site administration workflow depends on versioned, up-to-date GenericSetup configuration profiles and running import steps from the ZMI when production changes occur. We have too many production instances to point-and-click our way through the ZMI for each change that deploys to production.

Zope2-2.13.8-py2.7
zope.component-3.9.5-py2.7
Products.CMFCore-2.2.4-py2.7

To reproduce, I would use the package 'quintagroup.plonegooglesitemaps' which registers a browser layer upon installation.

Install this product (it's quite easy) and configure a news-sitemap.xml

Request the news-sitemap.xml through a browser request, and note it works just fine - you get some xml - empty is ok.

1. Export local compoent registry (componentregistry.xml) and re-import it: This aggrivates the bug that Patric Gerken describes. Essentially at this point, you cannot save and version your ZMI settings (like we do) because imports will not work anymore.
2. Next: Comment out the offending utility in componentregistry.xml and import again. Of course, now it will work.
3. Access the news-sitemap.xml you created above. Zope will spin at %100 cpu and can only be fixed through a hard restart of the instance.
4. Fix by running the browserlayer import step (return to step 1)

I have a few traceback / dumps from my spinning zope process that I can provide upon request.

Is it feasible to have a persisted historic information of when a specific upgradeStep was executed/imported?

First of all, we would like to thank @mauritsvanrees for the 1.8.2 release that really improved the usability of this package and the usage of upgradeSteps since because of that improvements we had the idea of this issue.

For the sake of trying to improve Products.GenericSetup even more, even with just suggestions (we're sorry we can't yet do PR's for plone and zope packages, we're still addressing some internal corporate issues with contributor's agreements), we would like to know if it's possible to have a historic record of when specific upgradeSteps were executed.

Why are we asking this? Because you can have 20 upgradeSteps, you can execute just the last one and it's impossible to know which upgradeSteps before the last one were run or not. You already have an upgradeStep internal id, the next step would be to have a persisted list of timestamps when that id was executed/imported.

Since an image is worth a thousand words, something like this comes to mind:

Of course the layout doesn't need to be like this, you can have a link in the upgradeStep title for example. We don't care which layout is chosen as long as the historic record of every upgradeStep import is shown.

Challenges: since we have systems that already run the upgradeSteps but we don't have it's historic, if a new version of Products.GenericSetup would implement this, "NOT EXECUTED" would be incorrect in that case, something along the lines of "Unknown" would be more appropriate.

Update to adapt to Zope 5 deprecations

Products.GenericSetup/Products/GenericSetup/browser/components.py

Line 17 in 88f9289

from Products.Five.browser.decode import processInputs

This line is causing my watchdog staging environment to fail, because the referenced file was removed at zopefoundation/Zope@f15d25c

Not sure how to best deal with this, maybe have an alternative ready and fall back to that if the module isn't there anymore with a comment / deprecation to to remove the fallback after zope 5 is released?

Why don't we pass the ImportContext to pre_handler and post_handler?

If I understand it correctly the main thing about pre_handler and post_handler is to replace the cumbersome registration of import-steps. Import-steps were not passed the GenericSetup-Tool but the Products.GenericSetup.context.DirectoryImportContext which has the nice helper _profile_path and readDataFile which allowed to read various files (e.g. a csv-file with data from which to create users or content). Now have to do this by hand which makes switching existing code from import-steps to post_handlers a little more work than would be nice.
I just want to ask if there is any special reason why we don't pass context in Products.GenericSetup.tool.SetupTool._runImportStepsFromContext to _doRunHandler and then pass that to the handler.

GS can't eat its own dog food.

situation
set up a plone 5.1a2 site.
create a snapshot in portal_setup.
change the day the week should start with in the control panel "Date and Time Settings".
create a snapshot in portal_setup again and make a diff between those two.

i then safed this changed settings to my registry.xml

  <record name="plone.first_weekday" interface="Products.CMFPlone.interfaces.controlpanel.IDateAndTimeSchema" field="first_weekday">
    <field type="plone.registry.field.Choice">
      <description xmlns:ns0="http://xml.zope.org/namespaces/i18n" ns0:domain="plone" ns0:translate="help_first_weekday">First day in the week.</description>
      <title xmlns:ns0="http://xml.zope.org/namespaces/i18n" ns0:domain="plone" ns0:translate="label_first_weekday">First weekday</title>
      <vocabulary>plone.app.vocabularies.Weekdays</vocabulary>
    </field>
    <value>3</value>
  </record>

now i set up another plone site and also install my package with this settings stored inside and get an error:

Traceback (innermost last):
  Module ZPublisher.Publish, line 138, in publish
  Module ZPublisher.mapply, line 77, in mapply
  Module ZPublisher.Publish, line 48, in call_object
  Module Products.CMFPlone.browser.admin, line 272, in __call__
  Module Products.CMFPlone.factory, line 168, in addPloneSite
  Module Products.GenericSetup.tool, line 379, in runAllImportStepsFromProfile
   - __traceback_info__: profile-akbild.site:default
  Module Products.GenericSetup.tool, line 1414, in _runImportStepsFromContext
  Module Products.GenericSetup.tool, line 1226, in _doRunImportStep
   - __traceback_info__: plone.app.registry
  Module plone.app.registry.exportimport.handler, line 47, in importRegistry
  Module plone.app.registry.exportimport.handler, line 90, in importDocument
  Module plone.app.registry.exportimport.handler, line 246, in importRecord
   - __traceback_info__: record name: plone.first_weekday
  Module plone.supermodel.utils, line 144, in elementToValue
  Module zope.schema._field, line 322, in fromUnicode
  Module zope.schema._bootstrapfields, line 183, in validate
  Module zope.schema._field, line 338, in _validate
ConstraintNotSatisfied: (u'0', '')

0 -> Monday (Europe), but every other number from 0 (monday) to 6 (sunday) generates the same error.

why does that happen?
the choice field (zope.schema) validates a given value (the one from the xml file -> u'0') against its vocabulary, using by_value, which is - in case of plone.app.vocabularies.datetimerelated.WeekdaysFactory - a dict of number (keys) and simpleterm (value) - pairs. but the value from the xml is a unicode string, not an int. -> so its not a key in the vocabulary. see zope.schema._field.Choice._validate() for that.

u'0' in {0:term_a, 1:term_b} == False

further down the rabbit hole...
most SimpleTerms in plone.app.vocabularies are instantiated by a value (string) and a title (term); only the Month and Weekday Vocabularies are build different. the come with a value, token and title. value is an index, token a string representation of that index and the title (term). this is, at least, inconsistent.

solutions

1. rewrite the vocabularies to have better key/value consistency and fix cmfplone Products.CMFPlone.setuphandlers.first_weekday_setup to use strings instead of integers.
2. extend the choice field to also handle value_type; like IDict and ICollection does.
3. make all GS setup files JSON instead of XML (no... just kidding... but would be awesome)

i will implement the first solution, as i am not able (too heavy) to extend the Choice field or reimplement the whole GS-setup infrastructure.

take a look at:
plone/Products.CMFPlone#1794
and
plone/plone.app.vocabularies#41

sadly they are cross depending ...

GenericSetup tests fail under Zope 5.3.

I first saw this in a Plone 6 test run when I switch from Zope 5.2.1 to 5.3.

Sample failure locally with GenericSetup tox:

Failure in test test__initProperties_nopurge_base (Products.GenericSetup.tests.test_utils.PropertyManagerHelpersTests)
Traceback (most recent call last):
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
    yield
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 593, in run
    self._callTestMethod(testMethod)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
    method()
  File "/Users/maurits/community/plone-coredev/6.0/src/Products.GenericSetup/src/Products/GenericSetup/tests/test_utils.py", line 669, in test__initProperties_nopurge_base
    self.assertEqual(obj.getProperty('lines3'), (b'Gee', b'Foo', b'Bar'))
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 831, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 1048, in assertTupleEqual
    self.assertSequenceEqual(tuple1, tuple2, msg, seq_type=tuple)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 1019, in assertSequenceEqual
    self.fail(msg)
  File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 670, in fail
    raise self.failureException(msg)
AssertionError: Tuples differ: ('Foo', 'Gee', b'Foo', b'Bar') != (b'Gee', b'Foo', b'Bar')

First differing element 0:
'Foo'
b'Gee'

First tuple contains 1 additional elements.
First extra element 3:
b'Bar'

- ('Foo', 'Gee', b'Foo', b'Bar')
?  ^^^^^^^

+ (b'Gee', b'Foo', b'Bar')
?  ^

So there is a mixup between bytes and strings.

With tox, the Python 2.7 and 3.5 tests pass, because they use Zope 4.x.
On the newer Pythons, 6 tests fail. They pass when I change buildout.cfg to extend the Zope 5.2.1 versions.cfg.
When I edit tox.ini to let Python 3.6, 3.7 and 3.8 use Zope4 (buildout4.cfg) those tests pass.

The failure above is for this line. The test imports _NOPURGE_IMPORT

Ignoring the test failure for lines3 for a moment, let's look at what happens with the lines1 field. I think the following is a bit suspicious:

• The test creates a lines field with (in Python 3 terms) strings.
• When you call getProperty, you indeed get a tuple of strings.
• Then the import is done, which purges these strings and puts back two new elements.
• And then getProperty gives a tuple of bytes.

I don't know yet if this is a problem in practice, or if the tests just need to be fixed.

I wonder what changed in Zope 5.3 for this. Maybe zopefoundation/Zope#962 is involved.

Fix Flake8 errors and deprecation warnings

There are many deprecation warnings issued by the tests and the flake8 check (tox -eflake8) complains about many violations.

It would be nice to get them fixed. (Flake 8 could even be tested on Travis CI to ensure Flake8 compatibility for the future.)

Unicode/Text support for writeDataFile: ValueError: Unicode text is not supported, even if it only contains ascii. Please encode your data. See GS 1.7.0 changes for more

See

plone/Products.CMFPlone#2784

I don't see why we need to raise an explicit error instead of converting to utf-8 if needed under the hood?

https://github.com/zopefoundation/Products.GenericSetup/blob/master/Products/GenericSetup/context.py#L441-L444

Fix deprecation warnings

Products/GenericSetup/components.py:26: DeprecationWarning: IComponentRegistry is deprecated. Import from zope.interface.interfaces
Products/GenericSetup/utils.py:365: DeprecationWarning: cgi.escape is deprecated, use html.escape instead
Products/GenericSetup/utils.py:373: DeprecationWarning: cgi.escape is deprecated, use html.escape instead

Components handler: make 'interface' attribute optional

In https://bugs.launchpad.net/zope-cmf/+bug/161726, Yvo Schubbe reported:

Like in ZCML, 'interface' should be pulled from the component if not specified.

minor error when running linter

The flake8-html plugin wants to write into a subdirectory of parts - but the parts directory is not present. The flake8 check runs, but obviously not html gets generated.

❯ tox -e=flake8
flake8 installed: flake8==3.8.1,flake8-debugger==3.2.1,flake8-html==0.4.1,importlib-metadata==1.6.0,Jinja2==2.11.2,MarkupSafe==1.1.1,mccabe==0.6.1,Products.GenericSetup==2.0.3.dev0,pycodestyle==2.6.0,pyflakes==2.2.0,Pygments==2.6.1,zipp==3.1.0
flake8 run-test-pre: PYTHONHASHSEED='1702982963'
flake8 run-test: commands[0] | - flake8 --format=html --htmldir=/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/parts/flake8 --doctests Products setup.py
Traceback (most recent call last):
  File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/bin/flake8", line 10, in <module>
    sys.exit(main())
  File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/cli.py", line 22, in main
    app.run(argv)
  File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/application.py", line 360, in run
    self._run(argv)
  File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/application.py", line 347, in _run
    self.initialize(argv)
  File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/application.py", line 333, in initialize
    self.make_formatter()
  File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/application.py", line 226, in make_formatter
    self.formatter = formatter_class(self.options)
  File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/plugins/manager.py", line 131, in execute
    return self.plugin(*args, **kwargs)  # pylint: disable=not-callable
  File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/formatting/base.py", line 51, in __init__
    self.after_init()
  File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8_html/plugin.py", line 89, in after_init
    os.mkdir(self.outdir)
FileNotFoundError: [Errno 2] No such file or directory: '/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/parts/flake8'
flake8 run-test: commands[1] | flake8 --doctests Products setup.py
_________________________________________________________________ summary _________________________________________________________________
  flake8: commands succeeded
  congratulations :)

New release? 1.8.3

Any chance of a new release? I would like to use the toolset.xml improvements from my merged pull request #26.

Release rights on PyPI for user maurits would also be gladly accepted. :-)

New release, please

Can we have a new release of GenericSetup please? da0a3749 would allow Plone to use the latest versions of setuptools and zc.buildout

Tool.getSortedImportSteps should return a deterministic order

In https://bugs.launchpad.net/zope-cmf-buildout/+bug/1418881, Dieter Maurer reported:

Products.GenericSetup 1.7.3

"GenericSetup" supports the declaration of step dependencies and "getSortedImportSteps" returns a sequence of steps respecting those declarations. However, the step order is otherwise non-deterministic. This makes it extremely hard to detect missing step dependency declarations.

https://dev.plone.org/ticket/13815 shows an example where a missing step dependency was not recognized for a very long time.

I have spent the last few days to track down another similar problem: I start Zope and get the following (unworking) sort order:

2015-02-05T08:50:19 INFO bfd.theme_bp.utilities.tracing Return Products.GenericSetup.tool.getSortedImportSteps -> (u'plone-difftool', u'plone.app.caching', u'properties', u'Products.Ploneboard.uninstallVarious', u'rolemap', u'sharing', u'ploneopenid-various', u'collective.z3cform.datetimewidget_various', u'toolset', u'reference_catalog', u'various', u'typeinfo', u'cmfeditions_various', u'repositorytool', u'kupu-setup', u'workflow', u'update-workflow-rolemap', u'placeful_workflow', u'placeful_marker', u'catalog', u'content', u'cookie_authentication', u'caching_policy_mgr', u'various-calendar', u'plone.app.theming', u'uid_catalog', u'componentregistry', u'portlets', u'contentrules', u'atcttool', u'factorytool', u'actions', u'plonetheme.sunburst-various', u'controlpanel', u'memberdata-properties', u'plonepas', u'mailhost', u'action-icons', u'solr', u'browserlayer', u'content_type_registry', u'mimetypes-registry-various', u'kss_mimetype', u'difftool', u'tinymce_various', u'plone.app.registry', u'tinymce_settings', u'plone-final', u'plone-content', u'skins', u'jsregistry', u'bfd.gui.various', u'cssregistry', u'jquerytools-various', u'kssregistry', u'propertiestool', u'viewlets', u'archetypes-various', u'archetypetool', u'plone_outputfilters_various'

I restart Zope (without any modification) and get the following (working) sort order:

2015-02-05T08:55:18 INFO bfd.theme_bp.utilities.tracing Return Products.GenericSetup.tool.getSortedImportSteps -> (u'properties', u'rolemap', u'sharing', u'toolset', u'uid_catalog', u'various-calendar', u'plone.app.theming', u'various', u'componentregistry', u'plone_outputfilters_various', u'action-icons', u'memberdata-properties', u'skins', u'bfd.gui.various', u'kssregistry', u'cssregistry', u'jquerytools-various', u'jsregistry', u'tinymce_settings', u'tinymce_various', u'browserlayer', u'content_type_registry', u'mimetypes-registry-various', u'kss_mimetype', u'mailhost', u'propertiestool', u'actions', u'plonetheme.sunburst-various', u'controlpanel', u'plonepas', u'viewlets', u'difftool', u'plone.app.registry', u'solr', u'archetypes-various', u'archetypetool', u'catalog', u'atcttool', u'cookie_authentication', u'typeinfo', u'factorytool', u'kupu-setup', u'cmfeditions_various', u'repositorytool', u'reference_catalog', u'caching_policy_mgr', u'workflow', u'content', u'portlets', u'plone-final', u'plone-content', u'contentrules', u'placeful_workflow', u'placeful_marker', u'update-workflow-rolemap', u'ploneopenid-various', u'collective.z3cform.datetimewidget_various', u'Products.Ploneboard.uninstallVarious', u'plone-difftool', u'plone.app.caching', u'ploneboard-various')

Of course, the real problem are missing dependency declarations for the Plone "plone-final" import step (it depends on "skins"). However, the non deterministic behaviour of "getSortedImportSteps" hides this problem quite efficiently.

[py3] Permissions are imported as bytes but expected to be strings

I was having a look at why some robot tests are failing on Plone 5.2 running on python 3.6 and noticed that the Plone controlpanel to manage actions was failing due to a permission missing.

On closer inspection, the ZMI seems to be also broken as there is no permission set on a given action.

Turns out though, that the permission is imported from actions.xml but stored on the action object as (b'View',) but when compared with the list of available permissions, none of them matches because that list is a list of strings rather than bytes.

Which one should prevail? Available permissions should be also bytes or rather permissions should be imported as strings? 🤔

Test that makes no sense?

When running the tests against current master under Python 2.7 using the standard bin/test script they all complete successfully. However, running them via tox produces non-deterministic errors. Example:

Failure in test test_runAllExportSteps_empty (Products.GenericSetup.tests.test_tool.SetupToolTests)
Traceback (most recent call last):
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/Users/jens/src/Products.GenericSetup/Products/GenericSetup/tests/test_tool.py", line 890, in test_runAllExportSteps_empty
    self.assertEqual(result['steps'][1], 'step_registries')
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
    raise self.failureException(msg)
AssertionError: u'toolset' != 'step_registries'

The test code (see https://github.com/zopefoundation/Products.GenericSetup/blob/master/Products/GenericSetup/tests/test_tool.py#L882) appears to assume that result from a call to tool.runAllExportSteps() has a fixed order of steps. But that makes no sense when I look at how the result is put together. In listExportSteps the steps are combined from two set instances, which obviously don't have a sorting order.

Is the assumption of a fixed order wrong or is the implementation wrong which disregards any order?

Don't list upgradeSteps when the profile isn't installed

We had problems runing a upgradeStep of profile not installed. See:

collective/sc.embedder#38

upgradeStep like this, are listed even if the profile isn't installed. Is it really necessary list upgradeSteps of profiles not installed?

1.8.0 creates breakage with plone.app.testing

This is probably not a bug of GS, but tbh I am not sure where this bug belongs, I'll add it to plone.app.testing as well :-)

Some context:
When using the PloneSandbox layer in tests, the layer will create a demostorage and wrap the current storage. it will also "reset" the zcml config, doesn't matter, as the layer will rerun the zcml config anyway.

With 1.8.0 this doesn't happen any more with nested PloneSandboxes.
Nested DemoStorages aren't a problem, zcml reconfigurations are also not a problem, the Layer will rerigster plone ande dependencies, but:
Persistent tools registered via componentregistry.xml are gone.
The tool is still available in the database, but the registration that has happened in the import step is gone.

Create a nicer UI for importing

This is related to pull request #16, which already updates the Import tab in the ZMI.

Perceived problem: the Import tab is too complicated.

Initial idea:

• Move the tarball import to a separate tab. Any checkboxes or other options on the current Import tab have no influence at all on importing a tarball. Tab name: Tarball or Archive.
• The rest of the Import tab can stay. The changes I did to the UI in pull request #16 can probably be reverted for this tab. Maybe remove 'Import all steps' and instead have a 'select/deselect steps' button/checkbox, although that might mean adding some javascript, which is currently totally not in use. Rename the 'Include dependencies?' checkbox to 'Include dependencies of selected steps?', because that is what this checkbox does. It has nothing to do with dependency profiles, if you ever thought that. In other words, the Import tab is for when you want to have full control over which steps are imported.
• Create a new tab, say titled Addons. Start with a list of profiles that have upgrade steps that can be run. Then have a list of profiles that can be applied. Add the new UI options from pull request #16 here, giving control over how dependency profiles are handled. Maybe we can actually show the list of dependency profiles and give their current status: installed / not installed / needs upgrade. Do not give the option to select individual import steps: this page is for importing a full profile.

Given the new Addons tab, it probably makes sense for the Import tab to not apply or upgrade any dependency profiles. We could warn when we see this is a problem, and point to the Addons tab.

I am willing to work on this.

Import always recreates catalog indices

In https://bugs.launchpad.net/zope-cmf/+bug/161682, @mauritsvanrees reported:

At Zest we have an own Plone product feedfeeder for a client (soon to be released to the collective I think). We make use of GS. Its profiles/default/catalog.xml is:

<?xml version="1.0"?>
<object name="portal_catalog" meta_type="Pluggable Plone Catalog Tool">
 <index name="getFeedItemAuthor" meta_type="FieldIndex">
  <indexed_attr value="getFeedItemAuthor"/>
 </index>
 <index name="getFeedItemUpdated" meta_type="DateIndex">
  <property name="index_naive_time_as_local">True</property>
 </index>
 <column value="getFeedItemAuthor"/>
 <column value="getFeedItemUpdated"/>
</object>

So we are adding some indexes. But on a reinstall those indexes get thrown away and readded. This means we need to reindex those indexes manually (or rather in code) so the catalog is up to date again.

We tried to get GS to leave those indexes alone if they are already there, but failed. It looks like a bug in GS, but we could also do something wrong of course.

We have a separate product with an Extentions/install.py which takes care of loading all the GS profiles of the products that we need and that handles some other install functions via the quickinstaller. The relevant part of the code:

    setup_tool = getToolByName(portal, 'portal_setup')
    for extension_id in config.GS_DEPS:
        setup_tool.setImportContext('profile-%s' % extension_id)
        setup_tool.runAllImportSteps(purge_old=False)

And the culprit is the purge_old in the last line. If we have followed the trail into GS right, this should make sure that GS does not remove those indexes, but in fact it does. The indexes are there in the end, but none of the brains in the catalog have any content for those fields. Reindexing in some other part of the code solves it:

    site.portal_catalog.manage_reindexIndex(ids=['getFeedItemAuthor','getFeedItemUpdated'])

Would this be a GS bug or are we doing something wrong?

Yvo Schubbe replied:

Not sure if this is a bug report or a feature request or if the bug is the fact the current limitations are not documented. Anyway it is a known issue.

I quote some suggestions I made on the mailing list:
( http://mail.zope.org/pipermail/zope-cmf/2006-August/024747.html )

• The adapters should compare the new settings with the existing ones and make sure indexes are only updated if they are actually changed. This is not implemented so far.
• The adapters should modify existing indexes if the profile contains new settings. This is not implemented for the 'extra' elements.
• Reindexing might be very expensive. That's the reason why GenericSetup does no reindexing, you have to do it manually. We need a better solution for that, maybe some bookkeeping which indexes need to be reindexed and a button in the setup tool that triggers reindexing.

@mauritsvanrees responded:

I was not sure either when I filed this. :) Some time has passed, I asked some questions and learned some more. I would now say it is a feature request.

I reported some ideas in this thread:

http://article.gmane.org/gmane.comp.web.zope.cmf/13371

@tseaver wrote:

Yuppie thinks he can see how to get the index adapters in GS to avoid clearing an index if the import does not change its properties.

@wichert wrote:

Unfortunately there is no API to either get or update the values of the extra index parameters. That suggests that if extra is specified in the profile the index needs to be recreated.

@stainsby wrote:

I recently reported this in the Plone tracker and it was suggested that I add a comment here instead. Here is my original report:

"Catalog indexes that are defined by a product (in the catalog.xml file of its profile) are cleared each time a product is reinstalled. In fact, just deleting the product directory and re-adding also seems to do the same thing. With a client's membership database that takes one or two days to reindex, this is a major pain."

@dnouri posted two patches:

https://bugs.launchpad.net/zope-cmf/+bug/161682/+attachment/169881/+files/CMFCore-2.1-branch.diff

https://bugs.launchpad.net/zope-cmf/+bug/161682/+attachment/169882/+files/GenericSetup-1.3-branch.diff

with comment:

Attached are two patches. One against GenericSetup in its 1.3 branch and one against CMFCore in the 2.1 branch.

The goal of this patch is to work towards a solution for one part of this issue, namely, citing yuppie: "The adapters should compare the new settings with the existing ones and make sure indexes are only updated if they are actually changed."

For this purpose I added a generally usable function decorator called clearIfChanged that you can stick onto existing _importNode methods. This decorator will compare self.node with the value of self.node after the node was imported. It does so to detect whether anything has changed; if anything has changed, it will call self.context.clear().

I added the tests for this to CMFCore; I must admit I was a bit confused as to where to put them, because CMFCore seems to contain in CMFCore/exportimport/tests/test_catalog.py tests that should maybe live in GenericSetup. I'm happy to move tests around if need be.

Note that the only two indexes that I added this decorator to right now is the DateIndex and the ZCTextIndex. I'll add it to more index adapters once we have consensus that this approach is good, and everything is going into the right place.

@wichert responded:

Looking at that patch it relies on being able to make a reliable XML export of the node so it can compare that with the new state. That is an assumption that you can not make though: only ZCTextIndex has an exporter that exports the extra options correctly. For all other index types this is not implemented. Until that has changed this patch can break sites.

@dnouri replied:

If the export doesn't deal with the same data as the import, then that's a bug in the current handlers, and one that's quite easily fixed, I believe.

I see no need to do the "interface change for catalog indexes in general" that you suggest in an e-mail. Since handlers are index specific, it's fine for them to introspect states of their indexes down to private attribute level.

I'll try and see if I can get all indexes that I'm interested in to export properly. In the meantime, please help me understand how this patch can break sites.

Yvo Schubbe wrote:

Thanks for the patch! 2 comments:

• The export/import adapters are currently cheating extensively. There is no way to write them using just the interface methods, but they claim to adapt the interfaces. Things don't become worse if we use non-API methods for extracting and updating the 'extra' settings - in fact we adapt a specific implementation.
• The tests in CMFCore/exportimport/tests/test_catalog.py are just additional integration tests for the CMF specific catalog tool. GenericSetup contains test_exportimport.py files for the indexes and ZCatalog. All these adapter tests in GS should be modified to test the new behavior.

Toni Mueller ([email protected]) wrote:

It would great to see some progress in this area. We're currently bitten by this bug as well, and re-indexing would take several hours each time. Before solving the problem automatically, I'd like to have just an extension to the syntax of catalog.xml:

Now we have:

<indexed_attr value="some_attribute"/>

It would imho be a big step forward if we had this instead:

<indexed_attr value="some_attribute" [preserve=True/False] [reindex=True/False] />

With those two optional parameters, the decision about how to proceed with indices would not have to be computed, but could simply be specified by the user.

@rochecompaan followed up:

I just hit this bug again today. I don't understand why the PluggableIndexNodeAdapter needs to ever clear the index in _importNode - it makes catalog.xml dangerous and unusable.

Actually not only PluggableIndexNodeAdapter but all the other IndexNodeAdapter in Products/Generic Setup/PluginIndexes/exportimport.py as well.

@djay wrote:

This feature still isn't implemented :(
I might be missing something but does the solution have to involve changing the index api?
Can't we just define the behaviour such that

• purge="True", the index will always get recreated.
• purge="False",if the index id already exists do nothing. if it doesn't then create it. If the extra attributes have changed bad luck.

ToolsetRegistry is not persistent enough

I did a PR #26 more than three years ago, which was released in GenericSetup 1.8.3. It made it possible to move a tool from the required list to the forbidden list, or the other way around.

I now see a problem with that. I have a toolset.xml which does exactly that, and it seems to work, but its resulting changes are not persisted: they are gone after a restart. If more changes are done to portal_setup before a restart, then things might be fine. But not when there is an immediate restart.

I can work on a PR for that.

Let me describe the situation:

• client code with Tool class inheriting from CMFCore.utils.UniqueObject/ImmutableId and id "correct_id".
• toolset = portal_setup.getToolsetRegistry()
• toolset.listForbiddenTools() is an empty list.
• toolset.listRequiredToolInfo() dictionary has one Tool class with correct_id and one with typo_id.
• When I look in the git history of the project, this situation has existed since at least 2012.
• A few months ago I was tired of this typo_id and I manually removed this object in the ZMI.
• Result now: any profile or upgrade with a toolset.xml causes GenericSetup to try to create the missing typo_id but this fails.

I wanted to fix that. I actually already had a correct toolset.xml which should fix it:

<?xml version="1.0"?>
<tool-setup>
  <forbidden tool_id="typo_id"/>
  <required tool_id="correct_id"
            class="client.code.Tool"/>
</tool-setup>

Applying this initially works:

• the forbidden tools list contains typo_id
• the required dictionary only has my class Tool with correct_id.

But after a restart, the old situation is there again, and any other toolset.xml that is applied, throws an error, just like before.
The reason is that the forbidden and required information is stored in a simple list and dictionary instead of in persistent data.
In my case this is in GenericSetup 1.8.9 on Plone 4.3.18, but I see it is the same on the master branch.

I have tried a few things:

• Set toolset._p_changed = True: does not help.
• Add self._forbidden = self._forbidden whenever changing self._forbidden. Same for self._required. Does not help.
• Set portal_setup._p_changed = True: helps.

I expected that one of the first two would help, because ToolsetRegistry inherits from Acquisition.Implicit. But that is maybe only when it inherits from Explicit.

The question is then where to fix it.

1. If we try to fix it in the ToolsetRegistry class, then we would need to call something like self.__parent__._p_changed = True which seems wrong: it accesses a private variable of the parent.
2. We could change it in the SetupTool class: after calling toolset.parseXML we can set self._p_changed to True. It might need to be done in more cases, but at least this is the main one.
3. Possibly we could change the list and dictionary to their persistent counterparts on the fly, but that seems more invasive, with more chance of introducing bugs.

I think I will go with number 2.

Drop support for `python setup.py test`

This allows to drop the eggtestinfo setup dependency.

@tseaver Are you okay with this suggestion?

Export/import of local utilities considered harmful

In https://bugs.launchpad.net/zope-cmf/+bug/281216, @cjw296 reported:

For background see: http://dev.plone.org/plone/ticket/8579

Summary of the problem: You cannot re-constitute local utilities just from a dotted name:

• the constructor may take parameters. Indeeed, imnsho, this is how utilities that require configuration should be configured.
• the utilities may be changed after registration, as is the case with PortletType utilities.

This is with Plone 3.1.1 and associated CMF and Zope versions.

GenericSetup upgrade does not work in cluster environments.

BUG/PROBLEM REPORT (OR OTHER COMMON ISSUE)

What I did:

Plone cluster with 9 instances. Run upgrade steps or profiles.

What I expect to happen:

Profiles or upgrades are applied

What actually happened:

They are not.

What version of Python and Zope/Addons I am using:

Python 3.9.8 Zope 5 latest, Plone

Analysis

GenericSetup uses keys generated by hash(). Since hash is only the same inside one running Python each instance has its own set of keys. In a cluster the keys of one instance are used in the form and processed by another instance with different keys it can not look up.
This is probably a Python3 change on hash

Solution

Use a stable hashing algorithm to build the hash.

Import step dependecies relying on circular dependencies may result in bad order

In https://bugs.launchpad.net/zope-cmf/+bug/314010, Bertrand Mathieu (@bmathieu?) reported:

an example is here: http://dev.plone.org/plone/ticket/8350

if 'my-step' must be executed before 'mandatory-step', and 'mandatory-step' is involved in a circular dependency chain (with 'circular-step' for example), then sorted imported steps will be: ['my-step', 'mandatory-step', 'circular-step'].

This is reproducible if you define 'my-step' with zcml, and the other steps in a profile (with import_steps.xml).

I have posted a monkey patch in plone ticket system #8350 (in order to help others with a hopefully good workaround). There is also doctest that should fail with current implementation and succeed with mine. The idea is to change final loop and insert all steps detected involved in a circular loop, and then trying to resolve remaining dependencies before eventually inserting unresolved steps.

I don't know if it is the best way to solve this problem, but at least it should be a good start.