zopefoundation / products.genericsetup Goto Github PK
View Code? Open in Web Editor NEWMini-framework for expressing the configured state of a Zope Site as a set of filesystem artifacts
License: Other
Mini-framework for expressing the configured state of a Zope Site as a set of filesystem artifacts
License: Other
Plone 5.2RC5, GS 2.0
setup_layer(options, layer, setup_layers)
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.testrunner-5.0-py3.7.egg/zope/testrunner/runner.py", line 787, in setup_layer
setup_layer(options, base, setup_layers)
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.testrunner-5.0-py3.7.egg/zope/testrunner/runner.py", line 792, in setup_layer
layer.setUp()
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/plone.app.testing-6.1.3-py3.7.egg/plone/app/testing/helpers.py", line 374, in setUp
self.setUpZope(portal.getPhysicalRoot(), configurationContext)
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/src/ugent.vacancy/src/ugent/vacancy/tests/base.py", line 37, in setUpZope
xmlconfig.file("configure.zcml", mod, context=configurationContext)
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.configuration-4.3.1-py3.7.egg/zope/configuration/xmlconfig.py", line 659, in file
context.execute_actions()
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.configuration-4.3.1-py3.7.egg/zope/configuration/config.py", line 800, in execute_actions
None, sys.exc_info()[2])
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.configuration-4.3.1-py3.7.egg/zope/configuration/_compat.py", line 30, in reraise
raise value.with_traceback(tb)
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/zope.configuration-4.3.1-py3.7.egg/zope/configuration/config.py", line 791, in execute_actions
callable(*args, **kw)
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0-py3.7.egg/Products/GenericSetup/registry.py", line 438, in registerStep
if already and already['version'] > version:
zope.configuration.config.ConfigurationExecutionError: File "/home/ajung/sandboxes/ugent-portaal-plone-4x/src/ugent.ldap/src/ugent/ldap/configure.zcml", line 92.2-97.6
<genericsetup:importStep
name="ugent.ldap-setuphandlers"
title="ugent.ldap Setuphandlers"
description="Additional import step for package ugent.ldap"
handler="ugent.ldap.setuphandlers.import_various"
/>
TypeError: '>' not supported between instances of 'NoneType' and 'NoneType'
The version
parameter and the already['version']
are both None
.
Possibly a bad package configuration. However the error message is not directly helpful here.
(Pdb) args
self = <Products.GenericSetup.registry.ImportStepRegistry object at 0x7efc0eae2860>
id = 'ugent.ldap-setuphandlers'
version = None
handler = <function import_various at 0x7efc06d16158>
dependencies = ()
title = 'ugent.ldap Setuphandlers'
description = 'Additional import step for package ugent.ldap'
(Pdb) pp already
{'dependencies': (),
'description': 'Additional import step for package ugent.ldap',
'handler': 'ugent.ldap.setuphandlers.import_various',
'id': 'ugent.ldap-setuphandlers',
'invalid': False,
'title': 'ugent.ldap Setuphandlers',
'version': None}
(Pdb)
While creating PR #110, I ran into code and tests that tried to support other encodings, for example iso-8859-1
, at least when importing an XML file. But some parts of the code ignore this on Python 3.
So I wonder: does GenericSetup on Python 3 only support utf-8?
If so, then some code could be simplified, for example the code in that PR.
But simplifying may need to wait until we drop support for Python 2, and/or we require Zope 5.3, where lines
properties are text instead of bytes.
I wonder the same about Zope 5: does that only support bytes encoded as utf-8, at least when stored in the Data.fs?
Note: since Plone 4.3, core Plone assumes all bytes that are read from Data.fs are utf-8
encoded (or simple ascii), where previously you could set a default_charset
property (preferably only set it once, never change it). See this commit from 2012.
Python 3.6, Products.PythonScripts-4.6-py3.6.egg
I get this error while importing an older workflow definition containing some PythonScripts in Plone 5.2:
<string>(3)installProducts()
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/AccessControl-4.0b6-py3.6-linux-x86_64.egg/AccessControl/requestmethod.py(85)_curried()
-> return callable(*args, **kw)
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.CMFQuickInstallerTool-4.0.0-py3.6.egg/Products/CMFQuickInstallerTool/QuickInstallerTool.py(710)installProducts()
-> omitSnapshots=omitSnapshots
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.CMFQuickInstallerTool-4.0.0-py3.6.egg/Products/CMFQuickInstallerTool/QuickInstallerTool.py(626)installProduct()
-> blacklisted_steps=blacklistedSteps,
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/tool.py(393)runAllImportStepsFromProfile()
-> dependency_strategy=dependency_strategy)
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/tool.py(1465)_runImportStepsFromContext()
-> message = self._doRunImportStep(step, context)
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/tool.py(1277)_doRunImportStep()
-> return handler(context)
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.CMFCore-2.4.0b8-py3.6.egg/Products/CMFCore/exportimport/workflow.py(128)importWorkflowTool()
-> importObjects(tool, '', context)
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/utils.py(890)importObjects()
-> importObjects(sub, path + '/', context)
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.GenericSetup-2.0b6-py3.6.egg/Products/GenericSetup/utils.py(886)importObjects()
-> importer.body = body
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.DCWorkflow-2.4.0b2-py3.6.egg/Products/DCWorkflow/exportimport.py(92)_importBody()
-> self.environ)
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.DCWorkflow-2.4.0b2-py3.6.egg/Products/DCWorkflow/exportimport.py(981)_initDCWorkflow()
-> _initDCWorkflowScripts(workflow, scripts, context)
> /home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.DCWorkflow-2.4.0b2-py3.6.egg/Products/DCWorkflow/exportimport.py(1150)_initDCWorkflowScripts()
-> script.write(file)
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.PythonScripts-4.6-py3.6.egg/Products/PythonScripts/PythonScript.py(416)write()
-> m = _nonempty_line.search(text, st)
File "/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.PythonScripts-4.6-py3.6.egg/Products/PythonScripts/PythonScript.py", line 416, in write
m = _nonempty_line.search(text, st)
TypeError: cannot use a string pattern on a bytes-like object
/home/ajung/sandboxes/ugent-portaal-plone-4x/eggs/Products.PythonScripts-4.6-py3.6.egg/Products/PythonScripts/PythonScript.py(419)write()
-> if not m:
(Pdb) args
self =
text = b'## Script (Python) "setPortalMessage"\n##bind container=container\n##bind context=context\n##bind namespace=\n##bind script=script\n##bind subpath=traverse_subpath\n##parameters=data\n##title=\n##\ncontext.plone_utils.addPortalMessage(u'Opgelet : door de status van dit item aan te passen naar concept haalt u deze pagina offline en is het niet meer toegankelijk', 'warn')\n'
Add a class variable zmi_show_add_dialog
to the GS tool class so no modal add dialog is shown during creation.
I have a package with two upgrade steps. For each upgrade step source = 1, destination = 1, and sortkey = 1. When I try to list the old upgrades, I get the following exception:
Traceback (innermost last):
Module ZPublisher.WSGIPublisher, line 142, in transaction_pubevents
Module ZPublisher.WSGIPublisher, line 295, in publish_module
Module ZPublisher.WSGIPublisher, line 229, in publish
Module ZPublisher.mapply, line 85, in mapply
Module ZPublisher.WSGIPublisher, line 57, in call_object
Module Shared.DC.Scripts.Bindings, line 330, in call
Module Shared.DC.Scripts.Bindings, line 367, in _bindAndExec
Module Products.PageTemplates.PageTemplateFile, line 141, in _exec
Module Products.PageTemplates.PageTemplate, line 83, in pt_render
Module zope.pagetemplate.pagetemplate, line 135, in pt_render
Module Products.PageTemplates.engine, line 90, in call
Module z3c.pt.pagetemplate, line 173, in render
Module chameleon.zpt.template, line 306, in render
Module chameleon.template, line 209, in render
Module chameleon.utils, line 75, in raise_with_traceback
Module chameleon.template, line 187, in render
Module 561e1882708296964a293b8943c621a1.py, line 685, in render
Module Products.GenericSetup.tool, line 1002, in listUpgrades
Module Products.GenericSetup.upgrade, line 270, in listUpgradeSteps
TypeError: '<' not supported between instances of 'dict' and 'dict'
I get this problem with Python 3, but not with Python 2. It looks like the method listUpgradeSteps tries to sort a list of upgrade steps by comparing dicts, which works in Python 2, but not in Python 3.
There seems to be an issue with #16
After this was merged and enabled in the Plone 5 coredev the following problem was observed the workflow 'comment_review_workflow' from plone.app.discussion
is not installed upon creating a new site.
Steps to reproduce:
In https://bugs.launchpad.net/zope-cmf/+bug/1227290, @seanupton reported:
If a GenericSetup base profile defines a set of roles for a permission in its rolemap.xml, it is impossible to supplement (append) roles to that permission from an extension profile. Rather, the extension profile rolemap.xml must include duplicative (all) roles (for each permission) listed in upstream base profile and cannot merely supplement.
I have verified the outcome of this looking at manage_access in a Plone site, and by reviewing the code.
AccessControl.rolemanager.RoleManager.manage_permission() is passed only what is explicitly in the rolemap.xml for an extension profile by Products.GenericSetup.rolemap.importRolemap(). Permission.setRoles() does a setattr() replacement of the roles tuple/list in each permission. The consequence of this is that it is impossible to merge/supplement/append roles.
The work-around is to copylift and duplicate all roles listed in upstream package profiles.
Note: ac_roles is fine, is supplemented correctly. This is a per-permission mapping issue.
Versions: Zope: 2.13.21, Products.GenericSetup 1.7.4, Products.CMFCore 2.2.7 on Python 2.7.3
Products.GenericSetup, v1.7.6:
In Products/GenericSetup/context.py, there are two comparisons of path[-1] to ‘/’, which must fail for empty strings. IMO, here the endswith method must be used.
The empty string seems to be a known special case for tarballs.
See pull request #13; this ticket is identical with https://bugs.launchpad.net/zope2/+bug/1479268.
Should that be something desirable? Plone 5 already made a strong move out of zope.formlib in favor of z3c.form and one of the only few packages that do still depend on it is Products.GenericSetup: https://github.com/zopefoundation/Products.GenericSetup/search?utf8=%E2%9C%93&q=formlib
I installed a package with two namespace levels foo.bar.baz
and inside i have an import step handler foo.bar.baz.exportimport.import_handler
as development package or from an sdist.
Generic Setup finds the function and uses it.
It does not and log about an invalid import handler.
It used to work in Plone 5.2/buildout based installation.
For more Info, the story behind this report and the (possible) solution see https://community.plone.org/t/dotted-name-resolving-fails/14810
A PR will evolve at #117
Run an instance of Plone 5.2 in a Python 3
When you run a snapshot comparison in ZMI - portal_setup you get an error
Seems related to the str - bytes change in python 3, code is expecting bytes, but getting str
example buildout.cfg:
[buildout]
parts = instance
extends = https://dist.plone.org/release/5.2-latest/versions.cfg
[instance]
recipe = plone.recipe.zope2instance
wsgi = on
eggs =
Plone
Pillow
traceback:
2019-09-24 10:08:13,548 INFO [GenericSetup.workflow:98][waitress] Workflow tool exported.
/Users/KP/projects/plone/minimalplone52/eggs/Zope-4.1.1-py3.7.egg/OFS/ObjectManager.py:463: ResourceWarning: unclosed file <_io.FileIO name=19 mode='rb+' closefd=True>
return [(id, self._getOb(id)) for id in self.objectIds(spec)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/KP/projects/plone/minimalplone52/eggs/Zope-4.1.1-py3.7.egg/OFS/ObjectManager.py:463: ResourceWarning: unclosed file <_io.FileIO name=20 mode='rb+' closefd=True>
return [(id, self._getOb(id)) for id in self.objectIds(spec)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/KP/projects/plone/minimalplone52/eggs/Zope-4.1.1-py3.7.egg/OFS/ObjectManager.py:463: ResourceWarning: unclosed file <_io.FileIO name=21 mode='rb+' closefd=True>
return [(id, self._getOb(id)) for id in self.objectIds(spec)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
/Users/KP/projects/plone/minimalplone52/eggs/Zope-4.1.1-py3.7.egg/OFS/ObjectManager.py:463: ResourceWarning: unclosed file <_io.FileIO name=23 mode='rb+' closefd=True>
return [(id, self._getOb(id)) for id in self.objectIds(spec)]
ResourceWarning: Enable tracemalloc to get the object allocation traceback
2019-09-24 10:08:13,605 INFO [GenericSetup.contenttypes:98][waitress] Content type registry exported.
2019-09-24 10:08:56,450 ERROR [Zope.SiteErrorLog:251][waitress] 1569312536.4484330.7780034037876496 http://localhost:8080/Plone/portal_setup/manage_showDiff
Traceback (innermost last):
Module ZPublisher.WSGIPublisher, line 155, in transaction_pubevents
Module ZPublisher.WSGIPublisher, line 337, in publish_module
Module ZPublisher.WSGIPublisher, line 255, in publish
Module ZPublisher.mapply, line 85, in mapply
Module ZPublisher.WSGIPublisher, line 61, in call_object
Module Shared.DC.Scripts.Bindings, line 335, in __call__
Module Shared.DC.Scripts.Bindings, line 372, in _bindAndExec
Module Products.PageTemplates.PageTemplateFile, line 145, in _exec
Module Products.PageTemplates.PageTemplate, line 85, in pt_render
Module zope.pagetemplate.pagetemplate, line 135, in pt_render
Module Products.PageTemplates.engine, line 88, in __call__
Module z3c.pt.pagetemplate, line 173, in render
Module chameleon.zpt.template, line 306, in render
Module chameleon.template, line 209, in render
Module chameleon.utils, line 75, in raise_with_traceback
Module chameleon.template, line 187, in render
Module 653a823b417046947bff895cf59cce43, line 714, in render
Module Products.GenericSetup.tool, line 464, in markupComparison
TypeError: startswith first arg must be bytes or a tuple of bytes, not str
- Expression: "python: context.markupComparison( comparison )"
- Filename: manage_showDiff
- Location: (line 129: col 29)
- Arguments: template: <ImplicitAcquisitionWrapper manage_showDiff at 0x11c1ea230>
here: <ImplicitAcquisitionWrapper portal_setup at 0x11036f050>
context: <ImplicitAcquisitionWrapper portal_setup at 0x11036f050>
container: <ImplicitAcquisitionWrapper portal_setup at 0x11036f050>
nothing: <NoneType - at 0x109854058>
options: {...} (1)
root: <ImplicitAcquisitionWrapper at 0x1152f80f0>
request: <WSGIRequest - at 0x11727e410>
modules: <_SecureModuleImporter - at 0x10c0e8d10>
user: <ImplicitAcquisitionWrapper - at 0x11c0d56e0>
default: <object - at 0x10a9baf40>
repeat: {...} (0)
loop: {...} (1)
wrapped_repeat: <SafeMapping - at 0x11c1eabe0>
target_language: <NoneType - at 0x109854058>
translate: <function translate at 0x11401f7a0>
lhs: snapshot-snapshot-20190924080710
rhs: snapshot-snapshot-20190924080812
mae: 0
ib: 0
mcc: <method manage_compareConfigurations at 0x1209fe0a0>
comparison: <bytes - at 0x7f91bf9c9600>
See changes in tool.py in macbook icts kim
/Users/u0045445/projects/plone/minimal52/eggs/Products.GenericSetup-2.0b5-py3.6.egg/Products/GenericSetup/tool.py
See also: python - startswith TypeError in function - Stack Overflow
After changing some str to bytes (b''
) and line to bytes(line)
Module chameleon.template, line 183, in render
Module 5f634d11d3bcaaa0faf3cdadf322dc59.py, line 704, in render
Module Products.GenericSetup.tool, line 500, in markupComparison
Module Products.GenericSetup.tool, line 500, in <listcomp>
Module html, line 19, in escape
TypeError: a bytes-like object is required, not 'str'
- Expression: "python: context.markupComparison( comparison )"
- Filename: manage_showDiff
- Location: (line 129: col 29)
- Arguments: template: <ImplicitAcquisitionWrapper manage_showDiff at 0x11225ca20>
here: <ImplicitAcquisitionWrapper portal_setup at 0x11225c8b8>
context: <ImplicitAcquisitionWrapper portal_setup at 0x11225c8b8>
container: <ImplicitAcquisitionWrapper portal_setup at 0x11225c8b8>
nothing: <NoneType - at 0x10091f810>
options: {...} (1)
root: <ImplicitAcquisitionWrapper at 0x1122dda20>
request: <WSGIRequest - at 0x119f865f8>
modules: <_SecureModuleImporter - at 0x10eba2518>
user: <ImplicitAcquisitionWrapper - at 0x11225c798>
default: <object - at 0x101e95a90>
repeat: {...} (0)
loop: {...} (1)
wrapped_repeat: <SafeMapping - at 0x11a1e3bc8>
target_language: <NoneType - at 0x10091f810>
translate: <function translate at 0x11338f8c8>
lhs: snapshot-snapshot-20190924141937
rhs: snapshot-snapshot-20190924142041
mae: 0
ib: True
mcc: <method manage_compareConfigurations at 0x1190a7748>
comparison: <bytes - at 0x1296e9000>
`
Refs: #64, plone/Products.CMFPlone#2228.
The multiple errors still occur, I think the culprit lies in getProfileDependenciesChain
, because it is calling itself recursively:
And the try/except-block in _runImportStepsFromContext
of where it's called initially, is masking this:
Probably because the first try succeeds and later an exception occurs from a recursive-call.
So, I tried raising an error explicitly in getProfileDependenciesChain
, but that also doesn't stop the execution of further code.
The only solution I came up with, is to return None in getDependenciesForProfile
for symbolizing dependencies are broken, like it was sugested in #64:
dd11374
That will cause the interpreter to raise a "not-iterable"-error in getProfileDependenciesChain
and fail hard, as wished.
@tseaver, @icemac, @sunew, if you think this is feasible, I'd like to provide another PR.
In https://bugs.launchpad.net/zope-cmf/+bug/501668, @dataflake reported:
When visiting a setuptool in the ZMI it shows all registered base profiles on the Profiles tab and all registered extension profiles on the Import tab. However, all those are registered for specific interfaces, and still show up even if the tool's container does not provide them.
The code that generates the list of available profiles should check the tool container's interfaces and only list those profiles that are appropriate for the container.
Zope5.2.1 on Mac OS 10.13.5 Python3.7.7
From the ZMI I choosed "Configured PAS" and the action generate the following error:
2021-06-09 11:27:16 ERROR [waitress:404][waitress-0] Exception while serving /oneMoreTestFolder/manage_addProduct/PluggableAuthService/addConfiguredPAS
Traceback (most recent call last):
File "/Applications/zope521bo/eggs/waitress-2.0.0-py3.7.egg/waitress/channel.py", line 397, in service
task.service()
File "/Applications/zope521bo/eggs/waitress-2.0.0-py3.7.egg/waitress/task.py", line 168, in service
self.execute()
File "/Applications/zope521bo/eggs/waitress-2.0.0-py3.7.egg/waitress/task.py", line 434, in execute
app_iter = self.channel.server.application(environ, start_response)
File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/httpexceptions.py", line 30, in __call__
return self.application(environ, start_response)
File "/Applications/zope521bo/eggs/Paste-3.5.0-py3.7.egg/paste/translogger.py", line 69, in __call__
return self.application(environ, replacement_start_response)
File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/WSGIPublisher.py", line 376, in publish_module
response = _publish(request, new_mod_info)
File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/WSGIPublisher.py", line 279, in publish
bind=1)
File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/mapply.py", line 85, in mapply
return debug(object, args, context)
File "/Applications/zope521bo/eggs/Zope-5.2.1-py3.7.egg/ZPublisher/WSGIPublisher.py", line 68, in call_object
return obj(*args)
File "/Applications/zope521bo/eggs/Products.PluggableAuthService-2.6.3-py3.7.egg/Products/PluggableAuthService/PluggableAuthService.py", line 1419, in addConfiguredPAS
tool.runAllImportStepsFromProfile('profile-%s' % base_profile)
File "/Applications/zope521bo/eggs/Products.GenericSetup-2.1.1-py3.7.egg/Products/GenericSetup/tool.py", line 405, in runAllImportStepsFromProfile
dependency_strategy=dependency_strategy)
File "/Applications/zope521bo/eggs/Products.GenericSetup-2.1.1-py3.7.egg/Products/GenericSetup/tool.py", line 1483, in _runImportStepsFromContext
message = self._doRunImportStep(step, context)
File "/Applications/zope521bo/eggs/Products.GenericSetup-2.1.1-py3.7.egg/Products/GenericSetup/tool.py", line 1295, in _doRunImportStep
return handler(context)
File "/Applications/zope521bo/eggs/Products.PluggableAuthService-2.6.3-py3.7.egg/Products/PluggableAuthService/exportimport.py", line 29, in importPAS
IFilesystemImporter(context.getSite()).import_(context, 'PAS', True)
File "/Applications/zope521bo/eggs/Products.GenericSetup-2.1.1-py3.7.egg/Products/GenericSetup/content.py", line 159, in import_
stream = cStringIO(objects)
TypeError: initial_value must be str or None, not bytes
Log file objects stored in Generic Setup Tool instances (and the tool instances themselves) don't have any specific permission settings applied, they just acquire all settings from their container. If anonymous users have e.g. View
permission on the enclosing container the log files are accessible to anonymous users if they can guess the file name. This is an information disclosure bug.
P.S.: The snapshots
folder is affected as well, but actual snapshot data is not visible, only the page templates can be listed.
In https://bugs.launchpad.net/zope-cmf/+bug/924721, @rpatterson reported:
It appears the ZCML handler for the upgradeSetp directive doesn't include the profile id in the discriminator. This means that if you register the same handler for two different profiles but with the same source and destination versions, you get a ConfigurationConflictError.
See PR #61 which temporarily fixes it by pinning Products.ZCatalog back to 4.0.1.
I looked at the test runs in GHA.
e. g. https://github.com/zopefoundation/Products.GenericSetup/actions/runs/4274869009/jobs/7441795588
Green tests.
Tests are broken probably due to changes in newer setuptools versions resp. its bundled packaging
module.
Cc-ing some contributors:
Todo:
The GenericSetup PyPI page points to documentation at http://packages.python.org/Products.GenericSetup/ and https://pythonhosted.org/Products.GenericSetup/. The first url redirects to the second one.
What we see there, is documentation for version 1.7.0. Does anyone know how to update this?
Alternatively, we could rearrange things and move to readthedocs.
In https://bugs.launchpad.net/zope-cmf/+bug/530943, @do3cc reported:
When trying to modfiy the peristent registry with the manage_components tab, it usually fails if browserlayers are registered as utilities.
The code tries to generate a xml configuration for the utility and assumes type(utility) as the factory for the BrowserLayers. The BrowserLayer utility is an interface, type resorts to zope.interface.interface.InterfaceClass. This cannot be called without arguments. Thus when loading the new utilities, it fails for Browserlayers.
and attached a patch for a testcase to reproduce the issue:
https://bugs.launchpad.net/zope-cmf/+bug/530943/+attachment/1173798/+files/patch
with the comment:
Here is a traceback:
Module plone.postpublicationhook.hook, line 74, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 42, in call_object
Module zope.formlib.form, line 769, in __call__
Module Products.Five.formlib.formbase, line 55, in update
Module zope.formlib.form, line 750, in update
Module zope.formlib.form, line 594, in success
Module Products.Five.formlib.formbase, line 69, in handle_edit_action
Module zope.formlib.form, line 528, in applyChanges
Module zope.schema._bootstrapfields, line 183, in set
Module Products.GenericSetup.browser.components, line 55, in _setBody
Module Products.GenericSetup.utils, line 544, in _importBody
Module Products.GenericSetup.components, line 76, in _importNode
Module Products.GenericSetup.components, line 173, in _initUtilities
TypeError: __init__() takes at least 2 arguments (1 given)
This code is version 1.4.5
I have written a test for it.
I am not sure what a clean way of fixing would be. It seems to me that the code tries to create its own xml serialization and fails. I am not sure whether having your own serialisation here makes sense.
Later @do3cc wrote:
I talked with hannosch about the issue. In the end, the view never really worked in all cases, but some objected to the removal.
Since it is easily feasable to get a list of all utilities and adapters, but not so easy to serialize and deserialize them, a potential fix would be to create a new simple ui that lists local adapters and utilities and offers a delete action for each.
Yvo Schubbe responded:
Thanks for the report! AFAICS this issue is not specific to the manage_components tab, your testcase is for the ComponentRegistryXMLAdapter that is primarily used for importing and exporting GenericSetup profiles.
ComponentRegistryXMLAdapter has its limitations and a real UI for five.localsitemanager site managers would be better than the current view, but that's a completely different issue.
Your testcase just shows that imports using InterfaceClass as factory don't work, but that's not really the problem. The real issue seems to be the export/serialization. And if you report issues with the ComponentRegistryXMLAdapter please report them against version 1.5+ because it has many components handler fixes 1.4.5 does not contain.
@flipmcf followed up:
sorry to raise the dead here, but I'm seeing this behavior also with an 3rd party add-on which will go unnamed.
I may have a workaround, but it's unconfirmed:
I think a browserlayer.xml like this will create the utility which exports in componentregistry with the bad factory:
<?xml version="1.0"?>
<layers>
<layer name="MyLayer"
interface="IMyLayer"/>
</layers>
But a browserlayer.xml like this:
<?xml version="1.0"?>
<layers name="MyLayers" meta_type="ComponentRegistry">
<layer name="MyLayer"
interface="IMyLayer"/>
</layers>
Seems to avoid the issue and doesn't muck about with componentregistry's export process.
Would love to get a confirmation of this from someone, and an official "Nod of the head" that the 2nd example of a browserlayer.xml generic setup profile is the "Correct" way to do it.
@flipmcf later wrote:
Please disregard previous message. I experimented and saw no difference.
and then again:
I would like to raise the severity of this issue as it seems related to causing zope to spin in some situations. Our site administration workflow depends on versioned, up-to-date GenericSetup configuration profiles and running import steps from the ZMI when production changes occur. We have too many production instances to point-and-click our way through the ZMI for each change that deploys to production.
Zope2-2.13.8-py2.7
zope.component-3.9.5-py2.7
Products.CMFCore-2.2.4-py2.7To reproduce, I would use the package 'quintagroup.plonegooglesitemaps' which registers a browser layer upon installation.
Install this product (it's quite easy) and configure a news-sitemap.xml
Request the news-sitemap.xml through a browser request, and note it works just fine - you get some xml - empty is ok.
- Export local compoent registry (componentregistry.xml) and re-import it: This aggrivates the bug that Patric Gerken describes. Essentially at this point, you cannot save and version your ZMI settings (like we do) because imports will not work anymore.
- Next: Comment out the offending utility in componentregistry.xml and import again. Of course, now it will work.
- Access the news-sitemap.xml you created above. Zope will spin at %100 cpu and can only be fixed through a hard restart of the instance.
- Fix by running the browserlayer import step (return to step 1)
I have a few traceback / dumps from my spinning zope process that I can provide upon request.
First of all, we would like to thank @mauritsvanrees for the 1.8.2 release that really improved the usability of this package and the usage of upgradeSteps since because of that improvements we had the idea of this issue.
For the sake of trying to improve Products.GenericSetup even more, even with just suggestions (we're sorry we can't yet do PR's for plone and zope packages, we're still addressing some internal corporate issues with contributor's agreements), we would like to know if it's possible to have a historic record of when specific upgradeSteps were executed.
Why are we asking this? Because you can have 20 upgradeSteps, you can execute just the last one and it's impossible to know which upgradeSteps before the last one were run or not. You already have an upgradeStep internal id, the next step would be to have a persisted list of timestamps when that id was executed/imported.
Since an image is worth a thousand words, something like this comes to mind:
Of course the layout doesn't need to be like this, you can have a link in the upgradeStep title for example. We don't care which layout is chosen as long as the historic record of every upgradeStep import is shown.
Challenges: since we have systems that already run the upgradeSteps but we don't have it's historic, if a new version of Products.GenericSetup would implement this, "NOT EXECUTED" would be incorrect in that case, something along the lines of "Unknown" would be more appropriate.
This line is causing my watchdog staging environment to fail, because the referenced file was removed at zopefoundation/Zope@f15d25c
Not sure how to best deal with this, maybe have an alternative ready and fall back to that if the module isn't there anymore with a comment / deprecation to to remove the fallback after zope 5 is released?
If I understand it correctly the main thing about pre_handler
and post_handler
is to replace the cumbersome registration of import-steps. Import-steps were not passed the GenericSetup-Tool but the Products.GenericSetup.context.DirectoryImportContext
which has the nice helper _profile_path
and readDataFile
which allowed to read various files (e.g. a csv-file with data from which to create users or content). Now have to do this by hand which makes switching existing code from import-steps to post_handlers a little more work than would be nice.
I just want to ask if there is any special reason why we don't pass context
in Products.GenericSetup.tool.SetupTool._runImportStepsFromContext
to _doRunHandler
and then pass that to the handler.
situation
set up a plone 5.1a2 site.
create a snapshot in portal_setup.
change the day the week should start with in the control panel "Date and Time Settings".
create a snapshot in portal_setup again and make a diff between those two.
i then safed this changed settings to my registry.xml
<record name="plone.first_weekday" interface="Products.CMFPlone.interfaces.controlpanel.IDateAndTimeSchema" field="first_weekday">
<field type="plone.registry.field.Choice">
<description xmlns:ns0="http://xml.zope.org/namespaces/i18n" ns0:domain="plone" ns0:translate="help_first_weekday">First day in the week.</description>
<title xmlns:ns0="http://xml.zope.org/namespaces/i18n" ns0:domain="plone" ns0:translate="label_first_weekday">First weekday</title>
<vocabulary>plone.app.vocabularies.Weekdays</vocabulary>
</field>
<value>3</value>
</record>
now i set up another plone site and also install my package with this settings stored inside and get an error:
Traceback (innermost last):
Module ZPublisher.Publish, line 138, in publish
Module ZPublisher.mapply, line 77, in mapply
Module ZPublisher.Publish, line 48, in call_object
Module Products.CMFPlone.browser.admin, line 272, in __call__
Module Products.CMFPlone.factory, line 168, in addPloneSite
Module Products.GenericSetup.tool, line 379, in runAllImportStepsFromProfile
- __traceback_info__: profile-akbild.site:default
Module Products.GenericSetup.tool, line 1414, in _runImportStepsFromContext
Module Products.GenericSetup.tool, line 1226, in _doRunImportStep
- __traceback_info__: plone.app.registry
Module plone.app.registry.exportimport.handler, line 47, in importRegistry
Module plone.app.registry.exportimport.handler, line 90, in importDocument
Module plone.app.registry.exportimport.handler, line 246, in importRecord
- __traceback_info__: record name: plone.first_weekday
Module plone.supermodel.utils, line 144, in elementToValue
Module zope.schema._field, line 322, in fromUnicode
Module zope.schema._bootstrapfields, line 183, in validate
Module zope.schema._field, line 338, in _validate
ConstraintNotSatisfied: (u'0', '')
0 -> Monday (Europe), but every other number from 0 (monday) to 6 (sunday) generates the same error.
why does that happen?
the choice field (zope.schema) validates a given value (the one from the xml file -> u'0') against its vocabulary, using by_value, which is - in case of plone.app.vocabularies.datetimerelated.WeekdaysFactory - a dict of number (keys) and simpleterm (value) - pairs. but the value from the xml is a unicode string, not an int. -> so its not a key in the vocabulary. see zope.schema._field.Choice._validate() for that.
u'0' in {0:term_a, 1:term_b} == False
further down the rabbit hole...
most SimpleTerms in plone.app.vocabularies are instantiated by a value (string) and a title (term); only the Month and Weekday Vocabularies are build different. the come with a value, token and title. value is an index, token a string representation of that index and the title (term). this is, at least, inconsistent.
solutions
i will implement the first solution, as i am not able (too heavy) to extend the Choice field or reimplement the whole GS-setup infrastructure.
take a look at:
plone/Products.CMFPlone#1794
and
plone/plone.app.vocabularies#41
sadly they are cross depending ...
I first saw this in a Plone 6 test run when I switch from Zope 5.2.1 to 5.3.
Sample failure locally with GenericSetup tox:
Failure in test test__initProperties_nopurge_base (Products.GenericSetup.tests.test_utils.PropertyManagerHelpersTests)
Traceback (most recent call last):
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 593, in run
self._callTestMethod(testMethod)
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 550, in _callTestMethod
method()
File "/Users/maurits/community/plone-coredev/6.0/src/Products.GenericSetup/src/Products/GenericSetup/tests/test_utils.py", line 669, in test__initProperties_nopurge_base
self.assertEqual(obj.getProperty('lines3'), (b'Gee', b'Foo', b'Bar'))
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 831, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 1048, in assertTupleEqual
self.assertSequenceEqual(tuple1, tuple2, msg, seq_type=tuple)
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 1019, in assertSequenceEqual
self.fail(msg)
File "/usr/local/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/unittest/case.py", line 670, in fail
raise self.failureException(msg)
AssertionError: Tuples differ: ('Foo', 'Gee', b'Foo', b'Bar') != (b'Gee', b'Foo', b'Bar')
First differing element 0:
'Foo'
b'Gee'
First tuple contains 1 additional elements.
First extra element 3:
b'Bar'
- ('Foo', 'Gee', b'Foo', b'Bar')
? ^^^^^^^
+ (b'Gee', b'Foo', b'Bar')
? ^
So there is a mixup between bytes and strings.
With tox, the Python 2.7 and 3.5 tests pass, because they use Zope 4.x.
On the newer Pythons, 6 tests fail. They pass when I change buildout.cfg
to extend the Zope 5.2.1 versions.cfg
.
When I edit tox.ini
to let Python 3.6, 3.7 and 3.8 use Zope4 (buildout4.cfg
) those tests pass.
The failure above is for this line. The test imports _NOPURGE_IMPORT
Ignoring the test failure for lines3
for a moment, let's look at what happens with the lines1
field. I think the following is a bit suspicious:
getProperty
, you indeed get a tuple of strings.getProperty
gives a tuple of bytes.I don't know yet if this is a problem in practice, or if the tests just need to be fixed.
I wonder what changed in Zope 5.3 for this. Maybe zopefoundation/Zope#962 is involved.
There are many deprecation warnings issued by the tests and the flake8 check (tox -eflake8
) complains about many violations.
It would be nice to get them fixed. (Flake 8 could even be tested on Travis CI to ensure Flake8 compatibility for the future.)
See
I don't see why we need to raise an explicit error instead of converting to utf-8 if needed under the hood?
Products/GenericSetup/components.py:26: DeprecationWarning: IComponentRegistry is deprecated. Import from zope.interface.interfaces
Products/GenericSetup/utils.py:365: DeprecationWarning: cgi.escape is deprecated, use html.escape instead
Products/GenericSetup/utils.py:373: DeprecationWarning: cgi.escape is deprecated, use html.escape instead
In https://bugs.launchpad.net/zope-cmf/+bug/161726, Yvo Schubbe reported:
Like in ZCML, 'interface' should be pulled from the component if not specified.
The flake8-html plugin wants to write into a subdirectory of parts - but the parts directory is not present. The flake8 check runs, but obviously not html gets generated.
❯ tox -e=flake8
flake8 installed: flake8==3.8.1,flake8-debugger==3.2.1,flake8-html==0.4.1,importlib-metadata==1.6.0,Jinja2==2.11.2,MarkupSafe==1.1.1,mccabe==0.6.1,Products.GenericSetup==2.0.3.dev0,pycodestyle==2.6.0,pyflakes==2.2.0,Pygments==2.6.1,zipp==3.1.0
flake8 run-test-pre: PYTHONHASHSEED='1702982963'
flake8 run-test: commands[0] | - flake8 --format=html --htmldir=/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/parts/flake8 --doctests Products setup.py
Traceback (most recent call last):
File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/bin/flake8", line 10, in <module>
sys.exit(main())
File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/cli.py", line 22, in main
app.run(argv)
File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/application.py", line 360, in run
self._run(argv)
File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/application.py", line 347, in _run
self.initialize(argv)
File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/application.py", line 333, in initialize
self.make_formatter()
File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/main/application.py", line 226, in make_formatter
self.formatter = formatter_class(self.options)
File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/plugins/manager.py", line 131, in execute
return self.plugin(*args, **kwargs) # pylint: disable=not-callable
File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8/formatting/base.py", line 51, in __init__
self.after_init()
File "/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/.tox/flake8/lib/python3.6/site-packages/flake8_html/plugin.py", line 89, in after_init
os.mkdir(self.outdir)
FileNotFoundError: [Errno 2] No such file or directory: '/home/jugmac00/Repos/zopefoundation/Products.GenericSetup/parts/flake8'
flake8 run-test: commands[1] | flake8 --doctests Products setup.py
_________________________________________________________________ summary _________________________________________________________________
flake8: commands succeeded
congratulations :)
Any chance of a new release? I would like to use the toolset.xml
improvements from my merged pull request #26.
Release rights on PyPI for user maurits
would also be gladly accepted. :-)
Can we have a new release of GenericSetup please? da0a3749 would allow Plone to use the latest versions of setuptools and zc.buildout
In https://bugs.launchpad.net/zope-cmf-buildout/+bug/1418881, Dieter Maurer reported:
Products.GenericSetup 1.7.3
"GenericSetup" supports the declaration of step dependencies and "getSortedImportSteps" returns a sequence of steps respecting those declarations. However, the step order is otherwise non-deterministic. This makes it extremely hard to detect missing step dependency declarations.
https://dev.plone.org/ticket/13815 shows an example where a missing step dependency was not recognized for a very long time.
I have spent the last few days to track down another similar problem: I start Zope and get the following (unworking) sort order:
2015-02-05T08:50:19 INFO bfd.theme_bp.utilities.tracing Return Products.GenericSetup.tool.getSortedImportSteps -> (u'plone-difftool', u'plone.app.caching', u'properties', u'Products.Ploneboard.uninstallVarious', u'rolemap', u'sharing', u'ploneopenid-various', u'collective.z3cform.datetimewidget_various', u'toolset', u'reference_catalog', u'various', u'typeinfo', u'cmfeditions_various', u'repositorytool', u'kupu-setup', u'workflow', u'update-workflow-rolemap', u'placeful_workflow', u'placeful_marker', u'catalog', u'content', u'cookie_authentication', u'caching_policy_mgr', u'various-calendar', u'plone.app.theming', u'uid_catalog', u'componentregistry', u'portlets', u'contentrules', u'atcttool', u'factorytool', u'actions', u'plonetheme.sunburst-various', u'controlpanel', u'memberdata-properties', u'plonepas', u'mailhost', u'action-icons', u'solr', u'browserlayer', u'content_type_registry', u'mimetypes-registry-various', u'kss_mimetype', u'difftool', u'tinymce_various', u'plone.app.registry', u'tinymce_settings', u'plone-final', u'plone-content', u'skins', u'jsregistry', u'bfd.gui.various', u'cssregistry', u'jquerytools-various', u'kssregistry', u'propertiestool', u'viewlets', u'archetypes-various', u'archetypetool', u'plone_outputfilters_various'
I restart Zope (without any modification) and get the following (working) sort order:
2015-02-05T08:55:18 INFO bfd.theme_bp.utilities.tracing Return Products.GenericSetup.tool.getSortedImportSteps -> (u'properties', u'rolemap', u'sharing', u'toolset', u'uid_catalog', u'various-calendar', u'plone.app.theming', u'various', u'componentregistry', u'plone_outputfilters_various', u'action-icons', u'memberdata-properties', u'skins', u'bfd.gui.various', u'kssregistry', u'cssregistry', u'jquerytools-various', u'jsregistry', u'tinymce_settings', u'tinymce_various', u'browserlayer', u'content_type_registry', u'mimetypes-registry-various', u'kss_mimetype', u'mailhost', u'propertiestool', u'actions', u'plonetheme.sunburst-various', u'controlpanel', u'plonepas', u'viewlets', u'difftool', u'plone.app.registry', u'solr', u'archetypes-various', u'archetypetool', u'catalog', u'atcttool', u'cookie_authentication', u'typeinfo', u'factorytool', u'kupu-setup', u'cmfeditions_various', u'repositorytool', u'reference_catalog', u'caching_policy_mgr', u'workflow', u'content', u'portlets', u'plone-final', u'plone-content', u'contentrules', u'placeful_workflow', u'placeful_marker', u'update-workflow-rolemap', u'ploneopenid-various', u'collective.z3cform.datetimewidget_various', u'Products.Ploneboard.uninstallVarious', u'plone-difftool', u'plone.app.caching', u'ploneboard-various')
Of course, the real problem are missing dependency declarations for the Plone "plone-final" import step (it depends on "skins"). However, the non deterministic behaviour of "getSortedImportSteps" hides this problem quite efficiently.
I was having a look at why some robot tests are failing on Plone 5.2 running on python 3.6 and noticed that the Plone controlpanel to manage actions was failing due to a permission missing.
On closer inspection, the ZMI seems to be also broken as there is no permission set on a given action.
Turns out though, that the permission is imported from actions.xml
but stored on the action object as (b'View',)
but when compared with the list of available permissions, none of them matches because that list is a list of strings rather than bytes.
Which one should prevail? Available permissions should be also bytes or rather permissions should be imported as strings? 🤔
When running the tests against current master under Python 2.7 using the standard bin/test
script they all complete successfully. However, running them via tox
produces non-deterministic errors. Example:
Failure in test test_runAllExportSteps_empty (Products.GenericSetup.tests.test_tool.SetupToolTests)
Traceback (most recent call last):
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 329, in run
testMethod()
File "/Users/jens/src/Products.GenericSetup/Products/GenericSetup/tests/test_tool.py", line 890, in test_runAllExportSteps_empty
self.assertEqual(result['steps'][1], 'step_registries')
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 513, in assertEqual
assertion_func(first, second, msg=msg)
File "/usr/local/Cellar/python/2.7.13/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 506, in _baseAssertEqual
raise self.failureException(msg)
AssertionError: u'toolset' != 'step_registries'
The test code (see https://github.com/zopefoundation/Products.GenericSetup/blob/master/Products/GenericSetup/tests/test_tool.py#L882) appears to assume that result
from a call to tool.runAllExportSteps()
has a fixed order of steps. But that makes no sense when I look at how the result is put together. In listExportSteps
the steps are combined from two set
instances, which obviously don't have a sorting order.
Is the assumption of a fixed order wrong or is the implementation wrong which disregards any order?
We had problems runing a upgradeStep of profile not installed. See:
upgradeStep like this, are listed even if the profile isn't installed. Is it really necessary list upgradeSteps of profiles not installed?
This is probably not a bug of GS, but tbh I am not sure where this bug belongs, I'll add it to plone.app.testing as well :-)
Some context:
When using the PloneSandbox layer in tests, the layer will create a demostorage and wrap the current storage. it will also "reset" the zcml config, doesn't matter, as the layer will rerun the zcml config anyway.
With 1.8.0 this doesn't happen any more with nested PloneSandboxes.
Nested DemoStorages aren't a problem, zcml reconfigurations are also not a problem, the Layer will rerigster plone ande dependencies, but:
Persistent tools registered via componentregistry.xml are gone.
The tool is still available in the database, but the registration that has happened in the import step is gone.
This is related to pull request #16, which already updates the Import tab in the ZMI.
Perceived problem: the Import tab is too complicated.
Initial idea:
Given the new Addons tab, it probably makes sense for the Import tab to not apply or upgrade any dependency profiles. We could warn when we see this is a problem, and point to the Addons tab.
I am willing to work on this.
In https://bugs.launchpad.net/zope-cmf/+bug/161682, @mauritsvanrees reported:
At Zest we have an own Plone product feedfeeder for a client (soon to be released to the collective I think). We make use of GS. Its profiles/default/catalog.xml is:
<?xml version="1.0"?>
<object name="portal_catalog" meta_type="Pluggable Plone Catalog Tool">
<index name="getFeedItemAuthor" meta_type="FieldIndex">
<indexed_attr value="getFeedItemAuthor"/>
</index>
<index name="getFeedItemUpdated" meta_type="DateIndex">
<property name="index_naive_time_as_local">True</property>
</index>
<column value="getFeedItemAuthor"/>
<column value="getFeedItemUpdated"/>
</object>
So we are adding some indexes. But on a reinstall those indexes get thrown away and readded. This means we need to reindex those indexes manually (or rather in code) so the catalog is up to date again.
We tried to get GS to leave those indexes alone if they are already there, but failed. It looks like a bug in GS, but we could also do something wrong of course.
We have a separate product with an Extentions/install.py which takes care of loading all the GS profiles of the products that we need and that handles some other install functions via the quickinstaller. The relevant part of the code:
setup_tool = getToolByName(portal, 'portal_setup')
for extension_id in config.GS_DEPS:
setup_tool.setImportContext('profile-%s' % extension_id)
setup_tool.runAllImportSteps(purge_old=False)
And the culprit is the purge_old in the last line. If we have followed the trail into GS right, this should make sure that GS does not remove those indexes, but in fact it does. The indexes are there in the end, but none of the brains in the catalog have any content for those fields. Reindexing in some other part of the code solves it:
site.portal_catalog.manage_reindexIndex(ids=['getFeedItemAuthor','getFeedItemUpdated'])
Would this be a GS bug or are we doing something wrong?
Yvo Schubbe replied:
Not sure if this is a bug report or a feature request or if the bug is the fact the current limitations are not documented. Anyway it is a known issue.
I quote some suggestions I made on the mailing list:
( http://mail.zope.org/pipermail/zope-cmf/2006-August/024747.html )
- The adapters should compare the new settings with the existing ones and make sure indexes are only updated if they are actually changed. This is not implemented so far.
- The adapters should modify existing indexes if the profile contains new settings. This is not implemented for the 'extra' elements.
- Reindexing might be very expensive. That's the reason why GenericSetup does no reindexing, you have to do it manually. We need a better solution for that, maybe some bookkeeping which indexes need to be reindexed and a button in the setup tool that triggers reindexing.
@mauritsvanrees responded:
I was not sure either when I filed this. :) Some time has passed, I asked some questions and learned some more. I would now say it is a feature request.
I reported some ideas in this thread:
@tseaver wrote:
Yuppie thinks he can see how to get the index adapters in GS to avoid clearing an index if the import does not change its properties.
@wichert wrote:
Unfortunately there is no API to either get or update the values of the extra index parameters. That suggests that if extra is specified in the profile the index needs to be recreated.
@stainsby wrote:
I recently reported this in the Plone tracker and it was suggested that I add a comment here instead. Here is my original report:
"Catalog indexes that are defined by a product (in the catalog.xml file of its profile) are cleared each time a product is reinstalled. In fact, just deleting the product directory and re-adding also seems to do the same thing. With a client's membership database that takes one or two days to reindex, this is a major pain."
@dnouri posted two patches:
https://bugs.launchpad.net/zope-cmf/+bug/161682/+attachment/169881/+files/CMFCore-2.1-branch.diff
with comment:
Attached are two patches. One against GenericSetup in its 1.3 branch and one against CMFCore in the 2.1 branch.
The goal of this patch is to work towards a solution for one part of this issue, namely, citing yuppie: "The adapters should compare the new settings with the existing ones and make sure indexes are only updated if they are actually changed."
For this purpose I added a generally usable function decorator called
clearIfChanged
that you can stick onto existing_importNode
methods. This decorator will compare self.node with the value of self.node after the node was imported. It does so to detect whether anything has changed; if anything has changed, it will callself.context.clear()
.I added the tests for this to CMFCore; I must admit I was a bit confused as to where to put them, because CMFCore seems to contain in
CMFCore/exportimport/tests/test_catalog.py
tests that should maybe live in GenericSetup. I'm happy to move tests around if need be.Note that the only two indexes that I added this decorator to right now is the DateIndex and the ZCTextIndex. I'll add it to more index adapters once we have consensus that this approach is good, and everything is going into the right place.
@wichert responded:
Looking at that patch it relies on being able to make a reliable XML export of the node so it can compare that with the new state. That is an assumption that you can not make though: only ZCTextIndex has an exporter that exports the extra options correctly. For all other index types this is not implemented. Until that has changed this patch can break sites.
@dnouri replied:
If the export doesn't deal with the same data as the import, then that's a bug in the current handlers, and one that's quite easily fixed, I believe.
I see no need to do the "interface change for catalog indexes in general" that you suggest in an e-mail. Since handlers are index specific, it's fine for them to introspect states of their indexes down to private attribute level.
I'll try and see if I can get all indexes that I'm interested in to export properly. In the meantime, please help me understand how this patch can break sites.
Yvo Schubbe wrote:
Thanks for the patch! 2 comments:
- The export/import adapters are currently cheating extensively. There is no way to write them using just the interface methods, but they claim to adapt the interfaces. Things don't become worse if we use non-API methods for extracting and updating the 'extra' settings - in fact we adapt a specific implementation.
- The tests in CMFCore/exportimport/tests/test_catalog.py are just additional integration tests for the CMF specific catalog tool. GenericSetup contains test_exportimport.py files for the indexes and ZCatalog. All these adapter tests in GS should be modified to test the new behavior.
Toni Mueller ([email protected]) wrote:
It would great to see some progress in this area. We're currently bitten by this bug as well, and re-indexing would take several hours each time. Before solving the problem automatically, I'd like to have just an extension to the syntax of catalog.xml:
Now we have:
<indexed_attr value="some_attribute"/>
It would imho be a big step forward if we had this instead:
<indexed_attr value="some_attribute" [preserve=True/False] [reindex=True/False] />
With those two optional parameters, the decision about how to proceed with indices would not have to be computed, but could simply be specified by the user.
@rochecompaan followed up:
I just hit this bug again today. I don't understand why the PluggableIndexNodeAdapter needs to ever clear the index in _importNode - it makes catalog.xml dangerous and unusable.
Actually not only PluggableIndexNodeAdapter but all the other IndexNodeAdapter in Products/Generic Setup/PluginIndexes/exportimport.py as well.
@djay wrote:
This feature still isn't implemented :(
I might be missing something but does the solution have to involve changing the index api?
Can't we just define the behaviour such that
- purge="True", the index will always get recreated.
- purge="False",if the index id already exists do nothing. if it doesn't then create it. If the extra attributes have changed bad luck.
I did a PR #26 more than three years ago, which was released in GenericSetup 1.8.3. It made it possible to move a tool from the required list to the forbidden list, or the other way around.
I now see a problem with that. I have a toolset.xml
which does exactly that, and it seems to work, but its resulting changes are not persisted: they are gone after a restart. If more changes are done to portal_setup
before a restart, then things might be fine. But not when there is an immediate restart.
I can work on a PR for that.
Let me describe the situation:
Tool
class inheriting from CMFCore.utils.UniqueObject/ImmutableId
and id "correct_id".toolset = portal_setup.getToolsetRegistry()
toolset.listForbiddenTools()
is an empty list.toolset.listRequiredToolInfo()
dictionary has one Tool class with correct_id
and one with typo_id
.typo_id
and I manually removed this object in the ZMI.toolset.xml
causes GenericSetup to try to create the missing typo_id
but this fails.I wanted to fix that. I actually already had a correct toolset.xml
which should fix it:
<?xml version="1.0"?>
<tool-setup>
<forbidden tool_id="typo_id"/>
<required tool_id="correct_id"
class="client.code.Tool"/>
</tool-setup>
Applying this initially works:
typo_id
correct_id
.But after a restart, the old situation is there again, and any other toolset.xml
that is applied, throws an error, just like before.
The reason is that the forbidden and required information is stored in a simple list and dictionary instead of in persistent data.
In my case this is in GenericSetup 1.8.9 on Plone 4.3.18, but I see it is the same on the master branch.
I have tried a few things:
toolset._p_changed = True
: does not help.self._forbidden = self._forbidden
whenever changing self._forbidden
. Same for self._required
. Does not help.portal_setup._p_changed = True
: helps.I expected that one of the first two would help, because ToolsetRegistry
inherits from Acquisition.Implicit
. But that is maybe only when it inherits from Explicit
.
The question is then where to fix it.
ToolsetRegistry
class, then we would need to call something like self.__parent__._p_changed = True
which seems wrong: it accesses a private variable of the parent.SetupTool
class: after calling toolset.parseXML
we can set self._p_changed
to True. It might need to be done in more cases, but at least this is the main one.I think I will go with number 2.
This allows to drop the eggtestinfo
setup dependency.
@tseaver Are you okay with this suggestion?
In https://bugs.launchpad.net/zope-cmf/+bug/281216, @cjw296 reported:
For background see: http://dev.plone.org/plone/ticket/8579
Summary of the problem: You cannot re-constitute local utilities just from a dotted name:
- the constructor may take parameters. Indeeed, imnsho, this is how utilities that require configuration should be configured.
- the utilities may be changed after registration, as is the case with PortletType utilities.
This is with Plone 3.1.1 and associated CMF and Zope versions.
Plone cluster with 9 instances. Run upgrade steps or profiles.
Profiles or upgrades are applied
They are not.
Python 3.9.8 Zope 5 latest, Plone
GenericSetup uses keys generated by hash(). Since hash is only the same inside one running Python each instance has its own set of keys. In a cluster the keys of one instance are used in the form and processed by another instance with different keys it can not look up.
This is probably a Python3 change on hash
Use a stable hashing algorithm to build the hash.
In https://bugs.launchpad.net/zope-cmf/+bug/314010, Bertrand Mathieu (@bmathieu?) reported:
an example is here: http://dev.plone.org/plone/ticket/8350
if 'my-step' must be executed before 'mandatory-step', and 'mandatory-step' is involved in a circular dependency chain (with 'circular-step' for example), then sorted imported steps will be: ['my-step', 'mandatory-step', 'circular-step'].
This is reproducible if you define 'my-step' with zcml, and the other steps in a profile (with import_steps.xml).
I have posted a monkey patch in plone ticket system #8350 (in order to help others with a hopefully good workaround). There is also doctest that should fail with current implementation and succeed with mine. The idea is to change final loop and insert all steps detected involved in a circular loop, and then trying to resolve remaining dependencies before eventually inserting unresolved steps.
I don't know if it is the best way to solve this problem, but at least it should be a good start.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.