cfpb/wagtail-inventory

Search Wagtail pages by the StreamField blocks they contain

wagtail-inventory

Search Wagtail pages by block type.

Wagtail Inventory adds the ability to search pages in your Wagtail site by the StreamField block types they contain. It adds a new report to the Wagtail admin site that allows you to search for pages that do or do not contain certain blocks. It supports searching both by Wagtail built-in blocks (like CharBlock) as well as any custom blocks you might define.

Setup

Install the package using pip:

$ pip install wagtail-inventory

This will also install django-autocomplete-light.

Add dal, dal_select2, and wagtailinventory as installed apps in your Django settings:

# in settings.py
INSTALLED_APPS = (
    ...
    'dal',
    'dal_select2',
    'wagtailinventory',
    ...
)

Run migrations to create required database tables:

$ manage.py migrate wagtailinventory

Run a management command to initialize database tables with current pages:

$ manage.py block_inventory

Admin users should now be able to search pages in the Wagtail admin site, under Reports > Block Inventory.

Other user groups may be granted access to the report by giving them the "Can view" "Page block" permission in Wagtail Group settings.

Compatibility

This code has been tested for compatibility with:

• Python 3.8+
• Django 3.2 (LTS), 4.2 (LTS), 5.0
• Wagtail 3.0+, including 5.2 (LTS) and 6.0

It should be compatible with all intermediate versions, as well. If you find that it is not, please file an issue.

Testing

Running project unit tests requires tox:

$ tox

To run the test app interactively, run:

$ tox -e interactive

Now you can visit http://localhost:8000/admin/ in a browser and log in with admin / changeme.

Open source licensing info

1. TERMS
2. LICENSE
3. CFPB Source Code Policy

wagtail-inventory's Issues

Search form layout regression in Wagtail 4.0+

Wagtail 4.0 changed the way fields are rendered in wagtail/wagtail@eac5e0b, specifically which broke the (admittedly hacky) hardcoded CSS that breaks how the includes/excludes button was being rendered next to the dropdown:

Wagtail 2.16	Wagtail 3.0	Wagtail 4.0

#63 (comment) includes a suggestion for fixing this with a table. Once we drop support for Wagtail < 4.0, we can use a simpler implementation that makes use of Wagtail's {% field_row %} template tag; see gist here.

Make this package compatible with Wagtail 1.13

Add support to setup.py and add unit testing against Wagtail 1.13.

Could we get a "--no-input" flag for the "manage block-inventory" command?

Right now the manage block_inventory command can lead to user prompts, where someone has to manually type "y" and hit enter, which makes it very hard to use in automated deployment, since there is no "user" available to type anything in that context.

A number of manage commands support the --no-input flag which automates the question and answer process, simply assuming the answer is supposed to be "yes" to every question, so that deployment scripts can add that flag wherever users would normally be prompted for confirmation, and it would be super useful to have that available for the inventory plugin command, too =)

Columns misaligned when using Wagtail 2.15

The upgrade to Wagtail 2.15 caused the columns in the results table to be misaligned. The header has an extra item at the beginning of the row (for selecting all for bulk actions). And it would appear that allow_navigation=0 to prevent users from navigating away from the page doesn't work since wagtail/wagtail@e93b210

Screenshots

I am honestly not sure what the expected behavior should be for these. I "fixed" the allow_navigation issue in my fork by overriding the _list_explore.html to comment out {% block page_navigation %}

And I elected to make bulk actions work by adding the following code to search.html:

{% block extra_js %}
   {{ block.super }}
   <script>
       window.wagtailConfig.BULK_ACTION_ITEM_TYPE = 'PAGE';
   </script>
   <script defer src="{% versioned_static 'wagtailadmin/js/bulk-actions.js' %}"></script>
{% endblock %}

And then in the results section, I added show_bulk_actions=1 to the _list_explore include and then added the last 2 lines to get the footer which appears when you select one or more of the bulk actions checkboxes.

{% include "wagtailinventory/_list_explore.html" with show_bulk_actions=1 show_parent=1 %}
{% include "wagtailadmin/shared/pagination_nav.html" with items=pages %}
{% trans "Select all pages in listing" as select_all_text %}
{% include 'wagtailadmin/bulk_actions/footer.html' with select_all_obj_text=select_all_text app_label='wagtailcore' model_name='page' objects=pages %}

If you would like to tell me how you would like these 2 items resolved, I could submit a PR - but it may be faster for your team to decide and then put in place whatever fix you have decided on.

Address pagination changes in Wagtail 2.5+

My current setup:

Wagtail 2.6.2
Django 2.2.6
wagtail-inventory 0.7

All manage.py functions display the following warning in the console:

/lib/python3.6/site-packages/wagtail/utils/pagination.py:10: RemovedInWagtail27Warning: wagtail.utils.pagination is deprecated. Use django.core.paginator.Paginator directly with get_page instead category=RemovedInWagtail27Warning)

This is most likely due to a change to Wagtail's pagination handling - the fix for this is detailed in the Wagtail 2.5 release notes. This change should be applied under /wagtailinventory/views.py.

As per Wagtail's depreciation policy, this change will be necessary for this package to be compatible with Wagtail 2.7.

setup requires specific version of tqdm

wagtail-inventory/setup.py

Line 5 in 8b57fb3

"tqdm==4.15.0",

We've got a newer version than this installed in our env, is it possible to expand it to range of expected versions?

Docs

Can we have a README? The notes at cfpb/consumerfinance.gov#2816 would be a great start.

Search form block dropdown is unsorted

The select fields in the inventory search form are not explicitly sorted, and so their order is arbitrary (depends on database ordering). This will not necessarily produce nicely sorted alphabetical blocks, which provides for a poor user experience.

The form list of blocks should be sorted alphabetically in the select dropdown.

Wagtail 3 compatibility

The package can be updated to work with Wagtail 3

Current behavior

The package has a version constraint for Wagtail of "wagtail>=2.15,<3"

Expected behavior

Be able to install the package when using Wagtail 3+

Steps to replicate behavior (include URLs)

Installing the package (I was using poetry) fails

Solution

I have submitted a pull request to upgrade the package. #58

dealing with "I have no idea how this happened" multiple page/block bindings

We were getting reports from our staff that some wagtail pages refused to publish and instead were throwing up server error 500s, and after investigating (in this issue) it turns out that somehow, due to whatever bizarre set of circumstances, some page/block type combinations have more than one row in the wagtailinventory table, so the get_or_create call in helper.py would end getting (rather than creating) more than one result, causing Django to crash because get_or_create expects zero or one results, not "zero or more" =)

So my (termporary) solution to this problem has been to change the helper.py code so that rather than a list comprehension:

    return [
        PageBlock.objects.get_or_create(page=page, block=block)[0]
        for block in page_blocks
    ]

it does the more verbose "check if there are any results, then depending on the number of results, do one of three things":

    list = []

    for block in page_blocks:
        bindings = PageBlock.objects.filter(page=page, block=block)

        if bindings.count() > 0:
            list.append(bindings.first())
            for index, entry in enumerate(bindings):
                if index > 0:
                    entry.delete()
        else:
            result = PageBlock.objects.create(page=page, block=block)
            list.append(result)

    return list

It's not as clean or efficient, but it does allow the code to do "what it should be doing" while also cleaning up any problem cases it runs into at the same time - while an unusual situation, it might be worth putting a solution like this in place in the official helper.py so that even if it runs into an unexpected database situation, it'll gracefully recover rather than trigger a server error 500

Feature request: also search by specific field values within a block

E.g., search for all "Image and Text 25-75" modules where the "Has rule" option is checked.

Exempt the PageBlock model from Wagtail's reference index

Wagtail 4.1 introduced a new reference index that keeps track of Django model instance usage. In versions 4.1 and 4.2, this tracks instances from any Django model -- in 5.0 this was changed so that only Wagtail-related models are tracked.

There's no need to track this project's PageBlock model class and, for sites with a large number of pages and/or blocks, the large number of PageBlock instances can increase the size and overhead of the reference index.

We should add wagtail_reference_index_ignore = True to the PageBlock model to prevent it from being indexed. We'll want this code for as long as Wagtail 4.1 and 4.2 are supported by this package.

Wagtail 2.5 Support

Hello, Wagtail 2.5 is out. Can we get an update to support it? As it stands, Pipenv can not generate a lockfile because wagtail-inventory requires wagtail<2.5

Select dialogs in inventory form are missing chevron icon

Short description explaining the high-level reason for the new issue.

Current behavior

In the wagtail inventory page, there are 6 select elements that have no visual cue indicating that they are select dialogs. The UI would suggest they are text boxes. When clicked, the dropdown appears and functionality works as expected.

Expected behavior

The UI of the select dialog should have some sort of visual element indicating that the field is a dropdown. The easy solution is to leverage wagtail's built in CSS styles to render a chevron icon on the right hand side.

Steps to replicate behavior (include URLs)

1. Go to the wagtail inventory page (http://localhost:8000/admin/inventory/)
2. Observe the UI of the 6 select elements at the top of the page

Screenshots

wagtail 2.1+ support?

It looks like wagtail-inventory is blocking upgrading to django 2.1 with wagtail 2.3/2.4, are there any true problems or is this simply a matter of updating the version requirements and publishing a 0.5.2 with those updated requirements?

There are No Permissions policies available!

Current behavior

the Inventory is shown to all users, there are no restrictions

Expected behavior

it has to be given at least the BASIC PERMISSIONS, to prevent certain groups of users from viewing and searching across the entire DB, basically this should be available for the SUsers only, unless the complex permissions policies provided(perhups in the future)..!

"Page block" in the group edit form?

Does everyone else see "Page block" in their group edit form? The docs say

Other user groups may be granted access to the report by giving them the "Can view" "Page block" permission in Wagtail Group settings.

But I don't see "Page block" in the "Object permissions" section of my group editing pages e.g. https://xyz.localhost/admin/groups/2078/ In fact I don't see the word "block" on that page at all. The report is working just fine once I assign the permission via the database and the hook for putting the permissions on the group admin form looks OK to me so I thought I would start by asking if it was just me.

I am using Wagtail 5.1.2, Django 3.2, and wagtail-inventory 2.0.0.

Please create a release for Wagtail 5

I just installed commit c331865 into a project that is running Wagtail 5.0rc1. Everything is working as expected. I would love to see a pypi release so I can use that rather than a github url in my requirements file.

Support for multiple StreamFields in same page model

Short description explaining the high-level reason for the new issue.

Current behavior

Catches the first streamfield only.

Expected behavior

I would expect it to catch multiple streamfields. I have some page types configured like the following:

@add_stream_field(
    "body",
    [
        # Body blocks
    ],
)
@add_stream_field(
    "sidebar",
    [
        # Sidebar blocks
    ],
)
class SomePage(Page):
    # My page model 

The inventory gets blocks from "body" only.

Steps to replicate behavior (include URLs)

Previous points are probably clear enough.

Screenshots

Nothing to show.

Document need to re-run block_inventory for renames

Renaming an existing StreamField block class doesn't automatically update the database table that this project uses to keep track of block usage, because class names are stored as strings. The documentation should mention that you need to manually run manage.py block_inventory to regenerate the inventory if you rename the block class.

Comptability with Wagtail 2?

Hi @chosak, any plans to make this compatible with Wagtail 2? Or any guidance on what might need updating in order to submit a PR for Wagtail 2 compatibility?

publish 0.4.3?

#11 landing is a pretty important improvement, and having a 0.4.3 available with those changes means a Pipfile.lock can point to that, rather than having to point to a master branch on git (which has been the source of some bug reports for pipenv... it doesn't seem to always work for everyone =S)

publish to pypi?

Hiya,

it looks like some code landed about 2 weeks ago that solves a potential security issue for anyone using wagtail-inventory, would it be possible to get the new version published to PyPi so that folks can safely update rather than linking directly to a git commit in their pip requirements?

AttributeError: module 'dal.autocomplete' has no attribute 'Select2ListView'

I was using the latest version 2.0.0 while upgrading an existing site to Wagtail v4.2.1

I noticed that when running the site is saw this error:

Traceback (most recent call last):
  File "./manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/test.py", line 23, in run_from_argv
    super().run_from_argv(argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/test.py", line 55, in handle
    failures = test_runner.run_tests(test_labels)
  File "/usr/local/lib/python3.8/site-packages/django/test/runner.py", line 728, in run_tests
    self.run_checks(databases)
  File "/usr/local/lib/python3.8/site-packages/django/test/runner.py", line 665, in run_checks
    call_command('check', verbosity=self.verbosity, databases=databases)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 181, in call_command
    return command.execute(*args, **defaults)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/check.py", line 63, in handle
    self.check(
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 419, in check
    all_issues = checks.run_checks(
  File "/usr/local/lib/python3.8/site-packages/django/core/checks/registry.py", line 76, in run_checks
    new_errors = check(app_configs=app_configs, databases=databases)
  File "/usr/local/lib/python3.8/site-packages/django/core/checks/urls.py", line 13, in check_url_config
    return check_resolver(resolver)
  File "/usr/local/lib/python3.8/site-packages/django/core/checks/urls.py", line 23, in check_resolver
    return check_method()
  File "/usr/local/lib/python3.8/site-packages/django/urls/resolvers.py", line 416, in check
    for pattern in self.url_patterns:
  File "/usr/local/lib/python3.8/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.8/site-packages/django/urls/resolvers.py", line 602, in url_patterns
    patterns = getattr(self.urlconf_module, "urlpatterns", self.urlconf_module)
  File "/usr/local/lib/python3.8/site-packages/django/utils/functional.py", line 48, in __get__
    res = instance.__dict__[self.name] = self.func(instance)
  File "/usr/local/lib/python3.8/site-packages/django/urls/resolvers.py", line 595, in urlconf_module
    return import_module(self.urlconf_name)
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/app/ousweb/urls.py", line 10, in <module>
    from wagtail.admin import urls as wagtailadmin_urls
  File "/usr/local/lib/python3.8/site-packages/wagtail/admin/urls/__init__.py", line 13, in <module>
    from wagtail.admin.api import urls as api_urls
  File "/usr/local/lib/python3.8/site-packages/wagtail/admin/api/urls.py", line 11, in <module>
    for fn in hooks.get_hooks("construct_admin_api"):
  File "/usr/local/lib/python3.8/site-packages/wagtail/hooks.py", line 112, in get_hooks
    search_for_hooks()
  File "/usr/local/lib/python3.8/site-packages/wagtail/hooks.py", line 106, in search_for_hooks
    list(get_app_submodules("wagtail_hooks"))
  File "/usr/local/lib/python3.8/site-packages/wagtail/utils/apps.py", line 23, in get_app_submodules
    yield name, import_module("%s.%s" % (name, submodule_name))
  File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/usr/local/lib/python3.8/site-packages/wagtailinventory/wagtail_hooks.py", line 11, in <module>
    from wagtailinventory.views import (
  File "/usr/local/lib/python3.8/site-packages/wagtailinventory/views.py", line 11, in <module>
    class BlockAutocompleteView(autocomplete.Select2ListView):
AttributeError: module 'dal.autocomplete' has no attribute 'Select2ListView'

I was upgrading wagtail-inventory from 1.6.0

The solution was to pin the wagtail-inventory requirement to 1.7.0

Remove wagtail max version in setup.py

Following up on a conversation in #36, I suggest the team consider removing the max supported version of Wagtail in the setup.py

Current behavior

When installing libraries into a python environment (using pip, pipenv, etc.), wagtail-inventory restricts the permitted versions of Wagtail with a minimum version (currently 2.7, inclusive) and a maximum version (currently 2.12, exclusive).

This means that when the next version of wagtail comes out (e.g. 2.12.0), projects that use wagtail-inventory will not be able to upgrade until wagtail-inventory releases an update to its setup.py max version.

This gives 2 potential routes for a project owner:

1. Remove wagtail-inventory from the project until an update is released
2. Wait to update wagtail, which assumes CFPB will maintain wagtail-inventory indefinitely and timely.

Expected behavior

Each project should have the freedom to pair wagtail-inventory with a new version of wagtail, even if CFPB has yet to test the new version.

This aligns with guidance in python.org documentation

It is not considered best practice to use install_requires to pin dependencies to specific versions, or to specify sub-dependencies (i.e. dependencies of your dependencies). This is overly-restrictive, and prevents the user from gaining the benefit of dependency upgrades.

Some workaround ideas:

• You can define your supported versions in the README and in the tox config. Of course, this is not programmatically enforced, but that's kind of my point from the perspective of the external community.
• Consider rephrasing the development standards to clarify that "unsupported versions" refers to known incompatibilities, but does not apply to untested future versions.

Steps to replicate behavior (include URLs)

Screenshots