oarepo/oarepo-model-builder

A utility library that generates Invenio 3.5+ data model files from a YAML/JSON5 specification file

CESNET, UCT Prague and NTK wrapper around invenio v3

This meta-package contains a curated and tested set of dependencies on Invenio and OArepo libraries. It should be considered as a base building block for creating any OA repository applications.

The package contains the following dependency bundles:

Bundle name	Description
deploy	All the dependencies needed for a basic repository deployment
deploy-es7	All the ElasticSearch 7 compatible dependencies needed for a basic repository deployment
openid	Dependencies needed for openid authentication
multisum	Support for multiple file checksums (deprecated)
files	Support for uploading files
acls	ACLs support
links	Enable linking to another records
models	Includes all the common data models (DCObject, multilingual fields, Invenio model,...)
includes	Adds support to compose ES mappings by including another mappings
taxonomies	Adds support for taxonomic trees
tests	Includes test dependencies
draft	Adds support for draft records
iiif	Adds support for file (image) preview generation
micro-api	A WSGI app to serve API-only apps under the /api prefix

They could be installed, depending on needs of each repository application, by running:

pip install oarepo[bundle-name]

oarepo-model-builder's Issues

Validate model via jsonschema

To minimize user errors

Create data model for document repository

Custom names for generated facets/filters

Add an option, for example

oarepo:facet:
  key: <myname>

to override generated facet/filter names

pridat generator editorconfigu

Alembic v bootstrapu by se mel volat jen kdyz jeste nejsou vytvorene migrace

Config should be instantiated in ext.py

    def init_resource(self, app):
        """Initialize vocabulary resources."""
        self.service = app.config["THESIS_SERVICE_CLASS"](
            config=app.config["THESIS_SERVICE_CONFIG"](),
        )
        self.resource = app.config["THESIS_RESOURCE_CLASS"](
            service=self.service,
            config=app.config["THESIS_RESOURCE_CONFIG"](),
        )

Facet support for ui contexts

Implement UI contexts into facet generator - will need to decide how this should be done as ui contexts are defined in a different module - oarepo-model-builder-ui

Add autoflake to remove unused imports

vytvořený repozitář nefunguje bez "invenio use"

Model builder by měl na tuto skutečnost upozornit "vhodným errorem".

Localized enums in facet bucket values

Same as #151 , but translate value labels using babelex/invenio_i18n

Invalid attribute in mapping json

oarepo_theses/records/mappings/v7/oarepo_theses/oarepo-theses-1.0.0.json
attribute: "type": "object"

{
    "mappings": {
        "properties": {
            "title": {
                "type": "text",
                "fields": {
                    "keyword": {
                        "type": "keyword",
                        "ignore_above": 50
                    }
                }
            }
        },
        "type": "object"
    }
}

bootstrap script - sed behaves differently on Mac and Linux

-i parameter requires backup extension on mac but not on linux → do not use in-place editing of alembic files

Merging call lists

Add support for automatic merging of parameters inside calls, such as

a = blah(b, c, d=[1])
a = blah(b, c, d=[2])

into

a = blah(b, c, d=[1, 2])

Problem with generated search for a theses schema

The commented-out searches bring listing errors when uncommented:

        "metadata_subjects": facets.metadata_subjects,
        # "metadata_publishers": facets.metadata_publishers,
        "metadata_subjectCategories": facets.metadata_subjectCategories,
        "metadata_languages": facets.metadata_languages,
        # "metadata_notes": facets.metadata_notes,
        "metadata_rights": facets.metadata_rights,

Check if field: "blah" works with oarepo-model-builder-multilingual

subject:
      type: multilingual
      required: true
      label.cs: Klíčová slova
      label.en: Keywords
      ui:
        detail: string
        marshmallow:
          field: MultilingualLocalizedUIField()
          imports:
            - import: oarepo_runtime.i18n.ui_schema.MultilingualLocalizedUIField

seems to be generated incorrectly

EDTFDateString namiesto Date

Pouzil som nasledujuce metadata:

model:
  properties:
    metadata:
      properties:
        title:
          type: fulltext
        updated:
          type: date
          required: true
          oarepo:sample:
            faker: iso8601
            skip: false

oarepo:use:
  - invenio

Trieda {Nazov}MetadataSchema po vygenerovani:

class ThesisMetadataSchema(
    ma.Schema,
):
    """ThesisMetadataSchema schema."""

    title = ma_fields.String()

    updated = ma_fields.Date(required=True)

Problem je, ze po vygenerovani pozadujeme pre field updated typ EDTFDateString (from marshmallow_utils.fields import EDTFDateString).

Poprosim upravit. Zaroven upravit aj generovanie sample dat.

Marshmallow classname used for generated Schema should allow relative addressing

For example:

Somewhere in model (will be generated to mymodel python package):

properties:
  person:
    oarepo:use: person

The person is defined elsewhere, for example in oarepo-model-builder-org plugin:

type: object
oarepo:marshmallow:
  generate: true
  class-name: .schema_person.Person
properties:
  ...

The overall model schema will be implicitly generated to mymodel/services/schema.py file.
The Person schema should get generated to mymodel/services/schema_person.py file.

Common values:

class-name: .Person                      // generate it to the schema.py file
class-name: .subschemas.Person           // generate it to this file in services directory
class-name: ..common.person.Person       // generate it to "common" package on the same level as "services"

Allow custom pidtype

currently, 'recid' is used, it would be nice if Record API could generate one from configuration.

Nekompatibilita s opensearch

Vygenerovany model je pevne svazany s pouzitim elasticsearche:

  File "/model/model/services/facets.py", line 3, in <module>
    from elasticsearch_dsl import Facet
ModuleNotFoundError: No module named 'elasticsearch_dsl'

Keep standard invenio sort options

In generated search.py, prepend:

    sort_options = {
        **InvenioSearchOptions.sort_options,
        # any custom sort options
    }

Generating over existing sources fails with Getting name of node class Expr not implemented yet

File "/Users/miroslavsimek/w/repository-3.4/.venv-builder/lib/python3.10/site-packages/oarepo_model_builder/utils/cst.py", line 60, in visit_Assign
el_name = self._node_name(el)
File "/Users/miroslavsimek/w/repository-3.4/.venv-builder/lib/python3.10/site-packages/oarepo_model_builder/utils/cst.py", line 147, in _node_name
raise NotImplementedError(f'Getting name of node type {type(name_element)} not implemented yet')
NotImplementedError: Getting name of node type <class 'libcst._nodes.statement.Expr'> not implemented yet

Use correct facet type for edtf-interval fields

TermsFacet aggregations cannot be used with fields with date_range mapping type

Wrongly generated package name

When package name is missing in model and --output-directory points to a different directory, package name is generated from the current directory and not from the output directory

vars.search_options_bases is not filled

At oarepo_model_builder/invenio/templates/invenio_record_search_options.py.jinja2 :

    sort_options = {
        {% if vars.search_options_bases %}
        **{{ vars.search_options_bases[0]|base_name }}.sort_options,
        {% else %}
        **InvenioSearchOptions.sort_options,
        {% endif %}

It seems that search_options_bases is not filled, see the last test function in tests/test_search_options.py - base class is defined, but sort options are not included from it.

Interactive mode for replacing/commenting changes to user code

When updating already existing source files, allow for updating already existing elements that were changed. For example, in marshmallow, if the generated field is different that an already existing field in schema file, notify user and let him select which action to perform:

• replace with the new value
• keep the previous value (noop)
• keep the previous value and add the new one to # TODO comment

Add a command-line option --resolve-conflicts to allow for automatic processing with values:

• replace
• keep
• comment

model builder core - release Q1 2023

Tasks

• pinned main
• tests
• release
• readme documentation
• dev documentation

Parts

• Facets
  • Special characters in property names (@v)
  • searchable "false"
  • invenio searchable

Required for

• NR
• MOSBRI

Invenio props are generated inside the metadata section, not on top-level

model:
  oarepo:use: invenio
  properties:
    title:
      type: fulltext+keyword

permissions should be in services folder, not resources folder

Vypnout pousteni cookiecutteru v testech

Sorting support

Add support for autogenerated sorters.

Suggestion for simple sorters:

model:
  ...
    creator:
      oarepo:sortable: true
    creator2:
      oarepo:sortable:
        key: creator2
        direction: asc|desc

And for complex sorts:

oarepo:sort:
  bytitle:
  - key: title.sort
     direction: asc
  - key: creator
     direction: asc

Nested fields should have the correct mapping generated automatically

Use field's ui definition label in get_facet implementation

https://github.com/oarepo/oarepo-model-builder/blob/master/oarepo_model_builder/datatypes/primitive_types.py#L36

Resolve facet value label for Vocabularies

Same way as #151

Localized facet values - datetime, date, edtf & edtf intervals

• Create subclass of LabelledTermsFacet
• Pass the correct value_labels param to constructor of subclass
• Format facet date values inside the value_labels function