epics-containers/ibek-support

ibek definitions

ibek-support's Issues

Add system tests

We need CI on this project to verify that all support modules will install into a container.
i.e. test install.sh in each subfolder

Also we should have some system tests to demonstrate that ibek can successfully render startup and DB for a range of interesting example IOC instance YAMLs.

Clashes between definition names

We can't build a global schema of all support yaml if two support yaml share a definition name. That is even though the module name is different.

This is perhaps not a real problem, except that we do this in the CI tests to make sure all submitted yaml is good.

This limitation makes the module a little brittle.

e.g.

generating all support schema
+ ibek ioc generate-schema /home/runner/work/ibek-support/ibek-support/tests/../ADAravis/ADAravis.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../ADCore-slim/ADCore.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../ADCore/ADCore.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../ADSample/ADSimDetector.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../ADSimDetector/ADSimDetector.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../_global/epics.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../asyn/asyn.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../autosave/autosave.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../iocStats/iocStats.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../modbus/modbus.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../motor/motor.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../motorMotorSim/motorMotorSim.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../pmac/pmac.ibek.support.yaml /home/runner/work/ibek-support/ibek-support/tests/../zebra/zebra.ibek.support.yaml --no-ibek-defs --output /tmp/all.ibek.ioc.schema.json
Traceback (most recent call last):

  File "/home/runner/.local/bin/ibek", line 8, in <module>
    sys.exit(cli())

  File "/home/runner/.local/lib/python3.10/site-packages/ibek/ioc_cmds/commands.py", line 72, in generate_schema
    ioc_model = ioc_create_model(definitions)

  File "/home/runner/.local/lib/python3.10/site-packages/ibek/gen_scripts.py", line 45, in ioc_create_model
    model = make_ioc_model(entity_models)

  File "/home/runner/.local/lib/python3.10/site-packages/ibek/ioc.py", line 178, in make_ioc_model
    class NewIOC(IOC):

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 182, in __new__
    complete_model_class(

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_model_construction.py", line 501, in complete_model_class
    schema = gen_schema.clean_schema(schema)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py", line 405, in clean_schema
    schema = _discriminated_union.apply_discriminators(schema)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 60, in apply_discriminators
    return simplify_schema_references(_core_utils.walk_core_schema(schema, inner))

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 431, in walk_core_schema
    return f(schema.copy(), _dispatch)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 48, in inner
    s = recurse(s, inner)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 207, in walk
    return f(schema, self._walk)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 48, in inner
    s = recurse(s, inner)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 210, in _walk
    schema = self._schema_type_to_method[schema['type']](schema.copy(), f)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 240, in handle_definitions_schema
    new_inner_schema = self.walk(schema['schema'], f)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 207, in walk
    return f(schema, self._walk)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 48, in inner
    s = recurse(s, inner)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 210, in _walk
    schema = self._schema_type_to_method[schema['type']](schema.copy(), f)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 219, in _handle_other_schemas
    schema['schema'] = self.walk(sub_schema, f)  # type: ignore

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 207, in walk
    return f(schema, self._walk)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 48, in inner
    s = recurse(s, inner)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 210, in _walk
    schema = self._schema_type_to_method[schema['type']](schema.copy(), f)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 354, in handle_model_fields_schema
    replaced_field['schema'] = self.walk(v['schema'], f)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 207, in walk
    return f(schema, self._walk)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 48, in inner
    s = recurse(s, inner)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 210, in _walk
    schema = self._schema_type_to_method[schema['type']](schema.copy(), f)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 336, in handle_json_or_python_schema
    schema['json_schema'] = self.walk(schema['json_schema'], f)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 207, in walk
    return f(schema, self._walk)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 48, in inner
    s = recurse(s, inner)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 210, in _walk
    schema = self._schema_type_to_method[schema['type']](schema.copy(), f)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 254, in handle_list_schema
    schema['items_schema'] = self.walk(items_schema, f)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_core_utils.py", line 207, in walk
    return f(schema, self._walk)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 57, in inner
    s = apply_discriminator(s, discriminator, definitions)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 99, in apply_discriminator
    return _ApplyInferredDiscriminator(discriminator, definitions or {}).apply(schema)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 194, in apply
    schema = self._apply_to_root(schema)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 234, in _apply_to_root
    self._handle_choice(choice)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 285, in _handle_choice
    self._handle_choice(self.definitions[choice['schema_ref']])

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 309, in _handle_choice
    self._set_unique_choice_for_values(choice, inferred_discriminator_values)

  File "/home/runner/.local/lib/python3.10/site-packages/pydantic/_internal/_discriminated_union.py", line 504, in _set_unique_choice_for_values
    raise TypeError(

TypeError: Value 'ADSimDetector.simDetector' for discriminator 'type' mapped to multiple choices

Error: Process completed with exit code 1.

install.sh missing dependency checks

install.sh is supposed to be the recipe for building a support module in the container environment.

However it is up to the Dockerfile to build dependencies before calling install.sh for a given module. This is fine, except that the install.sh should be able to give the developer information on the dependencies and check that they have been supplied.

I suggest:

ibek support check-dependencies Asyn busy ADCore

for example. How to check? We could just do it off of macros in global configure RELEASE or we could get more sophisticated and check that the module has actually been built. @coretl @GDYendell looking for suggestions of how to do this in a clean fashion.

This helps avoid unpicking build errors due to missing dependencies.

CI tests not ideal

The CI tests seem like an important thing to have as this module affects all generic IOCs.

At present they are fiddly and hard to maintain plus take ages to run.

Needs a major review and at the very least should be using ec commands like other project's CI.

Generic IOC image generation should require main branch of ibek-support

This issue is about generic IOCs and therefore could be in https://github.com/epics-containers/ioc-template but I'm puting it here because it's really an issue with management of this repo.

Because ibek-support is a submodule of every generic IOC and submodule versions are defined by commit hash only, there is a very likely scenario that individual branches of the this repo will proliferate on a per-generic-IOC basis. The only motivation to merge into main is when you want to re-use changes to a support module definition in a different generic IOC. At that point you may be in a world of merge conflict pain. giles as already experienced this when he is the only person working on the repo at present.

@GDYendell has suggested the following fix.

1. Change the generic IOC CI such that it will fail and not release push image to the container registry if it determines that the commit hash that it is using is NOT in the main branch. This therefore requires that the developer merges and pushes their changes before pushing the generic IOC source.
2. Allow that this check is switched off if the generic IOC tag that kicked off the CI is a beta tag

@coretl care to comment on this?