Metadata to ORM feature

[openenergyplatform]

Metadata to ORM feature

[christian-rli]

I agree with @MGlauer . The tool to handle and integrate metadata is omi. it's even in the name :)
oem-codegen seems like a reasonable name. Just to add another candidate here: oem2orm looks somewhat self-explanatory to me, but i might have been looking at acronyms for too long in my youth.

[MGlauer]

So I'm not sure if this feature should be part of OMI or oedialect.
Please discuss and improve this workflow!

IMO it should be an independent tool altogether or a dialect in OMI. But definitely not part of oedialect: oedialect is a facility to connect and communicate with an existing database. Changing things on the client-side is not part of any use-case. oedialect is a tool that is imported in a project to develop a tool A. Users of A have to install the additional oem-codegen functionality even though this has no bearing for the functionality of A. Users might want to generate an ORM schema but not use it with the OEP - but for their own, local database. We would limit the user base of md2orm by linking unnecessarily close to oedialect.

If it is part of OMI, there should also be a parser that tries to parse a given metadata schema and generates the corresponding metadata file (i.e. add an ORM dialect), because this is what OMI does. If we do not do this, I would argue for an independent tool.

And maybe rename md to oem (open energy metadata) as md might be associated with well-known markdown format. What about oem-codegen, as the proposed functionality is rather similar to sqlalchemy-codegen?

md2orm could be it's own command line application similar to OMI:

oem-codegen -t oep-1.4 medadata.json > schema.py

[Ludee]

i like oem2orm. That's what it is.
Is it ready for use?

I have an issue over there where I create metadata an want to upload it.

Can you please have a look @jh-RLI

[henhuy]

oem2orm - very good!
I don't know how much you tested it?!
Think we should test it with some more metadata and then it's ready

[jh-RLI]

I have already tested this with several metadata files. I'm now testing again if everything works with the oedialect and then we can do the package release.

[henhuy]

I think you could have made typer optional (check if import fails) and use input if not, but anyway... no problem for me.
I would be glad to see it on PyPI!
Could you add me to authors? Thx

[Ludee]

Similar toolchain for Data Download:

1. Register OEP schemas using saio
2. Retrieve ORM from OEP using saio
3. Download data using oedialect
4. Create ERM using ERAlchemy
5. Download metadata using oedialect

[Ludee]

So I'm not sure if this feature should be part of OMI or oedialect.
Please discuss and improve this workflow!

[jh-RLI]

I also think it would be practical if you could use the new tool on an independent basis. But I think the integration into OMI is useful to provide the user with all tools for metadata integration.

I like the idea of renaming the tool to oem-codegen .

I do not yet fully understand the parser part. The metadata itself should be created manually I think.
Or should the metadata string be validated by that?

[jh-RLI]

For the technical documentation of md2orm:
The tool is created as cli tool. This is implemented with Typer. All functions can also be imported individually.

To use it as a CLI tool a virtual environment (e.g. anaconda) should be created and all packages (requirements.txt) must be available there.
To run md2orm as a CLI tool the CMD must be started, the environment must be started and then the following command must be executed
python md_to_orm.py --engine=postgresql --database=name??? --host=localhost --port=5432 --log-level=debug --from-folder test_files

--database=name??? must be customized with the desired database name. If no value is set here, all tables will be created in the default postgresql database postgres.

The command shown above for md2orm uses the available test data from the test_files folder.

If the OEdialect is to be used, then the following values must be set --engine=postgresql+oedialect and --host=https://openenergy-platform.org/

To see all the options that can be used in md2orm you can use the CMD command:

python md_to_orm.py --help

[jh-RLI]

@henhuy as you mentioned in the past there are some data type issues with the oedialect. I moved the development to GitHub here. I removed the typer feature for convenience :). Recently I implemented the oep-restrictions. I would like to release this on PyPI.
@Ludee Do we have a OEP PyPi account?

[jh-RLI]

That would be possible, maybe I'll even put it back in. I have removed Typer because it has created confusion about how the tool is used. Still a great feature IMO. I will let you know when the release is done.
Sure, I'll add you as author.

[jh-RLI]

https://github.com/OpenEnergyPlatform/oem2orm

[henhuy]

We can close, right?

[jh-RLI]

Yes! But maybe we should document the diagrams that @Ludee created. I think he got them saved somewhere, but I'll put them in our "cloud" for now and then close this Issue.