Repository for the Open Metadata Integration (OMI). For metadata definition see metadata repo:
Home Page: https://github.com/OpenEnergyPlatform/metadata
License: GNU Affero General Public License v3.0
In file README.rst there is a link to https://metadata-tool.readthedocs.io/, however following the link you land on a page telling "SORRY This page does not exist yet."
OEP Metadata is now developed in its own repository. OMI should use the JSON-schema to parse the OEP metadata format.
The meta fields in the metadata JSON do not maintain their order when using the conversion. This originally is caused by the OEMetadata object that is created by omi´s compiler.
Example:
_comment is originally the last key in the oemetadata.
caused by this compiler method:
omi/src/omi/dialects/oep/compiler.py
Lines 339 to 370 in 27305e9
"_comment": {
"metadata": "Metadata documentation and explanation (https://github.com/OpenEnergyPlatform/organisation/wiki/metadata)",
"dates": "Dates and time must follow the ISO8601 including time zone (YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh)",
"units": "Use a space between numbers and units (100 m)",
"languages": "Languages must follow the IETF (BCP47) format (en-GB, en-US, de-DE)",
"licenses": "License name must follow the SPDX License List (https://spdx.org/licenses/)",
"review": "Following the OEP Data Review (https://github.com/OpenEnergyPlatform/data-preprocessing/wiki)",
"null": "If not applicable use: null",
"todo": "If a value is not yet available, use: todo"
},
"metaMetadata": {
"metadataVersion": "OEP-1.5.1",
"metadataLicense": {
"name": "CC0-1.0",
"title": "Creative Commons Zero v1.0 Universal",
"path": "https://creativecommons.org/publicdomain/zero/1.0/"
}
}Hi all,
I have around 30 JSON files I try to convert from OEP-1.3 to OEP-1.4.
Around ten files are failing, because they have in temporal.reference_date a none valid input (string: "none" instead of a datestring) and produce an error ("Unknown string format:", timestr).
If I have a look at the corresponding file [parse.py] (https://github.com/OpenEnergyPlatform/omi/blob/dev/src/omi/dialects/oep/parser.py) I can see that there are multiple instances in which parse_date is used without a safety net.
I would solve the issue in the concerning code as follows. Also for all the other occurrences of parse_date in parse.py.
BEFORE:
# filling the temporal section
temporal = structure.Temporal(
reference_date=parse_date(json_old["temporal"].get("reference_date")),
start=None,
end=None,
resolution=None,
ts_orientation=None,
)
AFTER:
# filling the temporal section
try:
temporal__reference_date = parse_date(json_old["temporal"].get("reference_date"))
except ValueError:
temporal__reference_date = json_old["temporal"].get("reference_date")
temporal = structure.Temporal(
reference_date=temporal__reference_date,
start=None,
end=None,
resolution=None,
ts_orientation=None,
)
What do you think? Is this approach feasible?
Currently, we have two independent developments of metadata functionalities and should integrate both in one independent tool that can be used by the oeplatform, oedialect and other users.
Omi is unable to parse this metadata string. As far as I can tell, the string contains a valid empty foreign key, but omi seems not to think so. Any idea as to why that is @MGlauer ?
omi translate -f oep-v1.4 cams_timeseries.json
Traceback (most recent call last):
File "/home/christianh/anaconda3/bin/omi", line 11, in <module>
sys.exit(main())
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/cli.py", line 49, in main
cli()
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/cli.py", line 35, in translate
obj = from_dialect.parse(infile.read())
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/dialects/base/dialect.py", line 44, in parse
return p.parse_from_string(string, *args, **kwargs)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/dialects/base/parser.py", line 69, in parse_from_string
**(parse_kwargs or {})
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/dialects/oep/parser.py", line 348, in parse
for field_name in fk.get("fields", [])
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/dialects/oep/parser.py", line 348, in <listcomp>
for field_name in fk.get("fields", [])
KeyError: None
A function would come in handy to split and join metadata from existing tables.
1st use-case: 1 metadata string with multiple tables in key resources. Create one metadata string for each resource and copy the general information to each individual new resource-metadata-string.
2nd use-case: reverse 1st use-case
When compiling an oemetadata json in version oem-v1.5, the keys for isAbout and valueReference are renamed to is_about and value_reference. This is a bug and causes errors when uploading and editing the metadata on the OEP.
Related to OpenEnergyPlatform/oeplatform#1549
I expected OMI to work in Python using translate(), I tried
from omi.cli import translate
translate('oep-v1.4', 'oep-v1.4', 'out.json', 'in.json')but it outputs
Usage: oep-v1.4 [OPTIONS] FILE_PATH
Try 'oep-v1.4 --help' for help.
Error: Got unexpected extra arguments (e p - v 1 . 4)
and exits Python.
What's wrong here?
JSON lists are inherently ordered. The RDF-based metadata use mostly properties which does not preserve orderings. We could use RDFS list semantics, but those are (imo) mostly unneeded for the metadata as is.
Are (explicit) orderings necessary or can they be omitted?
I ran into this error:
File "/home/christianh/Schreibtisch/omi/env/lib/python3.7/site-packages/omi-0.0.2-py3.7.egg/omi/cli.py", line 42, in translate
outfile.write(s.decode("utf-8"))
AttributeError: 'str' object has no attribute 'decode'
Stackoverflow suggests to simply drop the .decode("utf8"), because with python3 all strings are unicode objects already. Removing it worked for me and the cli worked afterwards. Going to push it in a new branch and create a PR.
When updating the structure.py with the latest metadata version (oem v1.5.0 and v1.5.1) I noticed that older versions are no longer supported when the structure is changed. For example in OEM v 1.5.0 the key temporal has been changed. Here the subkey timeseries is now a separate object.
Old until oem v1.4:
class Temporal(Compilable):
__compiler_name__ = "temporal"
def __init__(
self,
reference_date: datetime = None,
start: datetime = None,
end: datetime = None,
resolution: str = None,
ts_orientation: TimestampOrientation = None,
aggregation: str = None,
): # TODO: This should not be a string... maybe
# we should use datetime instead?
self.reference_date = reference_date
self.ts_start = start
self.ts_end = end
self.ts_resolution = resolution
self.ts_orientation = ts_orientation
self.aggregation = aggregationNew in oem v1.5:
class Timeseries(Compilable):
__compiler_name__ = "timeseries"
def __init__(self,
start: datetime = None,
end: datetime = None,
resolution: str = None,
ts_orientation: TimestampOrientation = None,
aggregation: str = None):
self.ts_start = start
self.ts_end = end
self.ts_resolution = resolution
self.ts_orientation = ts_orientation
self.aggregation = aggregation
class Temporal(Compilable):
__compiler_name__ = "temporal"
def __init__(
self,
reference_date: datetime = None,
timeseries_collection: Iterable[Timeseries] = None
# start: datetime = None,
# end: datetime = None,
# resolution: str = None,
# ts_orientation: TimestampOrientation = None,
# aggregation: str = None,
): # TODO: This should not be a string... maybe
# we should use datetime instead?
self.reference_date = reference_date
self.timeseries_collection = timeseries_collection
# self.ts_start = start
# self.ts_end = end
# self.ts_resolution = resolution
# self.ts_orientation = ts_orientation
# self.aggregation = aggregationIs there a way to keep the structure.py downward compatible without keeping a class for obsolete structure. Or would one structure.py have to be created for each OEM version.
@MGlauer as you implemented this could you help me out with this? Let me know if you understand my issue description.
I'm on an Ubuntu machine and not sure whether this is an actual bug or a result of my setup. Installing omi via pip or from source both result in an error, when using the following command in via command line:
omi translate -f oep-v1.3 -t oep-v1.4 metadata_v13.json
I used the sample json from within this repo. My error looks like this:
File "/home/christianh/anaconda3/bin/omi", line 11, in <module>
sys.exit(main())
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/cli.py", line 49, in main
cli()
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/cli.py", line 35, in translate
obj = from_dialect.parse(infile.read())
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/dialects/base/dialect.py", line 18, in parse
return p.parse(string, *args, **kwargs)
File "/home/christianh/anaconda3/lib/python3.7/site-packages/omi/dialects/oep/parser.py", line 59, in parse
old_spatial = json_old.get("spatial")
AttributeError: 'str' object has no attribute 'get'
I tried fiddling around with different branches and installing omi from there, but to no avail. Since it's mentioned in the documentation, I tried the same conversion with the command metadata-tool as well, which produced the same error.
Release omi v0.0.7 on Github and update PyPi package.
The current docs seem to be outdated (0.0.2, changelog says 0.0.0).
Also, the docs would really benefit from some more explanations and references.
Validation should be based on the schema.json. Write a new function that validates a metadata json based on the schema. This should work independent of the versions. But tests should include oemetadata versions 1.5.2, 1.6 and 2.0
See #99
You code seems to be written in Python2.x. According to our coding conventions we need at least python 3.5. Please make your code compatible
https://gitlab.com/dlr-ve/esy/ioprocmeta/
ioprocmeta is a library providing interfaces for writing and reading and processing meta data formats like the Open Energy Platform meta data format 1.5.0.
base, that must contain a python file, e.g. test.pyomi: from omi.dialects import get_dialectModuleNotFoundError: No module named 'omi.dialects.base.test'
the __import__ function gets confused because it searches in __path__ as well as the local path.
Maybe do the dynamic import manually, for instance by replacing code in in dialects/__init__.py:
import os
from omi.dialects.base.register import get_dialect
path = os.path.dirname(__file__)
for d in os.listdir(path):
if os.path.isfile(os.path.join(path, d, 'dialect.py')):
module_name = 'omi.dialects.' + d + '.dialect'
__import__(module_name)OMI has several issues. Most importantly it's difficult to maintain and needlessly convoluted. It should reliably perform a handful of tasks and be less of everything else. @henhuy now has time time to address this in a radical rewrite. This issue is a meta level discussion space and collects things to be done.
@wingechr : you are warmly invited to ask for features, make structural remarks and get involved if you like. The goal is to have a tool that you and all of us enjoy using.
Broad plan:
Things OMI should reliably do after the rewrite:
Related info:
I used omi and when I output each field, the order is not the same as in the schema and there is a difference between the namings. Furthermore, there are also more fields displayed in omi than in the schema.
Here is a list of what I found:
General Keys:
There is also a problem that omi only outputs fields with values, if values = null --> these are not shown
We use the OMI tool in eGon-data to check the metadata. It is quite helpful but lacks some speaking output:
omi.dialects.base.parser.ParserException: metadata string does not contain an id. In my opinion it'd be better to parse the entire metadata and give a list of errors/warnings, otherwise one might end up in a trial-and-error odyssey.dateutil.parser._parser.ParserError: String does not contain a date:. Pardon, which field?Should it be possible to add some user input to new fields like "is_about" or should omi just convert the metadata and leave the editing to tools like the MetaEditor on the OEP.
Example:
{
"name": "id",
"description": "Unique identifier",
"type": "serial",
"is_about": [
{
"name": "",
"path": ""
}
],
"value_reference": [
{
"value": "",
"name": "",
"path": ""
}
]
}To conform to DCAT-AP it is mandatory to have a dcat:Catalog (with its publisher) where the dataset belong to. I am not sure how to derive this information from the metadata. Who is the publisher of a dataset?
I tried to parse and compile this file which first fails with the error described in #12 . After changing the null values to dates for testing purposes, I can parse without a problem.
Compiling the parsed metadata back to v1.4 works. Compiling to 1.3 yields this error:
print(dialect1_3.compile(parsed))
File ".../omi/env/lib/python3.7/site-packages/omi-0.0.2-py3.7.egg/omi/dialects/base/dialect.py", line 13, in compile
c = self._compiler()
TypeError: 'NoneType' object is not callable
Reactivate CI by introducing github actions workflow automatization features:
Adds new bug and feature templates.
When omi processes json metadata it omits values and metadata that are null.
This is due when omi reads a json into a python dict, null is read in as None. This results in missing metadata fields when compiling with omi.
e.g. missing description field and unit field
input
"schema": {
"fields": [
{
"name": "fid",
"description": null,
"type": "int",
"unit": null,
"isAbout": [
{
"name": null,
"path": null
}
],
"valueReference": [
{
"value": null,
"name": null,
"path": null
}
]
}
output
"schema": {
"primaryKey": [],
"foreignKeys": [],
"fields": [{
"name": "fid",
"type": "int",
"isAbout": [{}
],
"valueReference": [{}
]
}
This is important as oem2orm throughs an error for some keys when generating tables from metadata for the OEP and they're missing, e.g. description.
ERROR:Could not generate tables from metadatafile:
KeyError Traceback (most recent call last)
<ipython-input-17-7e96dde4dbd2> in <module>
----> 1 tables_orm = oem2orm.collect_tables_from_oem(db, metadata_folder)
~\Miniconda3\envs\p_py38_oep-upload\lib\site-packages\oem2orm\oep_oedialect_oem2orm.py in collect_tables_from_oem(db, oem_folder_path)
272 for metadata_file in metadata_files:
273 try:
--> 274 md_tables = create_tables_from_metadata_file(db, metadata_file)
275 logging.info(md_tables)
276 except:
~\Miniconda3\envs\p_py38_oep-upload\lib\site-packages\oem2orm\oep_oedialect_oem2orm.py in create_tables_from_metadata_file(db, metadata_file)
217 column_type,
218 primary_key=field["name"] in primary_keys,
--> 219 comment=field["description"],
220 )
221 columns.append(column)
KeyError: 'description'
The current status makes automatic metadata processing and upload impracticable.
EDIT
After implementing omit_none=False results
output
"schema": {
"primaryKey": [],
"foreignKeys": [],
"fields": [{
"name": "fid",
"description": null,
"type": "int",
"isAbout": [{
"name": null,
"path": null
}
],
"valueReference": [{
"value": null,
"name": null,
"path": null
}
],
"unit": null
}
When parsing a metadata string, omi seems to just look for a few mandatory keys and accepts what comes with it. Keys that aren't defined in the metadata standard don't trigger a warning and are simply ignored. A warning would be quite helpful, for example in case a key was misspelled.
Right now, validation stores error report into file.
But with this implementation, I cannot inspect error report directly, but have to load report from file again.
Instead I suggest to return error report from validation function.
In case of no errors an empty error report, stating "no errors found" should be returned.
This is the way it is implemented in frictionless as well.
As validation function does not return anything ATM, this can be easily implemented.
Additionally, I would make storing report on filesystem an option (with default False) as this might not be wanted by users?
The field "resources" should be mandatory (This is also the specification for datapackages)
Hi @Bachibouzouk,
yesterday, you did send me JSON schema drafts for v1.3 and v1.4 of the OEP metadata, I included them in a gist so that other people know what we are talking about if they are reading this: link
I had a look at the getting started section on json-schema.org and realised that our draft version doesn't include descriptions (additionally to the types) and also doesn't define schema version and $id (which is obvious because we didn't decided that yet (OpenEnergyPlatform/organisation#26). I think the possibility of adding descriptions is very handy and shouldn't be missed, because we could get rid of https://github.com/OpenEnergyPlatform/examples/wiki/Metadata-Description and could point $id directly to a single JSON schema file (that is self describing).
What do you think? Shall I add the description of v1.4 into the schema draft?
And second question, where is the description of v1.3? Maybe @Ludee or @christian-rli know that ...
@areleu developed functions to validate metadata.
This should be included in OMI: OpenEnergyPlatform/oedatamodel#55
@jh-RLI please get in contact to coordinate the merge and implementation.
Conversion between different versions of the metadata needs to be extended or rewritten. Versions 1.5.2, 1.6, 2.0 need support. Only forward support is necessary. Include tests and a short documentation on how to use the conversion function.
See #99
In the scope of current projects, it's becoming necessary to annotate existing datasets on the OEP with the ontology. Unfortunately, these datasets are often still annotated with OEM141, which is not yet ready to work with the ontology. Therefore, instead of transforming the OEM141 strings manually to OEM151, a script could come in handy.
TODO
Since we have developed the metadata standard OEM further and have not adapted omi immediately, the metadata conversion from a low version to the latest version is currently not supported.
With version v1.5.0 the structure of the metadata was changed, essentially under the key "temporal" the subkey timeseries (already introduced in v1.4.1) was changed to the type list of objects :[{..},{..}]. Since omi works with the current version of the oem structure and does not use one structure definition per version, it is necessary to define in the parser classes or directly in the conversion how the current structure can be translated to the desired metadata version.
Also the other new keys have to be included in the conversion method.
If you install omi and try to convert a metadata file v1.4.1 to v.1.5.0 (using the metadata from the oemetadata repository) there is some output but the output is very limited and not correct.
omi translate -f oep-v1.4 -t oep-v1.5 ..\oemetadata\metadata\v141\example.json
Output:
{
"title": "Ludee",
"object": "metadata",
"comment": "Release metadata version OEP-1.3.0",
"date": "2019-07-09"
}
],
"resources": [
{
"profile": "tabular-data-resource",
"name": "model_draft.oep_metadata_table_example_v141",
"path": "http://openenergyplatform.org/dataedit/view/model_draft/oep_metadata_table_example_v141",
"format": "PostgreSQL",
"encoding": "UTF-8",
"schema": {
"primaryKey": [
"id"
],
"foreignKeys": [
{
"fields": [
"year"
],
"reference": {
"resource": "schema.table",
"fields": [
"year"
]
}
}
],
"fields": [
{
"name": "id",
"description": "Unique identifier",
"type": "serial"
},
{
"name": "year",
"description": "Reference year",
"type": "integer"
},
{
"name": "value",
"description": "Example value",
"type": "double precision",
"unit": "MW"
},
{
"name": "geom",
"description": "Geometry",
"type": "geometry(Point, 4326)"
}
]
},
"dialect": {
"decimalSeparator": "."
}
}
],
"metaMetadata": {
"metadataVersion": "OEP-1.5.0",
"metadataLicense": {
"name": "CC0-1.0",
"title": "Creative Commons Zero v1.0 Universal",
"path": "https://creativecommons.org/publicdomain/zero/1.0/"
}
}
}In Metadata v1.4 the value of the language attribute is described with the Standard: IETF (BCP47). What is a good way to get the correct entry of the MDR Languages Named Authority List, which is used as obligatory controlled vocabulary for dct:language in DCAT-AP.de?
In the course of a review in the oedatamodel Reop. was asked to allow YYYY as a valid timestamp.
Currently, we just support YYYY-MM-DD or YYYY-MM-DDThh:mm:ss±hh format.
Using metadata json structures that do not provide all fields of the oemetadata spec. are not handled as expected by OMI. For most fields this will not raise any braking errors but with the latest version of OMI (0.0.8) the _comment field was added to the OEMetadata class as keyword-argument reading the values from the input metadata. If the _comment field is empty in the input this will cause a breaking error when trying to compile from a metadata json/dict.
Example
meta = {"id": self.test_table}
using omi to compile then raises:
File "/home/runner/work/oeplatform/oeplatform/.tox/py3/lib/python3.8/site-packages/omi/dialects/oep/compiler.py", line 372, in visit_metadata
metadata=metadata.comment.metadata_info,
AttributeError: 'NoneType' object has no attribute 'metadata_info'
There is an API with checked licenses: https://opendefinition.org/licenses/api/
And here's how to use curl to access the CKAN license group:
curl https://licenses.opendefinition.org/licenses/groups/ckan.json
Check the licenses and give warnings, e.g. when the license is not open data.
The source code is cluttered and has a range of no longer useful bits.
Remove obsolete classes, specifically the numerous classes for every key in every version of the metadata. New functions will be based on the schema.json. Make a list of what can be reused or rewritten. Tidy up the repository with everything that will stay. Drop loose ends if necessary.
See #99
This might be an issue, because people may try to parse metadata with typos.
Will this be solved by #20?
When parsing a metadata string, omi seems to just look for a few mandatory keys and accepts what comes with it. Missing optional fields don't trigger a warning. These would be quite helpful, so there should be one.
Other projects already developed functions to generate parts of the OEMetadata string.
eGoN-data:
https://github.com/openego/eGon-data/blob/dev/src/egon/data/metadata.py
There is a package from open_ENTRANCE project called:
https://github.com/sentinel-energy/friendly_data
It is based on data packages and should be compatible.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.