projekt-opal/catfish

DCAT data cleaning (D4.2)

catfish's Issues

Check downloadUrl accessUrl of Distribution

If both not given, remove Distribution.
If then no Distribution available, clear model.

Cleaning of filetypes

The download url has to be analyzed to create format or mediaType like described in DCAT.

(This is required for filtering in the frontend.)

Cleaning of distributions

If a dataset has no distribution, it should be ignored -> empty model

Cleaning of themes

Themes should use a limited set of URIs.
If localized (german, french, ...) URIs are used, international (english) URIs should be added.
Alternatively, localized URIs could be replaced by international ones.

List of theme translations:
http://publications.europa.eu/resource/dataset/data-theme
(source: https://ec.europa.eu/isa2/dcat-ap-key-european-data-publishers-reach-data-consumers-and-create-european-open-data-ecosystem_en)

Cleaning of dates

If for a Dataset [Dataset] any temporal information is available, the following formats should be added, if not available:

• Set dct:issued for the Dataset
• Set dct:modified for the Dataset

If these temporal information is available in related Distributions, but not in the Dataset, it has to be checked if the information can be copied.

Regarding the format to use, it has to be [Dataset]:

• rdfs:Literal
• encoded using the relevant ISO 8601 Date and Time compliant string [ISO8601]
• typed using the appropriate XML Schema datatype [xmlschema-2]
• Example from [Dataset]: "2011-12-05"^^xsd:date

Additionally, and especially if not all information is available (e.g. only year), it has to be checked, if information in the following types can be made available. One soulution is to provide 3 triples for every predicate:

• xsd:gYear | Gregorian calendar year
• xsd:gMonth | Gregorian calendar month
• xsd:gDay | Gregorian calendar day of the month

A better way would be to choose the predicate type which is appropriate for the available information: [Datatypes]

• xsd:gYear | Gregorian calendar year
• xsd:gYearMonth | Gregorian calendar year and month
• xsd:date | Dates (yyyy-mm-dd) with or without timezone

Resources:

• [Dataset] https://www.w3.org/TR/vocab-dcat-1/#class-dataset
• [Datetime] ISO 8601 Date and Time compliant string http://www.w3.org/TR/NOTE-datetime
• [ISO8601] https://en.wikipedia.org/wiki/ISO_8601
• [Datatypes] https://www.w3.org/TR/rdf11-concepts/#xsd-datatypes

Cleaning of licenses