projekt-opal/converter

Spring cloud designed converter (D4.2)

converter's Issues

Update README.md

The current README.md seems to be incomplete.

Please check:

• WARNING: The DATASET_FOLDER variable is not set. Defaulting to a blank string.
• WARNING: The ES_INDEX variable is not set. Defaulting to a blank string.
• please add ex_logstash and bind q_logstash to it
• ERROR: for opal-converter_dataset-file-fetcher_1 Cannot create container for service dataset-file-fetcher: create .: volume name is too short, names should be at least two alphanumeric characters
• ERROR: for dataset-file-fetcher Cannot create container for service dataset-file-fetcher: create .: volume name is too short, names should be at least two alphanumeric characters

Successfully built c8f51b4e1f75
Successfully tagged opal-converter_elasticsearch-writer:latest
WARNING: The DATASET_FOLDER variable is not set. Defaulting to a blank string.
WARNING: The ES_INDEX variable is not set. Defaulting to a blank string.
Starting opal-converter_rabbitmq_1           ... done
Recreating opal-converter_discovery-server_1 ... done
Recreating opal-converter_es_1               ... done
==============
please add ex_logstash and bind q_logstash to it
==============
Press [ENTER] to continue,...: 
WARNING: The DATASET_FOLDER variable is not set. Defaulting to a blank string.
WARNING: The ES_INDEX variable is not set. Defaulting to a blank string.
opal-converter_rabbitmq_1 is up-to-date
opal-converter_es_1 is up-to-date
Recreating opal-converter_config-server_1 ... done
Starting opal-converter_logstash_1        ... done
Starting opal-converter_kibana_1          ... done
WARNING: The DATASET_FOLDER variable is not set. Defaulting to a blank string.
WARNING: The ES_INDEX variable is not set. Defaulting to a blank string.
opal-converter_discovery-server_1 is up-to-date
opal-converter_es_1 is up-to-date
opal-converter_rabbitmq_1 is up-to-date
opal-converter_config-server_1 is up-to-date
opal-converter_kibana_1 is up-to-date
opal-converter_logstash_1 is up-to-date
Recreating opal-converter_data-cleaner-service_1 ... 
Recreating opal-converter_opal-confirm-conversion-service_1 ... 
Recreating opal-converter_data-cleaner-service_1            ... done
Recreating opal-converter_opal-confirm-conversion-service_1 ... done
Recreating opal-converter_triplestore-writer_1              ... done
Recreating opal-converter_elasticsearch-writer_1            ... done
Recreating opal-converter_dataset-fetcher_1                 ... done
Recreating opal-converter_quality-metrics-service_1         ... done
ERROR: for opal-converter_dataset-file-fetcher_1  Cannot create container for service dataset-file-fetcher: create .: volume name is too short, names should be at least two alphanumeric characters

ERROR: for dataset-file-fetcher  Cannot create container for service dataset-file-fetcher: create .: volume name is too short, names should be at least two alphanumeric characters
ERROR: Encountered errors while bringing up the project.

Fetch complete dataset graphs

Please ensure that a complete graph of a dataset is fetched.

E.g. the code below would not include a o3:

?dataSet ?predicate ?object
?object ?p2 ?o2
?o2 ?p3 ?o3

        ParameterizedSparqlString pss = new ParameterizedSparqlString("" +
                "PREFIX dcat: <http://www.w3.org/ns/dcat#> " +
                "PREFIX dct: <http://purl.org/dc/terms/> " +
                "CONSTRUCT { " + "?dataSet ?predicate ?object . " +
                "?object ?p2 ?o2} " +
                "WHERE { " +
                ( this.queryWithGraph ? "  GRAPH ?g { " : "") +
                "    ?dataSet ?predicate ?object. " +
                "    OPTIONAL { ?object ?p2 ?o2 } " +
                ( this.queryWithGraph ?  "} " : "") +
                "}");

https://github.com/projekt-opal/converter/blob/developer/dataset-fetcher/src/main/java/org/diceresearch/datasetfetcher/utility/DataSetFetcher.java#L231

Use OPAL Common to create URIs

Since November 2019, OPAL Common provides a URI rewriter:
https://github.com/projekt-opal/common/blob/master/src/main/java/org/dice_research/opal/common/utilities/UriRewriter.java
This has to be used to rewrite URIs.

The current implementation of the converter is here:
https://github.com/projekt-opal/converter/blob/f466e2b033bd7bbb8cfa9a34fd5f23eb189145a6/opal-confirm-conversion-service/src/main/java/org/diceresearch/opalconfirmconversionservice/utility/OpalConfirmer.java
in methond generateOpalConfirmedUrl(...)

Both methods create the same hashes, see:
https://github.com/projekt-opal/common/blob/master/src/test/java/org/dice_research/opal/common/UriRewriterDemoTest.java