projekt-opal/batch

Batch processing of OPAL components (D4.2)

batch's Issues

Elasticsearch Writer: Check content length

To be examined:

java -jar opal-batch.jar DICE/git/opal-batch/local/elasticsearch/elasticsearch.properties
00:25:32.102 [main] INFO  org.dice_research.opal.batch.configuration.Cfg - Configuration: DICE/git/opal-batch/local/elasticsearch/elasticsearch.properties
00:25:32.173 [main] INFO  org.dice_research.opal.batch.Batch - Processing 27 files in DICE/Data/OPAL/OpalGraph/latest-2020-07-14/2020-07/opal-2020-07
Juli 23, 2020 12:25:49 VORM. org.apache.sis.referencing.factory.sql.EPSGFactory <init>
WARNUNG: The “SIS_DATA” environment variable is not set.
00:44:01.727 [main] ERROR org.dice_research.opal.batch.Batch - Error in processing DICE/Data/OPAL/OpalGraph/latest-2020-07-14/2020-07/opal-2020-07/opal-edp-3.ttl
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=illegal_argument_exception, reason=Document contains at least one immense term in field="distributions.description" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[34, -30, -128, -108, 32, 42, 32, 84, 105, 116, 101, 108, 32, 42, 58, 10, 68, 99, 95, 116, 105, 116, 108, 101, 32, 91, 116, 101, 120, 116]...', original message: bytes can be at most 32766 in length; got 43648]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1897) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1867) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1624) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1581) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1551) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:968) ~[opal-batch.jar:?]
	at org.dice_research.opal.batch.writer.ElasticsearchWriter.processModel(ElasticsearchWriter.java:48) ~[opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.processFile(Batch.java:256) ~[opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.processDirectory(Batch.java:220) [opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.execute(Batch.java:103) [opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.execute(Batch.java:86) [opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.main(Batch.java:57) [opal-batch.jar:?]
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://opaldata.cs.uni-paderborn.de:9200], URI [/opal-july/_doc?timeout=1m], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"distributions.description\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[34, -30, -128, -108, 32, 42, 32, 84, 105, 116, 101, 108, 32, 42, 58, 10, 68, 99, 95, 116, 105, 116, 108, 101, 32, 91, 116, 101, 120, 116]...', original message: bytes can be at most 32766 in length; got 43648"}],"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"distributions.description\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[34, -30, -128, -108, 32, 42, 32, 84, 105, 116, 101, 108, 32, 42, 58, 10, 68, 99, 95, 116, 105, 116, 108, 101, 32, 91, 116, 101, 120, 116]...', original message: bytes can be at most 32766 in length; got 43648","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"bytes can be at most 32766 in length; got 43648"}},"status":400}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1611) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1581) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1551) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:968) ~[opal-batch.jar:?]
		at org.dice_research.opal.batch.writer.ElasticsearchWriter.processModel(ElasticsearchWriter.java:48) ~[opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.processFile(Batch.java:256) ~[opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.processDirectory(Batch.java:220) [opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.execute(Batch.java:103) [opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.execute(Batch.java:86) [opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.main(Batch.java:57) [opal-batch.jar:?]
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=max_bytes_length_exceeded_exception, reason=bytes can be at most 32766 in length; got 43648]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496) ~[opal-batch.jar:?]
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407) ~[opal-batch.jar:?]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437) ~[opal-batch.jar:?]
	at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603) ~[opal-batch.jar:?]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:169) ~[opal-batch.jar:?]
	... 12 more
01:26:33.926 [main] ERROR org.dice_research.opal.batch.Batch - Error in processing DICE/Data/OPAL/OpalGraph/latest-2020-07-14/2020-07/opal-2020-07/opal-edp-5.ttl
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=illegal_argument_exception, reason=Document contains at least one immense term in field="distributions.description" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[34, -30, -128, -108, 32, 42, 32, 77, 111, 110, 97, 116, 32, 42, 32, 74, 97, 104, 114, 32, 42, 58, 10, 77, 111, 105, 115, 95, 97, 110]...', original message: bytes can be at most 32766 in length; got 34234]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:1897) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:1867) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1624) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1581) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1551) ~[opal-batch.jar:?]
	at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:968) ~[opal-batch.jar:?]
	at org.dice_research.opal.batch.writer.ElasticsearchWriter.processModel(ElasticsearchWriter.java:48) ~[opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.processFile(Batch.java:256) ~[opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.processDirectory(Batch.java:220) [opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.execute(Batch.java:103) [opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.execute(Batch.java:86) [opal-batch.jar:?]
	at org.dice_research.opal.batch.Batch.main(Batch.java:57) [opal-batch.jar:?]
	Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://opaldata.cs.uni-paderborn.de:9200], URI [/opal-july/_doc?timeout=1m], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"distributions.description\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[34, -30, -128, -108, 32, 42, 32, 77, 111, 110, 97, 116, 32, 42, 32, 74, 97, 104, 114, 32, 42, 58, 10, 77, 111, 105, 115, 95, 97, 110]...', original message: bytes can be at most 32766 in length; got 34234"}],"type":"illegal_argument_exception","reason":"Document contains at least one immense term in field=\"distributions.description\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[34, -30, -128, -108, 32, 42, 32, 77, 111, 110, 97, 116, 32, 42, 32, 74, 97, 104, 114, 32, 42, 58, 10, 77, 111, 105, 115, 95, 97, 110]...', original message: bytes can be at most 32766 in length; got 34234","caused_by":{"type":"max_bytes_length_exceeded_exception","reason":"bytes can be at most 32766 in length; got 34234"}},"status":400}
		at org.elasticsearch.client.RestClient.convertResponse(RestClient.java:283) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:261) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1611) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1581) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1551) ~[opal-batch.jar:?]
		at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:968) ~[opal-batch.jar:?]
		at org.dice_research.opal.batch.writer.ElasticsearchWriter.processModel(ElasticsearchWriter.java:48) ~[opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.processFile(Batch.java:256) ~[opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.processDirectory(Batch.java:220) [opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.execute(Batch.java:103) [opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.execute(Batch.java:86) [opal-batch.jar:?]
		at org.dice_research.opal.batch.Batch.main(Batch.java:57) [opal-batch.jar:?]
Caused by: org.elasticsearch.ElasticsearchException: Elasticsearch exception [type=max_bytes_length_exceeded_exception, reason=bytes can be at most 32766 in length; got 34234]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:496) ~[opal-batch.jar:?]
	at org.elasticsearch.ElasticsearchException.fromXContent(ElasticsearchException.java:407) ~[opal-batch.jar:?]
	at org.elasticsearch.ElasticsearchException.innerFromXContent(ElasticsearchException.java:437) ~[opal-batch.jar:?]
	at org.elasticsearch.ElasticsearchException.failureFromXContent(ElasticsearchException.java:603) ~[opal-batch.jar:?]
	at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:169) ~[opal-batch.jar:?]
	... 12 more
02:33:00.587 [main] INFO  org.dice_research.opal.batch.Batch - Finished. Results: /tmp/opal-batch