Add robustness against syntax errors?

[wouterbeek]

Add robustness against syntax errors?

[kurzum]

This is not possible for complex turtle. But if it is a one line per triple format with escaped \n skipping the line and continueing would be nice.
Overall, I needed several weeks now to get the HDT working for DBpedia, i.e. merging several bz2 files into one .gz and compiling the develop branch.

Then I had to rezip everything again to let rapper filter out bad triples, which was just an extra uneccessary step.

[wouterbeek]

@kurzum I can feel your pain having to deal with dirty data! Linked Data Quality is an important topic.

WRT parser fallback strategies, the ClioPatria parser collection has fallback strategies for N-Triples and N-Quads, but also for Turtle and RDF/XML (and maybe for RDFa as well, but I'm not sure). This feature is very important in LOD Laundromat, where the vast majority of datasets on the web turn out to be syntactically malformed.

[drobilla]

Hm. I'm not sure. I sympathise, but this strikes me as a never-ending rabbit hole... that said, I'll look into these heuristics. If something relatively simple to implement works well and isn't problematic I will give it a shot.

[drobilla]

There are already some lax parsing facilities in serd and an existing option for that, so statement-level recovery could latch on to that. Worth noting that anything that requires backtracking can never work in serd (which is strictly a streaming parser), though, so a "perfectly" malformed file could probably cause massive chunks of the file to be skipped. Some condensed real test cases would definitely help with this if you could provide any.

[kurzum]

@wouterbeek can point you to the dirty and clean lodlaundromat files.

Skipping lines until a new well formed statement appears covers 90% of use cases for ntriples, Quads and maybe turtle.

Btw although streaming, keeping a "previous 5 line" window enables some backtracking.
Not saying that it is worth the effort....

[drobilla]

8d954ab will skip to the next line when lax parsing. This works pretty well for the line-based formats (gets me through sketchy dbpedia dumps anyway), definitely will fail horribly in many cases for abbreviated syntaxes, but doing that well will require more work and an actual test suite and so on.

[wouterbeek]

Thank you for implementing the skip functionality for lax parsing mode.