Frog Chunker creates invalid FoLiA

[kosloot]

Frog Chunker creates invalid FoLiA

[kosloot]

OK. This is caused by the fact that Frog can process FoLiA sequential.
The <s> in the input bears the text: "Een Bug? Nee toch?

The tokenizer sees 2 sentences here, which is correct.
The first sentence "Een Bug?" generates 3 <w> nodes, that are appended to the <s>.
Then is is given to the Chunker which creates a a <chunking> layer with one Chunk.

Then the second sentence "Nee toch?" is analyzed. Again 3 words are generated and appended AFTER the Chunking Layer
Then the Chunker generates another 2 Chunks, which are added to the already provided <chunking> layer.
So BEFORE the last 3 words.
But the chunks carry Word references, therefore creating a problem.

Possible solutions:

• allow Frog/Tokenizer to create a <p> and 2 new sentences, replacing the original sentence. This seems nice, but creates a lot of problems. Al least it should be done as a Correction
• Detect this case, and just IGNORE that the tokenizer detected multiple sentences, gather all words in one list to append to the <s>, and also handle this list to the Chunker.
• Allow multiple Chunking layers interleaved with Words inside the Sentence

For now, i assume the second solution to be the "best" and lesser intrusive.

[kosloot]

As far as I can see, the fix, (using the second scenario,) is indeed sound and solves the problem.
Planning a new release RSN