improve tokenisation for Danish

[centre-for-humanities-computing]

improve tokenisation for Danish

[KennethEnevoldsen]

Extracting information from the screenshot these are some of the major issues:

• The tokenization of "efterår/forår" as one token in DaNE (e.g. 84)
• The issues with apostrophes in SpaCy (e.g. 552). This is probably because SpaCy isn't supplied with spacing when supplied the tokens. (supplying it with this information should solve the issue).
  EDIT: Actually it is a spelling error in the text (there is no spacing). What you would actually need here is text normalization (spelling correction) and/or better tokenization (e.g. neural tokenization using a character-based LSTM)
• the issues of punct being split from shorthands in SpaCy (e.g. 477)
• EOS punct not being split from numbers (e.g. 395)

One potential solution might be to use the tokenization lstm from stanza. Which should be accessible using the spacy integration. Would also be interesting to compare these.