Tokenizer outputs single characters per line

[tsproisl]

Tokenizer outputs single characters per line

[konstantinmiller]

While this works:

from somajo import SoMaJo
import io
tokenizer = SoMaJo("de_CMC")
sentences = tokenizer.tokenize_text_file(io.StringIO('Dies ist ein Text.'), SoMaJo._default_parsep)
for sentence in sentences:
    for token in sentence:
        print(token.text)

[peteruhrig]

Yes, the behaviour appears a bit odd. The reason is that the tokenize_text function expects a list of paragraphs. So you will have to write
sentences = tokenizer.tokenize_text(['Das ist ein Test.'])
instead.
(Edit: Otherwise the string will be used as a list, which is why every character is separated.)

[konstantinmiller]

Ah, got it! Thanks!

[tsproisl]

To make the intended usage more clear, tokenize_text raises an exception now if it is called with a string instead of an iterable of strings.