How do I split sentences but not words?

[tsproisl]

How do I split sentences but not words?

[tsproisl]

SoMaJo's sentence splitter operates on tokenized input. This means you cannot do sentence splitting without tokenization.

However, you could try to detokenize the output to some extent. This will not completely restore the original input (original whitespace will be normalized to a single space, things like control characters will be removed, etc.), but maybe it is close enough for your purposes:

#!/usr/bin/env python3

from somajo import SoMaJo


def detokenize(tokens):
    out = []
    for token in tokens:
        if token.original_spelling is not None:
            out.append(token.original_spelling)
        else:
            out.append(token.text)
        if token.space_after:
            out.append(" ")
    return "".join(out)


tokenizer = SoMaJo("de_CMC")

paragraphs = ["der beste Betreuer?\n-- ProfSmith! : )",
              "Was machst du morgen Abend?! Lust auf Film?;-)"]

sentences = tokenizer.tokenize_text(paragraphs)
for sentence in sentences:
    print(detokenize(sentence))

This produces the following output where the individual sentences are restored to their original form:

der beste Betreuer? 
-- ProfSmith! : ) 
Was machst du morgen Abend?! 
Lust auf Film?;-) 

[PhilipMay]

Thanks. Closing this again.