tokenize with continue multiple punctions

[tsproisl]

tokenize with continue multiple punctions

[tsproisl]

Wow, I didn't expect SoMaJo to be useful for Malaysian!

Unfortunately, I am not able to reproduce the problem. I get the following output which looks fine:

[['BE', 'kem', 'pertama', 'dalam', 'Bahasa', 'Melayu', '.'], ['350', 'Pax', 'Pemimpin', 'daripada', 'Malaysia', ',', 'Singapura', ',', 'Brunei', 'Dan', 'Indonesia', '!!!'], ['Marilah', 'kita', 'membawa', 'gelombang', '#BEInternational', 'ke', 'Pasaran', 'Melayu', '!!!', '🔥', '🔥', '🔥', '🔥', '🔥']]

How did you run the tokenizer? I tested it like this ("en_PTB" and "de_CMC" give the same results on this input – I don't know which one would be more appropriate for Malaysian):

from somajo import SoMaJo

tokenizer = SoMaJo("en_PTB")
paragraphs = ["BE kem pertama dalam Bahasa Melayu. 350 Pax Pemimpin daripada Malaysia, Singapura, Brunei Dan Indonesia!!! Marilah kita membawa gelombang #BEInternational ke Pasaran Melayu!!!🔥🔥🔥🔥🔥"]
sentences = tokenizer.tokenize_text(paragraphs)
print([[token.text for token in s] for s in sentences])