Very slow run time

[duydo]

Very slow run time

[Trey314159]

Or—I can upload the compressed corpus here! vi.5K.txt.zip

For anyone else who wants to download and use this corpus, since it is a derivative work from Vietnamese Wikipedia articles, it is licensed CC BY-SA 3.0 by me and contributors to Vietnamese Wikipedia.

[duydo]

Hi @Trey314159,
Thank you for spending time doing analysis with the plugin, I really appreciate what you've contributed to it.
After looking at all the issues you reported and reviewing whole source code of the original tokenizer I decided to stop hacking it, I will implement a new Vietnamese word segmentation from the scratch based on this paper and your suggestions.

By the way, is there somewhere I can download the articles you was using for testing?

[Trey314159]

Hi @duydo,

Glad to help. Sorry the result is abandoning the current tokenizer, but I'm very glad you are going to pursue a new implementation!

I can send you my corpus—it's 5000 Vietnamese Wikipedia articles, with all extra markup removed, and individual lines deduped. It's 3.7MB (1.1MB compressed), so I can email it to you if that's feasible.

[Trey314159]

Hmm. One last comment: I've also removed extra whitespace—extra blank spaces and blank lines, both of which caused problems for the old plugin/tokenizer.

[duydo]

@Trey314159 I've just downloaded the corpus, thanks for sharing. It would be great if you can upload the original corpus without removing extra spaces or blank lines for testing the new tokenizer.

[Trey314159]

@duydo I don't have the original corpus anymore. I just have a script that downloaded the text of 5000 Wikipedia articles.

[duydo]

@Trey314159 Ok, I think the corpus you provided is enough for now, I will crawl more if needed. Thanks.

Btw, I close this issue, it will be fixed in new tokenizer #37