CreateDtm() ignores all non-English words

[tommyjones]

CreateDtm() ignores all non-English words

[dselivanov]

Problem is here.
Before @TommyJones will fix it, you can try to construct DTM directly with text2vec::create_dtm function.

[dselivanov]

@TommyJones probably str_replace_all(doc_vec, "[^[:alnum:]]", " ") will work. See ?base::regex.

stringr::str_replace_all("advad ывмывм 134134 ,..?", "[^[:alnum:]]", " ")

"advad ывмывм 134134 "

[TommyJones]

Thank you for the feedback. As @dselivanov noted, the issue is with the
regular expression underlying the option remove_punctuation. In the short
run, you can set remove_punctuation = FALSE in the call to CreateDtm.
However, I believe Russian still has punctuation marks. So you'd have to
strip them out with a regular expression before a call to CreateDtm. (Note:
you would have to do the same if you call text2vec::create_dtm directly.)

Some example code:

character vector containing a Russian document

my_docs <- scan("https://habrahabr.ru/post/123845/", what="character",
sep="\n")

my_docs <- paste(my_docs, collapse = "\n")

remove punctuation - this regular expression will work with Russian only.

I'll find something more general to patch textmineR
my_docs <- stringr::str_replace_all(my_docs, "[^а-яА-ЯёЁa-zA-Z]", " ")

create DTM

my_dtm <- CreateDtm(my_docs, remove_punctuation = FALSE) # set other
options you may need here

I will keep this issue open until I find a general solution that works for
many languages.

On Wed, Apr 27, 2016 at 6:48 AM Dmitriy Selivanov [email protected]
wrote:

Problem is here
https://github.com/TommyJones/textmineR/blob/master/R/CreateDtm.R#L69-L74
.
Before @TommyJones https://github.com/TommyJones will fix it, you can
try to construct DTM directly with text2vec::create_dtm function.

—
You are receiving this because you were mentioned.

Reply to this email directly or view it on GitHub
#20 (comment)

[paskn]

Thanks for suggestions!
remove_punctuation = FALSE fixed the problem.

[TommyJones]

@dselivanov, I was playing with "_|[^\\w0-9]" but your solution is much more elegant.

[TommyJones]

I've tested this successfully with Russian, Japanese, Korean, Chinese, and English. I'm going to call that good enough for now and close the issue. Please re-open it if you run into a similar issue.