Error using cast_dtm command

[juliasilge]

Error using cast_dtm command

[juliasilge]

I just went through the example with the Associated Press data in the vignette here and it seems to be still working. What happens if you try this, with the example data?

data("AssociatedPress", package = "topicmodels")
ap_td <- tidy(AssociatedPress)
# the output here ap_td should look very very similar to your tibble above
ap_tdm %>% cast_dtm(document, term, count)

Do you get the same error as on your data?

One thought -- do you have the latest version of dplyr? It's possible we should require a later version of dplyr and don't right now.

[dsdesrosiers]

Julia -

Thanks for the response and suggestion.

Your sample code works, my code still fails. I downloaded the latest dplyr... no help. I've done the unlikely dumb stuff like cast the integer column to a double and rename the column 'n' to 'count'. I wrote out the data and there is nothing obviously wrong with it in terms of unexpected characters. I also tested a couple subsets using the words for a single case and I get the same result.

The data I am working with can be a bit unpredictable in that it's free form text entered by our support agents which may contain software error codes. Is there any cleansing steps I should try before I run the unnest_tokens command?

Danielle

[juliasilge]

Could it be the column CaseNumber? When it tries to make a DocumentTermMatrix, it needs a column of documents (case numbers, for you) in integer form (like 1, 2, 3...) to use as matrix indices. What data type and distribution are the values in CaseNumber? Will the function be able to use those values as matrix indices to make a DocumentTerm Matrix?

(Actually, now that I wrote this, I am not sure if it's right. Let me check...)

[juliasilge]

I don't think I am right that you need to pass an integer, but I am wondering if that is where the problem is, in converting the data you have in CaseNumber into row numbers for the matrix indices. (You can see where this is done right about here. What kind of data is being kept in CaseNumber? What type is it? Factor? How is it distributed?

[dsdesrosiers]

So, to expedite this, I just replaced the case numbers (which are literally the number of the case in our CRM) with 1, 2, 3 in the .CSV before reading it into the initial data frame. Same result.

But, then I wrote the data to a .CSV, re-read the same data and now it works. I have to set this aside for now and I don't want to take any more of your time at the moment since I have a workaround. It's very strange though - I am not doing anything particularly novel with the data processing.

Please let me know if you'd like me to supply some sample data/code. Thanks for your time and energy.

[juliasilge]

If you can make a small, reproducible example (i.e. not 40,000 lines) that errors, that would certainly be great. If not, best of luck! I'll let you close this issue if you feel like you are good for now.

[dgrtwo]

The problem is that subject.freq is grouped- I'm able to reproduce the error with:

mtcars %>% group_by(gear) %>% cast_dtm(cyl, am, mpg)

I think cast_dtm should ignore groups- if @juliasilge agrees I'll push a fix. (In the meantime an easier workaround than writing to a CSV and reading would be using ungroup() first).

Thanks for the report!

[juliasilge]

Ah, I didn't think of that! Yes, that sounds very sensible; I think cast_dtm (cast_sparse_?) should ignore groups.

[dsdesrosiers]

I had actually ungrouped and was still getting the issue, but I will try the updated version. If I am still getting the error I will supply some sample data and my R code. I just need identify a set of data that reproduced the error and keeps me in line with my employers data guidelines.

Thanks to both of you for helping with this issue.

[dsdesrosiers]

It worked!!! Happy weekend!

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.