Question about grouping text for pairwise cor and tf idf

[GabriellaS-K]

Question about grouping text for pairwise cor and tf idf

[juliasilge]

When it comes to tf-idf, by definition it applies to documents within a corpus, so you have to think about what you mean by "document" and "corpus" to have tf-idf be a meaningful statistic that is helpful in your analysis. If you are looking at your survey responses and you want to have a sense of the overall most important words (not differences by person) you may just want to look at word frequencies, the most common words used in the survey responses.

If you have survey responses, then that would be the "feature" that you would want to use for pairwise_count() and pairwise_cor(). You wouldn't want to count up or compute the correlation of words that separate survey respondents used. You'll end up with correlations or co-occurrences like this; that analysis was done with exactly this kind of per-survey-respondent counting.

[GabriellaS-K]

Thank you, that is very helpful!!