juliasilge / tidytext Goto Github PK
View Code? Open in Web Editor NEWText mining using tidy tools :sparkles::page_facing_up::sparkles:
Home Page: https://juliasilge.github.io/tidytext/
License: Other
Text mining using tidy tools :sparkles::page_facing_up::sparkles:
Home Page: https://juliasilge.github.io/tidytext/
License: Other
As quanteda will soon re-implement the innards of a corpus class object, the methods in tidy. corpus are likely to break soon. The code should be reimplemented using the stable accessor functions.
(tidytext is an awesome package, thanks for that!)
When purrr 0.1.0 is installed, tidytext loads fine, but tidy.Corpus
does not work, since purr::transpose
is not found. It was (see here) introduced only with version 0.1.1 of purrr -- so this should go into the DESCRIPTION of tidytext, I think.
I've started developing the widyr package (the docs and examples there are still rather primitive) that handles the common pattern of
tidydata -acast->
matrix -some operation->
matrix -melt->
tidydata
One notable example of this pattern (what got me onto the topic) is pair_count. I'd thus like to deprecate tidytext's pair_count
and move it there (and in the new package name it pairwise_count
just to avoid any future name clash). I think there's a variety of applications and I think it would be easier to experiment and develop them out of the way of text mining applications, especially since tidytext is on CRAN and has active users.
Does this sound reasonable?
Is this intended behaviour? I note that tidyr::unnest also destroys attributes, though tidyr::nest preserves them (#272 at tidyr). Makes it harder to use unnest_tokens with subclasses of data.frame, since you then have to take care to save the old attributes before calling unnest_tokens, and then restore them afterwards.
Repro:
library(data.table)
library(tidytext)
d <- data.table(a = c("hello world", "goodbye world"))
unnest_tokens(d, word, a)
Error:
Error in `[[<-.data.frame`(`*tmp*`, output_col, value = c("hello", "world", :
replacement has 4 rows, data has 0
... with the existing implementation as the default method.
This would allow alternative implementations, dispatched based on an additional item in the class() vector, while preserving use of the tidytext grammar.
I realize a common pattern is to look for frequencies that words cooccur in a document. Right now the best way to do that is to create a document-term matrix then multiply it with its own transpose, and even that has some gotchas that can catch you.
I realized there is a dplyr-like verb that can describe this, such that you could do books %>% cooccur(chapter, word)
and get out a tidy dataset.
Ideally I'd like to make it work in the group_by() syntax but haven't gotten that to work yet, so here's some preliminary work that I'll continue on later
#' Count pairs of items that cooccur within a group
cooccur_ <- function(d, group_col, item_col, sort = FALSE) {
requireNamespace("Matrix")
sparse_mat <- cast_sparse_(d, group_col, item_col, value_col = 1)
cooccur_mat <- t(sparse_mat) %*% sparse_mat
ret <- tidy(cooccur_mat) %>%
rename(value1 = row, value2 = column, count = value) %>%
tbl_df()
if (sort) {
ret <- arrange(ret, desc(count))
}
ret
}
mtcars %>% cooccur_("am", "carb")
Let's write tidiers for the topic models from stm. This package has no rJava dependency and the same input as the topicmodels LDA modeling.
bind_tf_idf()
iterates over the id
column when calculating TF. However, it gets confused if id
is numeric. This is due to how the underlying code for bind_tf_idf_()
performs iteration in the following piece of code:
[...]
tbl$tf <- tbl[[n_col]]/as.numeric(doc_totals[documents])
[...]
I solved it by by simply doing as.character()
on documents
before passing it as a selector to doc_totals
, like so:
tbl$tf <- tbl[[n_col]]/as.numeric(doc_totals[as.character(documents)])
But I'm sure there are more elegant solutions.
It should perhaps be mentioned that I discovered this by sampling from a bigger dataset where the id
column was indeed numeric. This is a very common thing in the databases I'm working with (SQL based).
Minimal reprex:
# Corpus data has an id column where max(id) > nrow(my_corpus)
my_corpus <- data_frame(
id = rep(c(2, 3), each = 3),
word = c("an", "interesting", "text", "a", "boring", "text"),
n = c(1, 1, 3, 1, 2, 1)
)
# This results in NA values in the TF column for the second document, with id == 3
my_corpus %>% bind_tf_idf(word, id, n)
Also, thanks for a great package! It has greatly reduced by work load for a number of text analysis tasks.
The deprecation here isn't really deprecation, since that's defined as:
While a deprecated software feature remains in the software, its use may raise warning messages recommending alternative practices; deprecated status may also indicate the feature will be removed in the future. Features are deprecated rather than immediately removed, to provide backward compatibility and give programmers time to bring affected code into compliance with the new standard.
pairwise_count
just refuses to be used; it's in effect entirely removed. Could I suggest either using .Deprecated
and allowing it to continue running, going for .Defunct
instead, or straight-up removing it?
Hi, thanks for developing this helpful package!
unnest_tokens seems to be using tokenizers from "tokenizers" package but how about making it possible to use a tokenizer users choose?
I'm from Japan and tokenizer for Japanese is totally different, so I want to use a custom tokenizer.
OK, the XML package is a "suggests" dependency of quanteda and it is not getting built on Travis currently for the tidying_casting.Rmd vignette
https://travis-ci.org/juliasilge/tidytext/builds/124871786
Here is XML on CRAN.
I think this has the answer of what we need to change, but I haven't tried yet.
Do we want to implement a tidy interface to a stemming algorithm? I am aware of the one in SnowballC, which we are already importing via dependency.
I suggest to add a function to bind BM25 score (which is based on a probabilistic term weighting model). It is useful in some cases as it gives control over:
It is commonly used as a ranking function by search engines.
I implemented a function bind_bm25
in the forked repo HERE
# bind_bm25 is given bare names -------------------
bind_bm25 <- function(tbl, term_col, document_col, n_col, k = 1.2, b = 1) {
bind_bm25_(tbl,
col_name(substitute(term_col)),
col_name(substitute(document_col)),
col_name(substitute(n_col)),
k = k,
b = b)
}
# bind_bm25_ is given strings -------------------------
bind_bm25_ <- function(tbl, term_col, document_col, n_col, k = 1.2, b = 1) {
terms <- tbl[[term_col]]
documents <- tbl[[document_col]]
n <- tbl[[n_col]]
doc_totals <- tapply(n, documents, sum)
avg_dl <- mean(doc_totals)
idf <- log(length(doc_totals) / table(terms))
tbl$tf_bm25 <- ((k+1)*n)/(n+(k*((1-b)+b*(as.numeric(doc_totals[documents])/avg_dl))))
tbl$idf <- as.numeric(idf[terms])
tbl$bm25 <- tbl$tf_bm25 * tbl$idf
tbl
}
I am working through your "Text Mining with R" and I am getting a 'bug' with the Loughran and McDonald sentiment lexicon in the "get_sentiments" function. Seems like the "loughran" lexicon is not recognized as an option in the get_sentiments function. I see you appear to be working on the matter, but I can't determine if you have 'solved' the issue. How can I incorporate the Loughran and McDonald lexicon in the get_sentiments function?
Thank you for writing such a cool tool and book! Just excellent!
Currently the naming overlaps with tidyr::unnest()
, which takes unwraps list columns. By contrast tidytext::unnest_tokens()
acts on a character vector column. For example, tokenise()
would avoid this collision of names / semantics, though at the cost of losing the sense that multiple rows are being produced for each input text
field. tokenise_to_rows
or tokens_to_rows
addresses this, but doesn't feel very elegant. I'm afraid I don't have the solution. But there must be a name that captures both that sense, without overlapping with tidyr::unnest()
?
Used tidytext to parse support case subjects into words...then grouped by case number and word to count,. Ended up with a tibble that looks like this sample
A tibble: 38,923 x 3
CaseNumber word n
1 20695703 backup 1
2 20695703 catalogs 1
When I try to execute subject.freq %>% cast_dtm(CaseNumber, word, n) I get this error
Error in as.lazy_dots(.dots) :
argument ".dots" is missing, with no default
I downloaded the latest dev version of tidytext but that does not alleviate the issue. Any help is appreciated.
On Twitter as @NevilleLeoniers
Hello,
First of all, many thanks for this wonderful package that really helps processing text data better and more efficiently!
It seems I am unable to load the loughran
lexicon. Packages are up to date.
What is the problem here?
library(tidytext)
library(tidyverse)
test = data_frame('text' = c('this is a nice test yay'))
# WORKS
test %>% unnest_tokens(word, text) %>%
anti_join(stop_words, by = "word") %>%
inner_join(get_sentiments('bing'), by = "word")
# A tibble: 2 × 2
word sentiment
<chr> <chr>
1 nice positive
2 yay positive
# FAILS
test %>% unnest_tokens(word, text) %>%
anti_join(stop_words, by = "word") %>%
inner_join(get_sentiments('loughran'), by = "word")
Error in match.arg(lexicon) :
'arg' should be one of “afinn”, “bing”, “nrc”
The sentence tokenizer isn't consistent in breaking the text into different sentences. From what I see, it sometimes fails to break the sentence at "periods(.)"
In unnest_tokens, we have method = "words", "sentences", etc. But really this should describe the unit we are tokenizing on. Maybe unit =, or something similar, is a better argument name?
(Sorry this was posted early, keyboard mishap)
Hi Julia!
I totally love your package and way to handle text data. Im working with topic models and was thinking of trying to start with a tidytopic package for topic models using the same format. Is tidytext stabile enough to build upon, you would say? Do you have any suggestions on API to enable additional packages?
You can reach me on mons.magnusson at gmail dot com if you want to discuss more in details.
I'm just thinking through what needs to be done before the next release to CRAN, submitting to JOSS, etc and this is one of those things. It looks awesome so far.
Would the addition of datasets of sentiment qualifiers like negations and adverbs be of interest to add in?
e.g.
negations
-------
cannot
could not
did not
does not
had no
have no
may not
never
no
not
nothing
was no
was not
will not
would not
ain't
aint
can't
cant
couldn't
didn't
doesn't
don't
dont
hasn't
haven't
won't
Compiled from datasets available from Sentiment Composition Lexicons
tokenizers
is about to change in a couple of ways with the next release:
n_min
field (which currently causes all sorts of interesting warnings when unnest_tokens
is used.I suspect we should sync up with @lmullen on release dates and conditions to make sure that we push out the next tidytext version at roundabouts the same time as the next tokenizers version, to avoid wailing and gnashing of teeth. Thoughts?
First of all I want to thank you and David Robinson. tidytext and Text Mining in R are absolutely brilliant! I have a feature suggestion, which may or may not be within the scope of the package.
I've been working with the apartment rental listing data from this Kaggle competition. It contains text tags for each apartment listing. I've been using tidytext to analyse, transform and cast to DTMs.
My goal is to predict the interest level of a listing given its feature tags. The cast_
functions have been tremendous for creating a set of training data for training a model, but I have a challenge. The training data produces an n x m DTM - so there are m terms in the set. I train a model with this matrix, but the test data won't necessarily have the same m terms. I could include both in the dataset for casting and then separate them, but then I have to recast every time I have a new listing to predict.
It would be great to constrain the set of m terms that get cast to the DTM (kind of like the opposite of stop words). Then I could do something like:
allowed_terms <- c("bright", "hardwood", "cosy")
dtm <- get_some_text() %>%
unnest_tokens(term, text) %>%
count(document, term) %>%
cast_sparse(document, term, n, allowed_terms = allowed_terms)
This would mean any new observation with a set of features could be standardised as input into the model (for example if a new observation contained none of the allowed terms, then it would be cast to a 1 x m matrix of zeroes).
I would be happy to work on this feature and make a pull request if you thought it was a reasonable feature to be included. Thanks again, your work is amazing!
Cheers,
DJ
If either tm or quanteda is loaded after tidytext, their stopwords
function can replace our dataset. In any case it means the same name appears across three text-mining packages.
Should we rename the stopwords dataset to stop_words to avoid this?
This is definitely not a bug, but I've run into a situation where I split up a document into characters using unnest_tokens
, but now I need to know whether any given character is at the beginning of a word or not. The easiest way to find out would be if I were able to ask the tokenizer to preserve spaces between characters and then I could check if lag(token)
is a space. Is that a possibility?
Have you thought about putting together a list of commonly done operations to clean text for analysis within tidytext? I know this is much more of a data cleansing task, but I think having some resources laid out such as packages like qdap, textclean, stringi/stringr can be helpful when prepping text for analysis purposes. The data below is wine reviews from here: http://snap.stanford.edu/data/cellartracker.txt.gz, which has almost 2 million wine reviews. It is relatively large, but might even be worth using as an example within you book. Consider this example of wine reviews from a raw txt file, and removing html tags, html unicode, and key value pairs of data:
wine/name: 2001 Le Clos du Caillou Côtes du Rhône Villages Tres Vieilles Vignes Reserve
wine/wineId: 3688
wine/variant: Red Rhone Blend
wine/year: 2001
review/points: 93
review/time: 1096243200
review/userId: 1
review/userName: Eric
review/text: This wine continues to wow me. The nose is sweet and herbal with garrigue pouring from the glass along with a laser-focused core of brambly raspberry and some duskier scents of licorice and, from the right glass, some roasted meat. On the palate this is impeccably balanced, pure and powerful. The wine seems to have closed up a tiny bit from prior tastings, yet it juggles well between a core of sweet fruit and some nice elements of minerality. There is a lush, silky, enveloping texture that just washes over the palate in a very inviting way. The wine finishes powerfully with some hints of sour cherry and lovely, dusty, tooth-coating tannins. Delicious!
I also decided to try an experiment to see which Riedel glass format would work best for the wine. I took out four glasses and filled each glass roughly 1/6th of the way up. (My wife wondered who else was coming over for dinner...) Generally speaking, the most significant difference was on the nose, but I was surprised at how much some of the differences actually carried onto the palate. It seems like that initial sip is heavily flavored by what you smell as you tip the glass, far more than I ever imagined:
- Riedel Vinum Chianti/Zin: Excellent all-purpose glass. The nose was nicely balanced with the raspberry really coming out and nice waves of garrigue.
- Riedel Vinum Syrah: Hands down the best glass. It was like a bigger, more intense, more focused version of the Chianti glass. The garrigue really comes forward, but the big, tall bowl and narrow nose seems to amplify the bouquet quite a bit. Also, complex elements of smoke and roasted meat come to life. Wow, this is almost like a different wine.
- Riedel Vinum Bordeaux: This one was puzzling. For some reason the nose on this glass was the least focused and most diffuse. I could smell all the same elements as in the prior two glasses, but the wine literally seemed to wander. It didn't know what it wanted to be. I had to work to smell the same things, and even then it was like I had to convince myself of what I was smelling. The one word summary is diffuse.
- Riedel Vinum Burgundy: This is the one glass that brings forward a sharp, almost painful note of alcohol. I really did not enjoy the wine from this glass.
I actually found this experiment a bit disturbing. I like my Riedel's a lot, but I never really expected the marketing hype to be as true to reality as I saw tonight. I have a bad feeling that in the future I will have more glassware to wash, as frankly I will likely try the same wine from a few glasses. It is highly educational!
DAY 2: Still going strong. Smooth as velvet, rich, deep and integrated. Excellent!
As much as I would love to start utilizing your package to analyze reviews like these, there is definitely more than one way to parse out the html tags, and convert unicode characters to make this file readable. However, it has been quite a challenge to find resources on tidying data like this for analysis work with tidytext.
Maybe adding a chapter to the tidytext mining book regarding text cleaning would be the best approach? Even character representation such as #244; can mess up the encoding of the file you want to read in, which gives file encoding errors in r.It might be worthwhile to have a brief section regarding file encoding, and how that impacts text data for analysis. Not to sound harsh, but this is simply my feedback. I look forward to working with your package for my analysis!
Of course this can be achieved with tidyr::nest(). Proposing a wrapper to allow tidytext users to operate within the grammar of the package.
Hi, I was using tidytext for a project where I was trying to make a text prediction algorithm. This involves tokenizing text into n-grams, and I noticed a difference between the results using tidytext vs using the tm package. Basically it seems that if you have multiple lines of text, tm computes n-grams for each line separately, whereas tidytext computes n-grams for all the lines pasted together (so you get n-grams that span lines). I'm not sure if there is a 'correct' method . If your text is liked the tidytext examples from gutenbergr, it makes sense because each line of text appears in order in the book. But in my case, each line was a different article or tweet for example, so an n-gram spanning lines wouldn't make sense (because those words never actually occur together in that order). But I thought it might be worth pointing out in a vignette or something since people might assume you get the same results as with tm. I made a little example of this in the attached file.
From #22 :
Include multilanguage options for stopwords. This is a useful thing with or without other internationalisation changes. So this would consist of:
stop_langs
to select languagesstop_languages()
that returns a set of which languages are supported by stop_words
.I will probably work on this over the next few days.
So the dictionary tidier is now broken. I'm not sure where the change happened (it's either in quantedata
or broom
- I suspect quantedata) but input values must be named. This means that, starting from the unit test's position of 'a list of named elements':
tidy
then rejects it.@dgrtwo any idea? I've got this patch I'm reworking for Andy and this is messing with it some :/.
require(tidytext)
ap_td<-tidy(AssociatedPress)
ap_td
ap_sentiments<-ap_td %>%inner_join(get_sentiments("afinn"), by=c(term="word"))
I see this:
Error in is.data.frame(y) : could not find function "get_sentiments"
Before the new release, I've been concerned about something. I use code like this a lot in my analyses, often as the only setup needed before a long text-mining pipe:
AFINN <- sentiments %>%
filter(lexicon == "AFINN") %>%
select(word, score)
When we were originally creating the sentiments data frame we wanted to combine them all in one tidy structure, in case someone wanted to do a cross-lexicon analysis. That's still a plausible use case, but overwhelmingly often I choose just one lexicon to use in an analysis. I think we should support that case more easily
I think we should either, in rough descending order of my preference:
get_sentiments
that supports get_sentiments("AFINN")
, get_sentiments("bing")
, etc that pre-perform thoseget_sentiments_afinn()
, get_sentiments_bing()
, etcI think it's important to get these out in this week's CRAN release, and not hard to write (I'm happy to do it)
install_github("juliasilge/tidytext")
Downloading GitHub repo juliasilge/tidytext@master
from URL https://api.github.com/repos/juliasilge/tidytext/zipball/master
Installing tidytext
--2016-11-15 12:43:46-- https://cran.rstudio.com/src/contrib/tokenizers_0.1.4.tar.gz
Resolving cran.rstudio.com... 52.84.129.209
Connecting to cran.rstudio.com|52.84.129.209|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50453 (49K) [application/x-gzip]
Saving to: “/tmp/RtmpbgvQJi/tokenizers_0.1.4.tar.gz”
0K .......... .......... .......... .......... ......... 100% 1.23M=0.04s
2016-11-15 12:43:46 (1.23 MB/s) - “/tmp/RtmpbgvQJi/tokenizers_0.1.4.tar.gz” saved [50453/50453]
Installing tokenizers
'/usr/lib64/R/bin/R' --no-site-file --no-environ --no-save --no-restore
--quiet CMD INSTALL '/tmp/RtmpbgvQJi/devtoolsb60b281702c/tokenizers'
--library='/home/R/x86_64-redhat-linux-gnu-library/3.2'
--install-tests
sessionInfo()
R version 3.2.3 (2015-12-10)
Platform: x86_64-redhat-linux-gnu (64-bit)
Running under: CentOS release 6.8 (Final)
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] devtools_1.12.0
loaded via a namespace (and not attached):
[1] httr_1.2.1 R6_2.2.0 tools_3.2.3 withr_1.0.2 curl_2.2
[6] memoise_1.0.0 knitr_1.11 git2r_0.15.0 digest_0.6.10
Thanks!
The chapters don't line up with the text for the last bit of the main vignette. Text needs to be edited.
I added codecov
today and things look pretty decent. If we wanted to get a bit higher on test coverage eventually, it would be good to write some more tests for sparse_casters.R
and some (one?) for dictionary_tidiers.R
.
Here's the link showing details:
https://codecov.io/gh/juliasilge/tidytext
A quick bug report:
In unnest_tokens
, to_lower = FALSE
is insufficient for token = 'ngrams'
and 'skip_ngrams'
without specifying the accompanying parameter (lowercase = FALSE
) for the function called from tokenizers
Reprex:
library(tidytext)
df <- data.frame(text = "Here's some sample text. Hello world!")
# all lowercase
unnest_tokens(df, ngram, text, token = 'ngrams', to_lower = FALSE)
#> # A tibble: 4 × 1
#> ngram
#> <chr>
#> 1 here's some sample
#> 2 some sample text
#> 3 sample text hello
#> 4 text hello world
# keeps case
unnest_tokens(df, ngram, text, token = 'ngrams', to_lower = FALSE, lowercase = FALSE)
#> # A tibble: 4 × 1
#> ngram
#> <chr>
#> 1 Here's some sample
#> 2 some sample text
#> 3 sample text Hello
#> 4 text Hello world
# all lowercase
unnest_tokens(df, ngram, text, token = 'ngrams', to_lower = FALSE)
#> # A tibble: 4 × 1
#> ngram
#> <chr>
#> 1 here's some sample
#> 2 some sample text
#> 3 sample text hello
#> 4 text hello world
# keeps case
unnest_tokens(df, ngram, text, token = 'skip_ngrams', to_lower = FALSE, lowercase = FALSE)
#> # A tibble: 6 × 1
#> ngram
#> <chr>
#> 1 Here's sample Hello
#> 2 some text world
#> 3 Here's some sample
#> 4 some sample text
#> 5 sample text Hello
#> 6 text Hello world
With lowercase
but without to_lower
is all lowercase, as well.
The other options for token
besides "ngram"
and "skip_ngram"
are ok (well, 'words'
and 'characters'
are mysteriously erroring out, but that's a different issue).
To reproduce,
library(tidytext)
.tidytext::get_sentiments()
.The problem is that sentiments
doesn't get loaded if tidytext isn't on the search path.
I think the fix is to add
data(list = "sentiments", package = "tidytext")
somewhere near the start of get_sentiments()
.
You might also need to specify the enivr
argument to data()
.
Are non-english languages in scope for tidytext?
I just discovered tidytext (looks great!), and was trying to follow along with the code from the readme with my own dataset, which is in dutch. stop_words
currently only has english words, so I had to improvise with stopwords()
from the tm package.
If you're interested in supporting other languages, would you want to add a language field to stop_words
, or might it be a better idea to depend on packages like tm that already have stopword lists, and generate the dataframe on the fly?
We need an example and something in a vignette somewhere on how to tokenize using regex. Specifically I was asked how to get out @-mentions and hashtags from twitter data.
Tf-idf (term frequency-inverse document frequency) can be calculated for a one-term-per-document-per-row tidy dataset using a few group_by and mutate statements, but we should add this as one or more verbs since it's a common operation.
One way I could imagine it would be a function d %>% tf_idf(term, document, n)
that adds three columns:
tf
for term frequency (n / total in doc)idf
for inverse document frequency (log(n_distinct(document) / # of docs with term))tf_idf
for the product of the aboveThere are variations that we'd have to explore and add options for
Let's combine and expand the vignettes and blog posts into a paper.md to submit to JOSS.
(Do I need to get an ORCID ID?)
e.g. using
if (requireNamespace("tm", quietly = TRUE))
I think it would be fairly easy to add support for the lsa package to tidytext and broom. See example below.
# Put some docs in a vector
library("dplyr")
doc1 <- c("pets dog cat ferret")
doc2 <- c("sandwiches turkey ham")
doc3 <- c("cat ferret cat bird")
doc4 <- c("turkey beef sandwiches")
myvector <- c(doc1,doc2,doc3,doc4)
mydf <- data_frame(id = 1:4, text = myvector)
# Create a corpus
library("quanteda")
mycorpus <- corpus(mydf, text_field = "text")
mytokens <- tokens(mycorpus)
mydfm <- dfm(mytokens)
# Perform LSA
mytdm <- convert(mydfm, to = "lsa")
mytdm_weighted = lw_logtf(mytdm) * gw_idf(mytdm)
myLSAspace = lsa(mytdm_weighted, dims=2)
# Here's how broom::augment could add
# factor scores back to the original data frame
factor_scores <- as_data_frame(myLSAspace$dk)
(augmented <- bind_cols(mydf, factor_scores))
# Here's how tidytext:tidy could tidy the factor loadings
library("tidyverse")
# as.data.frame is used to maintain row names until
# rownames_to_column can get them
loadings_tidy <- as.data.frame(myLSAspace$tk) %>%
rownames_to_column() %>%
rename(term = rowname) %>%
gather(factor, loading, # The new variables.
starts_with("V"), # These go into "loading".
-term) %>% # term is not "gathered".
arrange(factor, desc(loading)) %>% # Sort
select(factor, term, loading) # Change var order to enhance readablity.
print(loadings_tidy)
These are in tokenizers but we haven't tested them out or anything. I have had one question from a user (potential user?) about this so far.
Thanks a lot for the package and the book, really enjoying them.
My recommendation here does not require any new features or coding, but simply demonstrating a powerful way to use unnest_tokens()
in a special case, when documents in a corpus come each with a certain metric.
A simple example to illustrate my point. Assume we have a report for a set of keywords (the documents) with the impressions each generated:
library(tibble)
news_keywords <- data_frame(
keyword = c("us news", "us elections", "china news", "china updates", "latest china news", "china newspaper"),
impressions = c(500, 500, 100, 100, 100, 100)
)
news_keywords
#> A tibble: 6 × 2
#> keyword impressions
#> <chr> <dbl>
#> 1 us news 500
#> 2 us elections 500
#> 3 china news 100
#> 4 china updates 100
#> 5 latest china news 100
#> 6 china newspaper 100
Now we separate the words with tidytext::unnest_tokens()
library(tidytext)
library(magrittr)
news_words <- news_keywords %>%
unnest_tokens(output = words, input = keyword)
Counting the frequency
library(dplyr)
news_words %>% count(words, sort = TRUE)
#> A tibble: 7 × 2
#> words n
#> <chr> <int>
#> 1 china 4
#> 2 news 3
#> 3 us 2
#> 4 elections 1
#> 5 latest 1
#> 6 newspaper 1
#> 7 updates 1
We can see that the most frequently used words are "china", "news", then "us".
Now let's take a look at the weighted frequency
news_words %>%
group_by(words) %>%
summarise(impressions = sum(impressions)) %>%
arrange(desc(impressions))
#> A tibble: 7 × 2
#> words impressions
#> <chr> <dbl>
#> 1 us 1000
#> 2 news 700
#> 3 elections 500
#> 4 china 400
#> 5 latest 100
#> 6 newspaper 100
#> 7 updates 100
Here, we see a completely different view, where "us" is the most frequent (by number of impressions) and china is in fourth position.
While "china" occurs more often in the keywords, there are more times that people search for keywords that include "us", and therefore "us" is bigger in terms of searches.
Another simple example: let's say the most frequent word you use in your tweets is "dplyr", but when you take into consideration the number of impressions of your tweets, you might find that tweets that generated the most impressions where the ones with "ggplot" in them.
Text | Metrics |
---|---|
search engine keywords | impressions, clicks, conversions |
tweet text | impressions, engagements, retweets |
web page titles | pageviews, avg. session duration |
song* / movie / book title | downloads, sales, likes |
If you think this is useful, I'd be more than happy to expand, and share more detailed examples.
'* might work with your post https://www.washingtonpost.com/news/wonk/wp/2016/10/01/the-states-that-americans-sing-about-most/?postshare=5471475343908445&tid=ss_tw
Let's write a tidier for the topic models from mallet along the lines of the LDA tidier for topicmodels.
I have 4 columns in a tibble.
1st is the Date, then the other 3 are <chr>
type.
When I try to create ngrams by specifying the actual column of "Text" that I am interested in, it always selects a different column. I have tried rearranging the column positions but it still doesn't work.
I ended up having to select only the Date and the Text column in order to create ngrams.
Here is the final code:
consumer_ngrams <- consumer_small %>% select(Date, Text)%>% mutate(feedback_no = row_number())%>% unnest_tokens(ngram, Text, token = "ngrams", n = 2)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.