chocoluffy/redditqa

Explore some interesting NLP experiments with reddit comments data.

1-Predict-Topics

Run TF-IDF and LSI on existing subreddit comments, and given user's new comment, try predicting and recommending subreddit.

Predict Topics

2-PCA-Distribution-Plot

Build document-term matrix from BigQuery data, then run LDA to find topics distribution for each subreddit, and apply t-SNE dimension reduction with matplotlib visualization.

LDA visualization

3-Bipartite-Graph

Construct a bipartite graph between authors and topics, and propagate back and forth the labels to identify generalist/specialist among reddit authors for differnt community.

Bipartite Graph

4-LDA-On-TFIDF

Fine tune the model from week3, with TF-IDF weights applied on BOW matrix but keep in same magnitude.

Improved LDA

5-Model-Inspection

Examine the validity of models obtained from week4, and refine models by tuning hyper-parameters.

Model Inspection

6-Word2Vec

Apply non-semantic techniques(finding overlapping commenters), and semantic techniques(such as LSA, word2vec) to examine similarity between each subreddits.

Word2vec

redditqa's Issues

10.11 meeting

☑️ Use raw tf-idf algorithm to apply on document-term matrix. Look at the subreddit, and check the highest words. Then apply LDA on them. Pick the number of topics interested. Included the week 3's results.
☑️ Associate subreddits with topics using threshold like 10%.
☑️ Pick the top 5% of the heavy-comment user.

10.18 meeting

☑️ check topics, make sure not always those .
☑️ plot per user, generalist and specialist plot. Pick a few extremist.

• plot user to topic counter, above the cutoff.

9.26 meeting

☑️ try different topics numbers, N = 500
☑️ use tf-idf vector instead of bow vector
☑️ print top 10 words for each topic
☑️ examine the bi-graph on user and community, and by propagating the labels, eventually find out the percentage of specialist and generalist for each subreddit community (only examining the most active users.) ; and further, is the comments written by old person or new person
[ ] take log into account in 1. plotting.

10.4 meetiing

☑️ try update BOW matrix to TF-IDF weighted matrix. See if non-integer matrix could work.
☑️ pick more comments for each subreddit, say, 10000.
☑️ use > 10% for topic cutoff, and combine those topic vectors with their weights.

• use subreddit comment frequency to weight the similarity.
• find the most voted comments, and find that if they are written by generalist or specialist. find the distribution for each subreddit community. How to measure the "success" from the perspective of generalist and specialist.
• plots of "fraction of specialist" to the "average comment score", size by the number of comments.