akbindal/reddit-top-2.5-million

This is a dataset of the all-time top 1,000 posts, from the top 2,500 subreddits by subscribers, pulled from reddit between August 15–20, 2013.

Reddit Top 2.5 Million

What is this?

This is a dataset of top posts from reddit. It contains the top 1,000 all-time posts from the top 2,500 subreddits, so 2.5 million posts in total. The top subreddits were determined by subscriber count and are located in the manifest file within.

Ths data was pulled between August 15-20 of August 2013.

Each file is a CSV with the related subreddit as its filename. Each CSV file contains a header line.

A good example of the structure of this data is here.

Why?

I plan to use this for some data analysis I'm putting together for /r/serendipity. You're free to do whatever you like with it.

What use is this?

Who knows! Here's one simple example: This is a breakdown of the top voted domains on Serendipity. (Yes, as a pie chart.)

For my use, I'm considering finding important terms for a subreddit using TF-IDF.

Have other ideas? I'd love to hear what you do with this. Let me know.

Gotchas?

Yes, one: This only includes subreddits that are not marked NSFW. This is related to my particular use case, in which NSFW subreddits aren't useful for me. They'd be relatively easy to scrape if you're interested, you'll just need to get the list of NSFW subreddits from http://www.reddit.com/reddits.