Getting content from social media data for analysis can be kind of a nuisance. This project aims to make that collection process as simple as possible, by making some common-sense assumptions about what most researchers need, and how they like to work with their data. For example, tasks like grabbing all the posts and comments from a handful of Facebook pages, and dumping the results into a sqlite database.
Currently only Facebook is supported, but Twitter will follow shortly.
This isn't on pypi yet, so first clone this repo into the directory where you'll be working.
git clone https://github.com/Data4Democracy/collect-social.git
Then install the requirements using pip
.
pip install -r requirements.txt
If you haven't already, make sure to create a Facebook app with your Facebook developer account. This will give you an app id and app secret that you'll use to query Facebook's graph API.
Note that you'll only be able to retrieve content from public pages that allow API access.
You can retrieve posts using Facebook page ids. Note that the page id isn't the same as page name in the URL. For example Justin Beiber's page name is JustinBieber, but the page id is 67253243887
. You can find a page's id by looking at the source HTML at doing a ctrl+f (find in page) for pageid
. Here's a longer explanation.
from collect_social.facebook get_posts
app_id = <YOUR APP ID>
app_secret = <YOUR APP SECRET>
connection_string = 'sqlite:///full-path-to-an-existing-database-file.sqlite'
page_ids = [<page id 1>,<page id 2>]
get_posts.run(app_id,app_secret,connection_string,page_ids)
This will run until it has collected all of the posts from each of the pages in your page_ids
list. It will create post
, page
, and user
tables in the sqlite database created in/opened from the file passed in connection_string
.
This will retrieve all the comments (including threaded replies) for a list of posts. You can optionally provide a max_comments
value, which is helpful if you're grabbing comments from the Facebook page of a public figure, where posts often get tens of thousands of comments.
from collect_social.facebook import get_comments
app_id = <YOUR APP ID>
app_secret = <YOUR APP SECRET>
connection_string = 'sqlite:///full-path-to-an-existing-database-file.sqlite'
post_ids = [<post id 1>,<post id 2>]
get_comments.run(app_id,app_secret,connection_string,post_ids,max_comments=5000)
This will create post
, comment
, and user
tables in the sqlite database created in/opened from the file passed in connection_string
, assuming those tables don't already exist.
Reactions are "likes" and all the other happy/sad/angry/whatever responses that you can add to a Facebook post without actually typing a comment. The reaction author_id
and reaction_type
are saved to an interaction
table in your sqlite database.
from collect_social.facebook import get_reactions
app_id = <YOUR APP ID>
app_secret = <YOUR APP SECRET>
connection_string = 'sqlite:///full-path-to-an-existing-database-file.sqlite'
post_ids = [<post id 1>,<post id 2>]
get_reactions.run(app_id,app_secret,connection_string,post_ids,max_comments=5000)
More social media platforms coming soon. In the meantime, please let me know if there's anything in particular you'd like to see.