andrewts129/transcript-scraping

Script that can scrape the transcripts of every speech a politician has given (as long as it's been recorded on whatthefolly.com)

NOTICE

It appears that whatthefolly.com has shut down as of Feb 2018, so this script will no longer work.

Script that can scrape the transcripts of every speech a politician has given (as long as it's been recorded on whatthefolly.com)

Some assembly required. The script is currently set to scrape Donald Trump's speeches. If you want to get somebody else's transcripts, you'll need to modify the variables 'tagPageUrl' and 'names'. Directions on what to change them to are given in comments in the script.

You will need to have Python 3 with the modules requests and BeautifulSoup4 installed.

Python 3: https://www.python.org/downloads/

Requests: http://docs.python-requests.org/en/master/

BeautifulSoup:https://www.crummy.com/software/BeautifulSoup/

I've also run the script for a few prominent politicans and uploaded the output here. They may be out of date, however.

transcript-scraping's Issues

Speech of Donal trump Jr.

This needs to be removed from the dataset: https://github.com/andrewts129/transcript-scraping/blob/master/Donald%20Trump%20Speeches/Donald%20Trump%20Jr.%E2%80%99s%20speech%20at%20the%20Republican%20National%20Convention.txt