rasbt/datacollect

A collection of tools to collect and download various data.

datacollect

A collection of tools to collect and download various data.

Often, I write simple scripts and tools to collect data for various "data science" tasks. I thought that it might be worthwhile to collect them in a central repository since they might be useful to others!

Contents

• Collect Lyrics
• Twitter Timeline
• Collect Popular Music Tags
• PDB Info Table
• ZINC Molecule Downloader
• Collect English Premier League Soccer Data

Important Note
Please note that I developed and tested these tools in Python 3.x, and it could be possible that the scripts do not work flawlessly in Python 2.7.x due to the more challenging unicode handling.

Collect Lyrics

[back to top]

A command line tool to download song lyrics given artist names and song titles.

Twitter Timeline

[back to top]

A command line tool that downloads your personal twitter timeline in CSV format with optional keyword filter.

Tutorial for turning your twitter timeline into a word cloud.

Collect Popular Music Tags

[back to top]

A command line tool to download popular tags for a list of songs from last.fm, e.g., for various data mining projects.

PDB Info Table

[back to top]

A command line tool that creates an info table from a list of PDB files.

ZINC Molecule Downloader

[back to top]

A command line tool for downloading 3D structures of small chemical molecules from http://zinc.docking.org.

Collect English Premier League Soccer Data

[back to top]

A command line tool to Collect Fantasy Soccer data from the Premier League.

datacollect's Issues

Encoding issue when trying to run soccer collect py file

When i try and run collect_fantasysoccer it threw the following error below

SyntaxError: Non-ASCII character '\xc2' in file collect_fantasysoccer.py on line 415, but no encoding
declared; see http://www.python.org/peps/pep-0263.html for details

I then added the following line to the py file

#coding=UTF-8

Tried to run again, this time it got further but resulted in the following error below

Getting general statistics from https://www.dreamteamfc.com/statistics/players/ALL/ ...
Getting team standings from http://www.espnfc.com/barclays-premier-league/23/table ...
collect_fantasysoccer.py:197: UnicodeWarning: Unicode unequal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
res = [i.text for i in td.next_siblings if isinstance(i, bs4.element.Tag) and i.text!='\xa0']
Traceback (most recent call last):
File "collect_fantasysoccer.py", line 528, in
epl_data.get_all()
File "collect_fantasysoccer.py", line 41, in get_all
self.get_team_standings()
File "collect_fantasysoccer.py", line 199, in get_team_standings
values = [int(i) for i in res[1:]]
ValueError: invalid literal for int() with base 10: ''

Could you advise me as to how to get around this?

Appreciated

Links are not working

EXTREMELY uselful code. But I couldnt get it to work because https://www.dreamteamfc.com/c/statistics/players/ALL/
is outdated.

Links out of date

Shame that a lot of the links are out of date for the fantasy soccer scraper, this looks like a very useful project!

Installed dependencies, getting Syntax Error

$ python collect_fantasysoccer.py -h
File "collect_fantasysoccer.py", line 419
SyntaxError: Non-ASCII character '\xc2' in file collect_fantasysoccer.py on line 419, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details