The scraper is single threaded right now, this is lame

[rogersmark]

The scraper is single threaded right now, this is lame

[rogersmark]

Another note on this... We have to be a bit careful with how crazy we go with threading the scraper. I don't want to alert anyone to my presence if it's avoidable. Throwing everything onto a queue and having 2-4 workers in celery tackling scraping might be okay. Really would like to avoid getting a ban hammer if at all possible.

[dget]

Honestly, since there's not really that many games that need parsing (at least, not all at once), it might be not be so bad if it's just single threaded since it comes with the benefit of lying under the radar.

Another idea is to have an option to cache downloaded stats pages / run the scraper on them locally so you're not making a ton of requests while in dev.

[rogersmark]

Totally agree, it's really low priority. Ideally, I'd like this to be extensible to the point it could cover other leagues (making the name of this project really piss poor). Other leagues would hopefully benefit from a parser that could handle multiple games at a time.

[rogersmark]

You can now use the "-w" or "--workers" args to increase the number of threads. I don't really recommend going over 5, and think I've been throttled at 10 workers before. Anyways, makes crawling an entire past season a bit easier.