gschier/atp-crawler

Simple web crawler to fetch all show titles of ATP podcasts (atp.fm)

ATP.fm Web Crawler

Simple web crawler to fetch all show titles of ATP podcasts (atp.fm)

50 lines of code on top of ~165,000 lines of dependencies (for Casey)

What does it use?

• request (HTTP client)
• domino (server-side DOM)
• zepto-node (JQuery-like library)

How does it work?

1. fetch base URL of atp.fm
2. download HTML
3. build DOM object with Domino
4. select show titles with Zepto
5. select next page link with Zepto
6. repeat with URL of next page

Usage

$ npm install
$ npm start

Todo

• make it fetch more metadata

Author

@GregorySchier - schier.co