Simple CLI app to download Hacker News database using API from https://github.com/HackerNews/API
These instructions will get you a copy of the project up and running on your local machine for development and testing purposes.
-
Node.js
-
A MySQL server
Install the dependencies
npm install
Prepare database schema: Import database.sql
file that came with this repo
( Note: you must create an empty database first then USE
it )
Change database.js
according to your database credential
Update config.js
to suit your preference (optional)
concurrentFetch : 1000
maxRetry : 3
concurrentFetch
number of items to download before insert into databasemaxretry
if a network error occured, how many times it will attempt to retry before giving up
You are all set!
node index.js [largestId]
largestId
(optional) largest ID the script will download then stop, in case you just want to download a part of it.
Leave it blank to download the whole database (takes several hours!)
Warning: [Aug/2017] Whole database is nearly 7GB. Make sure you have enough room for it.
This project is licensed under the MIT License - see the LICENSE.md file for details