maluramichael/node-reddit-scraper

Node.js powered asynchronous scraper for reddit, scrapes imgur and gfycat images.

node-reddit-scraper

a node.js powered asynchronous scraper for reddit and imgur images.

Supports multiple subreddits in the config file. At the present moment only supports images from imgur.

---

Installation

Installation is straight forward

git clone https://github.com/mikedevita/node-reddit-scraper;
cd node-reddit-scraper;
cp config.example.js config.js
npm install;

Imgur v3 api requires a client-ID

Go here https://imgur.com/account/settings/apps and create a new app. Paste the Client-ID and Client-Secret into the config.js.

auth: {
  clientId: 'foo',
  clientSecret: 'bar'
},

Usage

This app is designed to be run via crontab or based on some sort of schedule.

Edit the config.js file to include your subreddits, use the example as a guidelines.

To invoke, just run npm run or node index.js.

You can use this on a crontab like so..

# run every day at midnight
0	0	*	*	*	node /path/to/node-reddit-scraper/index.js

Contributing

pull requests are welcomed, just please adhere to the .eslintrc and .editorconfig's provided. Also run tests and make sure they pass fully.

To run tests just use npm test to execute the tests.

Note Due to how long reddit takes to respond the Reddit unit tests might fail due to a timeout error, if thats the case re run it or skip that single test.