esqew/lirr-penn-station-departure-scraper

A rudimentary Node.js script to read and procedurally retrieve Penn LIRR Station train departure and track data from LIRR's API and store it in an MSSQL database for future use in data science projects.

lirr-traintime-scraper

A rudimentary Node.js script to read and procedurally retrieve train time and track data from LIRR's API and store it in an MSSQL database for future use in data science projects.

Issues

This isn't the greatest code in the world. I'm actively tracking all outstanding issues in the Issues tab. Pull requests are always welcome.

lirr-penn-station-departure-scraper's Issues

Handle potential Promise rejections

Promise rejections are currently unhandled, which can lead to crashes. Wrapping Promises with appropriate try/catch logic will increase the module/script's stability and align with generally-accepted best practices.

This was unfortunately yet another convention deprioritized for the initial MVP but could ultimately end up providing a huge amount of value.

Refactor to modular design

Increase re-usability in separate projects by decoupling database logic from scraper and exporting the scraper's functionality as a Node.js module. This was deprioritized for the initial MVP in favor of ensuring the scraper could be set up to collect continuous data as quickly as possible.

Update README with information on how to use pm2 to run the example script perpetually

Create database/table structure

Included scripts should have the ability to create the appropriate database/table structure to store the data

Update LICENSE to MIT

Add documentation

traintime.scrape.js should include more in-depth documentation to better facilitate future feature additions/improvements/bugfixes. This was deprioritized for the initial MVP but now that the core functionality is not weighing on the timeline as heavily it should be brought back into the backlog.