znmeb/gtfs-collector

A Docker image to collect General Transit Feed Specifcation (GTFS) data

gtfs-collector's Issues

Document hosting processes

Document existing code

Limit 1.0.0 scope to real-time data collector

The only piece of the puzzle that needs to be operating continuously on a server in the cloud is collecting the real-time data as it's published. Everything else can be done on the desktop or another server. This also reduces the space requirement dramatically.

Host-side and container-side tools to load a GTFS dataset into the database

This would have two pieces:

1. A script that runs in the container to run the glfsdb executable.
2. Host-side scripts for Windows, MacOS and Linux to pass the GTFS URL to the container-side script and execute it via "docker exec".

Auto-start the real-time collector

gtfsrdb mysterious Python crashes

Describe the bug
executing gtfsrdb sometimes works and sometimes doesn't. It's obviously an environment issue.

To Reproduce
Steps to reproduce the behavior:
Clone the repo and execute it. The error is something to do with the order in which it creates tables. If it tries to create one with a foreign key before the table with that key has been created, it crashes.

Expected behavior
Collecting data begins.

Additional context
It usually works with the native Python on Arch. But it needs to run in containers.

Explore Docker hosting options

Ideally, there would be something that could work in a "forever free" mode. The biggest hurdle is likely to be disk space; most free tier services are small SSDs.

Refactor into three containers

One for PostGIS, one for gtfsdb and one for `gtfsrdb. This will help isolate the crashes and other bugs.

Tools for managing disk space for the real-time data collection

The intended use case is that the user starts the container collecting data, then accesses it via a PostgreSQL client library from R, Python, Julia, or some other data science language. Since collection is supposed to be continuous (we wouldn't need a server otherwise), there needs to be a way to periodically back up the database and truncate it, or some kind of round-robin scheme with roll-ups, perhaps daily or weekly. Monthly seems too long - weekly feels right to me.

Host filesystem for PostgreSQL data

This is required to persist data in the event of an unplanned container shutdown.

Rename - make TriMet-specific

That way, we can use experimental TriMet feeds

Rebuild using R!!

There's a lot of good transit code out there in R, and it would relieve me of having to troubleshoot other peoples' Python code. And it would make a cool workshop for R and TriMet nerds in Portland. ;-)

Add a pgAdmin container

Not sure it's a necessity given psql, but it's nice to have.

Tools to configure and start the real-time data collection

The intended use case is that a user deploys this container, tells it the URLs for the GTFS real-time feed and any authentication secrets required to connect. Then the container starts collecting data into the PostgreSQL database.

Need a data volume for PostGIS

As of now, if the container dies, we lose the data. So we need a volume.

create a Windows 10 host-side version

Not sure where to prioritize this, but there needs to be a bare metal version at least to create the database populated by gtfsdb. I'm building the Linux version with Conda so it will be Windows-ready.

Memory killer??

On a Digital Ocean mini-droplet with 1 GB of RAM, the gtfsdb (non-real-time) collector crashes with "Killed". It runs on the workstation, and there appears to be a steady-state 2.5 GB requirement and a spike over 5 GB! So the plan for deployment is to just collect real-time data.