Use with existing MDS data?

[thekaveman]

Use with existing MDS data?

[morganherlocker]

@thekaveman To run the tooling out of the box, you need to have credentials to an active MDS endpoint (I have a feeling you have these, so feel free to email me for help setting up securely: [email protected]). There is a generic stream interface we use for status_changes and trips, and we have implemented a couple providers, with more on the way soon:

https://github.com/sharedstreets/sharedstreets-mobility-metrics/blob/master/src/providers/

The generic stream interface is subject to change a bit, but the basics are that each provider needs a changes and trips function that takes a stream to write to, a start timestamp, and a stop timestamp. Using this interface has made it easier to smooth over the auth and pagination quirks of each provider, as I've seen you and Hunter run into elsewhere. This project currently runs tests against a live MDS endpoint, which the stream interface has worked well to enable, but it also means only people with a specific set of provider creds can run the tests. This is far from ideal.

I have been working on another project called sharedstreets/trip-simulator, which aims to provide highly realistic synthetic trip data. I'm currently working to add support for MDS compliant data, with synthesized status_change events, in addition to the existing raw GPS. The goal is to be able to safely demo this for any city interested in implementing MDS, before having to get providers & legislators/regulators on board to acquire credentials. This would also allow for running tests against the metrics, guaranteeing that everyone (agencies & providers) is calculating regulatory heuristics with an aligned set of algorithms, regardless of implementation, helping with overall trust of the systems. All that said, trip-simulator essentially outputs giant line-delimitted JSON files, so I'll be digging into this shortly. Feel free to email me if you would like a hand writing a custom provider for the private data you have available for testing.

[morganherlocker]

Yep, something like this would totally work. Line delimitted is used by trip-simulator just to make it easier to generate and ingest very large datasets, but normal JSON works too (and can be streamed with a bit more effort).

Assuming you had two files in your root directory that were simply big arrays of MDS trips and changes, something like this should do the trick:

const trips = require('../../trips.json')
const changes = require('../../changes.json')

function trips(stream, start, stop) {
  trips
  .filter(trip => {
    return trip.start_time >= start && trip.start_time < stop
  })
  .forEach(trip => {
    stream.write(trip)
  })

  stream.end();
}

function changes(stream, start, stop) {
  changes
  .filter(change => {
    return change.event_time >= start && change.event_time < stop
  })
  .forEach(change => {
    stream.write(change)
  })

  stream.end();
}

module.exports.trips = trips;
module.exports.changes = changes;

Update: added time filter, so you can split a big file into daily aggregates

[thekaveman]

All that said, trip-simulator essentially outputs giant line-delimitted JSON files, so I'll be digging into this shortly. Feel free to email me if you would like a hand writing a custom provider for the private data you have available for testing.

Interesting plans indeed. I may take a stab at an implementation for JSON files. You mention line-delimited, what are your thoughts on support for something closer to the raw MDS output format? e.g. for /trips:

{
     "version": "x.y.z",
     "data": {
         "trips": [{
               "provider_id": "...",
               "device_id": "...",
          },
          {
              "provider_id": "...",
               "device_id": "...",
          }]    
     }
}

(or potentially an array of these objects, representing each of the "pages" of an API request)

[morganherlocker]

This is now officially supported. To use a local provider, config a provider with local paths for trips and status_changes, then change the type to "local". Data should be line delimited JSON items in the MDS record level format.

"Synth": {
      "type": "local",
      "trips": "./sample/trips.json",
      "status_changes": "./sample/changes.json",
      "enabled": true
    }