ferrytempo/ftserver

A NodeJS implementation of the FerryTempo service.

FerryTempo Server

This is the NodeJS server providing data for the FerryTempo. It fetches data from the WSDOT Traveler Information API, specifically the ferry vessel location endpoint. Then, it processes that ferry data into the FerryTempo data object format. Finally, it provides HTTP endpoints for fetching ferry data on a per-route basis.

Getting Started

If you'd like to run FTServer locally, follow these instructions:

1. Clone this repo (see clone instructions).
2. Install NodeJS (see install instructions).
3. Run npm i to install the dependencies.
4. Create a .env file in the project root containing WSDOT_API_KEY=<API KEY> (ask a team member for the key).
5. Run npm run dev to start the server in development mode with hot reloading. or
6. Run npm run start to start the server in production mode.

Schema Documentation

We auto-generate schema documentation based on the JSON Schema. That documentation is available in the /docs directory.

Contributing

To help contribute to FTServer, please read and follow the guidelines in Contributing.md.

ftserver's Issues

Provide server version in JSON response

Port ETA is incorrect

It seems that the Port WN and ES ETA are reversed. The Boat ETA is correct

BoatETA not consisent with PortETA.pdf

Port attributes are not always provided

PortES Data is missing. See here is JSONPretty output

PortWN have all the attributes present. But PortES is missing some.
Clients expect a set of constant attributes even if they are set to zero or are null.

{"boatData":{"boat1":{"ArrivingTerminalAbbrev":"P52","ArrivingTerminalName":"Seattle","AtDock":false,"BoatDepartureDelay":0.239,"BoatETA":1710529620,"DepartingTerminalName":"Bainbridge Island","DepartingTerminalAbbrev":"BBI","Direction":"ES","Heading":88,"InService":true,"LeftDock":1710527639,"OnDuty":true,"PositionUpdated":1710529423,"Progress":0.04713265184909391,"ScheduledDeparture":1710527400,"Speed":12.6,"VesselName":"Chimacum","VesselPosition":1},"boat2":{"ArrivingTerminalAbbrev":"P52","ArrivingTerminalName":"Seattle","AtDock":true,"BoatDepartureDelay":0,"BoatETA":0,"DepartingTerminalName":"Bainbridge Island","DepartingTerminalAbbrev":"BBI","Direction":"ES","Heading":280,"InService":true,"LeftDock":0,"OnDuty":true,"PositionUpdated":1710529417,"Progress":0,"ScheduledDeparture":1710530400,"Speed":0,"VesselName":"Tacoma","VesselPosition":2}},"routeID":"5","portData":{"portES":{"TerminalName":"Seattle","TerminalAbbrev":"P52","TerminalID":7},"portWN":{"BoatAtDock":true,"NextScheduledSailing":1710530400,"PortDepartureDelay":0,"PortETA":0,"TerminalName":"Bainbridge Island","TerminalAbbrev":"BBI","TerminalID":3}},"lastUpdate":1710529434}

Provide portData.NextScheduledSailing

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

Provide portData.PortDepartureDelay

Set the DepartureDelay based on the current time vs the scheduled departure time.
If the vessel in dock, set DepartureDelay as difference between scheduled departure and current time IF >0;
If the vessel not in dock, continue reporting last calculated DepartureDelay.

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

WSDOT data in debug table

Implement a view of the WSDOT data used to calculate each response. It should provide the full JSON object of the WSDOT data at the time of request.

This will require storing a history of X requests and responses, where X is determined by some cap. Maybe the Render.com limits?

OnDuty flag being set with terminal name

Need to fix the logic for the setting of the OnDuty flag to result in true/false and not have the terminal name... I didn't see this when running local on my windows machine, but this happens on the server.

It turns out that javascript interprets the true for InService as meeting the value and therefore returns the second value, which is the terminal name in the following code:
const onDuty = AtDock ? InService : (InService && ArrivingTerminalAbbrev);

Modifying logic to test ArrivingTerminalAbbrev for null.

Provide portData.PortArrivalTimeMinus

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

WSDOT API errors in debug table

Implement a view of any errors received when requesting data from the WSDOT upstream API. These errors should appear in the /debug table, indexed by the time they occurred.

This will require storing a history of X errors, where X is determined by some cap. Maybe the Render.com limits?

Error handling for invalid WSF API key

If WSF_API_KEY key is not defined or is invalid, the server code fails in FerryTempo.js with "TypeError: newFerryData is not iterable".

It would be nice to detect that WSF doesn't like the key and log a message to the console.

Export debug table to CSV

To help with debugging, the currently displayed data in the /debug page table should be easily exported to CSV. The exported file should contain all data within the table.

Send time in seconds instead of milliseconds

Figure out how to make sure we update all ports with minimal data

Right now we iterate through the vessel data structure to update each of the boats and then use that data to update the corresponding ports. However, if a route has only a single boat, the ports on that route will not get fully populated. Instead they will just get the data from the one boat, leaving the other data as null (e.g. one port will have an ETA while the other will have a Departure Delay).

This will have an impact as we start to compute values like the average departure delay for a given port. If we do not process that port in our iteration, then we will not have a chance to update that value. We need a way to make sure we touch all ports regardless of whether or not there are two boats on the route.

Error handling for getEpochSecondsFromWSDOT

See https://github.com/FerryTempo/FTServer/blob/main/src/Utils.js#L27

Error handling for OpRouteAbbrev

See https://github.com/FerryTempo/FTServer/blob/main/src/FerryTempo.js#L77

Remove WSDOT API key from repo

Unexpected reporting of "boat at dock" status

See rows 85 - 90

Defect - Boats arriving.xlsx

Validate and correct Direction reporting

"Direction" is defined as "Conveys travel towards one port or the other." If a boat is going from BBI to SEA, the expected direction is ES, as Seattle is the Eastern Port.

Looks like the boat direction being sent is incorrect. The heading is correct though.

FTData in debug table

Implement a view of the response data sent with each request. It should provide the full JSON object that was returned.

This will require storing a history of X requests and responses, where X is determined by some cap. Maybe the Render.com limits?

Create a Logger module that wraps Node.js Winston logging

Create a Logger module to use for standardizing logging across the FT Server application. This will wrap Winston logging for Node.js to make it easier to change logging in the future.

Bug with departure delay

I noticed the WN departure display light was showing when both boats were at dock and neither of them was exceeding the 300 second time. Looking at the port data, it was incorrect compared to the two boat values. See dataset pasted in the issue below.

Fix the OnDuty calculation

Today we calculate if a boat is OnDuty by looking at the InService flag and whether or not the ArrivalTerminal data is null. The problem with this is that there can be a delay in populating these fields whenever a boat arrives at port. So we end up marking these boats as off duty when they are not.

Provide portData.PortSheduleList

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

Provide boatData.VesselPositionNum

The FTData object currently providing VesselPosition, but not VesselPositionNum. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

Update server schema to match "FT Server Variables" sheet

Declan's algorithm for determining progress

Correct number schema types to integers

The only number types in the schema should be:
Progress
Speed (WSF says so)
Heading (WSF says so)

Provide portData.PortLastArrived

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

Add Contribution guidelines document

Docked vessel causes missing data

See https://ferry-tempo.slack.com/archives/C0316RBDM7U/p1703804644643379

Provide boatData.StopTimer

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

Provide portData.PortStopTimer

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

Validate data fields against schema

Prioritize ArrivalTimeMinus fields - needed for client dev.

DEFECT: Mis-spelling "DepartingTermnialAbbrev" key.

Device ID throttling

Based on a discussion on 5/31/24, we want to ensure that no device is pinging the service more than it should. Therefore, we want to identify devices which make calls to our endpoints. Those unique IDs should be used to track time between calls from devices based on their unique identifier.

If a particular device is calling more frequently than it should, we should throttle their calls and/or log a warning.

We may have to have devices provide a unique chip ID, or we could potentially fingerprint the calling device using something like https://www.npmjs.com/package/express-fingerprint.

Provide boatData.ArrivalTimeMinus

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

Debug page - requests table

Implement a table of received requests on the /debug page. It should display a row for each request made to the server, indexed by the timestamp used in the response.

This will require storing a history of X requests and responses, where X is determined by some cap. Maybe the Render.com limits?

Add /debug endpoint and page

Set up the debug page by adding an endpoint to the server at /debug. It can just be an empty HTML page for the purpose of this ticket.

Boat Progress - Wrong Direction

Boat progress seems to be counting down from 1 rather than counting up to 1.

Provide boatData.DepartureDelayAverage

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

The storage manager cache reset is not working

Due to an error in the logic, the cache is not getting properly reset at 3am.

Provide portData.PortDepartureDelayAverage

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

Provide portData.BoatAtDockName

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

Provide boatData.ArrivedDock

The FTData object is missing this attribute. See the FT Server Variables sheet for a description of the expected attributes.

This story will be satisfied once the described attribute exists in the FTData object for any given route.

BoatDepartureDelay isn't calculating if the boat is at the dock

Asynchronous source data fetching

Currently the NodeJS implementation of FTServer fetches data from its source in a reactive manner. It only retrieves updated vessel locations when it receives a request for data.

We need to shift this data fetching to a regular, asynchronous model where the data is constantly being kept up to date. This will ensure that calls from the clients are returned ASAP, without waiting for a call to a data source.

Boat and port data does not align

When two boats on the same route get too close to each other, they can report the same arrival and departure terminal. In this case, the port data is set using the latest of the two boats instead of the correct boat. See attached screen shot of invalid data for the bi-sea route.

lastUpdate is a float

Should be an integer representing the current epoch time in seconds.

Validation of newVesselData with error handling

See https://github.com/FerryTempo/FTServer/blob/main/src/WSDOT.js#L53

Error handling for the case of no upstream data

If our data source(s) go down, we need a way to notify our users/clients.