linkedconnections/linked-connections-server

Express based server that exposes Linked Connections.

linked-connections-server's Issues

graph.irail.be/sncb/stops does not have translated names

[http://irail.be/stations] has French, Dutch and English names (if applicaple) for most of the stops.

However, [http://graph.irail.be/sncb/stops] only gives the french names. As the latter one should be the preferred one (the former doesn't have all the stops), it would be useful to include the names as well.

Provide stops using trig and n-quads

Use correct ids to identify trips and vehicles, include direction

Currently a departure contains the following information:

> "@id": "#1510063200000884906488____%3A007%3A%3A8846201%3A8400219%3A3%3A1501%3A20171208",
> "@type": "Connection",
> "departureStop": "http://irail.be/stations/NMBS/008849064",
> "arrivalStop": "http://irail.be/stations/NMBS/008400219",
> "departureTime": "2017-11-07T14:00:00.000Z",
> "arrivalTime": "2017-11-07T14:01:00.000Z",
> "gtfs:trip": "http://irail.be/trips/88____%3A007%3A%3A8846201%3A8400219%3A3%3A1501%3A20171208",
> "gtfs:route": "http://irail.be/routes/327

• @id should be formatted as a departureConnection id: http://irail.be/connections/<station hafas id>/<yyyymmdd>/<train id>, for example http://irail.be/connections/8892007/20171107/IC532. Hafas id is simply the station id without the leading zeros.

• gtfs:trip should be the trip id, which is a train trip on a certain day (correct me if I'm wrong): http://irail.be/vehicle/{VehicleShortName}/{yyyymmdd} where VehicleShortNameis for example IC532

• gtfs:route should identify a route, with an identifier in the format http://irail.be/vehicle/{VehicleShortName}

• A field 'direction' should be included, containing a string which identifies the station name where this train is heading.

Update gtfs2lc

gtfs2lc has updated to 0.8.2. It contains some critical bug fixes. Can you update you linked-connections-server?

Write proper tests for AVL tree serving

Wrong arrivalTime in connection

https://graph.irail.be/sncb/connections?departureTime=2019-01-10T08:16:00.000Z

connection ID: http://irail.be/connections/8896008/20190110/IC409

This connection (Kortrijk - Harelbeke) has the wrong arrivalTime, that of Welkenraedt.

{
@id: "http://irail.be/connections/8896008/20190110/IC409",
@type: "Connection",
departureStop: "http://irail.be/stations/NMBS/008896008",
arrivalStop: "http://irail.be/stations/NMBS/008896115",
departureTime: "2019-01-10T08:17:00.000Z",
arrivalTime: "2019-01-10T11:11:00.000Z",
gtfs:trip: "http://irail.be/vehicle/IC409/20190110",
gtfs:route: "http://irail.be/vehicle/IC409",
direction: "Welkenraedt",
gtfs:pickupType: "gtfs:Regular",
gtfs:dropOffType: "gtfs:Regular",
departureDelay: 0,
arrivalDelay: 0
}

utils.js is not stateless

The utils class which contains a lot of the logic for the web server is not stateless making it hard to reuse the functions. Especially staticFragments gives a thight coupling to the functions making it hard to use a different datastructure for storage of the fragments. Removing this from the state would at least make it clear that the utils functions depend on this object.

Connections travel back in time

As mentioned earlier on slack, some connections travel back in time.

For example, in [https://graph.irail.be/sncb/connections?departureTime=2019-10-31T09:23:00.000Z], the connection with id http://irail.be/connections/8813003/20191031/S11981 seems to depart one minute before it arrived.

Note that this is probably related to incorrect delays.

As reference, the data I got from the server is copied below:

@id	"http://irail.be/connections/8813003/20191031/S11981"
@type	"Connection"
departureStop	"http://irail.be/stations/NMBS/008813003"
arrivalStop	"http://irail.be/stations/NMBS/008813037"
departureTime	"2019-10-31T09:25:00.000Z"
arrivalTime	"2019-10-31T09:24:00.000Z"
departureDelay	420
arrivalDelay	300
direction	"Nivelles"
gtfs:trip	"http://irail.be/vehicle/S11981/20191031"
gtfs:route	"http://irail.be/vehicle/S11981"
gtfs:pickupType	"gtfs:Regular"
gtfs:dropOffType	"gtfs:NotAvailable"

hydra:previous link missing between GTFS versions

The first page of a new processed GTFS source is not providing the corresponding hydra:previous link to the appropriate page in the previous data source.

URIs from GTFS are escaped

When the original GTFS file contains URIs, then those URIs are escaped when the GTFS is converted into Linked Connections.

The following configuration

{
    "companyName": "dsb",
    "keywords": ["Train", "Stops"],
    "geographicArea": "http://sws.geonames.org/2623032",
    "downloadUrl": "./gtfs/DSB.zip",
    "downloadOnLaunch": false,
    "updatePeriod": "0 0 0 0 1 *",
    "fragmentSize": 500000,
    "realTimeData": false,
    "baseURIs": {
    "stop": "{connection.departureStop}",
    "connection": "http://dk.lc.bertmarcelis.be/connections/dsb/{connection.departureTime(YYYYMMDD)}/{connection.$
    "trip": "http://dk.lc.bertmarcelis.be/vehicle/dsb/{trips.trip_id}/{connection.departureTime(YYYYMMDD)}",
    "route": "http://dk.lc.bertmarcelis.be/routes/dsb/{routes.route_id}"
            }

results in

 {
      "@id": "http://dk.lc.bertmarcelis.be/connections/dsb/20181123/http%3A%2F%2Fdk.lc.bertmarcelis.be%2F000008600566/55407198",
      "@type": "Connection",
      "departureStop": "http%3A%2F%2Fdk.lc.bertmarcelis.be%2F000008600566",
      "arrivalStop": "http%3A%2F%2Fdk.lc.bertmarcelis.be%2F000008600567",
      "departureTime": "2018-11-23T18:43:00.000Z",
      "arrivalTime": "2018-11-23T18:44:00.000Z",
      "gtfs:trip": "http://dk.lc.bertmarcelis.be/vehicle/dsb/55407198/20181123",
      "gtfs:route": "http://dk.lc.bertmarcelis.be/routes/dsb/11748_2",
      "direction": "Svendborg St.",
      "gtfs:pickupType": "gtfs:Regular",
      "gtfs:dropOffType": "gtfs:Regular"
    },

When the departureStop should be http://dk.lc.bertmarcelis.be/000008600566.

This is an issue as the GTFS file contains stops from 3 different base URIs, meaning that they have to be defined in the GTFS file. A suggested solution would be to not encode values if they are at the first position in the template (if they are at the position where you'd expect http), or to only encode them if the template starts with http, or to pass the base URIs as a nested object which specifies if encoding should be disabled, e.g. {"format": "{...}", "escape": false} .

Memento and next page strategy

Current state

If you follow next page links, while following, the page may already be out of date. Linked Connections Server needs to have a request header Accept-DateTime added in order to show the right next page for sure.

This means that in some cases, a wrong next page will return.

Suggested solution

Slightly redesign the next page strategy… When requesting a page:

• Redirect to the right ?departureTime=... page
• add a content-location header to the response which indicates the URL of this archived page

Npm start command was deleted which makes the README inconsistent.

The npm start command was deleted, unclear to me why. The README still mentions the start command to start the server.

Storage folder in datasets_config.json cannot be relative.

Just like the title says, it only works with an absolute path as the method preSortGTFS in dataset_manager.js expects an absolute path.

An in-range update of fast-csv is breaking the build 🚨

The dependency fast-csv was updated from 4.0.3 to 4.1.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

fast-csv is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ Travis CI - Branch: The build errored.

---

Commits

The new version differs by 4 commits.

• 682710d v4.1.0
• b9dd314 Merge pull request #327 from C2FO/v4.1.0-rc
• 22e4fb7 Added benchmarks for files of 1000 and 10000
• c0d8f72 Added headers event #321

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

[Low priority] Object can appear twice

On graph.irail.be, the following excerpt can be found in the result for https://graph.irail.be/sncb/connections?departureTime=2018-02-15T13:40:00.000Z

  {
    "@id": "http:\/\/irail.be\/connections\/8821717\/20180215\/IC4314",
    "@type": "Connection",
    "departureStop": "http:\/\/irail.be\/stations\/NMBS\/008821717",
    "arrivalStop": "http:\/\/irail.be\/stations\/NMBS\/008832458",
    "departureTime": "2018-02-15T13:43:00.000Z",
    "arrivalTime": "2018-02-15T13:49:00.000Z",
    "direction": "Hamont",
    "gtfs:trip": "http:\/\/irail.be\/vehicle\/IC4314\/20180215",
    "gtfs:route": "http:\/\/irail.be\/vehicle\/IC4314",
    "gtfs:pickupType": "gtfs:Regular",
    "gtfs:dropOffType": "gtfs:Regular"
  },
  {
    "@id": "http:\/\/irail.be\/connections\/8821717\/20180215\/IC4314",
    "@type": "Connection",
    "departureStop": "http:\/\/irail.be\/stations\/NMBS\/008821717",
    "arrivalStop": "http:\/\/irail.be\/stations\/NMBS\/008832458",
    "departureTime": "2018-02-15T13:43:00.000Z",
    "arrivalTime": "2018-02-15T13:49:00.000Z",
    "direction": "Hasselt",
    "gtfs:trip": "http:\/\/irail.be\/vehicle\/IC4314\/20180215",
    "gtfs:route": "http:\/\/irail.be\/vehicle\/IC4314",
    "gtfs:pickupType": "gtfs:Regular",
    "gtfs:dropOffType": "gtfs:Regular"
  }

Only one of those should be shown as they are identical

Differential LC generation for new static version updates

Make the static LC generation process to store only differential updates with respect to the latest available version (if any) and emit lc:UnscheduledConnections on LDES for removed Connections. Also make sure that overnight trips during the "cutoff date" are properly updated in the new version.

Don't use inconsistent date for folder naming in RT data

The realtime folder structure contains folders with names like '2018_4_12'. These names are hard to generate, especially now we don't use a fixed time interval anymore.
Instead, these folders should be named '2018-04-12', identical to the first part of each file it contains.
In other words, the folder name should be the shared prefix of every file it contains.

Example:
I have the static file 2018-04-07T17:34:00.000Z.jsonld.gz. I want to get the realtime data for this file.

I need to navigate to /real_time/{last_version}/{special_date}/2018-04-07T17:34:00.000Z.jsonld.
last_version is identical to the version field I used in the static file path. This is perfect, I can just reuse the same string.
Howevern, special_date requires me to take a substring of the filename, parse it as a date, print it as another date. By using the prefix of the file, I could just use a 10-digit substring.

Migrate moment.js to date-fns

See https://raygun.com/blog/moment-js-vs-date-fns/

how to use with sparse/query-only data sources?

I want to built Linked Connections wrapping sparse data sources, which I need to query for connections on demand. This means that:

• I need to be able to decide how to fetch the sparse data.
• There is nothing stored as files.
• The Linked Connections server should only let me fetch whatever is needed to answer the client's query.

How would this work with linked-connections-server?

Define a better strategy to expose a catalog for available datasets

context: missing gtfs:Regular and others

The URI definitions for gtfs:Regular and others are missing in the @context. When parsing the JSON-LD, the prefixes are kept as is right now, which results in wrong Linked Data.

Problem with etag header

When giving back the etag, take into account to content-type and return a different etag for every format

Set cache-control header to no cache for 404 responses

Doesn't work on windows

datasets doesn't work on windows as the path contains illegal characters like :.
In order to achieve maximum interopability, it might be a good idea to find a way around this, e.g by using dashes, underscores and dots as separators.

[High prio] Accept valid ISO8601 date stamps

Currently only dates formatted in UTC are accepted. Alternate timezones, which are accepted in ISO8601, should also be accepted.

Complete date plus hours, minutes and seconds:
YYYY-MM-DDThh:mm:ssTZD (eg 1997-07-16T19:20:30+01:00)
Complete date plus hours, minutes, seconds and a decimal fraction of a
second
YYYY-MM-DDThh:mm:ss.sTZD (eg 1997-07-16T19:20:30.45+01:00)

where:

 YYYY = four-digit year
 MM   = two-digit month (01=January, etc.)
 DD   = two-digit day of month (01 through 31)
 hh   = two digits of hour (00 through 23) (am/pm NOT allowed)
 mm   = two digits of minute (00 through 59)
 ss   = two digits of second (00 through 59)
 s    = one or more digits representing a decimal fraction of a second
 TZD  = time zone designator (Z or +hh:mm or -hh:mm)

https://www.w3.org/TR/NOTE-datetime

Currently UTC is a hardcoded requirement, even while the date might parse correctly:
https://github.com/julianrojas87/linked-connections-server/blob/master/src/routes/page-finder.js#L37

Example:
https://graph.irail.be/sncb/connections?departureTime=2018-03-09T12:00:00.000+01:00
won't be accepted and will be rewritten to the actual time
https://graph.irail.be/sncb/connections?departureTime=2018-03-09T18:20:00.000Z

Shape doesn’t have the right URI

https://github.com/linkedconnections/linked-connections-server/blob/development/lib/routes/router.js#L44

This should be configured instead of fetched through the req.protocol, as when hosted behind an nginx that takes care of the https, you will always have http here.

Weird bug in real-time processing of SNCB data

https://graph.irail.be/sncb/connections?departureTime=2018-10-31T07:05:00.000Z
→ in this page, this connection: http://irail.be/connections/8891553/20181031/L558 takes a long time to do a short trip...

This is the actual trip: https://irail.be/vehicle/L558/20181031 (8:06 tot 8:13)

The connection:

"@id": "http://irail.be/connections/8891553/20181031/L558"
"@type": "Connection"
arrivalStop: "http://irail.be/stations/NMBS/008891629"
arrivalTime: "2018-10-31T09:28:00.000Z"
departureStop: "http://irail.be/stations/NMBS/008891553"
departureTime: "2018-10-31T07:06:00.000Z"
direction: "Malines"
"gtfs:dropOffType": "gtfs:Regular"
"gtfs:pickupType": "gtfs:Regular"
"gtfs:route": "http://irail.be/vehicle/L558"
"gtfs:trip": "http://irail.be/vehicle/L558/20181031"

Document how GTFS-RT is processed

Provide a friendly explanation of the process and the logic behind processing GTFS-RT updates.

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.

An in-range update of gtfsrt2lc is breaking the build 🚨

The dependency gtfsrt2lc was updated from 1.2.0 to 1.2.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

gtfsrt2lc is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ Travis CI - Branch: The build failed.

---

Commits

The new version differs by 7 commits.

• eb1a2f7 Merge pull request #37 from linkedconnections/development
• 7cd5160 1.2.1
• 35027ca Fix for date-fns last version syntax
• eaad9e1 Merge pull request #36 from linkedconnections/master
• e8953d7 Merge pull request #35 from linkedconnections/greenkeeper/initial
• 510359a chore(package): update lockfile package-lock.json
• 467fb1d chore(package): update dependencies

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Bundle Linked Connections server using docker

Handle train cancellations from GTFS-RT updates

Fix semantics of Stops and Routes

• Use rdfs:label instead of schema:name.
• Change gtfs:Entrance_Exit to gtfs:EntranceOrExit.
• Remove gtfs:BoardingArea.
• Remove gtfs:route from Connections.
• Publish Trips and Shapes.

GTFS file sorting includes lock file

when calling readdir in getLatestGtfsSource, the lock file is also included. This file doesn't end on .zip, causing the entire name to be interpreted as a Date.

https://github.com/julianrojas87/linked-connections-server/blob/master/lib/utils/utils.js#L493

2018-06-22T12:13:31.613Z - error:  RangeError: Invalid time value
    at Date.toISOString (<anonymous>)
    at Utils.getLatestGtfsSource (linked-connections-server/lib/utils/utils.js:504:125)
    at <anonymous>

This doesn't seem to have a negative impact on the correct working, but should be resolved nonetheless. (Seems like low priority to me)

Fix catalog to handle new Linked Data serializations