derhuerst/berlin-gtfs-rt-server

Expose Berlin & Brandenburg transit data as a GTFS-RT feed.

berlin-gtfs-rt-server

Poll the VBB HAFAS endpoint to provide a GTFS Realtime (GTFS-RT) feed for Berlin & Brandenburg.

This project uses hafas-client & hafas-gtfs-rt-feed to fetch live data about all vehicles in the Berlin & Brandenburg area and build a live GTFS Realtime (GTFS-RT) feed from them.

Installing & running

Note: hafas-gtfs-rt-feed, the library used by this project for building the GTFS-RT feed, has more extensive docs. For brevity and to avoid duplication (with e.g. hamburg-gtfs-rt-server), the following instructions just cover the basics.

Prerequisites

berlin-gtfs-rt-server needs access to a Redis server, you can configure a custom host/port by setting the REDIS_URL environment variable.

It also needs access to a PostgreSQL 14+ server with PostGIS installed; Pass custom PG* environment variables if you run PostgreSQL in an unusual configuration.

It also needs access to a NATS Streaming server (just follow its installation guide); Set the NATS_STREAMING_URL environment variable if you run it in an unusual configuration.

It also needs the sponge command from the moreutils package to be installed.

The start.sh script requires at least Bash 5.0 to run (because it uses 5.0); macOS currently bundles Bash 3.2, so use brew install bash to install an up-to-date version.

Optional: dynamic local addresses from a range

As an optional feature, when polling the HAFAS API, berlin-gtfs-rt-server uses IP addresses from a pool defined in $RANDOM_LOCAL_ADDRESSES_PREFIX as local addresses. localaddress-agent is used for this, which currently only supports Linux.

localaddress-agent is defined as an optional dependency, so its installation will be attempted, but a failure won't cause the whole npm install command to fail. By setting the $RANDOM_LOCAL_ADDRESSES_PREFIX environment variable, you make its installation mandatory.

Refer to localaddress-agent's installation instructions for its required dependencies.

Cloning

git clone https://github.com/derhuerst/berlin-gtfs-rt-server.git
cd berlin-gtfs-rt-server
npm install

Building the matching index

npm run build

The build script will download the latest VBB GTFS Static data and import it into PostgreSQL. Then, it will add additional lookup tables to match realtime data with GTFS Static data. psql will need to have access to your database.

Running

Specify the bounding box to be observed as JSON:

export BBOX='{"north": 52.52, "west": 13.36, "south": 52.5, "east": 13.39}'

berlin-gtfs-rt-server uses hafas-gtfs-rt-feed underneath, which is split into three parts: polling the HAFAS endpoint (monitor-hafas CLI), matching realtime data (match-with-gtf CLI), and serving a GTFS-RT feed (serve-as-gtfs-rt CLI). You can run all three at once using the start.sh wrapper script:

./start.sh

In production, run all three using a tool that restarts them when they crash, e.g. systemctl, forever or Kubernetes.

via Docker

A Docker image is available as derhuerst/berlin-gtfs-rt-server.

Note: The Docker image does not contain Redis, PostgreSQL & NATS. You need to configure access to them using the environment variables documented above (e.g. NATS_STREAMING_URL).

export BBOX='{"north": 52.52, "west": 13.36, "south": 52.5, "east": 13.39}'
# build the matching index
docker run -e BBOX -i -t --rm derhuerst/berlin-gtfs-rt-server ./build.sh
# run
docker run -e BBOX -i -t --rm derhuerst/berlin-gtfs-rt-server

via docker-compose

The example docker-compose.yml starts up a complete set of containers (berlin-gtfs-rt-server, Redis, PostGIS/PostgreSQL, NATS Streaming) to generate a GTFS-RT feed

Be sure to set POSTGRES_PASSWORD, either via a .env file or an environment variable.

The environment may be started via

POSTGRES_PASSWORD=mySecretPassword docker-compose up -d

After starting, the GTFS-RT feed should be available via http://localhost:3000/.

inspecting the feed

Check out hafas-gtfs-rt-feed's inspecting the feed section.

metrics

Check out hafas-gtfs-rt-feed's metrics section.

License

This project is dual-licensed: My contributions are licensed under the Prosperity Public License, contributions of other people are licensed as Apache 2.0.

This license allows you to use and share this software for noncommercial purposes for free and to try this software for commercial purposes for thirty days.

Personal use for research, experiment, and testing for the benefit of public knowledge, personal study, private entertainment, hobby projects, amateur pursuits, or religious observance, without any anticipated commercial application, doesn’t count as use for a commercial purpose.

Buy a commercial license or read more about why I sell private licenses for my projects.

berlin-gtfs-rt-server's Issues

Observation: unrecognized value "on;" for "ON_ERROR_STOP": Boolean expected

Hi folks,
Have you ever noticed the unrecognized value "on;" for "ON_ERROR_STOP": Boolean expected feedback when calling npm run build? Here is a larger snippet.

...
ALTER TABLE
COPY 53855
COMMIT
+++ realpath ./build.sh
++ dirname /home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/build.sh
+ lib=/home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/lib
+ sponge
+ psql -b
+ NODE_ENV=production
+ node_modules/.bin/build-gtfs-match-index /home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/lib/hafas-info.js /home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/lib/gtfs-info.js
stops
routes
trip headsigns
unrecognized value "on;" for "ON_ERROR_STOP": Boolean expected
BEGIN
CREATE TABLE

COPY 579515
CREATE INDEX
...

Cheers!

correct content-type?

The TriMet TripUpdates GTFS-RT feed uses application/grtfeed:

curl 'https://developer.trimet.org/ws/V1/TripUpdate?appID=…' --head

HTTP/1.1 200 
Date: Tue, 22 Dec 2020 16:35:28 GMT
Content-Type: application/grtfeed
Content-Length: 39500
Access-Control-Allow-Origin: *
Connection: close
Strict-Transport-Security: max-age=31536000

allow importing into a custom DB schema

Edit @derhuerst:
The gist of this thread is here: #9 (comment)
gtfs-via-postgres's ability to import into a custom schema should be exposed, and match-gtfs-rt-to-gtfs should be adapted to support this too. Then, a schema switch can be added in here.

depends on derhuerst/match-gtfs-rt-to-gtfs#8

---

Hi folks,
I am wondering if someone has already successfully replicated this repository using the --schema switch.

I am failing with this feedback.

$ time npm run build

> [email protected] build
> ./build.sh && npm run docs

+ wget --compression auto -r --no-parent --no-directories -R .csv.gz -P gtfs -N https://vbb-gtfs.jannisr.de/latest/
--2024-02-12 13:08:57--  https://vbb-gtfs.jannisr.de/latest/
Resolving vbb-gtfs.jannisr.de (vbb-gtfs.jannisr.de)... 2001:41d0:701:1100::26d, 54.37.75.136
Connecting to vbb-gtfs.jannisr.de (vbb-gtfs.jannisr.de)|2001:41d0:701:1100::26d|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘gtfs/index.html’

index.html                                 [ <=>                                                                         ]     624  --.-KB/s    in 0s      

Last-modified header missing -- time-stamps turned off.
2024-02-12 13:08:57 (6.35 MB/s) - ‘gtfs/index.html’ saved [3222]

Loading robots.txt; please ignore errors.
--2024-02-12 13:08:57--  https://vbb-gtfs.jannisr.de/robots.txt
...

--2024-02-12 13:08:57--  https://vbb-gtfs.jannisr.de/latest/stops.csv
Reusing existing connection to [vbb-gtfs.jannisr.de]:443.
HTTP request sent, awaiting response... 304 Not Modified
File ‘gtfs/stops.csv’ not modified on server. Omitting download.

...

FINISHED --2024-02-12 13:08:57--
Total wall clock time: 0.4s
Downloaded: 1 files, 624 in 0s (6.35 MB/s)
+ env
+ grep '^PG'
...
+ NODE_ENV=production
+ node_modules/.bin/gtfs-to-sql -d --schema vbb --trips-without-shape-id --routes-without-agency-id -- gtfs/agency.csv gtfs/calendar.csv gtfs/calendar_dates.csv gtfs/frequencies.csv gtfs/routes.csv gtfs/stop_times.csv gtfs/stops.csv gtfs/transfers.csv gtfs/trips.csv
+ psql -b
+ sponge
is_valid_lang_code
is_timezone
calendar
  processed 2892 rows
calendar_dates
  processed 97563 rows
service_days
routes
  processed 1271 rows
trips
  processed 241801 rows
frequencies
  processed 0 rows
agency
  processed 33 rows
stops
  processed 41413 rows
stop_times
  processed 5748473 rows
transfers
  processed 53855 rows
NOTICE:  extension "postgis" already exists, skipping
CREATE EXTENSION
CREATE SCHEMA
BEGIN
CREATE FUNCTION
DO
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE TYPE
CREATE TABLE
COPY 2892
CREATE TYPE
CREATE TABLE
COPY 97563
CREATE INDEX
CREATE INDEX
SELECT 269243
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TYPE
CREATE TABLE
COPY 1271
CREATE INDEX
CREATE TYPE
CREATE TYPE
CREATE TABLE
COPY 241801
CREATE TYPE
CREATE TABLE
COPY 0
CREATE INDEX
CREATE INDEX
CREATE TABLE
COPY 33
CREATE TYPE
CREATE TYPE
CREATE TABLE
COPY 41413
ALTER TABLE
CREATE INDEX
CREATE TYPE
CREATE TYPE
CREATE TABLE
COPY 5748473
CREATE INDEX
CREATE INDEX
UPDATE 5748473
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE TYPE
CREATE TABLE
ALTER TABLE
COPY 53855
COMMIT
+++ realpath ./build.sh
++ dirname /home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/build.sh
+ lib=/home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/lib
+ NODE_ENV=production
+ node_modules/.bin/build-gtfs-match-index /home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/lib/hafas-info.js /home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/lib/gtfs-info.js
+ psql -b
+ sponge
stops
error: relation "stops" does not exist
    at Parser.parseErrorMessage (/home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/node_modules/pg-protocol/dist/parser.js:287:98)
    at Parser.handlePacket (/home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/node_modules/pg-protocol/dist/parser.js:126:29)
    at Parser.parse (/home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/node_modules/pg-protocol/dist/parser.js:39:38)
    at Socket.<anonymous> (/home/pacs/sib00/users/gtfsr_srv_vbb/git/berlin-gtfs-rt-server/node_modules/pg-protocol/dist/index.js:11:42)
    at Socket.emit (node:events:517:28)
    at addChunk (node:internal/streams/readable:368:12)
    at readableAddChunk (node:internal/streams/readable:341:9)
    at Readable.push (node:internal/streams/readable:278:10)
    at TCP.onStreamRead (node:internal/stream_base_commons:190:23) {
  length: 105,
  severity: 'ERROR',
  code: '42P01',
  detail: undefined,
  hint: undefined,
  position: '297',
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'parse_relation.c',
  line: '1392',
  routine: 'parserOpenTable'
}
unrecognized value "on;" for "ON_ERROR_STOP": Boolean expected
BEGIN
ERROR:  relation "stops" does not exist
STATEMENT:  CREATE TABLE stops_stable_ids (
	stop_id TEXT NOT NULL,
	FOREIGN KEY (stop_id) REFERENCES stops,
	stable_id TEXT NOT NULL,
	specificity INTEGER NOT NULL
);
ERROR:  current transaction is aborted, commands ignored until end of transaction block
STATEMENT:  COPY stops_stable_ids FROM STDIN csv;

I do not observe this behavior when I omit the --schema switch.

Cheers!

Question: Do I realy need to provide the GTFS Schedule feed at 2 places?

Hi!
I was wondering if I really need to provide the GTFS Schedule feed at 2 places.

In the database:

berlin-gtfs-rt-server/build.sh

Line 9 in 7e9e095

-P gtfs -N 'https://vbb-gtfs.jannisr.de/latest/'

Via URL:

berlin-gtfs-rt-server/start.sh

Line 29 in 7e9e095

--feed-url 'https://vbb-gtfs.jannisr.de/latest/' \

Is there a workaround for GTFS Schedule feeds that ar not provided via URL?

Cheers!

npm run build fails: sponge missing

Hi there,
According to this line I called this instruction

npm run build

, that came back with this reply.

npm run build

> [email protected] build
> ./build.sh && npm run docs

+ grep '^PG'
PGPORT=<secret>
PGPASSWORD=<secret>
PGUSER=<secret>
PGDATABASE=<secret>
PGHOST=<secret>
+ NODE_ENV=production
+ node_modules/.bin/gtfs-to-sql -d --trips-without-shape-id --routes-without-agency-id -- gtfs/agency.csv gtfs/calendar.csv gtfs/calendar_dates.csv gtfs/frequencies.csv gtfs/routes.csv gtfs/stop_times.csv gtfs/stops.csv gtfs/transfers.csv gtfs/trips.csv
+ psql -b
+ sponge
./build.sh: line 24: sponge: command not found
PostGraphile PostgreSQL user's password: <secret>
PostrREST PostgreSQL user's password: <secret>

I am wondering about this

./build.sh: line 24: sponge: command not found

and this

PostGraphile PostgreSQL user's password: <secret>
PostrREST PostgreSQL user's password: <secret>

snippet. Any hint why sponge is not found and the GTFS feed is not imported into PostgreSQL?

Cheers!

Question: Is there a checklist how to adjust this repository to a different GTFS Schedule feed & Hafas client?

Hi!
Thank you so much for providing and maintaining this repository.

I forked this repository to evaluate if it is possible to replicate it for a different GTFS Schedule feed and Hafas client.

I am wondering how a checklist could look like that provide an overview for the task ahead. I found the following places that are specific to https://github.com/derhuerst/berlin-gtfs-rt-server and need an adjustment.

• Provide GTFS Schedule feed to database:

  berlin-gtfs-rt-server/build.sh

  Line 7 in 7e9e095

  wget --compression auto \

  )
• Provide URL to GTFS Schedule feed:

  berlin-gtfs-rt-server/start.sh

  Line 29 in 7e9e095

  --feed-url 'https://vbb-gtfs.jannisr.de/latest/' \

  )

• berlin-gtfs-rt-server/package.json

  Line 48 in 7e9e095

  "normalize-vbb-station-name-for-search": "^1.0.0",

• How/Where do I provide the specific Hafas client?

Whats missing?

Cheers!

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@derhuerst/technical-docs-cli'

Dear you,
When calling npm run build have you ever met this reply?

...
"277889210","westvictorbur grundschule victorbur"
"277889213","zob"
"277889219","zob"
"277889209","suedbrookmerland moorhusen igs"

> [email protected] docs
> ./build-docs.js

node:internal/errors:496
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@derhuerst/technical-docs-cli' imported from /home/pacs/sib00/users/gtfsr_srv_fp/git/fp-gtfs-rt-server/build-docs.js
    at new NodeError (node:internal/errors:405:5)
    at packageResolve (node:internal/modules/esm/resolve:916:9)
    at moduleResolve (node:internal/modules/esm/resolve:973:20)
    at defaultResolve (node:internal/modules/esm/resolve:1193:11)
    at ModuleLoader.defaultResolve (node:internal/modules/esm/loader:403:12)
    at ModuleLoader.resolve (node:internal/modules/esm/loader:372:25)
    at ModuleLoader.getModuleJob (node:internal/modules/esm/loader:249:38)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:39)
    at link (node:internal/modules/esm/module_job:75:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Node.js v18.19.0
...

Cheers!

use station DHIDs from HAFAS for matching

das VBB-HAFAS gibt sogar, in einem extra-feld versteckt, die stations-DHID zurück, also bspw. für 900029371/S+U Rathaus Spandau (Berlin) [Bus Seegef. Str.] die stations-DHID de:11000:900029371, aber nicht die DHID der Haltestelle, also z.b. de:11000:900029371::1.