allow importing into a custom DB schema

[dancesWithCycles]

allow importing into a custom DB schema

[dancesWithCycles]

This is odd. When I am using the repository straight like this

npm i
./cli.js --schema <schema> --trips-without-shape-id --routes-without-agency-id -- gtfs/*.csv | sponge | psql -b -h <host> -p <port> -U <user> -d <database> > log.txt 2>&1

the schema switch works perfectly fine.

[dancesWithCycles]

I might have found the cause of my struggle. The GTFS Schedule feed is successfully imported as you can see from this log file.

less log-gtfs-to-sql.txt:

NOTICE:  extension "postgis" already exists, skipping
CREATE EXTENSION
CREATE SCHEMA
BEGIN
CREATE FUNCTION
DO
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE TYPE
CREATE TABLE
COPY 2892
CREATE TYPE
CREATE TABLE
COPY 97563
CREATE INDEX
CREATE INDEX
SELECT 269243
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE TYPE
CREATE TABLE
COPY 1271
CREATE INDEX
CREATE TYPE
CREATE TYPE
CREATE TABLE
COPY 241801
CREATE TYPE
CREATE TABLE
COPY 0
CREATE INDEX
CREATE INDEX
CREATE TABLE
COPY 33
CREATE TABLE
COPY 201
CREATE TYPE
CREATE TYPE
CREATE TABLE
COPY 41413
ALTER TABLE
CREATE INDEX
CREATE TYPE
CREATE TABLE
COPY 119239
CREATE TABLE
COPY 5282696
CREATE INDEX
CREATE INDEX
CREATE VIEW
CREATE TYPE
CREATE TYPE
CREATE TABLE
COPY 5748473
CREATE INDEX
CREATE INDEX
UPDATE 5748473
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE INDEX
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE VIEW
CREATE FUNCTION
CREATE TYPE
CREATE TABLE
ALTER TABLE
COPY 53855
COMMIT

However, the node_modules/.bin/build-gtfs-match-index does not find the imported feed as it is not looking in the respective schema.

less log-build-gtfs-match-index.txt:

unrecognized value "on;" for "ON_ERROR_STOP": Boolean expected
BEGIN
ERROR:  relation "stops" does not exist
STATEMENT:  CREATE TABLE stops_stable_ids (
        stop_id TEXT NOT NULL,
        FOREIGN KEY (stop_id) REFERENCES stops,
        stable_id TEXT NOT NULL,
        specificity INTEGER NOT NULL
);
ERROR:  current transaction is aborted, commands ignored until end of transaction block
STATEMENT:  COPY stops_stable_ids FROM STDIN csv;

Am I right or am I right? ;-)

[derhuerst]

The GTFS Schedule feed is successfully imported […]. […] However, the node_modules/.bin/build-gtfs-match-index does not find the imported feed as it is not looking in the respective schema.

Yes, I think that's correct!

build-gtfs-match-index comes from hafas-gtfs-rt-feed, which just calls match-gtfs-rt-to-gtfs's build-index, which currently doesn't support configuring a schema.

[derhuerst]

I'm wondering though: What's your use case for using a schema other than public? berlin-gtfs-rt-server (or rather hafas-gtfs-rt-feed, because berlin-gtfs-rt-server is just a thin shell around it) currently assumes that the PostgreSQL DB is exclusively used by it.

[dancesWithCycles]

My hosting is my use case!

I am using a Managed Server where I do not want to drop and create a database every time I update the GTFS feed. I rather drop and create the respective schema.

An alternative approach would be a script that cleans up an existing database without dropping it so that the update happens on a clean database.

At the end of the day I need to make sure to prepare a fresh environment for the GTFS feed import into PostgreSQL without dropping the database. How would you do it?

[derhuerst]

At the end of the day I need to make sure to prepare a fresh environment for the GTFS feed import into PostgreSQL without dropping the database. How would you do it?

I have created public-transport/gtfs-via-postgres#57 to discuss this further, because this topic is not specific to berlin-gtfs-rt-server, and because there have been similar discussions in the gtfs-via-postgres Issues before.

[derhuerst]

FYI: I'll rename this Issue to be a feature request.