bliksemlabs/bliksemintegration

bliksemintegration's Issues

extra data after last expected column

Hey guys,

I am trying to use the import since a few hours and neatly got it to work. Requirements.txt-contents are installed by pip and I adjusted a few URLs to point to the correct KV1-zips. After that I tried running the import.py with python and received the following error:
Traceback (most recent call last): File "/var/www/html/OVbusser/gtfs/bliksemlabs/bliksemintegration/importers/syntus.py", line 124, in sync download(url+link,filename) File "/var/www/html/OVbusser/gtfs/bliksemlabs/bliksemintegration/importers/syntus.py", line 109, in download import_zip('/tmp',filename,None) File "/var/www/html/OVbusser/gtfs/bliksemlabs/bliksemintegration/importers/syntus.py", line 55, in import_zip meta,conn = load(path,filename) File "/var/www/html/OVbusser/gtfs/bliksemlabs/bliksemintegration/importers/kv1_810.py", line 433, in load meta = importzip(conn,zip) File "/var/www/html/OVbusser/gtfs/bliksemlabs/bliksemintegration/importers/kv1_805.py", line 582, in importzip cur.copy_expert("COPY %s FROM STDIN WITH DELIMITER AS '|' NULL AS '' CSV ENCODING '%s'" % (table,encoding),f) DataError: extra data after last expected column CONTEXT: COPY dest, line 1: "DEST|1|I|SYNTUS|0|UNKNOWN|UNKNOWN|UNKNOWN|FALSE|UNKNOWN|UNKNOWN|UNKNOWN|UNKNOWN|UNKNOWN|UNKNOWN||"

Is there perhaps something outdated or did I miss anything else with running this?

Thank you. ;)

(Note: Yes, this issue was about something completely different, but I managed to fix that.)

add avv to import.py

Converting data from some format to GTFS

Do I correctly understand that the importer scripts are not converting data from, for example, DINO, to GTFS, but rather do some sort of operations directly on a database?

Do you have simply the tools to convert DINO to GTFS?
Thank you.

*_pool_geom and quays relations do not exist

I'm following the instructions at https://github.com/plannerstack/docs/blob/master/bliksem/Introduction.md
ARR, CXX and GVB import works for me, RET import gives an error:

relation "ret_pool_geom" does not exist

When looking at the source files, I can't find something that creates this relation. Same for the quays relation when trying to run the ns-import.py script:

psycopg2.ProgrammingError: relation "quays" does not exist

add ns to import.py

parse connexxion date from filename

Document how to use all the scripts

Document:

• Required privileges
• Expected params, it's meaning and optional values.

For all the python files containing an script. E.g: import.py

import-openov.py script

add an import-openov script

Initial import takes absurdly long

I started the import operation yesterday and after about 10 hours it has processed 2 KV1 files from Arriva. Is that normal for the first time?