hasadna/opentraincommunity

Opentrain repository for community data analysis of Israel Railways train delay data.

Online Project Documentation

Interactive documentation is available at http://opentraincommunity.readthedocs.org/en/latest/

Requirements

• Python 3.6
• Postgres (dumps postgres 10)

Our server (otrain.org) runs with ubuntu 18.04, postgres 10 and python 3.6.7

Developer Environment Setup

The project runs on Python 3 with POSTGRES, and we recommend using a virtual environment for development. You can use the following command to setup a virtualenv called opentrain:

mkvirtualenv -p $(which python3) opentrain

You might need to first install the virtualenv package, via:

sudo pip install virtualenv

Then activate the environment:

cd opentrain
source bin/activate

Then install all of the required modules into your virtualenv using pip:

cd OpenTrainCommunity/train2
pip install -r requirements.txt

Note: Installation of numpy via pip might require fulfilling certain prerequisites.

sudo apt-get install postgresql libpq-dev

Database setup and import

Best way to start is to download the POSTGRES dump file.

1. Download the latest sql dump from http://otrain.org/files/dumps/ (you can download the gz file)

python restore.py <name-of-sql-file>

Note that the script will gunzip the file. You might need to run,

pip install argparse

In case you have already set a postgres user name, you could face:

ERROR:  role "postgres" does not exist

In this case, create a user name postgres before running the script.

On mac platform you might also have to change l29 in clean_all.py to

postgres_cmd = "sudo -u your_macuser psql"

Alternativley, you can use fab and run the following commands:

fab create_db
fab restore_db:<path-to-sql-gz-file>

Now you can start the server.

python manage.py runserver 

Troubleshooting

• If you receive the error fatal error: xlocale.h: No such file or directory, it is likely due to a glibc header name change. One suggested solution is to manually create a symlink to the new name:

  ln -s /usr/include/locale.h /usr/include/xlocale.h
  

• If you receive the error symbol __res_maybe_init version GLIBC_PRIVATE not defined in libc.so.6 with link time reference, it is likely due to a psycopg bug and can likely be solved by upgrading psycopg2 to a later version like 2.7.3.1.

Files

• Excel files from Israeli railway: http://otrain.org/files/2017/, http://otrain.org/files/2018/
• Database dump: http://otrain.org/files/dumps/ (look for latest timestamp)

opentraincommunity's Issues

Provide way to treat two sub stations (different gtfs_id) as a single station

Example

1220: [u'לב המפרץ', u'חיפה לב המפרץ'], #  Leb Hmifratz
1230: [u'מרכזית מפרץ-עמק'], #  Merkazit Hmifratz - Emek (Lev Hmifratz)

These are in fact the same station although they have a different gtfs_id.
This causes a distortion of the data

Expected

Have a way to give alternative gtfs id's for a station.

Handle case where trip is not found in gtfs

Create Admin Models

Using Django's Admins to be able to view info in the browsers.

Different background colors

When initially loading the homepage we see a light blue background

Then when the page loads, it becomes dark blue:

Then when the route is being chosen, the background becomes light blue again

Seems like it should be more consistent...

Chatbor Visualization page

Add new UI page which displays:

1. Total cancellations per time period (today: since 00:00 today, past week, past month, total)
2. Chart displaying routes
3. Chart displaying hours of cancellations
4. Table with details of each report + any images or metadata

Clarify chatbot handles reports for current day (today) only

Chatbot text asks when is the cancelled train, but doesn't clarify it only handles report for the current day.

Secret Key In settings.py

https://github.com/hasadna/OpenTrainCommunity/blob/master/simple/train/train/settings.py

Not good for production. Will create PR

otrain.org currently only uses http, instead of https

In general there's a preference for sites to use HTTPS , though since there's no private user data this is not a pressing matter. Example: https://oknesset.org , https://kikar.org/

https://security.stackexchange.com/questions/142496/which-security-measures-make-sense-for-a-static-web-site

Analysis: scrape # of cancelled trpis from rail.co.il and compare to our data

http://diuk.myrail.co.il/?rel=topnav

Fix conditions when session is done

1. Mark session as complete when last message is sent
2. Terminate session after 2 hours

hasadna.redashapp.com no longer works

Documentation at
Accessing the data using redash (most up to date version of docs)

You can share the results with others.
Pretty cool tool!
http://hasadna.redashapp.com

hasadna.redashapp.com doesn't work.

Handle failure to download new GTFS data

1. GTFS data should be for several days ahead (currently only today is requested from partridge)
2. Chatbot should use yesterday's gtfs data if today's hasn't downloaded

This is a bigger change

Cannot report cancellation after 00:00 and before GTFS data is downloaded

Chatbot reports page: improve on mobile

Improve the reports page to look better on mobile (maybe hide some fields or display a bit differently)
https://next.otrain.org/#/reports

Front page drop down should only include destination stations to which routes exist

Example

1.Select בית שאן
2. Drop down includes all possible destinations, although the list of actual destinations to which routes exist is much shorted

Breadcrumb leads to a blank page

Reproduction steps:

1. Select source and destination stations.
2. Select from and to dates.
3. Click „הצג מידע”.
4. After the data will appear click on „מתחנת מוצא לתחנת יעד - חודש ושנה” at the top of the page.

The page turns blank at this step and I need to click Back in order to restore the page.

Thanks.

Web application uses #! even though it's deprecated

Visiting http://otrain.org redirects to http://otrain.org/#!/ (also, all of the breadcrumb links point to #!, etc.). Usage of #! is deprecated:
https://en.wikipedia.org/wiki/Fragment_identifier#Proposals
https://developers.google.com/web/progressive-web-apps/checklist
https://developers.google.com/webmasters/ajax-crawling/docs/specification?csw=1 (notice it's deprecated)

Bug: fix need to deserialize and serialize step data due to date format

Requirements page contains a duplicate entry

train2/requirements.txt has duplicate entries of

django-cors-headers==2.0.0

thereby yielding this error

Double requirement given: django-cors-headers==2.0.0 (from -r requirements.txt (line 16)) (already in django-cors-headers==2.0.0 (from -r requirements.txt (line 10)), name='django-cors-headers')

cannot proceed from here : (

Add "can we share your name?" message at the end

If yes, store the person's name or psid (verify it is not transient)

rest_framework_swagger's API doc generation can't handle query params

OpenTrainCommunity/train2/train2/urls.py

Line 23 in e6bfe77

schema_view = get_swagger_view(title='Pastebin API')

rest_framework_swagger generates docs at:
http://otrain.org/api/docs

API endpoints that require query parameters (as opposed to URL-supplied parameters) do not have those parameters generated in the documentation. Pressing the Try it out! button, for instance, will return a 500 error for:

http://otrain.org/api/docs/#!/general/general_goto_sleep
http://otrain.org/api/docs/#!/routes/routes_all_by_date
http://otrain.org/api/docs/#!/stats/stats_from_to_full
http://otrain.org/api/docs/#!/stats/stats_path_info_full
http://otrain.org/api/docs/#!/stats/stats_route_info
http://otrain.org/api/docs/#!/stops/stops_from_to
http://otrain.org/api/docs/#!/trips/trips_list_compact

This is a known issue with this package ever since YAML was deprecated. The various workarounds mentioned on the internet don't seem to work with list_route's defined on ViewSets, which is how the functions are defined. Also, the rest_framework_swagger package is not being maintained.

Some related discussions:
marcgibbons/django-rest-swagger#549
marcgibbons/django-rest-swagger#629
https://stackoverflow.com/questions/41104615/how-can-i-specify-the-parameter-for-post-requests-while-using-apiview-with-djang
https://stackoverflow.com/questions/28203070/how-to-show-query-parameter-options-in-django-rest-framework-swagger

redashapp link - broken

OpenTrainCommunity/docs/accessing_the_data_using_redash.rst says:

Pretty cool tool! Check it out:
http://hasadna.redashapp.com

but the link no longer works

UI: Remove "יציאה מהתחנה" columns

The delay with which the train left the station isn't interesting and can be removed.
The remaining three columns can be united into a single one, displaying ontime arrival in green, and delays in red. Early arrival to the station can be considered ontime arrival.

See here for screenshot: https://prnt.sc/kcv66z

Using Vagrant Ubuntu cannot import database

Couldn't get to run on first try. Will come up with more detailed issue. Will get directions from scratch.

When running command
python clean_all.py --restore BLAH

Unable to know if a station was skipped

@ekeydar We said previously that we are unable to know if a train station was skipped during a trip. Why is this so? Can't we simply compare it to the published train schedule? For instance, the schedule says train #29 is supposed to go on Monday from Bat Galim 10:09 to Hof HaCarmel 10:16 to Tel Aviv Univ 11:00. If train #29 doesn't arrive at Hof HaCarmel and later arrives at Tel Aviv Univ, then we can safely say that Hof HaCarmel was skipped and include this fact in our statistics.

The biggest problem I can see with this solution is station closures. Once in a while, a station is closed due to construction or rail improvements, in which case it would be difficult to distinguish these two cases. Is there a way to also request a formatted table of station closures, dates of closures, etc.? (Perhaps we can use a simple heuristic -- if > 10 train arrivals at station X are skipped, then we can assume the station is simply closed, instead of a single train needing to 'catch up' on its delayed route).

Even having a general statistic "this station did not provide service to train users X% of the time it was supposed to provide service (either due to construction or due to stops being skipped)" would be a very helpful statistic, even if we can't provide the exact reason.

Top 10 "route" delays this month

1. Choose 10 pairs of source and destination stations, manually as a first step.
2. Calculate delay for each route / timeslot / day permutation
3. Select top 10 routes with largest number of delays (delay > 5 mins)
4. Click on a route will display the matching route summary page and allow selecting a more specific route.

Add chatbot info page and link to it from first message

Create a page that explains what the chatbot does and what information is collected.
Add a link to it in the first message:

"אני בוט שאוסף דיווחים על רכבות שבוטלו"
"(קרא עוד כאן...)"

Save reported cancelled train in database

1. Save all information about the trip - all stops and times
2. Save reported stop
3. Save user info
4. Pointer to the chat session

Hide options after choosing one in quick replies

Example:
איזו מאילו?
ת"א השלום
ת"א מרכז
ת"א אוניברסיטה
..
User picks: ת"א מרכז
All quick replies should be hidded

Crossfilter: graphs should have the same dimensions on axis

Each delay graph has its own dimension, which prevents visual comparison between the max. delay and average delay graphs (for example).

The maximal dimension of the three graphs should be picked and used. This should be updated each time the filter is changed.

API is down

I afraid that there is a pretty long downtime.

1. No updates in Telegram since March 16.
2. Telegram bot is not responding to messages.
3. Website loads but fails to retrieve any data.

Add "מזכרת בתיה" GTFS code to database

Incorrect behavior with months containing two digits

The TimeParser.parseMonth function incorrectly handles inputs where the first digit of the month is not a leading zero (i.e. 10/11/12). As a result, any searches for those months are incorrectly shown to the user.

Steps to reproduce:

• From the homepage search for any origin and destination with either October, November, or December as the month.
• For example:
  • מוצא: ראשון לציון משה דיין
  • יעד: כפר סבא
  • December 2013
• Search and select any of the resulting routes
• Note that the displayed info is from February 2013 instead of December 2013 (example URL).

Due to the current parseMonth function, October 2015 becomes December 2014, November 2015 becomes January 2015, and December 2015 becomes February 2015.

Chabot reports page: change title and icon

https://next.otrain.org/#/reports
Title is: מדד הדיוק

Avoid sending 500 Internal Server Error on chatbot unexpected exceptions

facebook's messenger "punishes" the chatbot by delaying responses whenever a 500 Internal Server Error is received.

Handle all chatbot exceptions gracefully with an error message - log them and report them by slack.

Jupyter notebooks no longer accessible

https://github.com/hasadna/OpenTrainCommunity/blame/abe632211d4f903e85beac7f3d6944493d311f72/simple/train/data/analysis/data_analysis_readme.md#L17

references find_interesting_urls.ipynb, but this no longer exists. Here's the most updated version of the docs.

was deleted in
commit a56b04ce4a1f2916533fbd2aa7cb7ec343f2c637
Author: Ofer Bartal [email protected]
Date: Mon Nov 6 19:18:00 2017

Way to restart the session

Maybe send a termination message like "ביי" or "סוף"
Maybe add a permanent button

Trip ID gtfs shows wrong date

Example: Cancel report from 29/7
Trip ID is: 28918627_280719

Specific cancel report page: no back button to summary page

Route links containing query parameters (day, time) aren't taken into account

Reproduce:

Link below should open the route for Sunday 7:00-9:00, instead the filters are init to All
http://otrain.org/#/201805/select-route/2800/4600?day=1&time=7-9

fix names in admin to be short

Fix stops name in admin, to be shorter