osoc17/rideaway-data

The oSoc Brussels Mobility 1 team - Data validation and integration into OpenStreetMap

rideaway-data's Issues

Show last successful update of the map

A timestamp should show when the script last ran successfully and the map was updated.

Rotate log files

The log files should be cleaned after a certain amount of time to prevent the server of running out of disk space. Especially the debug.log file created by the Python processing script.

Finish and test scraper

Finish and test the scraping of the GFR and OSM data.

Preprocess BM data

Preprocess the Brussels Mobility data:

• Map digital colours on analogue colours using this specification: page 9
• Convert tags to correct OSM tags
• Convert coordinate system Lambert 72 to wgs84

Open in JOSM button

Add in an open in JOSM button.

Add link to original route-wide relation-object.

Example:

http://www.openstreetmap.org/relation/7397108

Initiate code

Start the project:

• Add Django website
• Add QGIS Python script
• Add necessary data files

Some ideas to help mappers map the bicycle network

• Create a map when you can choose to display only one road at a time (on top on OSM baselayer)
• Create a page with 2 synced maps (when you zoom/pan on one, it should zoom on the other one as well) to compare data :
  • Map on the left = Brussels bicycle network
  • Map on the right = OSM the data

It's possible with basic HTML/JavaScript so you can use GitHub pages to host it.

copy text

from qgis.core import *
from qgis.utils import iface
from qgis.analysis import QgsGeometryAnalyzer
from qgis.core import QgsMapLayerRegistry
import sys

from PyQt4.QtGui import *
app = QApplication([])
QgsApplication.setPrefixPath("/usr", True)
QgsApplication.initQgis()

Prepare processing framework

sys.path.append('/usr/share/qgis/python/plugins')
from processing.core.Processing import Processing
Processing.initialize()
from processing.tools import *

read location

OSM_INPUT_LOCATION="/home/mramachi/osm/rideaway-data/src/qgis_Scripts/datasources/osm.json"
GFR_INPUT_LOCATION="/home/mramachi/osm/rideaway-data/src/qgis_Scripts/datasources/gfr.geojson"

#output location
OSM_OUTPUT_LOCATION="/home/mramachi/osm/rideaway-data/src/qgis_Scripts/osm.geojson" #not used
GFR_OUTPUT_LOCATION="/home/mramachi/osm/rideaway-data/src/qgis_Scripts/GFR.geojson" #not used
GFRBUFFER_OUTPUT_LOCATION="/home/mramachi/osm/rideaway-data/src/qgis_Scripts/buffered_gfr.shp"

#load GFR data and osm data
osmlayer = QgsVectorLayer(OSM_INPUT_LOCATION,"osmgeojson","ogr")
#gfrlayer = QgsVectorLayer(GFR_INPUT_LOCATION,"gfrgeojson","ogr")

#write output
#writer = QgsVectorFileWriter.writeAsVectorFormat(osmlayer,r"/home/mramachi/osm/rideaway-data/src/qgis_Scripts/osm.shp","utf-8",None,"ESRI Shapefile")
#writer = QgsVectorFileWriter.writeAsVectorFormat(gfrlayer,r"/home/mramachi/osm/rideaway-data/src/qgis_Scripts/gfr.shp","utf-8",None,"ESRI Shapefile")

#read shape files
#osmshape = QgsVectorLayer("/home/mramachi/osm/rideaway-data/src/qgis_Scripts/osm.shp", "osmshape", "ogr")
#gfrshape = QgsVectorLayer("/home/mramachi/osm/rideaway-data/src/qgis_Scripts/gfr.shp", "gfrshape", "ogr")

#buffer
#QgsGeometryAnalyzer().buffer(gfrshape, GFRBUFFER_OUTPUT_LOCATION, 0.0003, False, False, -1)
#gfr_bufferedshape = QgsVectorLayer(GFRBUFFER_OUTPUT_LOCATION, "gfr_buffered", "ogr")

Exit applications

#QgsApplication.exitQgis()
#QApplication.exit()
#1 osm 2 gfr_buffered
#general.runalg('qgis:difference',osmshape,gfr_bufferedshape,False,"/home/mramachi/osm/rideaway-data/src/qgis_Scripts/finaloutput.shp")
#outputLayer2 = QgsVectorLayer("/home/mramachi/osm/rideaway-data/src/qgis_Scripts/finaloutput.shp", "gfrshape", "ogr")

#output as geojson
#QgsVectorFileWriter.writeAsVectorFormat(outputLayer2,'/home/mramachi/osm/rideaway-data/src/qgis_Scripts/output/output.geojson', 'utf-8', None, 'GeoJSON')

Simple false positive handling

A very simple way to add false positives would be to add two folders to this repo:

1. false-positives/osm
2. false-positives/bm

They both contain files like ref.geojson that are just added at the beginning of the pipeline to each corresponding route in the corresponding data source.

Feedback about data issues in BM source data

Collect issue here about BM source data on top of the false positives:

Navigation along the above route will never work because it's only valid in one direction. It should be in the source data that the signs are not pointing the cyclists down in this case.

Creating a page that provides a daily update of conflict in proportion of correct routes

Clearly visualising the progress that is made by the OSM community in correcting map data would allow people to easily observe the positive impact that our tool has made. This can be used by the OSM Belgian community to demonstrate the power of oSoc and vice versa. A graph can be shown that provides the total conflicts as a percentage and then breaks this down per route ID. The graph would feed off from a text file of this one a daily basis. The percentage could be calculated by the distance the total conflicted routes in proportion to the route length. This can also assist the community to prioritise their map editing work per route.

"Brussels Mobility Route" doesn't show errors when clicked

Data about flemish cycling nodes also available now.

http://data.toerismevlaanderen.be/tourist/routes/cycling_node_network_v2 => there is a shapefile there that wasn't there before.

Logging this issue here for future reference/work.

Automate data comparison

Automate the comparison between the Brussels data and the OpenStreetMap data using a Python script and QGIS.

Don't hardcode the route names

The names of the routes are currently hardcoded in a routes file, but it would be better if they're extracted from the reference data.

Controls are hidden by routes list

Map control (+/-) are hidden by the routes list in http://cyclenetworks.osm.be/brumob/

Brussels Data Preprocessing - New field

goal: writing an expression for 'field calculator' that combines 'icr' with 'part'

the field 'route' already exists

1. create a new field called 'route' - [ideas for a new name?]
2. combines the string 'icr' with a lower case 'part'

my simple solution of icr || lower(part) doesn't work with 'part' is NULL. fyi the fields of 'icr' and 'part' are in already in a string format.

new idea: can an expression be written that creates a new field of route. it would better automatic the process. thanks.

Set-up webserver

^ What it says in the title

Clicking output.geojson gets more data than expected.

The output.geojson contains all geometries and not just the selected ones.

Improve deployment of project

• The C# geometrical difference checker should be compiled while deploying instead of putting the compiled version on GitHub
• Install new Python dependencies via a requirements.txt file

Post process output

Post process the output files for the frontend.

• Route missing -> BM data with OSM tags
• Geographical difference (both directions) -> Diff with OSM tags
• Geographical match but wrong/missing tags -> OSM data with invalid tags
• Everything matches -> OSM data with tags

Downloaded files don't have .geojson extension.

Make sure the downloaded files have a geojson extension so tools recogize them.

Put Django in production mode

Django should be placed in production mode to prevent debug information to be shown to the public.

Process OSM data

Process the OSM data to extract the relations of the existing cycling network and output the underlying geographical data to GeoJSON for the automatic processing.

Manually compare data and update wiki

First do another manual comparison of the data and update the OSM wiki.

Also identify issues with the metadata and provide a detailed plan for fixing and completing the OSM data.

Metadata comparison

Make a component that compares the metadata of the BM data and OSM data.

Necessary tags:

• colour
• type = network
• route = bicycle
• name = Gewestelijke Fietsroutes - Itinéraires Cyclables Régionaux
• ref
• operator = Brussel Mobiliteit - Bruxelles Mobilité

Comparison script should be made more robust

Comparison script should be made more robust, either it should only accept geojson files with one geometry or it should merge all geometries into one.

Don't show false positives in output data

Now the false positives are also shown in the Brussels data or OSM data depending on the type of false positive. It would be better if the false positives are only used in the difference comparison.

Depends on #42.

Better server configuration

• Apache should be set up as a reverse proxy for Jenkins
• Allow browsers to cache files
  • This will also require a better handling of static files, e.g. minimization

Domain + HTTPS

Get a domain name for the website and add HTTPS

Integrate all scripts

Integrate the scraper, OSM processor and QGIS scripts

Edit here

Open favourite editor for the current mapview.