jbinto/bikeways

An exploration of the City of Toronto Bikeways open data set, using Rails & PostGIS

bikeways's Issues

Bikeways data doesn't line up with Google Maps

Everything is a perfect 10 or-so-metres west on the map. Consider using the WGS84 data instead of the local "MTM" (SRID 2019) data..

See http://staging.416.bike/bikeway_segments/1715
Or screenshot:

bikeway_segment entries have fragile IDs, can't rely on for assocations

As a result of #5, where we've been re-importing the Open Data dump over and over again, our IDs are incrementing. Previous links no longer work.

We should use a gem like friendly_id or sluggi to make slugs, so at least the URLs stable.

We could use the city_geo_id, or a hash of the geometry itself as a "stable" ID.

Disable scrollwheel on google maps

It's trivial to disable the scroll wheel in google maps.

https://stackoverflow.com/questions/2330197/how-to-disable-mouse-scroll-wheel-scaling-with-google-maps-api

This would allow us to put maps near the top.

Load Google Maps asynchronously

I ripped out turbolinks in an attempt to get things going quickly, but it was a hack around the fact that I'm loading the Google Maps scripts synchronously.

There's many advantages to loading the script async. (n.b. perhaps the following is an unrelated tangent, but:) We should be able to dynamically tell the map container to reload its KML without having to reload the page.

References:

• https://developers.google.com/maps/documentation/javascript/examples/map-simple-async
• https://stackoverflow.com/questions/16340529/loading-google-maps-asynchronously
• http://jsfiddle.net/doktormolle/7cu2F/

KML should use colors as per official city map

Cache KML output at all levels

Object cache of KML output. Make sure e-tags is used. Compress as much as possible.

Projections, or "Plate Carree considered harmful"

A little background:

The city supplies ESRI shapefiles in the following formats:

• Bikeways (MTM 3 Degree Zone 10, NAD27)
• Bikeways (WGS84 - Latitude / Longitude)

The former is SRID 2019 (Toronto/S. Ont. specific), the latter SRID 4326 (Plate Carree projection).

When using the SRID 2019 data and using PostGIS st_transform to convert to WGS84/4326, we get "reasonable" results but the misalignment to Google Maps is unacceptable (see #1).

When using the SRID 4326 data directly as a geometry (as opposed to a geography), we have fairly good matching with Google Maps. Some routes appear a little off, but closer examination with the satellite view proves the city's data more accurate than Google's. (n.b. this is an n=1 conclusion, further research needed)

However, the third option we haven't tried yet is storing lat/lng natively in a geography. According to Daniel Azuma on his GeoRails series:

You might be tempted to store latitude and longitude in a geometry type column. That is, to set up your PostGIS column with a geometry type, but use SRID=4326 (which is the EPSG number for WGS 84 latitude and longitude).

Don't do this.

I did this a few times in my naive youth, and it came back to bite me. What you're really doing here is employing a particular projection called Plate Carree, which simply maps latitude and longitude directly to x and y on the plane. Remember, any time you use geometry rather than geography, you are working with a flat coordinate system, and thus a projection. You might think you're working with latitude and longitude, but you're actually not.

Plate Carree is not a particularly useful projection (except that it is trivial to compute). It doesn't preserve distances, angles, directions, areas, or any other cartographically useful properties, and its distortion in polar regions is severe. In almost all cases, you can do much better with a different projection.

He suggests using a Mercator (EPSG:3785) projection that is used by Google Maps.

But here's the problem: My data is fixed. I am not creating any data myself. While I could transform the city's data to whatever projection I want, I currently only have two uses for the data:

• Displaying it on Google Maps (which takes real lat/lng as input)
• Calculating the lengths of the paths (more precise the better)

In hindsight, it seems obvious that KML was intended for Google Earth, a 3D application. That Google Maps accepts KML is surely a result of the synergies between the two, but it was easy to forget that Maps is a 2D application, despite accepting 3D coordinates.

I do not see any benefit to storing my data in EPSG:3785 when that is none of the following: the form my input data is in, the form my output data must be, nor the most efficient/accurate way to perform distance calculations.

This issue will remain open until I can wrap my head around the tradeoffs between these projections.

Set up pretty URLs for bikeways (routing, normalizing)

Think:

• https://416.bike/martin-goodman-trail/
• https://416.bike/the-queensway/
• https://416.bike/murrie-street/
• https://416.bike/wilket-creek-trail/
• https://416.bike/queens-greenbelt-trail/

Or, more restfully:

• https://416.bike/
• https://416.bike/routes/20
• https://416.bike/routes/20/broadway-duplex-lascelles
• https://416.bike/streets/murrie-st
• https://416.bike/trails/martin-goodman

Importing intersection data - how?

Not sure how I'm supposed to import the intersection data from the city's opendata site:
http://www1.toronto.ca/wps/portal/contentonly?vgnextoid=e659522373c20410VgnVCM10000071d60f89RCRD&vgnextchannel=1a66e03bb8d1e310VgnVCM10000071d60f89RCRD

I expected that the intersection table would look like this:

• Intersection_ID
• Cross_street_1
• Cross_street_2
• Geometry (point)

But instead, it's basically a duplicate of the existing bikeway data:

The only difference appears to be the existence of the fields fcode and fcode_desc (which I believe I deliberately omitted from the bikeways).

None of the city's geoid, rid, object_id, etc. match up on either table.

It's almost less than useless. Mainly because there doesn't appear to be any cross street information.

At this point I don't see any difference between the intersection and bikeway_segment tables.

Map enhancements

Look at e.g. http://www.canadatrailguide.com/map_results.php for inspiration

opendata:init task runs every time on cap deploy

For whatever reason, the Bikeways data file is not being detected, and the load process is happening on each deploy.

Perhaps vendor/opendata needs to be a shared (symlinked) directory on Cap deploys.

Determine correct GIS type to use in database for path field

When I imported the shapefiles manually using shp2pgsql, it created a geometry(MultiLineString,4326) column.

Using the RGeo::Shapefile::Reader``importer, it's stored as ageometry(Geometry,4326)`.

Determine if there's any real impact to this.

Also investigate why records are stored as MULTILINESTRING in the original data when they are all just individual LINESTRINGs.

Grouping by linear feature ID results in non-contiguous routes

See output from rake opendata:stats:walk_all_routes

Bayview Ave (lfn_id=24); id=1987; geoid=1903; 16 segments in db; 12 segments found by walking; BAD! counts DO NOT MATCH!

If you look at the data, you'll see it picks up geoid=1903. If you walk manually on the website from here, you see 12 segments on the Bayview Extension.

Where are the unaccounted 4? They are on Bayview Av near Lawrence, about 10 km away. They're not contiguous with the arbitrary segment we tried to walk.

This means that we can't just group by linear feature ID, because it could result in wonky data (e.g. two small bike lanes on opposite ends of town not at all connected).

If we do incorporate the SegmentWalker into the import process, it would need to be robust enough to not only handle non-contiguous paths, but forks in the road too.

Postgres user has SUPERUSER

The postgresql user bikeways_staging (et al) has SUPERUSER privilege. This is necessary to create the PostGIS extension.

This violates least privilege. We could fix this using the approach suggested in GeoRails:

You'll set up two users in the database. One is the normal database user that your Rails app will connect as. The other needs to be a superuser, and is used to create the databases initially (i.e. using rake db:create). Creating a PostGIS database requires superuser privileges unless you set up a special template for it, and for simplicity, we won't do that right now.

Curious about the template hint there. Not sure exactly what it means, but perhaps we could create a PostGIS template in the Ansible playbook.

Set colors for bikeway type on table

Bikeway table is destroyed/repopulated each time `cap deploy` is run

Make `opendata:init` process more robust

The initialization process should use some sort of heuristic (SHA hash, size of file, etc.) to determine whether or not to download the file.

Investigate leafly.js & geojson

The thrill of having a map that shows all the routes without chrome crashing has faded.

While the .kml approach works, it's slow and cumbersome.

Look into the following:

• getting useful GeoJSON output out of PostGIS via rails
• do a quick proof of concept using leafly.js with this GeoJSON
• assess whether we really even need server-side rendering of the GeoJSON, e.g.:
• test performance on a mobile device with 800+ elements, e.g. can we filter by individual items or is that pushing it.