kylebarron/all-transit

Interactive visualization of all transit in the Transitland database

Hi there! I'm Kyle 👋

I'm a software engineer passionate about fast geospatial data science.

I'm primarily developing the GeoArrow and GeoParquet ecosystems because I believe they portend a massive shift for faster, more interoperable geospatial data analysis.

Python:

Project	Role	Description
geopolars	author	Geospatial extensions for the Polars DataFrame library.
lonboard	author	Python library for fast, interactive geospatial vector data visualization in Jupyter.
geoarrow-rust	author	A Python library implementing the GeoArrow specification with efficient spatial operations.
suncalc-py	author	A Python port of suncalc.js for calculating sun position and sunlight phases.
pymartini	author	A Cython port of Martini for fast RTIN terrain mesh generation.
pydelatin	author	Python bindings to hmm for fast terrain mesh generation.
quantized-mesh-encoder	author	A fast Python Quantized Mesh encoder
usgs-topo-tiler	author	Python package to read Web Mercator map tiles from USGS Historical Topographic Maps
keplergl_cli	author	One-line geospatial data visualization using Kepler.gl

JavaScript:

Project	Role	Description
parquet-wasm	author	Rust-based WebAssembly bindings to read and write Apache Parquet data.
@geoarrow/deck.gl-layers	author	deck.gl layers for rendering GeoArrow data.
geoarrow-wasm	author	Efficient, vectorized geospatial operations in WebAssembly.
arrow-js-ffi	author	Zero-copy reading of Arrow data from WebAssembly.
geoarrow-js	author	TypeScript implementation of GeoArrow.
deck.gl	contributor	WebGL2 powered visualization framework.
deck.gl-raster	author	deck.gl layers and WebGL modules for client-side satellite imagery analysis.

Rust:

Project	Role	Description
geoarrow-rs	author	A Rust implementation of the GeoArrow specification and bindings to GeoRust algorithms for efficient spatial operations on GeoArrow memory.
geopolars	author	Geospatial extensions for the Polars DataFrame library.
geo-index	author	A Rust crate for packed, static, zero-copy spatial indexes.
arrow-wasm	author	Building block library for using Apache Arrow in Rust WebAssembly modules

Specifications:

Project	Role	Description
geoarrow	core contributor	Specification for storing geospatial data in Apache Arrow.
geoparquet	core contributor	Specification for storing geospatial vector data (point, line, polygon) in Parquet.

Other:

Project	Role	Description
National Scenic Trails Guide	author	A website and data tools for exploring and navigating the Pacific Crest Trail. After hiking the PCT, this project was the core of my effort to transition to a career in geospatial software engineering.
all-transit	author	Visualization of all transit routes in continental U.S.
vscode-jupyter-python	author	Run automatically-inferred Python code blocks in the VS Code Jupyter extension

all-transit's Issues

What is prevalence of ssp with frequencies

Mention animation scale

Client side requests to transit.land

You have route, operator IDs in the vector tiles already. You might be able to do something like "search times for route" using client requests. Or hover over a stop and see next scheduled departures

Upload for rest of world

If viewport is outside bbox of U.S. (or maybe bbox of U.S. plus bbox of Alaska plus bbox of Hawaii), say "Schedule data not yet available here"

Schedules not full resolution

Mention "animation using simplified data at this zoom" below the full resolution zoom

Make time 12 hr

Collapse options into a single accordion

Lower roads contrast at low zooms

Kinda hard to tell roads from transit. You could probably include the JSON style file in this repository, just import it, and pass the object to Deck.gl

Change fallback color to increase contrast

Maybe start by finding the hsl of the current color

Round to four decimal places

4 decimals should be about 10m precision. (presumably smaller precision at higher latitudes). Though jq doesn't support round?

https://gis.stackexchange.com/questions/8650/measuring-accuracy-of-latitude-and-longitude
https://wiki.openstreetmap.org/wiki/Precision_of_coordinates#Conversion_to_decimal

Filter by operators

Remove schedule stop pairs with speed above max speed

First I try matching to the route stop pattern, then I try matching to all route stop patterns identified from the route, then I match directly to the route. As noted in #35, sometimes the matched geometry is invalid, and appears to have a very high speed. I should automatically remove any matched schedule stop pairs where the imputed speed is above some max speed.

Option to hide stops

Add notice about schedule data not available in area

If zoom < minTripLayerZoom, say that you need to zoom in more to see trip schedules.

Otherwise, use onViewportLoad to tell if there's no data in the viewport. If no data, say something like "schedule data not available here"

Upload data dumps to S3 too share

Attribution control?

Where's the attribution control? Do I have to fix z values?

Fix route geometries

For Amtrak, try to snap to this dataset: https://osav-usdot.opendata.arcgis.com/datasets/4e32613ba4c9450880118b2fd639e8cb_0?geometry=-129.411%2C34.829%2C-57.824%2C44.916

Identify routes/route stop patterns by operator id that should be snapped to roads.

• Amtrak chartered vehicle (o-9-amtrakcharteredvehicle)
• Amtrak California (thruway buses)
• New York shuttle (o-dq-newyorkshuttle)
• NJ Transit (o-dr5-nj~transit)
• Peter Pan bus lines (o-dr-peterpanbuslines)
• Sunway charters o-dq0-sunwaycharters (NC)
• o-djj-annettbuslines (FL)
• o-dhv-martzfirstclasscoach (FL)
• o-9v-lonestarcoaches (TX)
• o-dnb-vandaliabuslines (MO, IL)

Host basemap of Alaska and Hawaii

I currently have transit data of Alaska, but no basemap:
(Anchorage):

Only show time when schedule animation enabled

I.e. don't show time there

Raise route opacity when schedule animation off

Put route labels above map labels

Since they're off by default, make them visible most often if turned on

Attach route information to schedule data

Attach route id, operator id, mode of transit. Maybe share as protobuf? Then when you select the filters in the left, the animation should update. Probably good to have the data filtering in a worker then? Or can you use deck's DataFilterExtension? I'm not sure if it works with strings...

Cache control headers

currently the schedule jsons have none and the openmaptiles tiles are only 5hours

Turn off labels by default

might help rendering go faster

First try matching against RouteStopPattern

I match ScheduleStopPairs against Routes instead of RouteStopPatterns because while manually looking through a few of them, the RouteStopPattern on the SSP was null. However it would be better if you first tried matching against the RouteStopPattern and only matched against the Route as a fallback.

When matching to Routes, sometimes the SSP is matched to the wrong direction of the Route. Usually that doesn't matter but with highways or any place where the route is different in each direction, it can look like the bus is driving on the wrong side of the road.
For example:
https://all-transit.com/#13.53/47.53084/-122.29016

Remove operators with bad geometries

There are many operators with very ugly geometries/bad data. I should remove these.

Generate data for other time slots

Only update operators list when filter box is open

Add mobile link preview

Have a ? Popup next to highlight routes by stop

Operators in area

Render transparent operators layer; queryRenderedFeatures on that layer

Include frequency-based schedules

It looks like frequency based schedules might only have one line per day, and it's my job to intersperse between start and end time

origin_onestop_id|destination_onestop_id|route_onestop_id|route_stop_pattern_onestop_id|operator_onestop_id|feed_onestop_id|feed_version_sha1|origin_timezone|destination_timezone|trip|tri
       │ p_headsign|block_id|trip_short_name|wheelchair_accessible|bikes_allowed|pickup_type|drop_off_type|shape_dist_traveled|origin_arrival_time|origin_departure_time|destination_arrival_time|de
       │ stination_departure_time|origin_dist_traveled|destination_dist_traveled|service_start_date|service_end_date|window_start|window_end|origin_timepoint_source|destination_timepoint_source|fr
       │ equency_start_time|frequency_end_time|frequency_headway_seconds|frequency_type|created_at|updated_at|service_days_of_week_0|service_days_of_week_1|service_days_of_week_2|service_days_of_w
       │ eek_3|service_days_of_week_4|service_days_of_week_5|service_days_of_week_6

s-dqcm21w9qc-benningrdand15thstne|s-dqcm237f1n-benningrdand19thstne|r-dqcm2-h~benning|r-dqcm2-h~benning-7270b2-b74f6b|o-dqcm2-dcstreetcar|f-dqcm2-dcstreetcar|8c2eaaf8e9750c849bbb61b193180│ 9977a05de22|America/New_York|America/New_York|1|Benning Rd|||true|true|||0|08:12:11|08:12:36|08:15:52|08:16:17|1898.5|2516.8|2017-09-16|2020-06-30|08:12:36|08:15:52|gtfs_exact|gtfs_exact|│ 08:00:00|26:00:00|720|exact|2018-06-28T02:00:13.586Z|2018-06-28T02:00:13.586Z|false|false|false|false|false|true|false

Fix flickering highlighting of routes

If you have "highlight routes by stop" set to on, then keep all routes at low (0.1) opacity when the mouse isn't hovering over anything?

Reset highlighted objects when zooming out

Incorrect matching of ScheduleStopPairs to route geometry

There are multiple instances (that are clear to see visually) where the transit vehicles are moving way too fast. For example, look at this on the loop in Chicago:

Notice how there are super long tails around the loop? That's from schedules that are incorrectly matched to routes.

I did some digging here, using the Brown line metro. Take a look at this ScheduleStopPair:

{
  "origin_dist_traveled": "546.2",
  "origin_onestop_id": "s-dp3wspsmsk-kedziebrown<30226",
  "route_onestop_id": "r-dp3w-brownline",
  "trip": "59151270686",
  "origin_timepoint_source": "gtfs_exact",
  "destination_dist_traveled": "1140.6",
  "route_stop_pattern_onestop_id": "r-dp3w-brownline-a3d334-f6e93f",
  "destination_timepoint_source": "gtfs_exact",
  "destination_onestop_id": "s-dp3wsr9tgt-francisco<30168",
  "destination_arrival_time": "17:49:00",
  "origin_departure_time": "17:48:00",
  "speed": 301.5832166384404,
  "dist": 18094.99299830642
}

(with operator id: o-dp3-chicagotransitauthority)

Here's a visualization of how I'm currently matching that to a route:

The two neighboring stops are in the top left. But since this is a loop route, the second stop is getting matched to the route on its way back. So it's assigning an 18km route between stops that are a minute apart.

Simplify schedule tiles at lower zooms

Then they could exist. Not sure how to find least dense features

Put below map labels

Binary tiles

Explore using binary tiles, possibly with Apache Arrow, to offload computation off the CPU.

"Use binary data" is a section of the deck.gl performance optimization guide.

Also see:
visgl/deck.gl#3467

Refactor Mapbox layers

Add modified mapbox gl style

Something like mapbox dark for openmaptiles

Only show clock timer when there's data in the viewport

All Transit not updated to Transitland v2

This is preventing All Transit from loading

https://www.transit.land/news/2023/09/14/shutting-down-v1-datastore-api

notes

When you hover over a route, show the dots for stops the route serves. Though that could be a lot of metadata, so only above a given zoom?
add
"operated_by_onestop_id"
"operated_by_name"
to attributes kept

Transit schedules

Unnecessary to transmit all schedule information for all route stop patterns for all times. What's the best way to condense this into info that could be represented by, e.g. the TripsLayer

You could have the frequency per hour of a route stop pattern? For each day of week

update API calls to page if necessary. Currently the max results is

https://github.com/kylebarron/transit-land-dump/blob/63606ffe2e5a82593f55776e8be54e9ad4654b98/code/download.py#L18-L23

---

This issue was generated by todo based on a TODO comment in 63606ff. It's been assigned to @kylebarron because they committed the code.

Check if DC metro uses frequency key in schedules

It's currently not animated, so I'm guessing that's why

Delays

Script to record real time gtfs
How to link to one stop id

Option to change routes colorscheme

By default, just use colors from data, but also have a choice for a separate color for each transit operator. You can get the list of operators from queryRenderedFeatures, and then on the fly generate new colors for each operator in the viewport. Note that you'll need operator_id present on the routes, but that might be included?

Explore stops by route or routes by stops

If you hover over a stop, highlight all the routes that serve that stop; if you hover over a route, highlight all the dots that the route stops at.

Write About page