OTv2: application to generate and update OSMLR segments (for use by the Open Traffic platform and others needing a stable linear-referencing system)
Home Page: https://mapzen.com/blog/osmlr-released-as-public-dataset/
License: Other
Given a bounding box (similar to script in Reporter). OSMLR tiles are in a public Open Traffic S3 bucket but listing contains a large number of files so this would be convenient.
Each OSMLR entry within a tile either has a valid segment definition or a Marker that indicates the segment is deleted / deprecated.
The Marker should be updated to add a version or date to indicate when the OSMLR segment was deprecated.
The Segment should be updated to add a version/date to indicate when the OSMLR segment was initially created.
GeoJSON (and perhaps pbf) could use 2 additional pieces of information to help with visualization:
Oneway - Flag indicating whether the segment is oneway travel or allows travel in both directions. This can be useful for logic that produces "offset geometry".
Side of Street Driving - Does the segment indicate drive on right (default) or drive on left. This also is important for producing offset geometry.
The OSMLR tile specification has new fields for tile creation date and OSM changeset ID. These help identify the tile age and rough association to OSM data. Need to actually set these values during OSMLR generation.
from @dnesbitt61:
Roundabouts generally contain several short Valhalla edges. Creating a separate OSMLR segment for each might not be the best way to represent a roundabout. Perhaps roundabouts are best ignored during OSMLR creation and the figure into the transition costs between OSMLR segments during routing?
Hello,
Are there some version constraints between osmlr and valhalla? Beacuse I can't generate osmlr segments..keep getting segfault error.
My valhalla.json:
{
"additional_data": {
"elevation": "/data/valhalla/elevation/"
},
"httpd": {
"service": {
"interrupt": "ipc:///tmp/interrupt",
"listen": "tcp://*:8002",
"loopback": "ipc:///tmp/loopback"
}
},
"loki": {
"actions": [
"locate",
"route",
"height",
"one_to_many",
"many_to_one",
"many_to_many",
"sources_to_targets",
"optimized_route",
"isochrone",
"trace_route",
"trace_attributes",
"transit_available"
],
"logging": {
"color": true,
"file_name": "path_to_some_file.log",
"long_request": 100.0,
"type": "std_out"
},
"service": {
"proxy": "ipc:///tmp/loki"
},
"service_defaults": {
"minimum_reachability": 50,
"radius": 0
},
"use_connectivity": true
},
"meili": {
"auto": {
"search_radius": 50,
"turn_penalty_factor": 200
},
"bicycle": {
"turn_penalty_factor": 140
},
"customizable": [
"mode",
"search_radius",
"turn_penalty_factor",
"gps_accuracy",
"sigma_z",
"beta",
"max_route_distance_factor",
"max_route_time_factor"
],
"default": {
"beta": 3,
"breakage_distance": 2000,
"geometry": false,
"gps_accuracy": 5.0,
"interpolation_distance": 10,
"max_route_distance_factor": 5,
"max_route_time_factor": 5,
"max_search_radius": 100,
"route": true,
"search_radius": 50,
"sigma_z": 4.07,
"turn_penalty_factor": 0
},
"grid": {
"cache_size": 100240,
"size": 500
},
"logging": {
"color": true,
"file_name": "path_to_some_file.log",
"type": "std_out"
},
"mode": "auto",
"multimodal": {
"turn_penalty_factor": 70
},
"pedestrian": {
"search_radius": 50,
"turn_penalty_factor": 100
},
"service": {
"proxy": "ipc:///tmp/meili"
},
"verbose": false
},
"mjolnir": {
"admin": "/data/valhalla/admin.sqlite",
"hierarchy": true,
"logging": {
"color": true,
"file_name": "path_to_some_file.log",
"type": "std_out"
},
"max_cache_size": 1000000000,
"shortcuts": true,
"tile_dir": "/data/valhalla_tiles",
"tile_extract": "/data/valhalla_tiles.tar",
"timezone": "/data/valhalla/tz_world.sqlite",
"transit_dir": "/data/valhalla/transit"
},
"odin": {
"logging": {
"color": true,
"file_name": "path_to_some_file.log",
"type": "std_out"
},
"service": {
"proxy": "ipc:///tmp/odin"
}
},
"service_limits": {
"auto": {
"max_distance": 5000000.0,
"max_locations": 20,
"max_matrix_distance": 400000.0,
"max_matrix_locations": 50
},
"auto_shorter": {
"max_distance": 5000000.0,
"max_locations": 20,
"max_matrix_distance": 400000.0,
"max_matrix_locations": 50
},
"bicycle": {
"max_distance": 500000.0,
"max_locations": 50,
"max_matrix_distance": 200000.0,
"max_matrix_locations": 50
},
"bus": {
"max_distance": 5000000.0,
"max_locations": 50,
"max_matrix_distance": 400000.0,
"max_matrix_locations": 50
},
"hov": {
"max_distance": 5000000.0,
"max_locations": 20,
"max_matrix_distance": 400000.0,
"max_matrix_locations": 50
},
"isochrone": {
"max_contours": 4,
"max_distance": 25000.0,
"max_locations": 1,
"max_time": 120
},
"max_avoid_locations": 50,
"max_radius": 200,
"max_reachability": 100,
"motor_scooter": {
"max_distance": 500000.0,
"max_locations": 50,
"max_matrix_distance": 200000.0,
"max_matrix_locations": 50
},
"multimodal": {
"max_distance": 500000.0,
"max_locations": 50,
"max_matrix_distance": 0.0,
"max_matrix_locations": 0
},
"pedestrian": {
"max_distance": 250000.0,
"max_locations": 50,
"max_matrix_distance": 200000.0,
"max_matrix_locations": 50,
"max_transit_walking_distance": 10000,
"min_transit_walking_distance": 1
},
"skadi": {
"max_shape": 750000,
"min_resample": 10.0
},
"trace": {
"max_best_paths": 4,
"max_best_paths_shape": 100,
"max_distance": 200000.0,
"max_gps_accuracy": 100.0,
"max_search_radius": 100.0,
"max_shape": 16000
},
"transit": {
"max_distance": 500000.0,
"max_locations": 50,
"max_matrix_distance": 200000.0,
"max_matrix_locations": 50
},
"truck": {
"max_distance": 5000000.0,
"max_locations": 20,
"max_matrix_distance": 400000.0,
"max_matrix_locations": 50
}
},
"thor": {
"logging": {
"color": true,
"file_name": "path_to_some_file.log",
"long_request": 110.0,
"type": "std_out"
},
"service": {
"proxy": "ipc:///tmp/thor"
},
"source_to_target_algorithm": "select_optimal"
}
}generated with valhalla_build_config --mjolnir-tile-dir ${PWD}/valhalla_tiles --mjolnir-tile-extract ${PWD}/valhalla_tiles.tar > valhalla.json
After I generate the valhalla tiles I run osmlr -T ${PWD}/osmlr_tiles -J ${PWD}/geojson_tiles --config valhalla.json
After that I get:
Are you sure you want to create new OSMLR data [Y|N]?
y
2018/06/07 11:56:38.237629 [INFO] Tile extract successfully loaded
Segmentation fault (core dumped)
thank you
Currently the creation of OSMLR GeoJSON output requires a separate execution of the OSMLR generation process. To support OSMLR segment migration/update processing, a new process will be created to generate the GeoJSON segment descriptions.
The anticipated approach will use the OSMLR protocol buffer tiles in conjunction with Valhalla traffic-enabled routing tiles to create OSMLR GeoJSON.
This will allow the GeoJSON generation process to use the most recent traffic-enabled Valhalla tiles (if desired) to show the OSMLR segments with geometry that matches any recent data changes to OSM (or underlying data). This can also show where OSMLR segments no longer match the underlying data as any current OSMLR segments that are not associated/matched within Valhalla tiles can be identified and marked.
Unmatched OSMLR segments will not have true geometry, but could have a simple linear representation between endpoints to potentially allow visualization of where OSMLR segments are potentially being deprecated in upcoming releases.
from @dnesbitt61:
There are several cases where OSMLR segments should be merged across intersections where only “intersecting roads” meeting specific criteria exist. Possibilities include:
Hi,
When is this platform available for public testing.
To start with we need API to find the distance between locations and Time to reach with traffic data.
We want to integrate Opensource Distance and Time (including traffic) API instead of Google with Traccar app(traccar/traccar#2629 (comment)) which is used for Fleet/logistic management in many countries.
With this integration we(Traccar and community) can contribute to Opentraffic in big way.
Regards,
Kiran.
I'm building it inside Docker. First of all it requires ubuntu:18.04 image, then old valhalla 2 dependency. May be it should be updated to valhalla3 and recent ubuntu? :) Or I miss something and recent osmlr replacement is included in other repository? Then may be this one should be marked as deprecated.
from @dnesbitt61:
Turn channels or turn lanes are short (maximum length = TBD) edges that allow turns to bypass the intersection of 2 edges.
Proposal:
Do the following rather than creating separate OSMLR segments for the turn lane edge and the 2 short edges between the turn lane endpoints and the intersection:
The goal would be to associate a trace along Snowden River Pkwy through the turn channel onto Waterloo Rd would “end” the OSMLR segment along Snowden River Pkwy and begin the segment along Waterloo Rd.
2017/10/03 13:59:44.000299 [INFO] Tile: 2/332383 Found 11708 of 11718 OSMLR segments
2017/10/03 13:59:44.260001 [INFO] Tile: 2/733330 Found 3656 of 3657 OSMLR segments
2017/10/03 13:59:44.559857 [ERROR] Single partial segment starts on this edge but does not use entire edge?
2017/10/03 13:59:44.561282 [ERROR] Could not find continuation for the segment!
2017/10/03 13:59:44.564827 [INFO] Tile: 2/542271 Found 525 of 526 OSMLR segments
terminate called after throwing an instance of 'std::runtime_error'
what(): Failed to close fd 39 because: Bad file descriptor
Aborted (core dumped)
Do I get it right that the OSM-LR and opentraffic work is now continued as http://sharedstreets.io ? Can you elaborate your post-mapzen situation in project documentations.
This generator application currently outputs GeoJSON in WGS84 using Valhalla's tile scheme. Some clients, like the Open Traffic Analyst UI, will more easily consume OSMLR GeoJSON geometries in Web Mercator using the /zoom/x/y tile scheme.
Questions on the generation side:
Questions on the consumer side:
//cc @dnesbitt61 @kevinkreiser @burritojustice @louh @meghanhade
Hello,
I try to run Basemap Generator on ubuntu 16.04 VM follow yours bash command and i have some problem:
sudo add-apt-repository -y ppa:valhalla-core/valhalla
sudo apt-get update
sudo apt-get install -y valhalla-binIt seems to work
2. No osmlr installed and i try to install osmlr with PPA: https://launchpad.net/~valhalla-core/+archive/ubuntu/opentraffic and when i running i got:
I try to find out what is wrong with my system and I'm still stuck. Have any ideas on how to fix those problems. Many thanks!
I was trying to build and run with help of 'Building and Running' docs, but I got this error, 'libvalhalla-dev has no installation candidate' and even after building Valhalla from the source I got 'osmlr not found'.
Can anyone tell me what is the issues and how to solve that?
At present, the OSMLR build process assigns whatever IDs it sees fit to the segments it creates. This works well for one-off builds, but presents a problem when there is an installed base of Reporters using them to feed data back to the central hub as it requires us to handle version mismatches.
For example, if we versioned the IDs then the segment with ID 0/1/1 generated in 2017Q1 would be called 2017Q1:0/1/1 or similar (probably more efficiently bit-packed, but that's irrelevant to the point). This means that anyone slow to update to new data would be sending old IDs, which the central process would have to either translate to the new IDs or drop the data. Dropping the data is obviously bad, and translating might be non-trivial. At the very least, translating would require us to keep a translation table whose size would grow with the number of OSMLR data releases.
Instead, we propose to have a single ID "namespace", so that 0/1/1 would always refer to the same segment so long as that segment didn't change. This requires the OSMLR generation process to begin by matching the previous segment descriptors against the current data, and keeping the IDs for those which continue to match. Since OSMLR descriptors are designed to be tolerant to some kinds of changes, we anticipate that the majority will continue to match.
The few which do not match will result in the old IDs being marked as "deleted" or "superseded" and new IDs being allocated for the new segments (possibly with forward / back references to aid in translation).
This means that most of the Reporters which are tardy in updating to the latest data release will still be sending mostly useful data, and the data which still uses old IDs might be of a small enough volume to make translation feasible - if not in real time, then at least for batch historical analysis.
Goals of this task:
from @dnesbitt61:
“Internal” edges are the edges between doubly digitized, separated carriageways that allow turns onto an intersecting road. In the image below it is the section of McGaw Road between the 2 separate carriageways of Snowden River Parkway. In reality these are often just large intersections.
Thought:
Traces that cross an internal intersection should somehow apply transition costs when going from one OSMLR segment onto another but should not actually be represented as a separate OSMLR segment.
When OSM ways overlap they can create duplicate Valhalla edges. This can also occur when a single way is drawn with sections that overlap. This results in multiple OSMLR segments. It also leads to splitting or breaking OSMLR segments at the nodes where duplicates occur (leading to shorter segments than are desirable).
Ideal solution is to detect these overlapping ways and not create duplicate edges in Valhalla, though this has some complications due to OSM relations. Better tools to detect and fix the underying OSML data can be helpful here as well.
A second possible approach is to not create duplicate OSMLR segments - though this does not fix the issue with splitting.
Fortunately the presence of overlapping or duplicate ways is somewhat small. While inconvenient and leading to more OSMLR segments than are needed, the overall impact is likely small.
Splitting segments over 1km long should occur at the same locations when there are opposing OSMLR segments (on 2-way driveable roads). Current logic moves in the forward direction and splits at appropriate locations such that no segment is over 1km long.
For single edges that are over 1km long this can be made consistent by considering the direction (forward/reverse) of the directed edge compared to the edge shape.
For multiple edges where the split occurs at intersecting nodes a different approach is needed.
Hi, we are discussing a future/definitive implementation of a "Permanent ID". OSMLR is one of the few OSM applications that is solving the problem of persistence of OSM's element identification...
We need some statiscs about "stability" of the real-world ways, the "stability" of the OSM's representation of that ways, and the changeset... There are some results of the OSMLR project? A CSV table or report summarizing this kind of data?
Main discussions and links:
We need to set up a self sustaining data creation environment. Thinking that we can use valhalla docker scripts. Should be able to use the following scripts for this environment:
scripts/minutely_update.sh -- Applies changesets to the latest planet.pbf file. Should be set up to run forever via lamdba.
scripts/associate_segments.sh -- runs valhalla_associate_segments on valhalla data using the latest osmlr...thinking it can run once a week.
scripts/cut_tiles.sh -- creates valhalla route tiles. Should be set up to run forever via lamdba.
We need to discuss as we are thinking that the results of minutely_update.sh should be saved in a mount point and cut_tiles will use that location to build valhalla tiles. The same goes for associate_segments.sh...it will use a mount point to get the latest version of the valhalla tiles.
Should OSMLR segments have a maximum length? When creating OSMLR this would mean that long edges - for example long stretches of highway between 2 exits - would be subdivided into multiple OSMLR segments.
Shorter segment length may be required for lower latency in a real-time system. If we rely on complete segment traversal to trigger a report on a traffic segment then shorter segments would decrease latency. Otherwise some "sub-segment" reporting mechanisms may be required to accumulate readings across multiple vehicles at different sections of a long segment in order to make reliable real-time speed measurements.
This would create OSMLR segment definitions on Valhalla level 2 (local tiles).
Questions and Issues:
Which road classifications to include: residential, unclassified, service, footways and cycleways?
Does this mean that OSMLR segments on arterials and highways should be split when crossing OSMLR segments on the local level? This would create a large increase in the number of OSMLR segments and would create a large number of short segments (which may cause accuracy issues and could complicate traffic influenced routing).
Note on Sizing: 60% of the Valhalla graph edges are on the local hierarchy level - so creating OSMLR segments for these would dramatically increase the OSMLR data size (and Valhalla tiles if association to OSMLR segments is required on level 2).
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.