giscience/oshdb

OpenStreetMap History Data Analysis Framework

OSHDB - OpenStreetMap History Data Analysis

High-performance spatio-temporal data analysis platform for OpenStreetMap full-history data. Developed by HeiGIT.

The OSHDB allows to investigate the evolution of the amount of data and the contributions to the OpenStreetMap project. It combines easy access to the historical OSM data with high querying performance. Use cases of the OSHDB include data quality analysis, computing of aggregated data statistics and OSM data extraction.

    OSHDBDatabase oshdb = /*…*/;

    Integer numberOfUsersEditingHighways = OSMContributionView.on(oshdb)
        .timestamps("2007-10-07", "2009-04-09")
        .filter("type:way and highway=*")
        .map(contribution -> contribution.getContributorUserId())
        .countUniq();

    System.out.println(numberOfUsersEditingHighways);

The main functionality of the OSHDB is explained in the first steps tutorial.

OpenStreetMap History Data

OpenStreetMap contains a large variety of geographic data, differing widely in scale and feature type. OSM contains everything from single points of interests to whole country borders, from concrete things like buildings up to more abstract concepts such as turn restrictions. OSM also offers metadata about the history and the modifications that are made to the data, which can be analyzed in a multitude of ways.

Because of it's size and variety, possibilities of working with OSM history data are limited and there exists a lack of an easy-to-use analysis software. A goal of the OSHDB is to make OSM data more accessible to researchers, data journalists, community members and other interested people.

Central Concepts

The OSHDB is designed to be appropriate for a large spectrum of potential use cases and is therefore built around the following central ideas and design goals:

• Lossless Information: The full OSM history data set should be stored and be queryable by the OSHDB, including erroneous or partially incomplete data.
• Simple, Generic API: Writing queries with the OSHDB should be simple and intuitive, while at the same time flexible and generic to allow a wide variety of analysis queries.
• High Performance: The OSM history data set is large and thus requires efficiency in the way the data is stored and in the way it can be accessed and processed.
• Local and Distributed Deployment: Analysis queries should scale well from data explorations of small regions up to global studies of the complete OSM data set.

The OSHDB splits data storage and computations. It is then possible to use the MapReduce programming model to analyse the data in parallel and optionally also on distributed databases. A central idea behind this concept is to bring the code to the data.

Data Model

The OSHDB uses its own binary data format that encapsulates the OSM history data available from planet.osm.org and is optimized for efficient storage and access to the history data. In order to allow parallelized data processing, the OSHDB data model also includes a data partitioning scheme.

See the data model section of the documentation for further information about the OSHDB data model.

API

The OSHDB offers a flexible, simple and intuitive application programming interface that provides helpful abstractions on top of the offered OSM data entities. It provides different views on the OSM history data that allow to either investigate the OSM data at specific points in time (as snapshots), or to investigate all contributions to the OSM data in their entirety. The OSHDB API also allows to filter the OSM data by arbitrary regions, time ranges and OSM properties such as tags, entity type, etc.

The API is based on the MapReduce programming model and offers powerful methods to aggregate and analyze the OSM history data. The OSHDB API is described in detail in the api section of the documentation.

Installation

The OSHDB is available as a pre-compiled maven library and can be incorporated easily in any maven project. If you're starting a new project, take a look at how your IDE handles maven projects (for example, here you find instructions how to create a new maven project using IntelliJ). Our first steps tutorial includes further information about how to add the OSHDB as a maven dependency to your projects.

Documentation

• first steps tutorial
• User Manual
• org.heigit.ohsome Javadoc

Examples

A few usage examples can be found in our oshdb-examples repository.

How to Cite

When using the OSHDB directly or indirectly (e.g. through the ohsome API) for a publication, please cite it using the doi:10.5281/zenodo.4146990 or a corresponding version-specific DOI found on zenodo.

For scientific papers or similar publications around the analysis of OSM history data, please consider to additionally also cite the technical paper describing the OSHDB: doi:10.1186/s40965-019-0061-3.

Publications and Presentations

• Raifer, Martin; Troilo, Rafael; Kowatsch, Fabian; Auer, Michael; Loos, Lukas; Marx, Sabrina; Przybill, Katharina; Fendrich, Sascha; Mocnik, Franz-Benjamin; Zipf, Alexander (2019): OSHDB: a framework for spatio-temporal analysis of OpenStreetMap history data. Open Geospatial Data, Software and Standards.
• Auer, Michael; Eckle, Melanie; Fendrich, Sascha; Kowatsch, Fabian; Marx, Sabrina; Raifer, Martin; Schott, Moritz; Troilo, Rafael; Zipf, Alexander (2018): Comprehensive OpenStreetMap History Data Analyses- for and with the OSM community. Talk at the State of the Map conference 2018, Milan.
• Auer, Michael; Eckle, Melanie; Fendrich, Sascha; Kowatsch, Fabian; Loos, Lukas; Marx, Sabrina; Raifer, Martin; Schott, Moritz; Troilo, Rafael; Zipf, Alexander (2018): Ohsome – eine Plattform zur Analyse raumzeitlicher Entwicklungen von OpenStreetMap-Daten für intrinsische Qualitätsbewertungen. AGIT ‒ Journal für Angewandte Geoinformatik.
• Auer, Michael; Eckle, Melanie; Fendrich, Sascha; Griesbaum, Luisa; Kowatsch, Fabian; Marx, Sabrina; Raifer, Martin; Schott, Moritz; Troilo, Rafael; Zipf, Alexander (2018): Towards Using the Potential of OpenStreetMap History for Disaster Activation Monitoring. Conference: Proceedings of the 15th ISCRAM Conference 2018, Rochester.
• Raifer, Martin (2017): OSM history analysis using big data technology. Talk at the State of the Map conference 2017, Aizuwakamatsu.

See Also

Projects using the OSHDB:

• https://api.ohsome.org
  A web api for executing common queries on OSM history data
• https://osm-measure.geog.uni-heidelberg.de/about
  A research project about data quality measures related to OSM data.

Other projects analyzing OSM history data:

• https://github.com/MaZderMind/osm-history-renderer
• https://github.com/zehpunktbarron/iOSMAnalyzer
• https://github.com/mojodna/osm2orc
• https://github.com/mapbox/osm-wayback, https://mapbox.github.io/osm-analysis-dashboard/
• https://osmstats.stevecoast.com

oshdb's Issues

improve oshdb-api processing parallelism

follow up from #26:

Ideally, one would like to start processing with the lower zoom cells, but have them evenly distributed across all threads in the pool (e.g. in an interleaved way: cell 1 -> thread 1, cell 2 -> thread 2, …, cell n -> thread n, cell n+1 -> thread 1, etc.), but I'm not sure if that's easy to implement using the normal java streams.

Separate ModificationTime logic from OSHxxx classes

Reduce the complexity of the OSH Classes and separate business logic from data container.

Automatic Code formatting

After the code-quality-week there are still a lot of problems in the code concerning code style. In addition leaving code formatting to the IDE might result in different rules for different IDEs in the future.

This maven plugin can check and automatically format the code when building: https://github.com/coveo/fmt-maven-plugin. This would leave code-formatting to a central place, same for all: maven.

Pros:

• one style, one format-mechanism
• don't bother on formatting, is automatically done when building
• imediately solves 3'000 checkstyle problems and java warnings

Cons:

• one major format commit on start that would have to be verified
• all horror-scenarios that come to your mind when thinking of a machine formatting your precious code
• its a machine, so no reasoning involved:
  user: writing aba is more readable than aab?
  code formatting machine: I DONT CARE, ITS NOT OUR STYLE!
• creates cpd and pmd warnings

I created a branch https://github.com/GIScience/oshdb/tree/automatedStyleFormat so you can see, what it would do to our code. You might also check what your IDE says to the new format.

Implement ZGrid

ZGrid is preferred over XYGrid. This needs to be fully implemented to replace it.
See
https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/67
https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/tree/transitionToNewIndex

Core OSHDB classes need javadoc

For unexperienced developers this: https://github.com/GIScience/oshdb/blob/master/oshdb/src/main/java/org/heigit/bigspatialdata/oshdb/osh/OSHRelation.java#L306 needs to get javadoc!

A) is this the right way to create an OSHEntity (in this example Type: Relation)
B) What are "nodes" and "ways"? All ways that were ever part of of the OSHRelation or just the current ones?

maven warnings

https://jenkins.ohsome.org/job/oshdb/job/master/169/maven/

Moving dependent code of Ignite/JDBC/H2 to own sub modules

The oshdb-api depends currently to Ignite and Jdbc/h2 due the adapters for those backends.
In concern of better dependencies it makes sense to move the code for ignite (OSHDBIgnite.java) and jdbc/h2 to their own submodules.
for example: oshdb-api-ignite/oshdb-api-jdbc/oshdb-api-jdbc-h2 .
This introduces more submodules but each module is more specific and creates room to support an unknown count of further backend adapters.

OSM-Tile-Logs

Similar to #84 the tile-logs (https://planet.openstreetmap.org/tile_logs/) are not directly linked to the data but might spark research.
Therefore they would be a good data set for our oshdb.
For reference:
http://lukasmartinelli.ch/python/2015/05/24/parsing-and-visualizing-osm-access-logs.html

Solve FindBugs Warnings

see #90

• etl
• oshdb
• oshdb-api
• oshdb-util
• oshpbf-parser

Make Jenkins deploy uber-jar (fat-jar, jar-with-dependencies) to artifactoy

For easy installation of oshdb to Ignite, a jar with dependencies is useful and should be stored in artifactory repository.

optimization: skip unchanged entries when entity/geometry is unchanged (OSMEntitySnapshotView)

For many analysis operations (e.g. calculation of histocounts), the result of a map function doesn't depend on the snapshot's timestamp. When this is the case, consecutive calls to the map function can be skipped and the previous result of the particular entity be copied as the next timestamp's result. This could speed up analysis by a great margin (especially calculations with small timestamp intervals, because then the entities are often unchanged between time snapshots).

opened by @tyrasd

I think implementing this could end up being a relatively low hanging fruit because the CellIterator already provides this optimization, and has only to be leveraged in the oshdb-api.

Only caveat: For some analysis, the assumption that the result of the mapper doesn't depend on the snapshot's timestamp may not be true (e.g. when interpreting OSM tags differently depending on the date), so this needs to be either behind a flag (active or inactive by default?), or work with some built in magic that detects if the mapper function actually accesses the snapshot's timestamp or not (e.g. by hooking into OSMEntitySnapshot::getTimestamp() -> if it's not called for a specific "state" of an entity one can assume that it's not being called by all further snaptshots/timestamps that share the same entity/geometry state -> the only case when this wouldn't be a valid assumption is when the map function is not pure, but for example depends on random numbers, e.g. when using some kind of monte carlo algorithm). 🤔

see https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/63

cleanup dependencies

• drop unused dependencies
• evaluate included dependencies and decide for each if it makes sense to use
• move dependencies to the module where they are actually needed
• eventually refactor code to reduce required dependencies
  • geotools: #5
  • apache commons #79
• upgrade JTS #75

OSM-Users

The user-name-resolver (https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/libs/username-resolver, formerly tracked in #59) is a first step but users have more information then their name that we could store:

• names can be changed over time, currently this only returns the latest name
• users may delete their account, this info is published here: https://planet.openstreetmap.org/users_deleted/
• users may agree or disagree to the new license: https://planet.openstreetmap.org/users_agreed/

request metadata output

opened by @tyrasd

we need some method to access some metadata like the following:

• "freshness" of the imported osm data (and/or minutely update status, see #6)
• source of the data
• license(s) of the returned data (i.e. ODbL and/or cc-by-sa in the case of OSM depending on requested time interval)
• spatial validity/completeness of the data in the oshdb instance (global or extent of regional extract)
• temporal validity/completeness of the returned data (e.g. that OSM's history data before October 2008 is incomplete!)
• general stats (amount of data)
• maxzoom parameter
• used grid index
• oshdb version
• keytables-"hash" for #105
• instead of a String getter Method should return a OSHDBMetadata-Object to simplify usage. Thereby a user directly knows what fields are available and their format.
• …

write getters for these properties in OSHDB class

todo:

• document existing metadata fields

see https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/64

ETL-Load deletes data if not persisted

https://github.com/GIScience/oshdb/blob/master/oshdb-tool/etl/src/main/java/org/heigit/bigspatialdata/oshdb/tool/importer/util/OSHDB2Ignite.java#L65
Turning the cluster off and on causes loss of non persisted data. This should be solved because persistence is not always necessary/desired and slows down import.

Ratio functionality

opened by @nb152

In our last osm data quality meeting, we realized that it would be useful to able to calculate the ratio between two groups of objects e.g. the ratio between residential roads ("highway"="residential") and all roads ("highway"). Right now one would have to make two separte queries to the oshdb and subsequently calculate the ratio.

Please see multiple comments at https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/147

improve error messages when ignite caches are not present

when executing queries on ignite and the respective caches are not present (e.g. when one used a wrong prefix), the resulting error messages are not very useful (for example a NullPointerException in the NumberUtils class). We should check this and return a dedicated error message when caches are not present on the ignite cluster.

OSHDBGeometryBuilder - history testdata: visible = false: geometry/geometrycollection is not empty

visible=false
nodes

• Test testVisibleChange()
  testdata
  A single node, visible changes between true and false. When visible is false, lat and lon are set 0.0.
  -> assertTrue(result.isEmpty()); fails

• Test testMultipleChanges()
  testdata
  A single node, various changes over time, visible changes between true and false. When visible is false, lat and lon are set 0.0.
  -> assertTrue(result.isEmpty()); fails

relation type not multipolygon

• Test testVisibleChangeOfWay()
  testdata
  relation type not multipolygon, with one way member: way 119 changes visible tag, when visible = false the geometrycollection is not emtpy
  -> assertTrue(result.isEmpty()); fails

Solve Checkstyle Warnings

As of now, 6533 CheckStyle Warnings are recorded. These are no code errors but rather coding style errors. This is a static analyses and issues may not be solved or be irrelevant.

The goal is to solve solve as may of these as possible and close the issue. After that, increases in reports can be handled e.g on each merge into master.

The following modules need checking:

• etl (2240)
• oshdb (919)
• oshdb-api -> #279
• oshdb-util -> #275, #283
• oshpbf-parser (will be removed, doesn't need to be fixed)

State: 2020-12-07, build master/367

Proposal for new ContributionTypes for CellIterator

The CellIterator is one of the key elements (and a great one, too!) of the DB. These are some proposals for ContributionTypes, to make the results more self-evident, especially when being used through the API:

• APPEARING and DISAPPEARING instead of CREATION and DELETION if an object is moved to or from the examined Sandbox but existed somewhere else before/after
• VERSIONONLY for contributions changing only the Versionnumber. The Objective is to never get empty results because every change should be monitored (this might also create the need for other Types I cannot think of now).
• Be more detailed on TAGCHANGE . The Code already exists here: https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/oshdb/apps/MMNepalAnalyses/blob/master/src/main/java/org/heigit/missingmaps/nepalanalyses/analyses/lambdaimplementations/followup/FMapper.java#L170 and might keep users from coding it again themselves

please consider the discussion on https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/118

A definite debate is needed on this topic!

API-tutorial: wrong base class use for keytables

Following the API-tutorial I get the following compilation error when trying to inlcude key-tables:

"The method keytables(OSHDBJdbc) in the type MapReducer is not applicable for the arguments (OSHDBDatabase)"

A previous version of the tutorial used OSHDB_JDBC tagsDB = new [...] which is now replaced by OSHDBDatabase tagsDB = [...]. Switching back to the (now renamed) OSHDBJdbc tagsDB = new [...] does not solve the problem.

Solve infer warnings

see #91
Results are stored here: https://reports.ohsome.org/oshdb/ (see e.g. https://reports.ohsome.org/oshdb/0.5.0-SNAPSHOT_master/204_c11a8e02/infer/bugs.txt)

• oshdb-tool
• oshdb-api

OSHDB2Ignite errors?

Using a default ignite-config.xml I get following StackTrace:

org.apache.ignite.IgniteException: Cache doesn't exist: grid_node
    at org.apache.ignite.internal.util.IgniteUtils.convertException (IgniteUtils.java:990)
    at org.apache.ignite.internal.cluster.IgniteClusterImpl.changeWalMode (IgniteClusterImpl.java:550)
    at org.apache.ignite.internal.cluster.IgniteClusterImpl.disableWal (IgniteClusterImpl.java:531)
    at org.heigit.bigspatialdata.oshdb.tool.importer.util.OSHDB2Ignite.doGridImport (OSHDB2Ignite.java:79)
    at org.heigit.bigspatialdata.oshdb.tool.importer.util.OSHDB2Ignite.load (OSHDB2Ignite.java:58)

but changing the order of https://github.com/GIScience/oshdb/blob/master/oshdb-tool/etl/src/main/java/org/heigit/bigspatialdata/oshdb/tool/importer/util/OSHDB2Ignite.java#L79 and https://github.com/GIScience/oshdb/blob/master/oshdb-tool/etl/src/main/java/org/heigit/bigspatialdata/oshdb/tool/importer/util/OSHDB2Ignite.java#L81 does not help because in dafault-config, persistance is disabled. So I guess WAL-Mode should be checked before disabeling it? I just don't know how to :-(

org.apache.ignite.IgniteException: Cannot change WAL mode because persistence is not enabled for cache(s) [caches=[grid_node], dataRegion=null]
    at org.apache.ignite.internal.util.IgniteUtils.convertException (IgniteUtils.java:990)
    at org.apache.ignite.internal.cluster.IgniteClusterImpl.changeWalMode (IgniteClusterImpl.java:550)
    at org.apache.ignite.internal.cluster.IgniteClusterImpl.disableWal (IgniteClusterImpl.java:531)
    at org.heigit.bigspatialdata.oshdb.tool.importer.util.OSHDB2Ignite.doGridImport (OSHDB2Ignite.java:79)
    at org.heigit.bigspatialdata.oshdb.tool.importer.util.OSHDB2Ignite.load (OSHDB2Ignite.java:58)

Literature suggests error in Contributions pre 2011

This article (Neis and Zipf (2012) Analyzing the Contributor Activity of a Volunteered Geographic Information Project — The Case of OpenStreetMap, https://www.mdpi.com/2220-9964/1/2/146) suggests that users might have increased the version of an element without touching the element (p155 bottom)¹. The text is rather unclear on what editors are affected and what the error exactly does.

• It should therefore be investigated, what this error does and if the users of oshdb can be notified/prevented from this error.

¹ "[...] It is important to note that for this particular method to determine the activity area polygon of a member, only Nodes that a member created were included, no edited Nodes or deleted Nodes were considered. Initial calculations that included all Nodes showed some irregularities, which were based on a software error in the OSM editors in the past (before 2011). This error increased the version number of a Node although the object was not changed in any way by any user directly, but because the Node would fall into the range of a certain changeset. Thus, the database would count a change to a Node, although the member did not actually edit the data. It is important to consider these errors when conducting similar studies to [32,40,43,44], in which the versions of an OSM object should be based on real changes and not primarily on the number of editors and the absolute version number. [...]"

Update Changelog

Document changes of v0.5.0 in Changelog so users can easily update.

• breaking changes
• new features
• performance
• other changes

Solve JavaDoc Warnings

As of now there are 36 JavaDoc warnings recognised by jenkins.
Note: Fixing these does not mean we have god javadoc. Nor does it make our javadoc doclint compatible.
The following modules need fixing:

• oshpbf-parser
• oshdb

Add getters to MapReducer/MapAggregator

opened by @franz-benjamin with title mapReducer.getTimestamps

It would be nice to have mapReducer.getTimestamps(), mapReducer.getAreaOfInterest(), etc.

see https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/150

Archive GitLab repository

clone of https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/153

• resolve all issues,
• mergerequests,
• branches etc. around gitlab repository
• merge tag 0.5.0 into GitLab master as last state
• close last issue on GitLab
• archive repository

add warning when versions of an osh entity are missing

opened by @tyrasd

due to some data redactions, there might still be some missing data versions in our .osh.pbf files (and therefore in our oshdb files). For example, due to https://www.openstreetmap.org/changeset/59473904 we're missing versions 67 to 187 of https://www.openstreetmap.org/relation/192734:

r192734 v66 dV c13067600 t2012-09-11T10:43:56Z i714035 unamuori …
r192734 v188 dD c59473904 t2018-06-02T00:23:36Z i5164 uwoodpeck T M
r192734 v189 dV c59490832 t2018-06-02T17:47:36Z i5164 uwoodpeck …

see https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/148

write tests for CellIterator methods

opened by @tyrasd

related to https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/52:

write tests that individually check the capabilities of the two CellIterator functions:

• iterateByTimestamps
• iterateContributions

Needs to check all functionality, e.g.

• area of interest filter
• oshentity prefilter
• osmentity filter
• geometry building
• …

see https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/53

Verify correct behaviour of benchmarks

Benchmark b2 has different checksums for Ignite and H2 (see db-1 postgres server benchmark results). This may point to an error in the code we should investigate.

solve java warnings

Static Code Analyses reports 10 Errors in modules etl and ospbf-parser: https://jenkins.ohsome.org/job/oshdb/job/master/171/java/
most of the are resource leaks.

• solve etl
• solve oshpbf-parser

query progress updates from ignite

opened by @tyrasd

via kafka or ignite event subscription system

see: https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/80

changesets

see https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/7

• set up service to import changesets (with comments) into postgres: https://github.com/ToeBee/ChangesetMD
• add query-class like TagTranslator

Solve PMD Warnings

see #91

• etl
• oshdb
• oshdb-api
• oshdb-util
• oshpbf-parser

Fix handling of special geometry cases in OSHDBGeometryBuilder

See: https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/81

• move specific branches to GitHub
• rebase branches
• fix issues -> https://github.com/GIScience/oshdb/projects/2
• merge

OSHDBGeometryBuilder - timestamp before first appearance

timestamp before first appearance
Test 1 testdata: A single node, lat lon changed.
Trying to get coordinates before first timestamp fails:
java.lang.AssertionError: cannot produce geometry of entity for timestamp before this entity's version's timestamp

spatial "proximity" queries

opened by @tyrasd

Feedback from today's workshop: spatial proximity queries are a very important topic for geographers. An example query would be: calculate all benches that are inside of a park and in the vicinity of a tree and have an ice cream shop nearby 🍨

//cc @franz-benjamin @redfrexx

!!! please consider the detailed comments: https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/104 !!!

The following steps need to be taken, to merge this:

• More test-cases (e.g. a mapping of OSHDBSnapshot-> XYZ before the match() would result in an Exception and needs to be asserted)
• reduce number of introduced functions: we propose one function e.g. .spatialJoin(...) that takes the type of join as an argument (instead of one method per type)
• clean up code
  • there is a lot of commented out code
  • assure the new code meets the quality standards set out in jenkins
• Implementation for SnapshotView has priority. A solution for ContributionView should be thought of (there is one in the code which needs to be reviewed in detail)
• possible other open questions should be discussed in a personal meeting
• this issue needs an assignee and a timestamp (milestone)
• "harden" the API against potential misuse (e.g. returning a non matching mapreducer in the spatial relation callback functions); or if possible: change the API to not allow such cases

Solve CPD Warings

see #91

• etl
• oshdb
• oshdb-api
• oshdb-util
• oshpbf-parser

[WIP] Updater

Write update-package that fulfils framework set out here: https://confluence.gistools.geog.uni-heidelberg.de/display/oshdb/Update+process+framework

Missing implementations are:

Updater Package

• implement different backends (ignite, h2 etc)
• insert new tags into tagtranslator-db BEFORE creating new entity
  • update TagTranslator cache if new tag or role is inserted to keytables or handle dublicate insertion exception
• account for back-references of OSHEntites (e.g. ways for nodes for changed ways and changed nodes)
  • account for back-references of backreferences of OSHEntites (e.g. relations depending on ways depending on changed nodes)
  • Relations are excluded for now, but might get important in a later update? @rtroilo
• implement flushing of updates to real database
• implement Kafka promoter (do we actually want this, its not hard to do but also a feature with no use-case so far)
• collect OSHEntites before processing to avoid multiple calls to same method
  • test if this actually is an improvement
• collect OSHEntites into insertCells before flushing to prevent multiple overwriting of the same Cell
• resolve comments in code
• assure concurrency (most of all, check if locking in flusher is safe)
• optimise Code (very important last step!)
• analyse and clean dependencies
• extensive testing!
• codeStyle, staticCodeAnalyses etc (resrouce handling)
• documentation
• correct errors introduced by 2ab2f88
  • check if functionality was preserved
• there might be a mixup of CellId.getId() and CellId.getLevelId() in the Flusher and possibly elswhere! Should be investigated!

OSHDB-API

• write class to append entities from update-db to stream when querying over OSHDB-API (I need help to point me to the right place where this would be implemented @tyrasd ?)
  • implement ScanQuery and LocalPeek OSHDB-API-Ignite Backends
  • move streaming of updates up do MapReducer-Class. It is only copy code now.
  • group 100 or 1'000 entities together in one 0,0 GridCell in OSHDB-API to avoid overhead of CellCreation (and calling of celliterator methods)
• improve testData for updates to speed them up (Area to big atm)

ETL-Backend

• implement ETLFileReaderClass (help needed from @rtroilo )

Dependencies

• check if ignite-geospatial is actually necessary as a dependency or only needs to be available at cluster startup on the server side. If so, remove artifact from artifactory and remove dependency.
• check if h2gis is necessary as a dependency

Newer JTS-Version available

Most of our code uses vividsolutions JTS stopping at version 1.14 but locationtechs (who baught the project) jts is at 1.16

Should we update?

OSHDBContribution evaluator

A Class that takes OSHDBContributions returns more detailed information like the type of tagChange or memberlistChange etc.

spin-off from https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/118

see: https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/120

Helpful Methods:

• evaluateTags to get a detailed view of the tags changed
• evaluateGeometries get a detailed view of the geometryChanges. This should be rather generic. Default methods are
  • area changes
  • length changes
  • movement
  • objectnr. (geomCollect.)
  • experimental!: rounder, more squared (helpful for building analyses)
  • other ideas? ...

OSM-Notes

In addition to OSM-Data, notes (with a T) are comments on the map by registered or non registered users similar to the fixme-tag. Similar to changesets (#52) they have a geographic location, text and comments.
Although their number is limited they might be interesting for future research.

useful link to load data into postgres: https://github.com/mapbox/osm-comments-parser

Standardised and extraordinary test data

OSHDB-API holds OSM data for unit-tests. This resource should be

• standardised (one type (xml), one place, one file...)
• have a standard transformation into OSH and GridOSH-Objects
• extended with use-cases from ETL

Solve SpotBugs Warnings

see #91

• etl
• oshdb
• oshdb-api
• oshpbf-parser
• oshdb-util

stream results

Sometimes, one just wants to stream non-aggregated data (e.g. to write it out into a file of DB), where the current workaround (of .collect()-ing all results into a temporary list and then streaming it out) is not perfect because it unnecessarily requires quite a lot of memory. There could be a .stream() method in the MapReducer which could return (chunks of) results as soon as they are ready.

Maybe this could also expanded to the MapAggregator class, where instead of a single stream one would get a set of streams?!

Implement Unit-Tests

Like #91 we use as static analyses to check for code that is not covered by unit-tests.
Results can be seen here: https://reports.ohsome.org/oshdb/0.5.0-SNAPSHOT_master/204_c11a8e02/jacoco/

• oshdb
• oshdb-api
• oshdb-tool
• oshdb-util

Better mapping of Way/Relation to Cells

opened by @tyrasd

Currently, the bounding box will be calculated according to the bounding boxes for all of his nodes(over the complete history). This could lead to big bounding boxes for a way.
for example if a node before or after the lifespan of the way is used in an other location.

Better: calculate the boundingbox only for the lifespan of the way/relation.

see https://gitlab.gistools.geog.uni-heidelberg.de/giscience/big-data/ohsome/oshdb/issues/8

Endless loading in OSHRelation.getModificationTimestamps()

When sending a simple request over the ohsome-API about bicycle routes for a small bbox in Rheinland-Pfalz (Germany), the code seems to be somehow stuck (or needs far too much time) within the OSHRelation.getModificationTimestamps() method. The reason seems to be that the relation https://www.openstreetmap.org/relation/88376#map=9/49.9658/7.3759 is quite big and has lots of versions and members with loads of different timestamps.

This is the request URI: /elements/count?time=2012-06-15&showMetadata=true&keys=route&values=bicycle&bboxes=7.6563,49.4635,7.7727,49.513&types=relation

OSHDBGeometryBuilder - history testdata: invalid MultiPolygon: GeometryCollection is empty

empty geometryCollection
Test testGeometryCollection()
testdata
relation, not valid, should be a not empty geometryCollection and not instanceOf MultiPolygon.