openhistoricaldatamap/ohdmconverter

ohdmconverter's Issues

bug import relation

There is malformed insert statement when extracting relations. See last berlin log in importa/berlin/logs/berlin.i2d

not an integer in admin_level: 9.5

Some boundaries are declared as non integer like 9.5 or 8,9. Guess, we should simly round it down. Sometimes just names are used like "Kent" "civil_parish". Strings an only be ignored I'm afraid

test task osm archive

test

old_names

Partially, there are old_names tag describing former names of a building or most usually places. Validity of old names are described by years or dates like 1789-1794. Those tags clearly indicate the existing of that building etc. and could be used to create OHDM entries.

See osm wiki

remove count(osm_id) in set up

First activity in inter2ohdm step is to figure out min and max id on e.g. nodes table.
And we also count number of lines. That takes (even with index) up to half an hour.
That counter is only required for log messages - find a better way.

double tagging

In some cases, osm elements are tagged with more than one feature class. Currently, the last feature class rules. We should remember all feature classes in intermediate db. During extraction, for each feature class an line in geoobject_geometry table will be produced - each for each feature class.

import osm changes

We should import osm change reports beside that update process. Qicker reponse to users who maybe added historic information.

Performance

We threads over a single JDBC connection to increase performance. That's not really satisfying. It does not really scale. More than one JDBC connection would be better - maybe also several processes instead of several threads. I have the impression that scaling / scheduling of those long-running processes can be optimized better based on processes that threads. Just a guess.

Current situation is hardly bearable.

hanging process

chunk import: some processes are hanging.. rarley happens but happens. 1st step - get aware of that situation e.g. in the 5 minute report?

polygons to huge

There are cases in which buildings somehow being combined to a huge polygon...
Guess, it is a extraction problem but no sure. see apparement close to WH campus.

Build this tool please

We have some prototype projects - webbased apps mainly which integrate shapefiles etc. into ohdm. I think we should deconstruct this process even more to allow integration into other (batch) processes. First, we need an extension to this converter tool that allows importing geometries from files and some metainformation into an OHDM database.
Would it be a good idea to separate OHDM DB which is filled with OSM data from imported data. I do think so. Geometries can be combined in the rendering, or whatever databases... It would at least offer more flexibility. We would even have a way from shapefile to GeoSPARQL.

sqlStatement too long during planet import..

Happend while importing relations into intermediate db.
Apparently, the sql statement was too long in exec.

OK, that import variant shall be replaced by dump import.
Anayway, it can cause problems also in inter2ohdm import.
Check max length of an sql statement for prepareStatement.

Also: Check when an exec thread is created from sql queue.

That's the tail of the err log of import ended 2017, Jan, 16th.

import from osm file into intermediate db
Exception in thread "Thread-446720582" java.lang.ArrayIndexOutOfBoundsException
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:597
)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at org.postgresql.core.Parser.parseSql(Parser.java:1026)
at org.postgresql.core.Parser.replaceProcessing(Parser.java:972)
at org.postgresql.core.CachedQueryCreateAction.create(CachedQueryCreateA
ction.java:41)
at org.postgresql.core.CachedQueryCreateAction.create(CachedQueryCreateA
ction.java:17)
at org.postgresql.util.LruCache.borrow(LruCache.java:115)
at org.postgresql.core.QueryExecutorBase.borrowQuery(QueryExecutorBase.j
ava:266)
at org.postgresql.jdbc.PgConnection.borrowQuery(PgConnection.java:143)
at org.postgresql.jdbc.PgPreparedStatement.(PgPreparedStatement.ja
va:88)
at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:1
256)
at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:1
622)
at org.postgresql.jdbc.PgConnection.prepareStatement(PgConnection.java:4
15)
at util.SQLExecute.doExec(SQLExecute.java:41)
at util.SQLExecute.run(SQLExecute.java:62)

unclosed polygons / ways

There is a reasonable number of relations describing polgyons which are not closed. It seems the endpoint are quite close but not identically. Would be a good idea to create a polygon if both endpoints are not that far away. What means "that far away?".

administrative boundaries import / extraction buggy

Adminstrative boundaries are mainly maintained by relations. There are relations in Intermediate DB and also in ODHM but no related ways... 1. Import must recognize admin level which should actually already work. Does it? 2. Extraction does not handle non-multipolygons correctly.

Polygons not valid

Some polygons are not valid - DB OHDM / table polygons. Apparently, import from intermediate is buggy

remove log message null value in uid / user

places of country / city names

We have to calculate the position of country names in some way. Unfortunately, boundaries are quite often described by line instead of polygons.

There are two options:
a) Produce a polygon out of those linestrings.
b) find another source to solve that problem.

I would vote for a) right now.

big geometries?

OHDM takes OSM geometries descriptions and creates OGC geometries out of it. It does no change those fain grained geometries, though. Especially, relations sometimes describe huge geometries like borders of a whole country. I wonder if it would be a good idea to transform those huge collection of little geometries into a single huge geometry

addNodes - smarter implementation?

AddNodes in OSMExtractor add nodes to ways. That makes perfect sense during way extraction.
A way geometries might already exist during relation extraction. Could that geometry be used instead of re-constracting that way from nodes. Could e.g. increase speed of addNodes which is rediculous bad (takes almost a second (relation import July, 2nd Germany (inter)-> integration db)