claeis/iox-ili

implementation of IOX API

iox-ili's Issues

Iox2wkb only generates binaries for 3D geometries in Extended Well-Known Binary format

In the class Iox2wkb, on line 912 is added the flag WKBConstants.ewkbIncludesZ when the geometry is 3D. But that means it is always added the Extended Well-Known Binary format.

Would it be possible to choose the standard Well-Known Binary format?

review the focus of the surface/area tests

Some of the tests
Area10Test
Area23Test
Surface10Test
Surface23Test
report multiple "cut edge" problems. Is this the intention or a bug or a bad test?

implement an XTF24 reader

remove string comparison in asserts in Iox2jtsTest

log message: LineRing not closed, is output on the wrong place

Falls die RingLine nicht geschlossen wurde, wird die Konvertierung von iox nach jts wird aufgrund der
Klasse: jts.geom.LinearRing und dessen Methode: validateConstruction() und
Ausgabe: Points of LinearRing do not form a closed linestring, wird richtig durchgeführt.

Die Fehlermeldung soll zu späterer Zeit in der Klasse: ItfSurfaceLinetable2Polygon in der Meldung: invalid ring line... ausgegeben werden.

Support remote plugins repository

Will be useful to have a centralized plugin repository, similar to the models.interlis.ch repository. It can have the same logic implemented on RepositoryAccess. Has this been discussed before?

create a new logErr output message, if 2 linesegments overlay each other

implement a GML reader

implement eCH-0118-2.0 encoding rules

implement generic/meta configuration from data repositories

Example

[CONFIGURATION]
baseConfig=ilidata:remoteBaseConfigBasketId;ilidata:otherRemoteBaseConfigBasketId;file:/path/otherBaseConfigLocalFile
org.interlis2.validator.config=ilidata:ilivalidatorConfigBasketId
ch.interlis.referenceData=ilidata:catalogBasketId;file:/path/localFile.xml

• Remote file are referenced by ilidata: and basketId
• multiple files can be referenced. It's up to the tool (e.g. ilivalidator) how it handles it. Default behaviour should be: later definitions override earlier ones.

implement result caching in areAreas(), areAreas2(), areAreas3()

unnecessary output of 2 different logErr messages by intersection of 2 boundaries

in the following tests

• Area10Test: twoInnerboundaryLinesCutEachOther_Fail
• Area23Test: twoInnerboundaryLinesCutEachOther_Fail

are 2 different logErr messages reported:

• 2 intersection tid messages
• 2 intersection overlap messages
  which are the correct output messages?

Area23Test.onePolygon_1Boundary_Area_RoundingTo3DecPoints_Ok

improve so that correct rounding is verified by input that is only rounded valid but not rounded invalid

implement a CSV reader

Does not compile on macOS sierra with Oracle Java 1.8

Find attached a patch.

Btree.diff.zip

add GenericReader

I can't compile with instructions

I'm reading this docs
https://github.com/claeis/iox-ili/blob/master/doc/README.src.txt
But i can't complete successfully the compilation with "$ ant jar " command.

improve asserts in Xtf23Reader tests

unnecessary "overlap null" text in intersection log message if no overlap detected

expected:

• wenn overlap grösser als 0.0: Intersection overlap 0.1, coord1 (500000.0, 77000.0, NaN), tid1 o1, tid2 o2.
• wenn overlap gleich null: Intersection, coord1 (500000.0, 77000.0, NaN), tid1 o1, tid2 o2.

actual:

• wenn overlap grösser als 0.0: Intersection overlap 0.1, coord1 (500000.0, 77000.0, NaN), tid1 o1, tid2 o2.
• wenn overlap gleich null: Intersection overlap null, coord1 (500000.0, 77000.0, NaN), tid1 o1, tid2 o2.

Reading multiple BAG OF Struct Values fails in XTF24Reader

ilivalidatorlog:

Info: ilivalidator-1.11.12-5186e3407dc72a6947c33d84a3ac86c20c555cc2
Info: ili2c-5.2.3-3ec7a10e9bacc87e149a14a7f11a6035e0481c69
Info: iox-ili-1.21.8-a6aee68136447c0616995e56888fa9c7ef362a77
Info: User <pluethi>
Info: Start date 2021-11-29 13:35
Info: maxMemory 253440 KB
Info: dataFile <C:\Users\pluethi\Desktop\TestDataBagOf.xtf>
Info: modeldir <%ITF_DIR;http://models.interlis.ch/;%JAR_DIR/ilimodels;C:\Projects\geow-interlis-functions\src\test\data\GetLength\SetConstraintBagOf.ili;C:\Projects\geow-interlis-functions\src\model>
Info: lookup model <TestSuite> 2.4 in repository <C:\Users\pluethi\Desktop/>
Info: ilifile <C:\Users\pluethi\Desktop\SetConstraintBagOf.ili>
Info: validate data...
Info: assume unknown external objects
Info: first validation pass...
Info: C:\Users\pluethi\Desktop\TestDataBagOf.xtf: TestSuite.FunctionTestTopic BID=TestSuite.FunctionTestTopic
Error: line 5: unexpected element: {http://www.interlis.ch/xtf/2.4/TestSuite}polylineAttribute in TestSuite.FunctionTestTopic.BaseClass
Info: ...validation failed

TestData.zip

fix mismatch between newVertexDistance and minimalVertexDistance

Disabling multiple multiplicity constraints are not logged but first

When disabling multiple multiplicity constraints in a toml file, only the first one is logged.

TOML file:

["ModelA.TopicA.ClassA.AttributeA"]
multiplicity="off"

["ModelA.TopicA.ClassA.AttributeB"]
multiplicity="off"

ILI file:

CLASS ClassA =
 AttributeA: MANDATORY TEXT*20;
 AttributeB: MANDATORY TEXT*20;
END ClassA;

resulting log file:

...
Info: validate data...
Info: first validation pass...
Info: ModelA.TopicA.ClassA.AttributeA not validated, validation configuration multiplicity=off
Info: second validation pass...
...
Info: ...validation done

So before the second validation pass I expect an additional line in the log file that says that AttributeB is not validated because multiplicity=off.

I looks like the reason for this behavior is the way the recently introduced HashSet validationConfigOff is used.
As can be seen in these lines in Validator.java, the string "multiplicity" is added to the HashSet validationConfigOff and all subsequently processed attributes are therefore ignored (regarding logging of multiplicity).

Changing this behavior to adding the qualified name of the attribute to validationConfigOff , e.g.

validationConfigOff.add(attrQName + ":" + ValidationConfig.MULTIPLICITY);

and afterwards checking for the status for exactly this attribute

if(!validationConfigOff.contains(attrQName + ":" + ValidationConfig.MULTIPLICITY)) ...

will resolve the problem.

Probably there are more elegant ways to do this. But this has worked for me without altering the definition of the HashSet.

The same issues persists with other validation checks, e. g. role cardinality (and more?)

Thanks,
Stefan

implement a CSV Writer

two different numbers of logErr output messages by cutting of 2 boundaries

• Surface10Test:
  in this test is only reported 1: intersection tids o1, o1 message.

• Surface23Test:
  in this test are reported 2: intersection tids o1, o1, intersection tids o1, o1 messages.

which output is the correct one?

Iox2jtsext und Iox2jts erzeugen doppelte Koordinaten

Falls ein IomObjekt mehrere Polylines hat, erzeugen die beiden Klassen doppelte Koordinaten:

        Iom_jObject objSurface = new Iom_jObject(ILI_CLASSE, OBJ_OID1);
        IomObject multisurfaceValue = objSurface.addattrobj("gebietseinteilung", "MULTISURFACE");
        IomObject surfaceValue = multisurfaceValue.addattrobj("surface", "SURFACE");
        IomObject outerBoundary = surfaceValue.addattrobj("boundary", "BOUNDARY");

        IomObject polylineValue1 = outerBoundary.addattrobj("polyline", "POLYLINE");
        IomObject segments1 = polylineValue1.addattrobj("sequence", "SEGMENTS");
        IomObject startSegment1 = segments1.addattrobj("segment", "COORD");
        startSegment1.setattrvalue("C1", "0.000");
        startSegment1.setattrvalue("C2", "0.000");
        IomObject endSegment1 = segments1.addattrobj("segment", "COORD");
        endSegment1.setattrvalue("C1", "0.000");
        endSegment1.setattrvalue("C2", "10.000");

        IomObject polylineValue2 = outerBoundary.addattrobj("polyline", "POLYLINE");
        IomObject segments2 = polylineValue2.addattrobj("sequence", "SEGMENTS");
        IomObject startSegment2 = segments2.addattrobj("segment", "COORD");
        startSegment2.setattrvalue("C1", "0.000");
        startSegment2.setattrvalue("C2", "10.000");
        IomObject endSegment2 = segments2.addattrobj("segment", "COORD");
        endSegment2.setattrvalue("C1", "5.000");
        endSegment2.setattrvalue("C2", "10.000");

        IomObject polylineValue6 = outerBoundary.addattrobj("polyline", "POLYLINE");
        IomObject segments6 = polylineValue6.addattrobj("sequence", "SEGMENTS");
        IomObject startSegment6 = segments6.addattrobj("segment", "COORD");
        startSegment6.setattrvalue("C1", "5.000");
        startSegment6.setattrvalue("C2", "10.000");
        IomObject endSegment6 = segments6.addattrobj("segment", "COORD");
        endSegment6.setattrvalue("C1", "10.000");
        endSegment6.setattrvalue("C2", "10.000");

        IomObject polylineValue7 = outerBoundary.addattrobj("polyline", "POLYLINE");
        IomObject segments7 = polylineValue7.addattrobj("sequence", "SEGMENTS");
        IomObject startSegment7 = segments7.addattrobj("segment", "COORD");
        startSegment7.setattrvalue("C1", "10.000");
        startSegment7.setattrvalue("C2", "10.000");
        IomObject endSegment7 = segments7.addattrobj("segment", "COORD");
        endSegment7.setattrvalue("C1", "10.000");
        endSegment7.setattrvalue("C2", "0.000");

        IomObject polylineValue8 = outerBoundary.addattrobj("polyline", "POLYLINE");
        IomObject segments8 = polylineValue8.addattrobj("sequence", "SEGMENTS");
        IomObject startSegment8 = segments8.addattrobj("segment", "COORD");
        startSegment8.setattrvalue("C1", "10.000");
        startSegment8.setattrvalue("C2", "0.000");
        IomObject endSegment8 = segments8.addattrobj("segment", "COORD");
        endSegment8.setattrvalue("C1", "0.000");
        endSegment8.setattrvalue("C2", "0.000");

POLYGON ((0 0, 0 10, 0 10, 5 10, 5 10, 10 10, 10 10, 10 0, 10 0, 0 0))

Korrekt wäre:

POLYGON ((0 0, 0 10, 5 10, 10 10, 10 0, 0 0))

improve PlausibilityConstraints23Test

Only four tests are required.

• 1 ok >=: 80% valid objects, 60% required
• 1 failing >=: 40% valid objects, but 60% required
• 1 ok <=: 40% valid objects, 60% required
• 1 failing <=: 80% valid objects, but 60% required

Remove the other, superfluous, ones.

Fix comments about what the tests test.

fix asserts in SURFACE/AREA tests (and improve description)

AdditionalConstraints23Test. mandatoryConstraint_OverlappedSurface_False
Area10Test. area2dInnerboundariesOverlapEachOther_Fail
Area10Test. area2dInnerboundaryIntersectOuterboundary_Fail
Area10Test. area2dOverlapOf2Lines_Fail
Area10Test. area3dInnerboundaryIntersectOuterboundary_Fail
Area10Test. area3dIntersectionOfInnerboundaries_Fail
Area10Test. area3dOverlapOf2Lines_Fail
Area23Test. area2dInnerIntersectOuterboundary_Fail
Area23Test. area2dIntersectionOf2Lines_Fail
Area23Test. area2dIntersectionOfInnerboundaries_Fail
Area23Test. area3dIntersectionOfInnerboundaries_Fail
Area23Test. area3dIntersectionOfOuterboundaries_Fail
Area23Test. area3dIntersectionOfOwnOuterboundary_Fail
Surface10Test. surface3dIntersectionErrs_Fail
Surface10Test. surface3dInvalidRingLines_Fail
Surface10Test. surface3dOverlapOf2Lines_Fail
Surface10Test. surfaceIntersectionErrs_Fail
Surface10Test. surfaceInvalidRingLines_Fail
Surface10Test. surfaceOverlapOf2Lines_Fail
Surface23Test. surface2DIntersectionErrs_Fail
Surface23Test. surface2dIntersectionOfInnerboundaries_Fail
Surface23Test. surface2dOverlabOf2Lines_Fail
Surface23Test. surface3dIntersectionOfInnerOuterBoundary_Fail
Surface23Test. surface3dInvalidRingLines_Fail
Surface23Test. surface3dOverlabOf2Lines_Fail

switch config file reader to pure ini file syntax

but allow for " around names and values

too many cut-edge messages are emitted by cutting of different boundaries

Tests:

• Surface10Test: lineOfInnerBoundaryLiesOverLineOfOuterBoundary_Fail
• Surface23Test: lineOfInnerBoundaryLiesOverLineOfOuterBoundary_Fail
• Area10Test: lineOfInnerBoundaryLiesOverLineOfOuterBoundary_Fail
• Area23Test: lineOfInnerBoundaryLiesOverLineOfOuterBoundary_Fail

validator: fix UNIQUE contsraint with PathEl to container object

see UniqueConstraints23Test.uniqueAttrValuesOfAttrPath_InEmbeddedAssociation_Fail() test case

Area23Test.area2dIntersectionOfBoundaries_False fails

improve event handling of Xtf24Reader like Xtf23Reader

automate tests

Xtf23Reader: fix BLACKBOX parser

implement a new Xtf23Reader

handle missing/no main obj in itf

Validator: failed to detect wrong type in COORD/POLYLINE/SURFACE attr

implement a GML writer

implement eCH-0118-2.0 encoding rules

Xtf23Reader/Xtf24Reader: support embedded association with attributes

add multigeometry converters to Iox2jts

add to class Iox2jts in folder of path: 'ch.interlis.iox_j.jts.Iox2jts'
•multicoord2JTS
•multipolyline2JTS
•multisurface2JTS

missing custom functions should not report an error

but a warning

CsvReader cannot read tab separated files

Eventuell auch eine Konstellation von unterschiedlichen Betriebssystem und exotischen Encodings. Sollten Tabs grundsätzlich funktionieren?

task validateCsv(type: CsvValidator, dependsOn: 'downloadData') {
    models = "SO_AGI_MOCheckSO_20200715"
    firstLineIsHeader = true
    valueDelimiter = null
    valueSeparator = '\t'
    encoding = "windows-1252"
    dataFiles = [pathToTempFolder.toString()+"/mocheckso_errors.csv"]
}

Auch habe ich versucht den Tab aus Notepad++ zu kopieren, was aber auch nicht funktioniert. Anbei die CSV-Datei.

mocheckso_errors.zip

No custom warnings message tests found in Configuration23Test class

In der Klasse Configuration23Test wurden keine Tests gefunden die folgendes Scenario testen:

• downgrade von Error --> Warnung
• ValidationConfig.MSG = ""
• ValidationConfig.MSG = "not empty"

folgende Tests müssen ergänzt werden mit:

• additional constraints
• existence constraints
• mandatory constraints
• plausibility constraints
• set constraints
• unique constraints
  sowie functions in mandatory constraints.

CsvWriterTest: replace use of CsvReader with java.io.LineNumberReader

add multigeometry converters to Jts2iox

add to class Iox2jts in folder of path: 'ch.interlis.iox_j.jts.Jts2iox'
•JTS2multicoord
•JTS2multipolyline
•JTS2multisurface

fix CurvePolygonOverlayTest

• twoPolygon_2ArcsOfDiffPolygonOnEachOther_Clockwise_False()
• twoPolygon_2ArcsOfDiffPolygonOnEachOther_Counterclockwise_False()

improve focus of SET constraint tests

Unknown WKB type 8

Exporting a dataset with MultiSurface from postgis, ili2pg errors out when a curvepolygon contains a circularstring directly (not wrapped in a compoundcurve)

Error: line 0: Administrative_Einteilung_LV95_V1_0.Administration.Einteilung: tid 731: Attribute Geometrie has wrong number of values
Error: Object 732: failed to convert surface/area
Error:   com.vividsolutions.jts.io.ParseException: Unknown WKB type 8
Error:       ch.ehi.ili2pg.converter.PostgisColumnConverter.toIomMultiSurface(PostgisColumnConverter.java:450)
Error:       ch.ehi.ili2db.toxtf.ToXtfRecordConverter.addAttrValue(ToXtfRecordConverter.java:629)
Error:       ch.ehi.ili2db.toxtf.ToXtfRecordConverter.convertRecord(ToXtfRecordConverter.java:441)
Error:       ch.ehi.ili2db.toxtf.TransferToXtf.dumpObjHelper(TransferToXtf.java:1104)
Error:       ch.ehi.ili2db.toxtf.TransferToXtf.dumpObject(TransferToXtf.java:1080)
Error:       ch.ehi.ili2db.toxtf.TransferToXtf.doBasket(TransferToXtf.java:564)
Error:       ch.ehi.ili2db.toxtf.TransferToXtf.doit(TransferToXtf.java:293)
Error:       ch.ehi.ili2db.base.Ili2db.transferToXtf(Ili2db.java:2300)
Error:       ch.ehi.ili2db.base.Ili2db.runExport(Ili2db.java:1720)
Error:       ch.ehi.ili2db.base.Ili2db.run(Ili2db.java:225)
Error:       ch.ehi.ili2db.AbstractMain.domain(AbstractMain.java:554)
Error:       ch.ehi.ili2pg.PgMain.main(PgMain.java:70)
Error:     Unknown WKB type 8
Error:         ch.interlis.iox_j.wkb.Wkb2iox.readGeometry(Wkb2iox.java:187)
Error:         ch.interlis.iox_j.wkb.Wkb2iox.readCurvePolygon(Wkb2iox.java:386)
Error:         ch.interlis.iox_j.wkb.Wkb2iox.readMultiSurface(Wkb2iox.java:469)
Error:         ch.interlis.iox_j.wkb.Wkb2iox.readGeometry(Wkb2iox.java:185)
Error:         ch.interlis.iox_j.wkb.Wkb2iox.read(Wkb2iox.java:131)
Error:         ch.interlis.iox_j.wkb.Wkb2iox.read(Wkb2iox.java:112)
Error:         ch.ehi.ili2pg.converter.PostgisColumnConverter.toIomMultiSurface(PostgisColumnConverter.java:448)
Error:         ch.ehi.ili2db.toxtf.ToXtfRecordConverter.addAttrValue(ToXtfRecordConverter.java:629)
Error:         ch.ehi.ili2db.toxtf.ToXtfRecordConverter.convertRecord(ToXtfRecordConverter.java:441)
Error:         ch.ehi.ili2db.toxtf.TransferToXtf.dumpObjHelper(TransferToXtf.java:1104)
Error:         ch.ehi.ili2db.toxtf.TransferToXtf.dumpObject(TransferToXtf.java:1080)
Error:         ch.ehi.ili2db.toxtf.TransferToXtf.doBasket(TransferToXtf.java:564)
Error:         ch.ehi.ili2db.toxtf.TransferToXtf.doit(TransferToXtf.java:293)
Error:         ch.ehi.ili2db.base.Ili2db.transferToXtf(Ili2db.java:2300)
Error:         ch.ehi.ili2db.base.Ili2db.runExport(Ili2db.java:1720)
Error:         ch.ehi.ili2db.base.Ili2db.run(Ili2db.java:225)
Error:         ch.ehi.ili2db.AbstractMain.domain(AbstractMain.java:554)
Error:         ch.ehi.ili2pg.PgMain.main(PgMain.java:70)

Here is a short example to reproduce the problem
MULTISURFACE(CURVEPOLYGON(CIRCULARSTRING(0 0, 4 0, 4 4, 0 4, 0 0),(1 1, 3 3, 3 1, 1 1)))

factor interlis functions out of class Validator

fix failing Xtf23Reader tests

xtf23: create (in)valid overlap surface/area tests