GML Application Schema toolbox is a QGIS plugin allowing to manipulate OGC Complex Features
Home Page: https://brgm.github.io/gml_application_schema_toolbox/
License: GNU General Public License v2.0
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
![pre-commit-ci[bot] avatar](https://avatars.githubusercontent.com/in/68672?v=4 "pre-commit-ci[bot]")
Using plugin 1.1.0.zip
Importing the following URI (http://ressource.brgm-rec.fr/data/Piezometre/06512X0037/STREMY.2) using GMLAS triggers
1°/ this stack trace
note that contrary to the message '20170711_EF.sqlite' is created and
RuntimeError: SQLite driver failed to create D:/Temp/20170711_EF.sqlite and all the tables appear in the QGIs project
Traceback (most recent call last):
File "C:/Users/grellets/.qgis3/python/plugins\gml_application_schema_toolbox\gui\import_gmlas_panel.py", line 280, in on_convertButton_clicked
self.translate(self.import_params())
File "C:/Users/grellets/.qgis3/python/plugins\gml_application_schema_toolbox\gui\gmlas_panel_mixin.py", line 65, in translate
res = gdal.VectorTranslate(**params)
File "C:\OSGEO41\apps\Python36\lib\site-packages\osgeo\gdal.py", line 718, in VectorTranslate
return wrapper_GDALVectorTranslateDestName(destNameOrDestDS, srcDS, opts, callback, callback_data)
File "C:\OSGEO41\apps\Python36\lib\site-packages\osgeo\gdal.py", line 3025, in wrapper_GDALVectorTranslateDestName
return _gdal.wrapper_GDALVectorTranslateDestName(*args)
RuntimeError: SQLite driver failed to create D:/Temp/20170711_EF.sqlite
2°/ this error message when clicking on the point created using 'identify'
'Access violation - no RTTI data !'
I don't know if 100% related but part of the behaviour is the same
Importing that URI (https://forge.brgm.fr/svnrepository/epos/trunk/instances/BoreholeView.xml) in GMLAS triggers the stack trace below eventhough the sqlite file is created and the project loaded in the GUI (but I can't do nothing with the content).
Stack trace :
File "C:/Users/grellets/.qgis3/python/plugins\gml_application_schema_toolbox\gui\import_gmlas_panel.py", line 288, in on_convertButton_clicked
'postgres' if self.databaseWidget.format() == 'PostgreSQL' else 'spatialite')
File "C:/Users/grellets/.qgis3/python/plugins\gml_application_schema_toolbox\core\load_gmlas_in_qgis.py", line 79, in import_in_qgis
raise RuntimeError("SQL error")
RuntimeError: SQL error
add a save button to export the temporary file on disk
Hi there, first of all, this is realy great stuff you're doing here!
I'm trying to use the GMLAS toolbox / driver on the german specific wastewater data format "ISYBAU Austauschformate Abwasser (XML)" - see here - and it works realy quite well.
But I came across some questions when reading ISYBAU files and I know this one relates only to the specific ISYBAU.xsd (and therefore might be off topic for you) but maybe you can give me a clue anyway.
When reading ISYBAU files (with 'metadaten.xsd' reference added to the header) the following error is shown:
RuntimeError: C:/Users/TS/Downloads/isybau2qgep-master/1302_ISYBAU_XML_Beispieldaten\../1302_ISYBAU_XML_Schema\.\1302-referenzlisten.xsd:7056:48 global type 'simpleType:EinheitZielgroesseType' declared more than once or also declared as complexType
Unfortunately I don't understand this supposed multiple declaration, the types in 1302-referenzlisten.xsd are refernced form the other XSDs, but to me these are no declarations.
This is from 1302-hydraulikdaten.xsd where EinheitZielgroesseType is mentioned (line 12):
<xsd:complexType name="ZielgroesseType">
<xsd:sequence>
<xsd:element name="Zielgroessenkennung">
<xsd:simpleType>
<xsd:restriction base="xsd:string">
<xsd:maxLength value="5"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:element>
<xsd:element name="HydObjektTyp" type="HydraulikObjektTypType"/>
<xsd:element name="ArtZielgroesse" type="ArtZielgroesseType"/>
<xsd:element name="EinheitZielgroesse" type="EinheitZielgroesseType"/>
<xsd:element name="WertZielgroesse">
<xsd:simpleType>
<xsd:restriction base="xsd:decimal">
<xsd:totalDigits value="9"/>
<xsd:fractionDigits value="3"/>
<xsd:maxExclusive value="1000000"/>
</xsd:restriction>
</xsd:simpleType>
</xsd:element>
</xsd:sequence>
</xsd:complexType>
Probably my xml knowledge is insufficient, but I'm also not able to reproduce this error with e.g. xmlspy.
But maybe the GMLAS driver is (too) picky on this??
The failure with OGR (OSGeo4W Shell) is unspecific, by the way. ERROR 3: Cannot open...
A workaround to make the opening/reading with GMLAS run is to comment the include of 'referenzlisten.xsd' out: 1302-metadaten.xsd, line 10
<!-- <xsd:include schemaLocation=".\1302-referenzlisten.xsd"/> -->
In "convert" tab, GMLAS import of the following url:
http://ressource.brgm-rec.fr/data/Piezometre/00463X0036/H1.2
It gives "Cannot open /vsicurl_streaming/http://ressource.brgm-rec.fr/data/Piezometre/00463X0036/H1.2"
However, with ogrinfo it works well:
ogrinfo -ro -al "GMLAS:/vsicurl_streaming/http://ressource.brgm-rec.fr/data/Piezometre/00463X0036/H1.2"
And in a python3 console:
from osgeo import gdal
gdal.OpenEx("GMLAS:/vsicurl_streaming/http://ressource.brgm-rec.fr/data/Piezometre/00463X0036/H1.2")However in the Python console of QGIS
gdal.OpenEx("GMLAS:/vsicurl_streaming/http://ressource.brgm-rec.fr/data/Piezometre/00463X0036/H1.2")
Traceback (most recent call last):
File "/usr/lib/python3.5/code.py", line 91, in runcode
exec(code, self.locals)
File "<input>", line 1, in <module>
File "/usr/local/lib/python3.5/dist-packages/osgeo/gdal.py", line 2995, in OpenEx
return _gdal.OpenEx(*args, **kwargs)
RuntimeError: Cannot open /vsicurl_streaming/http://ressource.brgm-rec.fr/data/Piezometre/00463X0036/H1.2@arnaud-morvan any idea ?
Text values from XML elements have encoding issues (on Windows)
An issue to where to add material for the workshop.
(at least)
content from our Groundwater Information Network (GIN): piezometers (INSPIRE EF) and associated WaterML2 TimeSeries
ex URI : http://ressource.brgm-rec.fr/data/Piezometre/06512X0037/STREMY.2
ex WFS : https://wfspoc.brgm-rec.fr/geoserver/ows?service=wfs&version=2.0.0&request=GetFeature&typenames=ef:EnvironmentalMonitoringFacility&count=10
content from EU H2020 EPOS Research Infrastructure : EU BoreholeIndex.
ad-hoc OGC SimpleFeature model linking to GIN above and also boreholeLogs (OGC:GroundWaterML2)
Example file : https://forge.brgm.fr/svnrepository/epos/trunk/instances/BoreholeView.xml
WFS query : http://resource.europe-geology.eu/service/wfsBorehole?service=wfs&version=2.0.2&request=GetFeature&typenames=gsmlp:BoreholeView&filter=gsmlp:drillerAGIP&count=1
Sylvain
When trying to import
https://github.com/BRGM/gml_application_schema_toolbox/blob/master/samples/BRGM_raw_database_observation_waterml2_output.xml
Maybe it lacks a way to set the source SRS ...
GMLAS mode: add customviewer icon in the legend, next to the layer icon
Allow the user to configure the language and document type negociated with the server
Allow to set the timeout
Redeploy the Database Model Viewer widget (was in the first ProofOfConcept)
Using :
https://www.emidius.eu/services/wfs?
The plugin calls 'https://www.emidius.eu/services/wfs?...' instead of 'http://www.emidius.eu/services/sheec/wfs' provided by the GetCaps
<ows:Operation name="GetFeature">
ows:DCP
ows:HTTP
<ows:Get xlink:href="http://www.emidius.eu/services/sheec/wfs"/>
<ows:Post xlink:href="http://www.emidius.eu/services/sheec/wfs"/>
</ows:HTTP>
</ows:DCP>
Related issue on QGIS
Stacktrace generated. No editFormConfig found
using V 1.1.1.zip
on 'http://ressource.brgm-rec.fr/data/Piezometre/06512X0037/STREMY.2'
with 'gmlasconf-inspire.xml' (note that I don't have this with 'gmlasconf.xml'
target : SQLITE
2017-07-13T16:52:55 1 Traceback (most recent call last):
File "C:/Users/grellets/.qgis3/python/plugins\gml_application_schema_toolbox\gui\import_gmlas_panel.py", line 308, in on_loadLayersBtn_clicked
import_in_qgis(source, self.databaseWidget.format(), schema)
File "C:/Users/grellets/.qgis3/python/plugins\gml_application_schema_toolbox\core\load_gmlas_in_qgis.py", line 61, in import_in_qgis
l = ds.ExecuteSQL(sql)
AttributeError: 'NoneType' object has no attribute 'ExecuteSQL'
From the various presentations done this year (FOSS4G-E, OGC TC, INSPIRE conf, French Water Information System) there is a clear feedback that the current GUI is too technical for the standard user.
There is a need to be able to launch the plugin in more simple mode(s) yet keeping the 'advanced' mode activable on demand.
An option would be to have the possibility to define a profile where one ticks what he wants to have displayed and save it.
The relational mode currently writes a QGIS project file "by hand" which is then loaded. This loading overwrite the current project and prevent from having more than one instance loaded.
Convert, Load & Import in one action and move the 'Load and configure layers' dans un autre onglet ou dans download
When trying to display items provided by the plugin over a WMS layer, I found that items added with the plugin are not well projected.
This point : http://ressource.brgm-rec.fr/data/Piezometre/00634X0147/PZ1.2 now appears near Somalia as shown on the screenshot.
I suppose projection are not well defined and then a mismatch with lon lat is done when trying to project data on the map (inversion of lon and lat).
In the plugin the link behind 'Check the plugin documentation to learn how to use it'. should point to 'https://github.com/BRGM/gml_application_schema_toolbox/blob/master/README.md' instead
When resolving an external XML element, it should be possible to update the XML tree so that it gets stored permanently.
The GUI refactoring described in #36 is still too complex for the average user.
Moreover, there are too much information displayed on the same panel.
After some discussions with @sgrellet it seems that using a QWizard to assist the opening would be a better approach. Each page of the wizard would correspond more or less to a group of options of the current panel.
When doing 'Resolve external', the initial information seed is enriched by new content (vocabulary, feature).
Enable to store this locally for further reuse in both modes (XML and DB).
In DB this requires to flag what does not need to be exported back in case the 'Export' tab is used afterwards.
The "download" tab does not work with 1.x servers
(for instance https://api.kuntatietopalvelu.fi/demo/wfs?service=wfs )
Adding a new attribute mapping in XML mode currently requires to
1/ Open the XML widget
2/ Copy the XPath of an element
3/ Close the dialog and open the creation dialog
4/ Add an attribute mapping
A new option could be added to the context menu "add as new column" on an element
When loading the following URI (http://ressource.brgm-rec.fr/data/EntiteHydroGeol/121AT30) the feature is correctly identified but it is not possible to retrieve information from it with the 'identify' function.
As this resource is related to another issue #29, I removed manually the un-necessary FeatureCollection / Members.
See file attached (remove the trailing .txt for import):
EntiteHydroGeol.121AT30.xml.txt
But the problem remains.
Is it linked to the complex geometry that may cause issue to store it temporarily in SpatialLite ?
Put it before the 'attribute mapping' section
FeatureCollection / Member should just be considered as containers of domain features
When loading the following URIs in XML Mode
it considers the information source is the FeatureCollection.
But the first one is empty (related to a Geoserver app-schema configuration issue).
When loading this 3rd URI : http://ressource.brgm-rec.fr/data/EntiteHydroGeol/121AT30
The first feature collection contains something, then the feature is correctly displayed (not sure it is properly handled see other issue #30 )
Note that in mode GDAL GMLAS, GMLAS drivers identifies properly the information source with those 3 URIs and does not propose FeatureCollection nor Member
Trying to download gmlp:BoreholeView from http://resource.europe-geology.eu/service/wfsBorehole results in this error message: "Feature type :g unknown"
Since HTML5 and .js are supported by QGIS, try another representation of the Borehole Log (GW_GeologyLogCoverage).
Ex: a pseudo-3D rendering.
When clicking on the attribute table after "load and configure layers"
Hi there,
Allow me to introduce: I'm Tim-Hinnerk Heuer, a geo software developer at Landcare Research New Zealand. As far as I'm aware Landcare Research has an ongoing relationship with BRGM.
I was told that this plugin works and has quite a lot of advanced features. However, I was unable to get anything other than the GetCapabilities feature to work. For this I had to change this line:
https://github.com/geekdenz/gml_application_schema_toolbox/blob/1135eb7bb8c17449f41521b578cde2583709bc8d/gml_application_schema_toolbox/gui/download_wfs2_panel.py#L52
Seems like I have a later version of the owslib library in Python 3 that may have changed their defaults.
I'm using a very recent version of QGIS:
https://github.com/qgis/QGIS/tree/f579f1a449b0bf21902570a32dafc9546cb414bd
or f579f1a
I cannot figure out how to get this plugin to work properly. I tried your WFS:
https://wfspoc.brgm-rec.fr/geoserver/ows
from the download tab.
I get this error:
2017-10-19T17:21:04 WARNING Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 377, in _make_request
httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 379, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.5/http/client.py", line 1197, in getresponse
response.begin()
File "/usr/lib/python3.5/http/client.py", line 297, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.5/http/client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.5/socket.py", line 575, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.5/ssl.py", line 929, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.5/ssl.py", line 791, in read
return self._sslobj.read(len, buffer)
File "/usr/lib/python3.5/ssl.py", line 575, in read
v = self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 247, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 686, in reraise
raise value
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
body=body, headers=headers)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 381, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
requests.packages.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='wfspoc.brgm-rec.fr', port=443): Read timed out. (read timeout=30)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/users/heuert/dev/cpp/QGIS/build-master/output/python/plugins/gml_application_schema_toolbox/gui/download_wfs2_panel.py", line 98, in on_downloadButton_clicked
out = self.download()
File "/home/users/heuert/dev/cpp/QGIS/build-master/output/python/plugins/gml_application_schema_toolbox/gui/download_wfs2_panel.py", line 155, in download
response = wfs.getfeature(**params)
File "/usr/local/lib/python3.5/dist-packages/owslib/feature/wfs200.py", line 221, in getfeature
username=self.username, password=self.password)
File "/usr/local/lib/python3.5/dist-packages/owslib/util.py", line 179, in openURL
**rkwargs)
File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 449, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='wfspoc.brgm-rec.fr', port=443): Read timed out. (read timeout=30)
Is anything working in this plugin yet? Do I need specific versions of things?
owslib.__version__
'0.15.0'
We are considering contributing to this plugin a XML/WFS browse facility, which I thought was at least partially implemented.
Any help will be greatly appreciated.
The custom widget declared in import_gmlas_panel.ui cannot be found:
Traceback (most recent call last):
File "/home/hme/src/QGIS/build/output/python/qgis/utils.py", line 338, in startPlugin
plugins[packageName] = package.classFactory(iface)
File "/home/hme/.qgis3/python/plugins/gml_application_schema_toolbox/__init__.py", line 35, in classFactory
from .main import MainPlugin
File "/home/hme/src/QGIS/build/output/python/qgis/utils.py", line 647, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/hme/.qgis3/python/plugins/gml_application_schema_toolbox/main.py", line 60, in
from .gui.dockwidget import DockWidget
File "/home/hme/src/QGIS/build/output/python/qgis/utils.py", line 647, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/hme/.qgis3/python/plugins/gml_application_schema_toolbox/gui/dockwidget.py", line 27, in
from gml_application_schema_toolbox.gui.import_panel import ImportPanel
File "/home/hme/src/QGIS/build/output/python/qgis/utils.py", line 647, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/hme/.qgis3/python/plugins/gml_application_schema_toolbox/gui/import_panel.py", line 10, in
from gml_application_schema_toolbox.gui.import_gmlas_panel import ImportGmlasPanel
File "/home/hme/src/QGIS/build/output/python/qgis/utils.py", line 647, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
File "/home/hme/.qgis3/python/plugins/gml_application_schema_toolbox/gui/import_gmlas_panel.py", line 50, in
os.path.dirname(__file__), '..', 'ui', 'import_gmlas_panel.ui'))
File "/usr/lib/python3/dist-packages/PyQt5/uic/__init__.py", line 197, in loadUiType
exec(code_string.getvalue(), ui_globals)
File "", line 258, in
File "/home/hme/src/QGIS/build/output/python/qgis/utils.py", line 647, in _import
mod = _builtin_import(name, globals, locals, fromlist, level)
ImportError: No module named 'qgsprojectionselectionwidget'
I tried to change the <header> element of import_gmlas_panel.ui, but without success.
@arnaud-morvan any idea ?
Not a generic one.
Thanks.
Enable the user to do 'Resolve external' from the forms when DB structure was generated by the GMLAS driver mode
GDAL GMLAS consider om:result as unparsed XML data. They are stored as a TEXT field ...
https://github.com/BRGM/gml_application_schema_toolbox/blob/master/samples/BRGM_raw_database_observation_waterml2_output.xml
https://github.com/BRGM/gml_application_schema_toolbox/blob/master/samples/BRGM_raw_database_observation_PointTimeseriesObservation_output.xml
@rouault normal or not ?
Stacktrace to come
'Stored Queries' tab does not list the Stored Queries provided by the server (ex : test on https://wfspoc.brgm-rec.fr/geoserver/ows? )
Previous versions did this (WFS ListStoredQueries operation) but did not allow to 'interact' with a Stored Query.
Example of such interaction would be
Stacktrace generated.
Details to come
Importing one of the 3 URIs below
http://ressource.brgm-rec.fr/data/EntiteHydroGeol/113AI01
http://ressource.brgm-rec.fr/data/EntiteHydroGeol/121AU
http://ressource.brgm-rec.fr/data/EntiteHydroGeol/121AT30
GDAL GMLAS identifies properly the various information elements.
However the GUI is not able from the 'identify' action on the mappedfeature the link to the other tables.
Is the driver capable of handling the <gsmlb:specification xlink:href="# structure (ex : <gsmlb:specification xlink:href="#EntiteHydroGeol.121AT30"/> in the last URI ) ?
When trying to export back a SQLite db to an XML (boreholeview):
Traceback (most recent call last):
File "/home/hme/.qgis3/python/plugins/gml_application_schema_toolbox/gui/export_gmlas_panel.py", line 163, in on_exportButton_clicked
self.translate(self.reproject_params(temp_datasource_path))
File "/home/hme/.qgis3/python/plugins/gml_application_schema_toolbox/gui/gmlas_panel_mixin.py", line 65, in translate
res = gdal.VectorTranslate(**params)
File "/usr/local/lib/python3.5/dist-packages/osgeo/gdal.py", line 709, in VectorTranslate
return wrapper_GDALVectorTranslateDestName(destNameOrDestDS, srcDS, opts, callback, callback_data)
File "/usr/local/lib/python3.5/dist-packages/osgeo/gdal.py", line 3122, in wrapper_GDALVectorTranslateDestName
return _gdal.wrapper_GDALVectorTranslateDestName(*args)
RuntimeError: Terminating translation prematurely after failed
translation of layer geom_cols_ref_sys (use -skipfailures to skip errors)
Allow to convert to Geopackage format.
Known drawbacks:
Display a polygon that corresponds to the bounding box when the information is present in the source
The current plugin uses kind of a "hack" to "augment" the identify dialog and display a custom XML tree QT widget.
QGIS apparently allows to declare custom editor widgets, even in Python. See for example
https://github.com/elpaso/qgis-formawarevaluerelationwidget/blob/master/FormAwareValueRelationWidget.py#L163
This should result in a better, more maintainable code.
Allow to update / delete a linked layer that has been linked by an href resolution.
See #38 for previous discussion
Enable the custom widgets from the forms when DB structure was generated by the GMLAS driver mode (was in the first ProofOfConcept).
Enable them also from the Database Model Viewer widget
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.