codeforberlin/wfs-downloader

Downloads GML files from a set of WFS service in a pseudo-paginated way using bounding boxes.

wfs-downloader's Issues

urllib.error.HTTPError: HTTP Error 500: Internal Server Error

config.yml:
url: https://inspire-hessen.de/ows/services/org.2.dd63cf0f-cf8a-4a69-be4c-b5c056b8528b_wfs
layer: cp:CadastralParcel

bbox:
west: 777247.0
south: 4939527.0
east: 1023641.0
north: 5165779.0

size: 10000
outputfile: flurstck_hessen.xml
projection: EPSG:25832
tmpdir: C:/Users/user/Programmieren/Pycharm

I'll get the following error:

(venv) PS C:\Users\user\Programmieren\Pycharm> wfs-downloader config.yml

fetching C:/Users/user/Programmieren/Pycharm\flurstck_hessen_777247.0_4939527.0.xml
Traceback (most recent call last):
File "", line 198, in _run_module_as_main
File "", line 88, in run_code
File "C:\Users\user\Programmieren\Pycharm\venv\Scripts\wfs-downloader.exe_main.py", line 7, in
File "C:\Users\user\Programmieren\Pycharm\venv\Lib\site-packages\wfs_downloader\download.py", line 32, in main
download_files(config)
File "C:\Users\user\Programmieren\Pycharm\venv\Lib\site-packages\wfs_downloader\download.py", line 73, in download_files
urlretrieve(url, filename)
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 240, in urlretrieve
with contextlib.closing(urlopen(url, data)) as fp:
^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 215, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\userAppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 521, in open
response = meth(req, response)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 630, in http_response
response = self.parent.error(
^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 559, in error
return self._call_chain(*args)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 492, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "C:\Users\user\AppData\Local\Programs\Python\Python312\Lib\urllib\request.py", line 639, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 500: Internal Server Error

error because wfs:boundedBy/gml:Envelope/gml:lowerCorner does not exist

In my case this attribute does not exist so it errors out.

Ease usage: Print help by default

If someone invokes the script without parameters the help should be automatically printed as in:

$ ./download.py

manage throttle or closing connections by server

when too many requests are made some servers close connection. may be use a delay between requests? also handle this kind of exceptions.

when file is empty an error occurs

change axis order in config when EPSG 4326

For some reason, in config file I have to switch south/north to longitude and east/west to longitude to get data from a wfs service. Have to try other services to understande if it's just my service. For instance:
bbox:
west: 37.621
south: -8.410
east: 38.602
north: -7.311

When in fact this is latlon notation, not west/south (lonlat).

The real coordinates are: -8.410 37.621 for minx miny, and -7.311 38.602 for maxx maxy.

Otherwise I get no data...

Error when downloading Baumbestand Berlin

https://daten.berlin.de/datensaetze/baumbestand-berlin-stra%C3%9Fenb%C3%A4ume-wfs

remove duplicates

some how remove duplicates, since bbox will sometimes download the same features, with lines or polygons.