bart-lg/geocropper

Download and crop/clip Sentinel and Landsat tiles for geolocations.

geocropper's Issues

grouping in combined previews

It would be nice if the rows could be optionally groupable.
E.g. one row for all images with same description.

Check if pathlib is used for all paths

Handle corrupt downloaded zip files

Script exits with error when downloaded zip file is corrupt.
It should delete the zip file and try to download it again several times.
If it is still not possible to download a good zip file => skip download and create warning message.

Make copy of metadata optional

DB internal: description doesn't get copied from CSVInput to PointOfInterests

Save more information about tiles in database

Check for incomplete LANDSAT download

The currently implemented package to download landsat data does not support a resume function. Therefore check if the size of an existing tar file is lower than expected. If yes, remove the tar file and start a new download.

Create combined RGB image sets of e.g. 5x5 cropped tiles

Value checking in create_random_crops probably not working well

The following code in the function create_random_crops should check if the random point has valid pixel values.

                    for file in HQ_dir.glob("*_B*_10m.jp2"):
                        dataset = gdal.Open(str(file))
                        x_index = (random_x - lower_left_x) / xres
                        y_index = (random_y - lower_left_y) / ( yres * -1 )
                        x_index, y_index = int(x_index + 0.5), int(y_index + 0.5)
                        array = dataset.ReadAsArray()
                        pixel_val = array[y_index, x_index]
                        if pixel_val > 0:
                            has_values = True

In a previous version this already worked well. Now the function also creates some totally black crops, so it is probably not working anymore.

Check inputs and create error messages

Import of csv files is very slow

Save metadata from search response?

The respond of the sentinel and landsat api to search queries include some meta information.
Shall this meta information be saved? If yes, store it in the database.

Check for clouds and lacks in cropped tiles

Update env.yml file

Permission denied: unlink

Fresh installation...

redundancies of coordinate transformations

Reduce transformation steps where possible!

Create a separate function for the transformation of coordinates:

# open raster image file
img = rasterio.open(str(path))    
dataset = gdal.Open(str(path))

toTargetCRS = partial(pyproj.transform,
    pyproj.Proj('+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs '), pyproj.Proj(img.crs))

# transform corner coordinates for cropping
poi_transformed = transform(toTargetCRS, Point(lon,lat))
poi_x = poi_transformed.x
poi_y = poi_transformed.y

Check code for TODO comments

firstDownloadRequest column name in db table Tiles misleading

It should read tileCreated

Only the column lastDownloadRequest is representing the download request datetimes

Check code for compliance with PEP8 style

Check code for compliance with PEP8 style
https://www.python.org/dev/peps/pep-0008/

Resolve cross references of modules

Move utility functions from geocropper class to a new file util.py

combined preview sometimes miss a few images

Handle different HTTP Codes in replies of API requests

Search for Sentinel data:
lat: 31.277945
lon: -101.142479
From: 01.07.2020
To: 30.09.2020
Platform: Sentinel-1
Tile-limit: 1
producttype: GRD

Traceback (most recent call last):
File "scripts/import_all_CSVs.py", line 6, in
geocropper.importAllCSVs()
File "../geocropper/geocropper/geocropper.py", line 50, in importAllCSVs
csvImport.importAllCSVs(delimiter, quotechar)
File "../geocropper/geocropper/csvImport.py", line 42, in importAllCSVs
loadImportedCSVdata()
File "../geocropper/geocropper/csvImport.py", line 140, in loadImportedCSVdata
width = item["width"], height = item["height"], tileLimit = item["tileLimit"], **kwargs)
File "../geocropper/geocropper/geocropper.py", line 908, in downloadAndCrop
products = self.downloadSentinelData(dateFrom, dateTo, platform, poiId=poiId, tileLimit=tileLimit, **kwargs)
File "../geocropper/geocropper/geocropper.py", line 179, in downloadSentinelData
products = self.sentinel.getSentinelProducts(self.lat, self.lon, dateFrom, dateTo, platform, limit=tileLimit, **kwargs)
File "../geocropper/geocropper/sentinelWrapper.py", line 37, in getSentinelProducts
result = self.api.query(footprint, date=(dateFrom, dateTo), platformname=platform, **kwargs)
File "/home/bleongruchalski/.conda/envs/env/lib/python3.7/site-packages/sentinelsat/sentinel.py", line 147, in query

File "/home/bleongruchalski/.conda/envs/env/lib/python3.7/site-packages/sentinelsat/sentinel.py", line 272, in _load_query
# Allow None to be used as a unlimited bound
File "/home/bleongruchalski/.conda/envs/env/lib/python3.7/site-packages/sentinelsat/sentinel.py", line 304, in _load_subquery
order_by: str, optional
File "/home/bleongruchalski/.conda/envs/env/lib/python3.7/site-packages/sentinelsat/sentinel.py", line 1027, in _check_scihub_response
if paths:
sentinelsat.sentinel.SentinelAPIError: HTTP status 500 Internal Server Error: SolrServerException : Server refused connection at: http://172.30.175.85:8983/solr/dhus
Makefile:3: recipe for target 'importall' failed
make: *** [importall] Error 1

Default config values cannot be loaded always

Default config values cannot be loaded if only parts of a block are defined in user-config.ini

E.g.:

If this content is part of the user-config.ini ...

[Combined Preview Images]
combinedPreview = True
previewBorder = 5
previewBackgroundR = 100
previewBackgroundG = 0
previewBackgroundB = 0
previewTextOnImage = True
previewImageFontSize = 12
previewImagesCombined = 100

... then the value of combinedPreview can be fetched

If the whole [Combined Preview Images]-Block is missing in user-config.ini then the values of default-config.ini are taken. But if the content in user-config.ini is reduced to ...

[Combined Preview Images]
previewBorder = 5
previewBackgroundR = 100
previewBackgroundG = 0
previewBackgroundB = 0
previewTextOnImage = True
previewImageFontSize = 12
previewImagesCombined = 100

... then the value of combinedPreview cannot be fetched. It should be taken from default-config.ini in this case.

Error when tileLimit is not defined in CSV

The problem in the following SQL statement is this:
AND width = None AND height = None AND description = '' AND tileLimit = None

= None needs to be replaced before the query with IS NULL

[ERROR]: database.py(fetch_first_row_query:394) >> Error in query [SELECT rowid, * FROM PointOfInterests WHERE groupname = 'BA' AND lat = -14.622934159 AND lon = -42.573899197 AND dateFrom = '2019-01-01' AND dateTo = '2021-03-17' AND platform = 'Sentinel-2' AND width = None AND height = None AND description = '' AND tileLimit = None AND polarisationmode IS NULL AND producttype IS NULL AND sensoroperationalmode IS NULL AND swathidentifier IS NULL AND cloudcoverpercentage IS NULL AND timeliness IS NULL AND orbitdirection IS NULL AND filename IS NULL] [values: None]: DatabaseLockedError('Database locked or other error. Please see the log-file for details. Timeout:30 Retries:5')

Out of bounds error in create_random_crops()

Traceback (most recent call last):
File "generate_random_crops.py", line 2, in
geocropper.utils.create_random_crops()
File "/data/projects/windturbine-identification-sentinel/geoc-symlinks/geocropper/geocropper/utils.py", line 386, in create_random_crops
pixel_val = array[y_index, x_index]
IndexError: index 10980 is out of bounds for axis 1 with size 10980

Improve resume function by checking outstanding downloads or croppings

Create RGB image of cropped tile

combined preview images - sometimes empty row in last preview image

That results also in an inapprorpiate behaviour of the Visual Selector, since the height of the individual images is wrong calculated.

Default config and user config

Create possibility to define user configurations additionally to API user credentials.

Download of archived Sentinel tiles not working well

Currently a cronjob more often than once every hour leads to a 429 http response code (too many requests).
No documentation found yet which explains the limit leading to a 429 return code.

Another issue is that probably the communication within the script regarding the last send request (which is currently done using the database) seems not to work smoothly. This might be because of a delay in the database, so that written timestamps will not be returned by the select statement immediately (race condition?). This is not tested. I only had the feeling that multiple download requests had been sent out within the 30 minutes - which should not be the case.

However, maybe it is better to read the last timestamp at the beginning in a global variable and to update the global variable while the script is running.

Another thing is, that currently two requests are sent for one download request. The first request determines if the tile is online. If not then a download request is sent if the last download request is not younger than 30 minutes. This multiple requests could also be leading to the 429 return code. Maybe this could be optimized as well.

Test on Windows machine

rename lat_lon_set to lon_lat_set

Name of crop folders is id_lon_lat therefore it is better if the set variables in Geocropper are renamed respectively.

Test Landsat: download & crop

Test also:

• interrupted downloads
• existing big tiles