cs-si / eodag Goto Github PK

Earth Observation Data Access Gateway

License: Apache License 2.0

Python 99.69% Shell 0.03% Dockerfile 0.15% Smarty 0.13%

remote-sensing spatial-imagery catalog search satellite-imagery earth-observation sentinel copernicus geographic-data landsat

eodag's People

Contributors

Stargazers

Watchers

Forkers

beaulima saiplanner maximlt apparell jpv-costa ahuarte47 waseem-aidash drnextgis geogubd griffinbabe nicolaimpe coronath3beer robert-werner simrit1 floriandeboissieu meriamlahsaini alexm98 operationcanard trellixvulnteam myfreebrain australiansdaf aoyono perezgau remi-braun lalbizua catchsheep diegoalarc choumingzhao thapawan peterpong alambare-csgroup ansotoo zivankaraman sandeep2995 leso-kn bjgiraudon james-hinton gasparakos xiaoguo123456

eodag's Issues

Add EOProduct filename attribute

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

Add a way to get the path to the product in one time, or at least set an attribute with the name of the product (e.g product.filename = '<product_basename>.SAFE')

Support EOCloud data provider

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

Some information are here: https://eocloud.cloudferro.com/knowledgebase.php?action=displayarticle&id=66

Metadata mapping

Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).

We need a way to express how eodag metadata map to each provider metadata. This is how EOProduct properties are built from provider search results.

Download doesn't work for scihub

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

Since EOProduct id has changed in 2a256a9, eodag send wrong id to sentinelsat download section, resulting in error when we want to download on SciHub provider.

how-to-fix-it : in eodag.plugin.apis.sentinelsat l.75 & 78, replace product.id by product.properties['provider_id'].

get_data should work when the product already exists

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

If a get_data on a product has been done one time, re-trying the operation, by doing search and get_data on the same product, doesn't work and eodag can't find path to product bands.

Find a way to link get_data to product informations if the product is already found on local.

StopIteration issue

Original report by orhygine (Bitbucket: orhygine, GitHub: orhygine).

#!python

[2018-06-07 05:34:18,767] INFO - product:download:99 : downloading product
Downloading products
[2018-06-07 05:34:19,011] DEBUG - core:__get_downloaders:207 : Looking for the appropriate Download instance to use for product: EOProduct(id=a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a, producer=peps)
[2018-06-07 05:34:19,011] DEBUG - instances_manager:__get_instances:96 : Creating configured *DOWNLOAD* plugin instances
[2018-06-07 05:34:19,011] DEBUG - instances_manager:__get_instances:103 : Creating 'DOWNLOAD' plugin instance with name 'peps'
[2018-06-07 05:34:19,011] DEBUG - instances_manager:instantiate_plugin_by_config:130 : Creating 'DOWNLOAD' plugin instance with config '{'base_uri': u'https://peps.cnes.fr/resto', u'extract': None, 'dl_url_params': {'issuerId': u'peps'}, 'plugin': u'HTTPDownload'}'
[2018-06-07 05:34:19,011] DEBUG - http:__init__:34 : Images will be downloaded to directory /tmp
[2018-06-07 05:34:19,011] DEBUG - instances_manager:__get_instances:96 : Creating configured *API* plugin instances
[2018-06-07 05:34:19,012] DEBUG - instances_manager:__get_instances:114 : Skipping 'API' plugin creation for instance 'peps': not an instance of 'API'
[2018-06-07 05:34:19,012] DEBUG - core:__get_downloaders:220 : Found 1 Download instance(s) for product EOProduct(id=a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a, producer=peps) (ordered by highest priority): [HTTPDownload(instance_name=peps, priority=0, topic=Download)]
[2018-06-07 05:34:19,012] DEBUG - core:__download:143 : Using interface for download : HTTPDownload on instance *peps*
[2018-06-07 05:34:19,012] DEBUG - core:__get_authenticator:174 : Authentication initialisation for instance peps
[2018-06-07 05:34:19,012] DEBUG - instances_manager:instantiate_plugin_by_config:130 : Creating 'AUTH' plugin instance with config '{'auth_uri': u'https://peps.cnes.fr/resto/api/users/connect', 'credentials': {'password': None, 'email': None}, 'plugin': u'GenericAuth'}'
[2018-06-07 05:34:19,012] DEBUG - core:__get_authenticator:183 : Initialized GenericAuth(instance_name=peps, priority=0, topic=Authentication) Authentication plugin for instance 'peps'
[2018-06-07 05:34:19,012] DEBUG - http:download:43 : Download url: https://peps.cnes.fr/resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download
[2018-06-07 05:34:19,016] DEBUG - connectionpool:_new_conn:824 : Starting new HTTPS connection (1): peps.cnes.fr
[2018-06-07 05:34:19,398] DEBUG - connectionpool:_make_request:396 : https://peps.cnes.fr:443 "GET /resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download?issuerId=peps HTTP/1.1" 404 44

 https://peps.cnes.fr/resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download?issuerId=peps
[2018-06-07 05:34:19,404] ERROR - http:download:75 : Error while getting resource : 404 Client Error: Not Found for url: https://peps.cnes.fr/resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download?issuerId=peps
[2018-06-07 05:34:19,406] ERROR - product:main:215 : 
Traceback (most recent call last):
  File "/home/00154_7f48a9e7e/wps-jobs/product.py", line 198, in main
    process_cfg['params']['inputFile'] = download(input_dir)
  File "/home/00154_7f48a9e7e/wps-jobs/product.py", line 103, in download
    product_path = next(eodag_api.download_all(product))
StopIteration

Support SRTM/DEM data download

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

Related to SNAP feature

the DEM data is very useful in several applications.
Several DEM are publicly available:

SRTM 1s : https://en.wikipedia.org/wiki/Shuttle_Radar_Topography_Mission
ALOS World 3D : https://www.eorc.jaxa.jp/ALOS/en/aw3d/index_e.htm

Search parameters mapping

Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).

We need search parameters to be uniform with eo products metadata throughout eodag project (cli and api). The providers config file should advertise which metadata is queryable by indicating how it is mapped to its search parameters.

Eodag should provide a way for plugin developers to retrieve the search parameters suitable to their plugin, using only the information available in the mapping.

Retrieve feature and merge with vito_download project

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

https://github.com/DHI-GRAS/vito_download

Enables `EOProduct` to remember its remote address (for subsequent downloads)

Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).

This would enable the EOProduct to trigger effective download in the situation when a previous download was done (changing the location attribute to the file system path) and the said product was deleted from the file system before a subsequent call to EOProduct.download, leading to a more intelligent object which knows exactly when to download itself .

Verification of the existence of a product before a download should avoid re-downloading it

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

In the case where a product has been already downloaded but we make a new search on it, eodag don't retrieve it and download it again. The verification should be done at least on zipfile existence. Or, for the check of the existence of de-zipped product, the .downloaded folder should contain informations like archive depth, filename, etc.

As the use of eodag goes hand in hand with non-manipulation of files, these checks on the default download directory should be sufficient.

Possibly linked to #39

Support ESA Scihub data provider

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

Information about API
https://scihub.copernicus.eu/userguide/5APIsAndBatchScripting

Support other Copernicus Mirror

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

Support

‌

Add eocloud for L2A products

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

S2_MSI_L2Ap product type is not yet searched on eocloud.

Add something like that (works on my side) in providers.yml in eocloud section :

#!yaml


 S2_MSI_L2Ap:
      product_type: L2A
      collection: Sentinel2

Modifier caractère optionnel de startDate pour les EOProducts + robustesse test pour création eoproduct a partir d'un geojson

Original report by clement cazorla (Bitbucket: ccazorla, GitHub: ccazorla).

Support CODE-DE provider

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

CODEDE is another collaborative ground segment which provide Sentinel Data and other data.
https://code-de.org/

Add a way for get_data to check local products before downloading

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

For all providers that give the choice to select way to access to data (file or http), get_data don't look if a product is already downloaded, and the download is always done.

Add a way for get_data to check local products before downloading it

handle asynchronous downloads

Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).

On some providers, downloading some products is an asynchronous process: when the client makes a get request to download the product, the provider returns a response with status code 202 (accepted).
Right now, the http download plugin does not understand this response and treat it as it was a 200 (Success) response, resulting in a downloaded "product" not containing any data (the payload of a 202 response is an empty bytes).

The plugin must better behave to enable getting the actual product by either :

waiting for the amount of time specified in the response header RETRY-AFTER, not exceeding a certain amount of time (roughly half-an hour)
checking the url specified in the response header Location for the status of the download

Example url leading to a 202 response: https://peps.cnes.fr/resto/collections/S2/420901c4-2f0e-59ad-a792-9043de6453fe/download?issuerId=peps (make sure you are connected to peps to reproduce). Exemple request with curl:

#!bash

curl --user id:pass "https://peps.cnes.fr/resto/collections/S2/420901c4-2f0e-59ad-a792-9043de6453fe/download?issuerId=peps"

Support ASF to retrieve EO data

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

Support ASF to retrieve EO data, mainly SAR one.

A python package is available for this provider https://github.com/asfadmin/Discovery-asf_search under BSD-3-Clause license (compatible with eodag).
pypi: https://pypi.org/project/asf-search/
conda-forge: https://anaconda.org/conda-forge/asf_search
Documentation: https://docs.asf.alaska.edu/asf_search/basics/
A python api plugin will be preferred (faster development time and easier maintainability), but API is also documented:

API documentation: https://docs.asf.alaska.edu/api/basics/
search example: https://api.daac.asf.alaska.edu/services/search/param?intersectsWith=POLYGON((-2.671%2042.2314,4.3776%2042.2314,4.3776%2045.9707,-2.671%2045.9707,-2.671%2042.2314))&platform=SENTINEL-1&instrument=C-SAR&start=2020-05-02T22:00:00Z&end=2020-05-20T21:59:59Z&maxResults=250&output=jsonlite2

{
  "results": [
    {
      "b": [],
      "bm": "IW",
      "d": "Sentinel-1A",
      "du": "https://datapool.asf.alaska.edu/RAW/SA/{gn}.zip",
      "f": 146,
      "fd": "ASCENDING",
      "fl": null,
      "fn": "{gn}.zip",
      "fr": null,
      "gid": "S1A_IWDV_0146_0153_032652_030",
      "gn": "S1A_IW_RAW__0SDV_20200520T175617_20200520T175649_032652_03C822_715C",
      "i": "C-SAR",
      "in": false,
      "mn": null,
      "o": [
        "32652"
      ],
      "on": null,
      "p": 30,
      "pa": null,
      "pid": "{gn}-RAW",
      "po": "VV+VH",
      "pt": "RAW",
      "ptd": "L0 Raw Data (RAW)",
      "s": 1494.7302169799805,
      "ss": null,
      "st": "2020-05-20T17:56:17.000000",
      "stp": "2020-05-20T17:56:49.000000",
      "t": null,
      "w": "POLYGON((0.742400 45.103700,-2.362300 44.824200,-2.905900 46.772000,0.311000 47.055600,0.742400 45.103700))",
      "wu": "POLYGON ((0.742400 45.103700, -2.362300 44.824200, -2.905900 46.772000, 0.311000 47.055600, 0.742400 45.103700))"
    },

"description" field in Theia product properties no longer exists

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

EOPRODUCT_PROPERTIES contains a "description" field that will induce a KeyError for a Theia search, as it is no longer provided. Remove it.

Add geojson representation method for a SearchResult

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

Add a method that return geojson object from a SearchResult:

#!python

    def as_geojson_object(self):
        return {
            'type': 'FeatureCollection',
            'features': [product.as_dict() for product in (
                self.__final_result if self.__crunch_calls_count > 0 else self.__original
            )]
        }

    @property
    def __geo_interface__(self):
        """Implements the geo-interface protocol.

        See https://gist.github.com/sgillies/2217756
        """
        return self.as_geojson_object()

Find a way to decouple AmazonWS and eocloud providers

Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).

At the moment, AwsSearch plugin is a subclass of RestoSearch AND AmazonWS which implements AwsSearch for search is in fact a doublon of eocloud provider when it comes to search. This means that if in the same search eocloud and AmazonWS providers are used, we will end up having partially duplicated products in our result (in terms of id for example).

Furthermore, plugins are supposed to be independent from each other, and these two are not.

Support partial mirror

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

Support:

‌

progress bar with tqdm

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

See to replace the direct use of tqdm by a python callback that allows to choose the progress bar type (Using tqdm_notebook allows to display the download progress on a single bar).

Support .netrc file

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

https://docs.python.org/2/library/netrc.html

Use Python2.7 with future capability

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

Support Google API to retrieve Sentinel2 and Landsat data

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

https://cloud.google.com/storage/docs/public-datasets/sentinel-2
https://cloud.google.com/storage/docs/public-datasets/landsat

Add scihub for L2A products

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

#!yaml

S2_MSI_L2Ap:
  product_type: S2MSI2Ap

Add console entry point to the eodag cli

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

get_data with airbus-ds on S2-MSI-L1C doesn't work

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

error :

#!python

UnsupportedDatasetAddressScheme: eo product EOProduct(id=5619902d-d6de-41ca-a295-e4d9efe970c2, provider=airbus-ds) is accessible through a location scheme that is not yet supported by eodag: {base}/5619902d-d6de-41ca-a295-e4d9efe970c2

Add new methods for specifying geographic extent

Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).

e.g: from a shapefile, wkt, geojson, ...

Find a key to sort products that share same extent

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

Sort products by wkb_hex is not optimal. There are a lot of cases where products doesn't share exactly the same extent.Below, the products of first and last index of sorted_products have the same tile id, but are not sorted in the same SearchResult.

#!python

product_type = 'S2_MSI_L1C'
extent = {
'lonmin': -106.130161,
'lonmax': -106.102524,
'latmin': 39.627243,
'latmax': 39.647336
}
dag.set_preferred_provider(provider='airbus-ds')

sorted_products = dag.sort_by_extent([
    dag.search(product_type,
               startTimeFromAscendingNode='2018-05-01', 
               completionTimeFromAscendingNode='2018-05-18', 
               geometry=extent),
    dag.search(product_type,
               startTimeFromAscendingNode='2018-06-05',
               completionTimeFromAscendingNode='2018-06-19',
               geometry=extent)
])

find a better discriminant to sort products.

add some library features to the command line interface

Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).

Currently, eodag list command only list available product types for search. It should also list available providers.

More generally, the provider's priority configuration of an eodag install should be customizable from the command line:

possibility to set the preferred provider for one search only (through a --preferred-provider option to search sub command for example)
possibility to change the default preferred provider through a set-preferred-provider sub command
possibility to fine tune the provider search order through a default-provider-order sub command or/and a --provider-order option on search sub command for example

Default eocloud product_location_scheme

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

Actually product_location_scheme is set to 'file' for eocloud search, making eodag searching in /eodata/...

Default parameter should be 'http' for general users.

Footprint search parameter doesn't work with SciHub provider

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

sentinelsat API must have the footprint as a area parameter, and not as a keyword argument.

In eodag.plugins.apis.sentinelsat , replace l.108 with :

#!python

 query['area'] = geometry.box(*box_values).to_wkt()

Enhance configuration file deployment

Original report by orhygine (Bitbucket: orhygine, GitHub: orhygine).

Update setup.py to deploy "user_conf_template.yml" into standard user configuration folder.

Use the mechanism for automatically discovering plugins at runtime provided by setuptools

Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).

See documentation of this feature here:
https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins

Support AWS API to retrieve Sentinel2 and Landsat data

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

For Sentinel-2:
http://sentinel-pds.s3-website.eu-central-1.amazonaws.com/
https://aws.amazon.com/fr/public-datasets/sentinel-2/

Warning it could necessary to regenerate the SAFE format:
https://github.com/sinergise/sentinelhub/

For Landsat:
https://aws.amazon.com/fr/public-datasets/landsat/

Add quicklooks as EOProduct attribute/properties field

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

Add quicklooks to properties EOProduct attribute, or directly as a new attribute. A simple look on the data allows to quickly check if the product correspond to what the user want before downloading it.

In an IDE like Jupyter notebook, it can be a real plus as the images are very well embedded to the structure.

Peps S2 L1C collection depends of the date

Original report by Baptiste Meylheuc (Bitbucket: bmeylheuc, ).

Since 2016-12-06, Sentinel-2 collection is S2ST as the products are provided per tiles, and S2 for earlier dates. Actually, only S2 is managed and search requests won't find any products on Peps.

Find a way to manage the two collections S2 & S2ST depending of the date (at least, set default to S2ST as a longer period is covered)
Check if no other provider is impacted

Handle pagination at the cli and api levels

Original report by Oyono (Bitbucket: aoyono, GitHub: aoyono).

Support Greek Copernicus Ground Segement to retrieve Copernicus data

Original report by Mickaël Savinaud (Bitbucket: savmickael, GitHub: savmickael).

https://sentinels.space.noa.gr/dhus/#/home

Give a way to ignore cloudCoverage criteria into Resto search plugin

Original report by orhygine (Bitbucket: orhygine, GitHub: orhygine).

Download issue

Original report by orhygine (Bitbucket: orhygine, GitHub: orhygine).

#!python

[2018-06-07 04:43:40,761] INFO - product:download:99 : downloading product
Downloading products
[2018-06-07 04:43:42,850] DEBUG - core:__get_downloaders:207 : Looking for the appropriate Download instance to use for product: EOProduct(id=a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a, producer=peps)
[2018-06-07 04:43:42,851] DEBUG - instances_manager:__get_instances:96 : Creating configured *DOWNLOAD* plugin instances
[2018-06-07 04:43:42,851] DEBUG - instances_manager:__get_instances:103 : Creating 'DOWNLOAD' plugin instance with name 'peps'
[2018-06-07 04:43:42,851] DEBUG - instances_manager:instantiate_plugin_by_config:130 : Creating 'DOWNLOAD' plugin instance with config '{'base_uri': u'https://peps.cnes.fr/resto', u'extract': None, 'dl_url_params': {'issuerId': u'peps'}, 'plugin': u'HTTPDownload'}'
[2018-06-07 04:43:42,851] DEBUG - http:__init__:34 : Images will be downloaded to directory /tmp
[2018-06-07 04:43:42,851] DEBUG - instances_manager:__get_instances:96 : Creating configured *API* plugin instances
[2018-06-07 04:43:42,851] DEBUG - instances_manager:__get_instances:114 : Skipping 'API' plugin creation for instance 'peps': not an instance of 'API'
[2018-06-07 04:43:42,852] DEBUG - core:__get_downloaders:220 : Found 1 Download instance(s) for product EOProduct(id=a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a, producer=peps) (ordered by highest priority): [HTTPDownload(instance_name=peps, priority=0, topic=Download)]
[2018-06-07 04:43:42,852] DEBUG - core:__download:143 : Using interface for download : HTTPDownload on instance *peps*
[2018-06-07 04:43:42,852] DEBUG - core:__get_authenticator:174 : Authentication initialisation for instance peps
[2018-06-07 04:43:42,852] DEBUG - instances_manager:instantiate_plugin_by_config:130 : Creating 'AUTH' plugin instance with config '{'auth_uri': u'https://peps.cnes.fr/resto/api/users/connect', 'credentials': {'password': None, 'email': None}, 'plugin': u'GenericAuth'}'
[2018-06-07 04:43:42,852] DEBUG - core:__get_authenticator:183 : Initialized GenericAuth(instance_name=peps, priority=0, topic=Authentication) Authentication plugin for instance 'peps'
[2018-06-07 04:43:42,852] DEBUG - http:download:43 : Download url: https://peps.cnes.fr/resto/collections/S1/a3d0fb48-82be-5b0b-8ccb-5ad483aa2f8a/download
[2018-06-07 04:43:42,853] ERROR - product:main:215 : [Errno 17] File exists: '/tmp/.downloaded'
Traceback (most recent call last):
  File "/home/00154_7f48a9e7e/wps-jobs/product.py", line 198, in main
    process_cfg['params']['inputFile'] = download(input_dir)
  File "/home/00154_7f48a9e7e/wps-jobs/product.py", line 103, in download
    product_path = next(eodag_api.download_all(product))
  File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/site-packages/eodag/api/core.py", line 119, in download_all
    for path in self.__download(product):
  File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/site-packages/eodag/api/core.py", line 152, in __download
    for local_filename in maybe_generator(iface.download(product, auth=auth)):
  File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/site-packages/eodag/utils/__init__.py", line 87, in maybe_generator
    for elt in obj:
  File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/site-packages/eodag/plugins/download/http.py", line 49, in download
    os.makedirs(download_records)
  File "/home/00154_7f48a9e7e/anaconda2/envs/wps/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/tmp/.downloaded'