yannforget/landsatxplore

Search and download Landsat scenes from EarthExplorer.

landsatxplore's Issues

How can i add in datamodel.py additionalcriteria to filter only NIGHT products

i add this line but when i look for data, the script give me only DAY

def additionalcriteria(daynight):
    """additionalCriteria data model.  

    Parameters
    ----------
    daynight : string
        DAY NIGHT ALL 

    Returns
    -------
    additionalcriteria : dict
        additionalcriteria data model as a dictionnary.
    """
    return {
        'filterType': 'and', 
        'childFilters': [
            {'filterType': 'value','fieldId': 10187,'value': daynight},
        ]           
    }

Download fails

(env) ➜  usgs-test landsatxplore search --dataset landsat_ot_c2_l2 --location 31.9378 -109.0803  --start 2021-09-01 --end 2022-02-02
LC08_L2SP_035038_20220120_20220127_02_T1
LC08_L2SP_034038_20220113_20220123_02_T1
LC08_L2SP_035038_20220104_20220113_02_T1
LC08_L2SP_034038_20211228_20220105_02_T1
LC08_L2SP_035038_20211219_20211223_02_T1
LC08_L2SP_034038_20211212_20211222_02_T1
LC08_L2SP_035038_20211203_20211209_02_T1
LC08_L2SP_034038_20211126_20211201_02_T1
LC08_L2SP_035038_20211117_20211125_02_T1
LC08_L2SP_034038_20211110_20211117_02_T1
LC08_L2SP_035038_20211101_20211109_02_T1
LC08_L2SP_034038_20211025_20211103_02_T1
LC08_L2SP_035038_20211016_20211026_02_T1
LC08_L2SP_034038_20211009_20211019_02_T1
LC08_L2SP_035038_20210930_20211013_02_T1
LC08_L2SP_035038_20210914_20210924_02_T1
LC08_L2SP_034038_20210907_20210916_02_T1

(env) ➜  usgs-test landsatxplore download LC08_L2SP_035038_20210914_20210924_02_T1                                                 
Traceback (most recent call last):
  File "/Users/henrykironde/Documents/GitHub/python-virtual-environments/env/bin/landsatxplore", line 10, in <module>
    sys.exit(cli())
  File "/Users/henrykironde/Documents/GitHub/python-virtual-environments/env/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/henrykironde/Documents/GitHub/python-virtual-environments/env/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/henrykironde/Documents/GitHub/python-virtual-environments/env/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/henrykironde/Documents/GitHub/python-virtual-environments/env/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/henrykironde/Documents/GitHub/python-virtual-environments/env/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/henrykironde/Documents/GitHub/python-virtual-environments/env/lib/python3.7/site-packages/landsatxplore/cli.py", line 165, in download
    scene, output_dir, dataset=dataset, timeout=timeout, skip=skip
  File "/Users/henrykironde/Documents/GitHub/python-virtual-environments/env/lib/python3.7/site-packages/landsatxplore/earthexplorer.py", line 150, in download
    filename = self._download(url, output_dir, timeout=timeout, skip=skip)
  File "/Users/henrykironde/Documents/GitHub/python-virtual-environments/env/lib/python3.7/site-packages/landsatxplore/earthexplorer.py", line 92, in _download
    raise EarthExplorerError(error_msg)
landsatxplore.errors.EarthExplorerError: Download is not available

Add “path” and “row” to serach

Hello, Thanks for the great tool of API.
Sometimes I want to select images by Path/Row, so I did a small change to your code.

def search(self, dataset, longitude=None, latitude=None, bbox=None, max_cloud_cover=None, start_date=None, end_date=None, months=None, wrs_path=None, wrs_row=None, max_results=10000,):
    .
    .
    .
    r = self.request(
        "scene-search",
        params={
            "datasetName": dataset,
            "sceneFilter": scene_filter,
            "maxResults": max_results,
            "metadataType": "full",
        },
    )
    # where the change were did
    scenes = [_parse_metadata(scene) for scene in r.get("results")]

    if wrs_path and wrs_row:
        return [scene for scene in scenes if wrs_path == scene['wrs_path'] and wrs_row == scene['wrs_row']]
    else:
        return scenes

Limit to 20 images

Hi,

Any kind of search that I do, only throws 20 images like maximun. So, if for instance, I try to look all the Landsat_TM_C1 collection, only get the first 20 from the beginning.

Is there anyway to skip this limitation?

Thank you so much

Downloading Sentinel data

Hello,
Thanks for the great tool! A follow-up question for this issue:
Where exactly do these hashes (?) for building the download URL come from?

DATASETS = {
    "LANDSAT_TM_C1": "5e83d08fd9932768",
    "LANDSAT_ETM_C1": "5e83a507d6aaa3db",
    "LANDSAT_8_C1": "5e83d0b84df8d8c2"

I am currently trying to also download Sentinel-2 data using landsatxplore, because downloading it from ESA's scihub (and sentinelsat etc.) is now harder as old data moved into a longterm archive. Searching for S2-Data already works by using the "Sentinel_2A" dataset name. However, I cannot figure out the constant for S2-Data.
Thanks,
Guido

Originally posted by @griembauer in #12 (comment)

Band LandSat 8

You know if it's possible to change the band quantity that will be downloaded? Because in my case i just need the pure image, without any filter. You have any suggestion in this case?

Problem of Downloading Landsat8 Collection2 Product

I download Landsat8 Collection2 Data, bue get a Collection1 Data
The downloading file doesn't match the local file
I don't know why it happens......

The function of downloading is paste here as following：

def Download_from_Landsatexplore(dataset,scene_list):
    if len(scene_list) > 0:
        for scene in scene_list:
            if dataset.lower() == "landsat_8_c1":
                output_dir = r'G:\landsat_download\landsat_8_c1' 
            elif dataset.lower() == "landsat_ot_c2_l1":
                output_dir = r'G:\landsat_download\landsat_ot_c2_l1' 
            elif dataset.lower() == "landsat_ot_c2_l2":
                output_dir = r'G:\landsat_download\landsat_ot_c2_l2' 
            else:
                output_dir = r'G:\landsat_download\other'
            
            if not os.path.isdir(output_dir):
                os.mkdir(output_dir)
            
            output_dir_demon = output_dir+'\\'+str(scene['acquisition_date'].year)
            if not os.path.isdir(output_dir_demon):
                os.mkdir(output_dir_demon)
            ee = EarthExplorer(username, password)
            print("Downloading: "+scene['display_id'])
            ee.download(identifier=scene['entity_id'], output_dir=output_dir_demon)

problem use landsatxplore and pylandsat

hello，
when i used landsatxplore,i found i only can searched less than 20 data,and the end_data also have some problem.

 Besides,when i used pylandsat,i meet an error wen i used catalog.search,which told me 'sensors=['ETM','LC08'] can not find model'

using bbox

Hi, I would like to download all Landsat 8 scenes that intersect a bounding box. Is this provided by the bbox option? If so, can you explain the syntax as I cannot get it to work.

I've tried: landsatxplore search --dataset LANDSAT_8_C1 --bbox -49.73 64.2 -49.48 64.3 --start 2018-03-01 --end 2018-09-01 --output csv > results.csv
which returns no results.

However: landsatxplore search --dataset LANDSAT_8_C1 --location 64.2 -49.73 --start 2018-03-01 --end 2018-09-01 --output csv > results.csv
returns many results

Thanks,

Ben

EE: Could not find api key

Hi!

When I try to download a set of scenes, I have this error:

py3.6.egg/landsatxplore/api.py", line 41, in request raise EarthExplorerError('EE: {}'.format(response['error'])) landsatxplore.exceptions.EarthExplorerError: EE: Could not find api key

landsatxplore download the first image but, the other fail with that error.

I am trying to download:

LC08_L1TP_220078_20181211_20181226_01_T1 and LC08_L1TP_220078_20190128_20190206_01_T1

KeyError: 'content-disposition'

Hi,
first first of all I want to express, that I now, that there is an issue with the swame title just thought the problem there, might be different.

I have some problems downloading data from the earth explorer. after a while the file didn't start to load down, this error is thrown:

0%| | 0.00/341 [00:00<?, ?B/s]
Traceback (most recent call last):
File "E:___SVN\COP4EE-2\Landcover\Code\venv\lib\site-packages\landsatxplore\earthexplorer.py", line 77, in _download
local_filename = r.headers['Content-Disposition'].split('=')[-1]
File "E:___SVN\COP4EE-2\Landcover\Code\venv\lib\site-packages\requests\structures.py", line 54, in getitem
return self._store[key.lower()][1]
KeyError: 'content-disposition'

The code I use is:

for year, month in observation_period:
    # Request
    scenes = api.search(
        dataset='LANDSAT_8_C1',
        bbox=bbox,
        start_date=str(date(year, month, 1)),
        end_date=str(date(year, month, monthrange(year, month)[1])),
        max_cloud_cover=20)

    gdf = to_geodataframe(scenes).to_crs(32633).sort_values(by='cloudCover', ascending=True)
    filtered = filter_by_min_intersect(gdf, local_vector)

    save_path = Path(args.save_path) / str(year) / str(month)
    if not save_path.exists():
        makedirs(save_path)
    for entry, name in zip(filtered.entityId.values, filtered.displayId.values):
        if type(entry) == str:
            if name not in [x.stem.split(".")[0] for x in save_path.iterdir()]:
                print("Start loading {}.".format(name))
                ee.download(scene_id=entry, output_dir=save_path)

ee.logout()
api.logout()

api search only 20 scenes

Hi,

I am trying api search and the scenes only return 20 scenes. It seems like the first 20 images.
Here are my codes:
scenes = api.search(
...: dataset='SP_TILE_DSWE',
...: latitude=41.33,
...: longitude=-72.80,
...: start_date='2004-01-01',
...: end_date='2018-12-31',
...: max_cloud_cover=10)
Yue

Issue downloading LandSat 8 Collection 2

Hello,
I encountered an issue downloading some LandSat 8 Collection 2 images with landsatxplore API.
The output / error written is :

    raise EarthExplorerError(error_msg)
landsatxplore.errors.EarthExplorerError: Download is not available

I checked on usgs website (earth explorer) that the tile I am looking for is available.
Thank you for your help

Error when running the download script. TypeError: download() got an unexpected keyword argument 'scene_id'

When I run the script below (which I copy-pasted from landsatxplore 0.13.0 page), it returns 'TypeError: download() got an unexpected keyword argument 'scene_id''.

from landsatxplore.earthexplorer import EarthExplorer

ee = EarthExplorer(username, password)

ee.download(scene_id='LT51960471995178MPS00', output_dir='./data')

ee.logout()

"KeyError: 'content-disposition'"

I'm getting a weird content disposition error with my output, not entirely sure where I've gone wrong. I'm using a similar code to that which has been posted before:

from landsatxplore.earthexplorer import EarthExplorer

ee = EarthExplorer("UUUUNNNN", "PPPPPPPP")
i = 1
with open("tiles.txt", 'r') as input_data:
    for scene in input_data:
        ee.download(
            scene_id=str(scene),
            output_dir='./output'
        )
ee.logout()

The error is as follows:

Traceback (most recent call last):
  File "LSX_DL.py", line 10, in <module>
    output_dir=r'./output'
  File "/home/user/.local/lib/python3.7/site-packages/landsatxplore/earthexplorer.py", line 96, in download
    filename = self._download(url, output_dir)
  File "/home/user/.local/lib/python3.7/site-packages/landsatxplore/earthexplorer.py", line 77, in _download
    local_filename = r.headers['Content-Disposition'].split('=')[-1]
  File "/home/user/.local/lib/python3.7/site-packages/requests/structures.py", line 54, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'content-disposition'

Landsat 9 updating

Thank you so much for your hard work.
I would like if you are thinking of including the dataset id for the new Landsat 9 product in the code.

Best regards,
Jesús.

surface reflectance data and meta data only

surface reflectance:

We would like to only download SR scenes from landsat_ot_c2_l2. But landsat_ot_c2_l2 downloading does not work (says download not available).

Meta data:
Is it possible to have only the metadata so that we can select which products to download.
We want to reduce the files and size of the product downloaded. We want to filter by surface reflectance before downloading to reduce the size.

Ref #38

scenes found, but "EarthExplorerError: Download is not available" error comes while downloading

import json
from landsatxplore.api import API
from landsatxplore.earthexplorer import EarthExplorer

Initialize a new API instance and get an access key

usgs_username = ""
usgs_password = ""

api = API(usgs_username, usgs_password)

Search for Landsat TM scenes

scenes = api.search(
dataset='landsat_ot_c2_l1',
latitude=30.9,
longitude=75.85,
#bbox= (70, 28, 80, 32), #(xmin, ymin, xmax, ymax),
start_date='2017-01-01',
end_date='2017-04-01',
max_cloud_cover=10
)

print(f"{len(scenes)} scenes found.")

Process the result

for scene in scenes:
print(scene['acquisition_date'].strftime('%Y-%m-%d'))
# Write scene footprints to disk
fname = f"{scene['landsat_product_id']}.geojson"
with open(fname, "w") as f:
json.dump(scene['spatial_coverage'].geo_interface, f)
print(f)

api.logout()

output

3 scenes found.
2017-03-19
<_io.TextIOWrapper name='LC08_L1TP_148038_20170319_20200904_02_T1.geojson' mode='w' encoding='cp1252'>
2017-03-19
<_io.TextIOWrapper name='LC08_L1TP_148039_20170319_20200904_02_T1.geojson' mode='w' encoding='cp1252'>
2017-02-15
<_io.TextIOWrapper name='LC08_L1TP_148039_20170215_20200905_02_T1.geojson' mode='w' encoding='cp1252'>

Downloading Scenes

ee = EarthExplorer(usgs_username, usgs_password)

ee.download('LC08_L1TP_148038_20170319_20200904_02_T1', output_dir=r'D:\RF_Paper\JAM\MODIS')

ee.logout()

error

---

EarthExplorerError Traceback (most recent call last)
in
2 ee = EarthExplorer(usgs_username, usgs_password)
3
----> 4 ee.download('LC08_L1TP_148038_20170319_20200904_02_T1', output_dir=r'D:\RF_Paper\JAM\MODIS')
5
6 ee.logout()

~\Anaconda3\lib\site-packages\landsatxplore\earthexplorer.py in download(self, identifier, output_dir, dataset, timeout, skip)
148 data_product_id=DATA_PRODUCTS[dataset], entity_id=entity_id
149 )
--> 150 filename = self._download(url, output_dir, timeout=timeout, skip=skip)
151 return filename

~\Anaconda3\lib\site-packages\landsatxplore\earthexplorer.py in _download(self, url, output_dir, timeout, chunk_size, skip)
90 error_msg = r.json().get("errorMessage")
91 if error_msg:
---> 92 raise EarthExplorerError(error_msg)
93 download_url = r.json().get("url")
94

EarthExplorerError: Download is not available

Unable to login EE

I am able to login for "landsatxplore" but unable to login for "EarthExplorer". Surprisingly, previously I was able to login.
EarthExplorerError: EE: login failed.

Error when running search script. AttributeError: module 'dateutil.parser' has no attribute 'ParserError'

When I run the script below (which I copy-pasted from landsatxplore 0.13.0 page), it returns 'AttributeError: module 'dateutil.parser' has no attribute 'ParserError''.

import json
from landsatxplore.api import API

# Initialize a new API instance and get an access key
api = API(username, password)

# Search for Landsat TM scenes
scenes = api.search(
    dataset='landsat_tm_c1',
    latitude=50.85,
    longitude=-4.35,
    start_date='1995-01-01',
    end_date='1995-10-01',
    max_cloud_cover=10
)

print(f"{len(scenes)} scenes found.")

# Process the result
for scene in scenes:
    print(scene['acquisition_date'])
    # Write scene footprints to disk
    fname = f"{scene['landsat_product_id']}.geojson"
    with open(fname, "w") as f:
        json.dump(scene['spatialCoverage'], f)

api.logout()

AttributeError: module 'dateutil.parser' has no attribute 'ParserError'

Is someone facing this error?

~\Downloads\landsatxplore-master\landsatxplore\api.py in _to_date(src_string)
324 try:
325 return parser.parse(src_string)
--> 326 #except parser.ParserError:
327 except Exception:
328 try:

AttributeError: module 'dateutil.parser' has no attribute 'ParserError'

error correction

def _parse_metadata(response, parse_browse_field=False):
"""Parse the full response returned by the API when requesting metadata."""
metadata = {}
for key, value in response.items():
name = _camel_to_snake(key)
if key == "browse":
if parse_browse_field:
metadata[name] = _parse_browse_metadata(value)
else:
continue
elif key == "spatialCoverage":
metadata[name] = shape(value)
elif key == "spatialBounds":

        metadata[name] = shape(value)
    elif key == "temporalCoverage":
        start, end = value["endDate"], value["startDate"]
        metadata[name] = [_to_date(start), _to_date(end)]
    elif key == "metadata":
        metadata.update(_parse_metadata_field(value))
    else:
        # Do not parse numeric IDs. Keep them as strings.
        if name.endswith("_id"):
            metadata[name] = str(value).strip()
        else:
            metadata[name] = _parse_value(value)
if "acquisition_date" not in metadata:
    metadata["acquisition_date"] = metadata["temporal_coverage"][0]
return metadata

download: EarthExplorerError: EE: login failed (csrf token not found).

Download operation fails with

landsatxplore.exceptions.EarthExplorerError: EE: login failed (csrf token not found).

Tested via API and command line tools. Note that search (LandsatAPI) works with the same username and password.

Search

LANDSATXPLORE_USERNAME=XXX LANDSATXPLORE_PASSWORD=XXX landsatxplore search --dataset LANDSAT_TM_C1 --location 12.53 -1.53 \
>     --start 1995-01-01 --end 1995-12-31
LT51950511995123MPS00
LT51950511995139MPS00
LT51950511995155MPS00
LT51950511995171MPS00
LT51950511995203MPS00
LT51950511995219MPS00

Download

LANDSATXPLORE_USERNAME=XXX LANDSATXPLORE_PASSWORD=XXX landsatxplore download LT51960471995178MPS00 
Traceback (most recent call last):
  File "/usr/local/bin/landsatxplore", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/dist-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/landsatxplore/cli.py", line 96, in download
    ee = EarthExplorer(username, password)
  File "/usr/local/lib/python3.7/dist-packages/landsatxplore/earthexplorer.py", line 45, in __init__
    self.login(username, password)
  File "/usr/local/lib/python3.7/dist-packages/landsatxplore/earthexplorer.py", line 58, in login
    csrf, ncform = _get_tokens(rsp.text)
  File "/usr/local/lib/python3.7/dist-packages/landsatxplore/earthexplorer.py", line 32, in _get_tokens
    raise EarthExplorerError('EE: login failed (csrf token not found).')
landsatxplore.exceptions.EarthExplorerError: EE: login failed (csrf token not found).

System Info

• Debian GNU/Linux stable
• Python 3.7.3
• landsatxplore 0.6 (pip)

Problems with login for EarthExplorer

Using my credentials, I can create an instance of API. But using the same credentials, it fails to create an instance of EarthExplorer. Can it be connected to my credentials, or is that something with the usage/code?

>>> from landsatxplore.api import API
>>> api = API(username, password)
>>> from landsatxplore.earthexplorer import EarthExplorer
>>> ee = EarthExplorer(username, password)
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 726, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 410, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 734, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 670, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 426, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 421, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1322, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 303, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 272, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/site-packages/landsatxplore/earthexplorer.py", line 55, in __init__
    self.login(username, password)
  File "/usr/local/lib/python3.8/site-packages/landsatxplore/earthexplorer.py", line 73, in login
    rsp = self.session.post(EE_LOGIN_URL, data=payload, allow_redirects=True)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 581, in post
    return self.request('POST', url, data=data, json=json, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 498, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

Using landsatxplore == 0.13.0 and requests == 2.22.0 (tested also with 2.25.1).

scenes retrieved do not correspond to bbox provided

Hello,

I'm trying to write a wrapper for GRASS GIS using landsatxplore. See i.landsat.download. It is really basic right now.

However in my tests using landsatxplore api and passing a bbox like a tuple obtained either from the extent of a vector map or the region settings in GRASS GIS: ('-78.77428134', '35.68792712', '-78.60830318', '35.80960938'), I get scenes located in Antarctica (while the coordinates belong to an area in North Carolina state). This is the script I use just to test:

#!/usr/bin/env python

import landsatxplore.api

# Initialize a new API instance and get an access key
api = landsatxplore.api.API('user', 'password')

# Request
scenes = api.search(
    dataset='LANDSAT_ETM_C1',
    bbox = ('-78.77428134', '35.68792712', '-78.60830318', '35.80960938'), #(xmin, ymin, xmax, ymax)
    start_date='2000-01-01',
    end_date='2000-07-01',
    max_cloud_cover=10)

print('{} scenes found.'.format(len(scenes)))

print('ID', 'DisplayID', 'Date', 'Clouds')
for scene in scenes:
    print(scene['entityId'], scene['displayId'], scene['acquisitionDate'], scene['cloudCover'])

api.logout()

The bbox corresponds to a small area in North Carolina state, however, I get scenes with path and row from Antartica

5 scenes found.
ID DisplayID Date Clouds
LE71311172000001EDC00 LE07_L1GT_131117_20000101_20170215_01_T2 2000-01-01 0.00
LE71381162000002SGS00 LE07_L1GT_138116_20000102_20170215_01_T2 2000-01-02 0.00
LE71361162000004SGS00 LE07_L1GT_136116_20000104_20170215_01_T2 2000-01-04 0.00
LE71361172000004SGS00 LE07_L1GT_136117_20000104_20170215_01_T2 2000-01-04 0.00
LE71331172000015SGS01 LE07_L1GT_133117_20000115_20170215_01_T2 2000-01-15 0.00

Is there a problem with the bbox I'm passing or is it in the api?
When I use the options longitude and latitude instead of bbox, I get the right path and rows:

# all the rest the same
scenes = api.search(
    dataset='LANDSAT_ETM_C1',
    longitude=-78.77428134,
    latitude=35.68792712,
    start_date='2000-01-01',
    end_date='2000-07-01',
    max_cloud_cover=10)

# result
10 scenes found.
ID DisplayID Date Clouds
LE70160352000011EDC00 LE07_L1TP_016035_20000111_20161002_01_T1 2000-01-11 0.00
LE70160352000027EDC00 LE07_L1TP_016035_20000127_20161002_01_T1 2000-01-27 0.00
LE70150352000036EDC00 LE07_L1TP_015035_20000205_20161003_01_T1 2000-02-05 7.00
LE70150352000052EDC00 LE07_L1TP_015035_20000221_20161002_01_T1 2000-02-21 0.00
...

Any help is highly appreciated :)

Landsat 4 partly supported?

I try to get Lansat 5 data and get an error: landsatxplore.errors.LandsatxploreError: Failed to guess dataset from identifier.

import landsatxplore.api
from landsatxplore.earthexplorer import EarthExplorer

bb=('-0.49799280', '5.48261312', '0.67396018', '6.03887914')

scenes = landsat_api.search(dataset='landsat_tm_c2_l2', bbox=bb,start_date='1990-10-01', end_date='1991-01-31', max_cloud_cover='15', max_results=50)
# scenes are: [{'cloud_cover': 15.0, 'entity_id': 'LT41930561990359XXX03', ...]

ee.download(identifier="LT41930561990359XXX03",  output_dir=outdir, timeout=int(options["timeout"]))
*** landsatxplore.errors.LandsatxploreError: Failed to guess dataset from identifier.

It seems that with landsat_tm_c2_l2 I get scenes from landsat 4 and 5 (

landsatxplore/landsatxplore/util.py

Line 80 in 2a97d28

"satellite": scene_id[2],

), but 4 is not yet supported

landsatxplore/landsatxplore/util.py

Line 90 in 2a97d28

def landsat_dataset(satellite, collection="c1", level="l1"):

.

Is my assumption correct, or does the error come from somewhere else?
And is it possible to integrate landsat 4 completely?

EarthExplorer should create directories if it can't find them

For example, if I typed

ee.download(scene_id='LT51960471995178MPS00', output_dir="C:/Users/MyUser/Desktop/MyData/test_data")

It should create the folder test_data if it already doesn't exist, in case the user forgot to put it there. Prevents them from spending time bughunting.

return the downloaded file name

It will be useful (to me) that the download procedure in python return the filename of the tar.gz download.

Previous API for search now retired

landsatxplore search now returns an exception
command : landsatxplore search --dataset LANDSAT_TM_C1 --location 12.53 -1.53 --start 1995-01-01 --end 1995-12-31
exception :
...landsatxplore/api.py", line 50, in login
raise EarthExplorerError('EE: {}'.format(response['error']))
landsatxplore.exceptions.EarthExplorerError: EE: This API has been retired and replaced by the API at https://m2m.cr.usgs.gov

question about EE_folder

Hi Yann,

Just have a quick question about the "EE_folder" and "EE_size" in your code.

EE_FOLDER = {
'LANDSAT_TM_C1': '12266',
'LANDSAT_ETM_C1': '12267',
'LANDSAT_8_C1': '12864'
}
SIZES = {
'LANDSAT_TM_C1': 150 * 10242,
'LANDSAT_ETM_C1': 235 * 10242,
'LANDSAT_8_C1': 919 * 1024**2
}
I wonder where did you find such information?

Thank you very much for your help.

Best,
Tong

Error: "KeyError: 'content-disposition'"

I'm trying to download several scenes with the following python script::

from landsatxplore.earthexplorer import EarthExplorer

ee = EarthExplorer(LX_USERNAME, LX_PASSWORD)
i = 1
with open(FILE_SCENES, 'r') as input_data:
    for scene in input_data:
        print "[%s] -> %s" % (i, scene)
        ee.download(
            scene_id=str(scene), 
            output_dir='/home/usuario/descargas'
        )
        time.sleep(30*60) # Delay for 30 minute (1800 seconds).
ee.logout()

However, I keep receiving this error:

Traceback (most recent call last):
  File "./script_descarga.py", line 34, in <module>
    output_dir=str(OUTPUT_DIR.absolute())
  File "/home/usuario/.local/lib/python2.7/site-packages/landsatxplore/earthexplorer.py", line 97, in download
    self._download(url, output_dir, file_size=SIZES[dataset])
  File "/home/usuario/.local/lib/python2.7/site-packages/landsatxplore/earthexplorer.py", line 82, in _download
    local_filename = r.headers['Content-Disposition'].split(
  File "/home/usuario/.local/lib/python2.7/site-packages/requests/structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
KeyError: 'content-disposition'

It's like USGS is blocking the download, however, if I execute the request from the command line, landsatxplore works. Any suggestions?...

Use getpass for API

I have a suggestion regarding credentials for API. Since passing user and pass as plain texts is not secure, you can use a Python built-in library called getpass. Here's an example:

import getpass as gp

urs = "https://ers.cr.usgs.gov/register"

prompts = {
    "username": "\n".join(
        [
            "Enter EarthExplorer username",
            f"(or create an account at {urs}): ",
        ]
    ),
    "password": "Enter EarthExplorer password: ",
}

user_data = {p: getpass(prompt=msg) for p, msg in prompts.items()}

This will prompt the user to enter the user and pass and will show asterisks (or bullets) as the user types the info.

Then you can carry on with login:

self.login(**user_data)

I can submit a PR if interested.

broken links in readme for registration

correct url to get access https://ers.cr.usgs.gov/login?redirectUrl=https://ers.cr.usgs.gov/profile/access

Issue with download function in EarthExplorer class

I have been using the download function multiple times but today I encounter this error...

There was some maintenance in the USGS portal yesterday and I am not sure if something went wrong because of that... I will be thankful for your kind help.

Download does not reach the end

Hello Yann,

Thanks so much for this module; helped me so much. Here's the thing: I'm running the code for download landsat_8 images, simple as you described in the description for usage; but my downloads don't finish... They stop in 96%, do you know why this is happening ?

Problem with Scene Search and EE downloads

Hi @yannforget , I've encounter these issue very recently. I have included the error below. Any chance its due to the USGS upgraded?
The error for scene search:

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The error for EE logins:

EarthExplorerError: EE: login failed (csrf token not found).

Collection 2 support?

Hello, I am sorry if I missed this elsewhere but are there any plans to support Collection 2 data? I just found this module and it would be highly useful in a pipeline I am created but we are using Collection 2 imagery.

Thank you!

Improvement parameters T1, 2 and RT ?

Hi,
Thank you a lot for your api and repos.
I would like to know if you are planning to improve the landsatxplore library with parameters selecting the category of images from the Collection 1 of LandSat 8 such that we can select Tier 1, Tier 2 or Real-Time images.
Also, will it be possible to return only images that are concerned by SR treatment ?
@jacquesmoati

Multiple scenes download

Hi! I appreciate the efforts for this package. It says download one or several scenes. Could let us know if the package can be used to download data parallelly and how?

Thanks!

acquisition_date seems to not exist in metadata

While updating the GRASS GIS addon i.landsat.download to support the recent changes in landsatxplore (Thanks much!!!), I found out that metadata regarding date has changed in both collection 1 and 2 Landsat data. There's not such thing as acquisitionDate anymore nor acquisition_date, but now there is a dictionary temporalCoverage with keys start and end date or a different key called publishDate. See OSGeo/grass-addons#450.

Where did you find acquisition_date variable when updating the README f44f122 ? If I perform the search and print scenes, I do not see it.

Download skip / continue

Hello, great job on your APIs, congrats!

I think that the download method does not support skipping, if the file already exists and is correct.
Another interesting feature would be the continuation of incomplete downloads, in case of errors etc.

What are your thoughts? I could help if you are interested in these features.

Problem with Download

Hi,
I have problem with open_dir with your script , cause i have some errors with it. Can u help me?
I want use this script to download landsat images by python2.7 .
this is my code:
import landsatxplore.api
from landsatxplore.earthexplorer import EarthExplorer

ee = EarthExplorer("*******, "********")
ee.download(scene_id='LT51960471995178MPS00', output_dir= 'I:\mmine\FILE\SCenes')

ee.logout()

and i have this erros:
Traceback (most recent call last):
File "C:/Users/Przemo/Desktop/US2.py", line 6, in
ee.download(scene_id='LT51960471995178MPS00', output_dir= 'I:\mmine\FILE\SCenes')
File "C:\Python27\ArcGIS10.6\lib\site-packages\landsatxplore\earthexplorer.py", line 96, in download
filename = self._download(url, output_dir)
File "C:\Python27\ArcGIS10.6\lib\site-packages\landsatxplore\earthexplorer.py", line 76, in _download
with self.session.get(url, stream=True, allow_redirects=True) as r:
AttributeError: exit

TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

below is my script for downloading landsat 8 data:

import landsatxplore.api
from landsatxplore.earthexplorer import EarthExplorer

# Initialize a new API instance and get an access key
api = landsatxplore.api.API(username='xxxxx', password='xxx')

# Request
scenes = api.search(
    dataset='LANDSAT_8_C1',
    latitude=38.7,
    longitude=115.5,
    start_date='2019-01-01',
    end_date='2020-01-01',
    max_cloud_cover=50)

print('{} scenes found.'.format(len(scenes)))

ee = EarthExplorer(username='xxxxx', password='xxx')
for scene in scenes:
    print(scene['acquisitionDate'], scene['entityId'])
    ee.download(scene_id=scene['entityId'], output_dir='.')


api.logout()
ee.logout()

and the error msg looks like this:

20 scenes found.
2019-01-05 LC81230332019005LGN00
Traceback (most recent call last):
  File "download.py", line 21, in <module>
    ee.download(scene_id=scene['entityId'], output_dir='.')
  File "/home/user/anaconda3/lib/python3.7/site-packages/landsatxplore-0.6-py3.7.egg/landsatxplore/earthexplorer.py", line 95, in download
    filename = self._download(url, output_dir)
  File "/home/user/anaconda3/lib/python3.7/site-packages/landsatxplore-0.6-py3.7.egg/landsatxplore/earthexplorer.py", line 75, in _download
    file_size = int(r.headers.get("Content-Length"))
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

Could you please give me a hint on this error?

unable to download c2l1 data prior to December 2020

Hello,
Thanks for creating this great API. I'm unfortunately having trouble downloading any collect 2 level 1 data prior to December 2020. For example, I would like to download LC08_L1TP_044035_20200130_20200823_02_T1, a c2l1 scene from January 2020. landsatxplore download --username <USERNAME> --password <PASSWORD> LC08_L1TP_044035_20200130_20200823_02_T1 errors out with landsatxplore.errors.EarthExplorerError: Download is not available, as does attempting to download with the entity id using landsatxplore download --username <USERNAME> --password <PASSWORD> --dataset landsat_ot_c2_l1 LC80440342020030LGN00. I have also tried going directly to the download URL, https://earthexplorer.usgs.gov/download/5e81f14ff4f9941c/LC80440342020030LGN00/EE/, and encountered the same error. This issue does not affect data from December 2020 or later.

bbox with api

Hi. I'm trying to use your api with a bbox parameter instead of longitude and latitude but i keep getting an empty list. Am i missing something?

scene could not be searched: LC08_L1TP_123040_20210502_20210502_01_RT

for instance, no scenes found,returned by the following command
landsatxplore search --dataset LANDSAT_8_C1 --location 29.4 113.1 --clouds 20 --start 2020-05-01 --end 2021-05-08

However, this scene LC08_L1TP_123040_20210502_20210502_01_RT exists and can be downloaded by landsatxplore download

Downloading Sentinel 2a

What is the keyword for downloading Sentinel 2a? In Landsat, the dictionary(scenes) contain the Landsat_scene_id which is used in the
ee.download('LT51960471995178MPS00', output_dir='./data') . But the information is missing in the sentinel 2a. What is the equivalent of Landsar_scene_id when downloading sentinel images

Missmatch results usgs vs landsatxplore

There seems to be divergent results when using the max_cloud_cover between landsatxplore and the results from website earthexplorer from usgs.
For example, filtering for landsat 8 l1c1, cloud_coverage=0,from aug.2013 to 19.11.2021 I got 250 scenes found using this landsatexplore, but using the usgs ee i got 32 results.
So, after some research I ve found out that the cloud_coverage with 0(no clouds) isn t working at all with landsatxplore(like no filtering),
and in the doc code itself sometimes the min value is 1 sometimes is 0.

JSONDecodeError using API(username, password)

File "C:\Users\mas108\AppData\Local\Continuum\anaconda3\lib\site-packages\landsatxplore\earthexplorer.py", line 46, in init
self.api = API(username, password)

File "C:\Users\mas108\AppData\Local\Continuum\anaconda3\lib\site-packages\landsatxplore\api.py", line 27, in init
self.key = self.login(username, password)

File "C:\Users\mas108\AppData\Local\Continuum\anaconda3\lib\site-packages\landsatxplore\api.py", line 47, in login
response = requests.post(self.endpoint + 'login?', data=data).json()

File "C:\Users\mas108\AppData\Local\Continuum\anaconda3\lib\site-packages\requests\models.py", line 900, in json
return complexjson.loads(self.text, **kwargs)

File "C:\Users\mas108\AppData\Local\Continuum\anaconda3\lib\json_init_.py", line 348, in loads
return _default_decoder.decode(s)

File "C:\Users\mas108\AppData\Local\Continuum\anaconda3\lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())

File "C:\Users\mas108\AppData\Local\Continuum\anaconda3\lib\json\decoder.py", line 356, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None

JSONDecodeError: Expecting value

I think the download URL in EE has changed.

The source code below is the source code I added the status_code code to check. Is there any place to know the information of the download URL? Or does anyone know?

url = "https://earthexplorer.usgs.gov/download/12864/LC81150342020112LGN00/STANDARD/EE" def _download(self, url, output_dir, chunk_size=1024): """Download remote file given its URL.""" with self.session.get(url, stream=True, allow_redirects=True) as r: if r.status_code == 404: raise "404 not found..." file_size = int(r.headers['Content-Length']) Common.Parameter.FileSize = file_size with tqdm(total=file_size, unit_scale=True, unit='B', unit_divisor=1024) as pbar: local_filename = r.headers['Content-Disposition'].split('=')[-1] local_filename = os.path.join(output_dir, local_filename) with open(local_filename, 'wb') as f: for chunk in r.iter_content(chunk_size=chunk_size): if chunk: f.write(chunk) pbar.update(chunk_size) return local_filename