Giter Site home page Giter Site logo

flaviostutz / sentinelloader Goto Github PK

View Code? Open in Web Editor NEW
39.0 6.0 18.0 1.09 MB

Sentinel-2 satellite tiles images downloader from Copernicus. Minimizes data download and combines multiple tiles to return a single area of interest.

License: MIT License

Dockerfile 0.46% Python 36.42% Jupyter Notebook 63.13%
copernicus satellite-imagery sentinel-2 hacktoberfest

sentinelloader's Introduction

sentinelloader

Sentinel-2 satellite tiles images downloader from Copernicus.

With this utility you can specify the desired polygon, image resolution, band name and aproximate dates and it will do the best effort to find all tiles needed to satisfy your requirement. Then it will download minimal data by selecting just the needed .jp2 files inside Products, combine downloaded tiles, crop the combined tiles image to the polygon and cache the results, returning a GeoTIFF image with raster for the selected area.

All API calls are in ESP:4326 reference.

Background

Granules are packages containing data taken from Sentinel-2 satellite for a region on the globe in a specific time. They contain a lot of data for that area (13 bands in different resolutions and other derived bands and quality data). Level-2A products, for example, have ~1GB of data for a single tile (100km2 x 100km2).

With this utility you can select which bands/resolutions to download. For example, if you need only the TCI band (true color) tile at 60m resolution, you will can use the utility to download just ~3MB of data (instead of 1GB!). For max resolution(10m), each band will have ~120MB. Some caching will be applied to avoid re-downloading of data that is already present in disk.

Usage

Docker example

  • Create docker-compose.yml
version: '3.3'
services:
  sentinelloader:
    image: flaviostutz/sentinelloader
    environment:
      - COPERNICUS_USER=auser
      - COPERNICUS_PASSWORD=apass
    ports:
      - 8686:8888
  • Create an account in Copernicus and change info in docker-compose.yml accordingly

  • Run docker-compose up -d

  • Open your browser at http://localhost:8686/

  • Open Jupyter notebook "example.ipynb" and press "Run"

  • You should see something like this

Python example

  • To install the latest version from GitHub
pip install git+https://github.com/flaviostutz/sentinelloader
  • To install the latest version from pypi
pip install sentinelloader
import logging
import os
from osgeo import gdal
import matplotlib.pyplot as plt
from sentinelloader import Sentinel2Loader
from shapely.geometry import Polygon

sl = Sentinel2Loader('/notebooks/data/output/sentinelcache', 
                    'mycopernicususername', 'mycopernicuspassword',
                    apiUrl='https://apihub.copernicus.eu/apihub/', showProgressbars=True, loglevel=logging.DEBUG)

area = Polygon([(-47.873796, -16.044801), (-47.933796, -16.044801),
        (-47.933796, -15.924801), (-47.873796, -15.924801)])

geoTiffs = sl.getRegionHistory(area, 'TCI', '60m', '2019-01-06', '2019-01-30', daysStep=5)
for geoTiff in geoTiffs:
    print('Desired image was prepared at')
    print(geoTiff)
    os.remove(geoTiff)

For a Jupyter example, click here

API

def getRegionHistory(self, geoPolygon, bandOrIndexName, resolution, dateFrom, dateTo, daysStep=5, ignoreMissing=True, minVisibleLand=0, visibleLandPolygon=None, keepVisibleWithCirrus=False, interpolateMissingDates=False):
        """Gets a series of GeoTIFF files for a region for a specific band and resolution in a date range. It will make the best effort to get images near the desired dates and filter out images that have poor land visibility due to cloudy days"""

minVisibleLand - a value from 0 to 1 indicating the percentage of land that must be visible on the image (according to cloud coverage at the time)

sl = SentinelLoader('/notebooks/data/output/sentinelcache', 'mycopernicususername', 'mycopernicuspassword', apiUrl='https://scihub.copernicus.eu/apihub/', showProgressbars=True, loglevel=logging.DEBUG)

desired_region = Polygon([(-47.873796, -16.044801), (-47.933796, -16.044801),(-47.933796, -15.924801), (-47.873796, -15.924801)])

geoTiffs = sl.getRegionHistory(desired_region, 'TCI', '60m', '2019-01-06', '2019-01-30', daysStep=5)

  • In this example, sentinelloader will connect to Coperrnicus with your account and try to get various images in the band "TCI" of the desired region at a resolution of 60m fom 2019-01-06 to 2019-01-30 (if still available in Copernicus Hub) each 5 days (it will try to get the closes image to the days selected, because not every day we have images for every places).

  • Supported band names

    • All bands that are part of Sentinel 2 products at Copernicus Hub (SCL, TCI, B01-08, B1A etc)
    • Sintetic indexes implemented by this tool: NDVI, NDWI, NDWI_MacFeeters or NDMI
    • If you implement a newer one, please send a PR with it!

Publishing package to pypi

python3 -m pip install --upgrade build

python3 -m build

python3 -m twine upload dist/*

https://packaging.python.org/en/latest/tutorials/packaging-projects/#uploading-the-distribution-archives

sentinelloader's People

Contributors

amoldosanjh avatar ashnair1 avatar devantler avatar flaviostutz avatar landswellsong avatar tiagostutz avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar

sentinelloader's Issues

Import error when running docker

Hiya, I just set up to user your docker container because my kernel suddenly started dying when running getRegionHistory (yesterday it worked fine, I triggered the LTA retrieval, and today the files are shown to be online but the kernel dies during execution of getREgionHistory),

But when importing sentinelloader I get the following error:

Traceback (most recent call last):

  File "/usr/local/lib/python3.5/dist-packages/IPython/core/interactiveshell.py", line 3291, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-1-9268305e3050>", line 4, in <module>
    from sentinelloader import Sentinel2Loader

  File "<frozen importlib._bootstrap>", line 969, in _find_and_load

  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked

  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible

  File "/usr/local/lib/python3.5/dist-packages/sentinelloader-1.1.1-py3.5.egg/sentinelloader/__init__.py", line 1, in <module>
    from .sentinel2loader import Sentinel2Loader

  File "<frozen importlib._bootstrap>", line 969, in _find_and_load

  File "<frozen importlib._bootstrap>", line 958, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 664, in _load_unlocked

  File "<frozen importlib._bootstrap>", line 634, in _load_backward_compatible

  File "/usr/local/lib/python3.5/dist-packages/sentinelloader-1.1.1-py3.5.egg/sentinelloader/sentinel2loader.py", line 1, in <module>
    from sentinelsat import SentinelAPI

  File "/usr/local/lib/python3.5/dist-packages/sentinelsat-1.1.0-py3.5.egg/sentinelsat/__init__.py", line 4, in <module>
    from . import sentinel

  File "/usr/local/lib/python3.5/dist-packages/sentinelsat-1.1.0-py3.5.egg/sentinelsat/sentinel.py", line 287
    self.logger.info(f"Found {count:,} products")
                                               ^
SyntaxError: invalid syntax

Sentinelcache

Sorry in advance for a basic support issue! I am interested in running the python example. Could you please provide some explanation on how to get started with setting up the sentinelcache file? I am not sure how to run sentinelloader using my own Copernicus hub credentials in this part:

sl = SentinelLoader('/notebooks/data/output/sentinelcache', 
                    'mycopernicususername', 'mycopernicuspassword',
                    apiUrl='https://scihub.copernicus.eu/apihub/', showProgressbars=True, loglevel=logging.DEBUG)

segmentation fault when running python example

❯ python loader.py
INFO:sentinelloader:Getting region history for band TCI from 2019-01-06 to 2019-01-30 at 60m
DEBUG:sentinelloader:2019-01-06 00:00:00
DEBUG:sentinelloader:Getting contents. band=TCI, resolution=60m, date=2019-01-06
DEBUG:sentinelloader:Querying API for candidate tiles
DEBUG:sentinelloader:Querying remote API
DEBUG:sentinelsat.SentinelAPI:Running query: order_by=None, limit=None, offset=0, query=beginPosition:[2019-01-01T00:00:00Z TO 2019-01-06T00:00:00Z] cloudcoverpercentage:[0 TO 80] platformname:Sentinel-2 producttype:S2MSI2A footprint:"Intersects(POLYGON ((-47.933796 -15.924801, -47.933796 -16.044801, -47.873796 -16.044801, -47.873796 -15.924801, -47.933796 -15.924801)))"
DEBUG:sentinelsat.SentinelAPI:Sub-query: offset=0, limit=None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): scihub.copernicus.eu:443
DEBUG:urllib3.connectionpool:https://scihub.copernicus.eu:443 "POST /apihub/search?format=json&rows=100&start=0 HTTP/1.1" 200 None
[1]    23643 segmentation fault (core dumped)  python loader.py

I have set my copernicus credentials, running on arch linux, python 3.8.2

core dump

Process 23643 (python) of user 1000 dumped core.

Stack trace of thread 23643:
#0  0x00007fe27642a310 __libc_free (libc.so.6 + 0x8c310)
#1  0x00007fe2714d541f inflateReset2 (libcfitsio.so.8 + 0x12241f)
#2  0x00007fe26e170ef4 inflateInit2_ (libz.so.1 + 0xbef4)
#3  0x00007fe275e3da4f zlib_decompressobj_impl.isra.4 (zlib.cpython-38-x86_64-linux-gnu.so + 0x5a4f)
#4  0x0000558fa0189c34 cfunction_vectorcall_FASTCALL_KEYWORDS (python3.8 + 0x14ac34)
#5  0x0000558fa0229fb1 _PyObject_Vectorcall (python3.8 + 0x1eafb1)
#6  0x0000558fa01d227e function_code_fastcall (python3.8 + 0x19327e)
#7  0x0000558fa01d25c3 _PyObject_Call_Prepend (python3.8 + 0x1935c3)
#8  0x0000558fa01d277a slot_tp_init (python3.8 + 0x19377a)
#9  0x0000558fa01860dc type_call (python3.8 + 0x1470dc)
#10 0x0000558fa0229eea _PyObject_Vectorcall (python3.8 + 0x1eaeea)
#11 0x0000558fa01d1c7b function_code_fastcall (python3.8 + 0x192c7b)
#12 0x0000558fa02256c6 _PyObject_Vectorcall (python3.8 + 0x1e66c6)
#13 0x0000558fa01d1421 _PyEval_EvalCodeWithName (python3.8 + 0x192421)
#14 0x0000558fa01d1d9e _PyFunction_Vectorcall (python3.8 + 0x192d9e)
#15 0x0000558fa0225889 _PyObject_Vectorcall (python3.8 + 0x1e6889)
#16 0x0000558fa026f029 gen_send_ex (python3.8 + 0x230029)
#17 0x0000558fa0225ca9 _PyEval_EvalFrameDefault (python3.8 + 0x1e6ca9)
#18 0x0000558fa026f029 gen_send_ex (python3.8 + 0x230029)
#19 0x0000558fa0225ca9 _PyEval_EvalFrameDefault (python3.8 + 0x1e6ca9)
#20 0x0000558fa026f029 gen_send_ex (python3.8 + 0x230029)
#21 0x0000558fa01c5b64 list_extend (python3.8 + 0x186b64)
#22 0x0000558fa01c5de2 PySequence_List (python3.8 + 0x186de2)
#23 0x0000558fa01c5e54 PySequence_Fast (python3.8 + 0x186e54)
#24 0x0000558fa01c5ebd stringlib_bytes_join (python3.8 + 0x186ebd)
#25 0x0000558fa01b82fb method_vectorcall_O (python3.8 + 0x1792fb)
#26 0x0000558fa0225889 _PyObject_Vectorcall (python3.8 + 0x1e6889)
#27 0x0000558fa01d1f88 function_code_fastcall (python3.8 + 0x192f88)
#28 0x0000558fa0187c65 _PyObject_GenericGetAttrWithDict (python3.8 + 0x148c65)
#29 0x0000558fa0225b2a _PyEval_EvalFrameDefault (python3.8 + 0x1e6b2a)
#30 0x0000558fa01d0cac _PyEval_EvalCodeWithName (python3.8 + 0x191cac)
#31 0x0000558fa01d1de8 _PyFunction_Vectorcall (python3.8 + 0x192de8)
#32 0x0000558fa01ae0d2 _PyObject_Vectorcall (python3.8 + 0x16f0d2)
#33 0x0000558fa0185a2f PyVectorcall_Call (python3.8 + 0x146a2f)
#34 0x0000558fa02271e7 do_call_core (python3.8 + 0x1e81e7)
#35 0x0000558fa01d0cac _PyEval_EvalCodeWithName (python3.8 + 0x191cac)
#36 0x0000558fa01d1de8 _PyFunction_Vectorcall (python3.8 + 0x192de8)
#37 0x0000558fa01ae0d2 _PyObject_Vectorcall (python3.8 + 0x16f0d2)
#38 0x0000558fa0185a2f PyVectorcall_Call (python3.8 + 0x146a2f)
#39 0x0000558fa02271e7 do_call_core (python3.8 + 0x1e81e7)
#40 0x0000558fa01d0cac _PyEval_EvalCodeWithName (python3.8 + 0x191cac)
#41 0x0000558fa01d1de8 _PyFunction_Vectorcall (python3.8 + 0x192de8)
#42 0x0000558fa01adf45 _PyObject_Vectorcall (python3.8 + 0x16ef45)
#43 0x0000558fa0226727 _PyObject_Vectorcall (python3.8 + 0x1e7727)
#44 0x0000558fa01d0cac _PyEval_EvalCodeWithName (python3.8 + 0x191cac)
#45 0x0000558fa01d1d45 _PyFunction_Vectorcall (python3.8 + 0x192d45)
#46 0x0000558fa0225889 _PyObject_Vectorcall (python3.8 + 0x1e6889)
#47 0x0000558fa01d0cac _PyEval_EvalCodeWithName (python3.8 + 0x191cac)
#48 0x0000558fa01d1d45 _PyFunction_Vectorcall (python3.8 + 0x192d45)
#49 0x0000558fa0225889 _PyObject_Vectorcall (python3.8 + 0x1e6889)
#50 0x0000558fa01d0cac _PyEval_EvalCodeWithName (python3.8 + 0x191cac)
#51 0x0000558fa01d1de8 _PyFunction_Vectorcall (python3.8 + 0x192de8)
#52 0x0000558fa01adf45 _PyObject_Vectorcall (python3.8 + 0x16ef45)
#53 0x0000558fa0226727 _PyObject_Vectorcall (python3.8 + 0x1e7727)
#54 0x0000558fa01d1c7b function_code_fastcall (python3.8 + 0x192c7b)
#55 0x0000558fa0225889 _PyObject_Vectorcall (python3.8 + 0x1e6889)
#56 0x0000558fa01d1c7b function_code_fastcall (python3.8 + 0x192c7b)
#57 0x0000558fa0225889 _PyObject_Vectorcall (python3.8 + 0x1e6889)
#58 0x0000558fa01d0cac _PyEval_EvalCodeWithName (python3.8 + 0x191cac)
#59 0x0000558fa01d1de8 _PyFunction_Vectorcall (python3.8 + 0x192de8)
#60 0x0000558fa01adf45 _PyObject_Vectorcall (python3.8 + 0x16ef45)
#61 0x0000558fa0226727 _PyObject_Vectorcall (python3.8 + 0x1e7727)
#62 0x0000558fa01d0cac _PyEval_EvalCodeWithName (python3.8 + 0x191cac)
#63 0x0000558fa01d1b44 PyEval_EvalCodeEx (python3.8 + 0x192b44)

Error, could not load metada

Hello,

Im trying the python example provided in the readme, everything is fine until it tries to download the metadata then i get this error message

DEBUG:urllib3.connectionpool:https://apihub.copernicus.eu:443 "GET /apihub/odata/v1/Products('3b235be9-dc40-45c8-b06e-ed806cd24f94')/Nodes('S2B_MSIL2A_20230123T110249_N0509_R094_T29SQU_20230123T123318.SAFE')/Nodes('MTD_MSIL2A.xml')/$value HTTP/1.1" 500 535 INFO:sentinelloader:Couldn't get data for 2023-01-26 using the specified filter. err=Could not get metadata info. status=500

any help

Docker space issue

Hi Flavio Stutz
Thanks for this very useful repository. I am using it with Docker. But when I go for a larger area of interest and months the docker space goes up and deleted all the output .tif files in Jupyter notebook. However the space in docker not released. I know it is more of docker related issue but have you come across this problem and found any solution?
Also, I tried to clone the git code in the local system but I am having an issue with installing gdal versions (near black utility not available in recent gdal) but when I go older versions such as GDAL 2.2.2 I am facing FIONA incompatibility.

[question] minLandVisible seems not to be effective

Could you please explain the minLandVisible parameter?

Is it 0 to 100% expressed as 0 to 1 or is it percentage 0 to 100 ?

EDIT:

it seems to not do anything except giving a warning

I expect an image to be not included into the results when passing a certain threshold (e.g. I want to have 90% land visible) to not have false NDVI values in my results. No matter what I set the minLandVisible parameter, the result will contain cloud images

image

limiting the number of tiles downloaded?

Hi Flavio!

I have a question, do you think it would be possible to limit the number of tiles getRegionHistory will download?
I need to download a large number of tiles of different regions, and as of now when I call getRegionHistory for 1 area, it returns a lot of possible tiles. I then pass it on to a function that requests LTA retrieval of those that are offline, saturating my LTA retrieval limit. Some of my area's need multiple sentinel images to be covered, but I think because it finds so many matches it by chance doesn't request just the tiles I need to get 1 combined tile matching my area.
I'm trying to explain this the best that I can, I'm sorry if I'm unclear!!
Ultimately, I'd like to be able to select just those sentinel tiles so I can have 2-3 tiles corresponding to my area to choose from, so I can limit how many sentinel api requests I do so I can download all the tiles of different areas I need in as little times as possible.

I'd be glad to hear your thoughts on this!

Add McFeeters NDWI index

https://en.wikipedia.org/wiki/Normalized_difference_water_index

image

Needed changes (just a quick extension to my local files, you can probably implement it in a nice way) :-)

elif indexName=='NDWI2':
            #get band 03
            b03,geoTransform,projection = self._getBandDataFloat(geoPolygon, 'B03', resolution, dateReference)
            #get band 08
            b08,_,_ = self._getBandDataFloat(geoPolygon, self.nirBand, resolution, dateReference)
            #calculate
            ndwi = ((b03 - b08)/(b03 + b08))
            #save file
            tmp_file = "%s/tmp/ndwi-%s.tiff" % (self.dataPath, uuid.uuid4().hex)
            saveGeoTiff(ndwi, tmp_file, geoTransform, projection)
            return tmp_file

and

if bandOrIndexName in ['NDVI', 'NDWI1', 'NDWI2', 'NDMI']:
                regionFile = self.getRegionIndex(geoPolygon, bandOrIndexName, resolution, dateRefStr)

Docker Usage Sample

Hi, I want to run this project on docker image.

When I run "docker-compose up", it give error "ERROR: Couldn't find env file: /home/User/sentinelloader/.env"

I decided to run with Dockerfile "docker build -t flaviostutz/sentinelloader ." and it created docker image.

So, I need a command sample to run this project on docker image for downloading "Level 1C tci.jp2".

Could you help ?

Thanks.

Error processing the downloaded tile

Hi
I am trying to execute the sentinelloader python example.
I could able to download the 60m data. But in processing, I am getting the following error. For your info I have already installed gdal.

DEBUG:sentinelloader:Removing near black compression artifacts
INFO:sentinelloader:Couldn't get data for 2020-01-01 using the specified filter. err=gdal nearblack utility was not found in the system. install it

Error During Installation

I am trying to install via using 'Pip install sentinelloader". However, I am continuously coming across this error. I have tried downgrading python, try installing Gdal separately but couldn't success. Could please anyone if look into this and provide some feedback regarding the solution. Thanks
Screenshot from 2022-08-24 22-06-07

Selecting view and sun angles

I tried the Sentinelloader for my study site, and works well, thank you. I am in fact not interested in downloading the optical bands but the raster layers of sun and view angles: view_zenith_mean, view_azimuth_mean, sun_zenith, sun_azimuth.
Is it possible to download those layers instead of the band selection?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.