mapbox/sentinel-tiler

A serverless Sentinel-2 tiles server using AWS Lambda

sentinel-tiler

Create a serverless tile server for Amazon's Sentinel-2 Public Dataset based on rasterio python library.

This project is linked to the landsat-tiler projected introduced in https://blog.mapbox.com/combining-the-power-of-aws-lambda-and-rasterio-8ffd3648c348

Sentinel data on AWS

Since 2016 ESA Sentinel-2 data is hosted on AWS and can be freely accessed.

Each file is its own object in the sentinel-s2-l1c Amazon S3 bucket. The data are organised into tiles using the Military grid system. The basic data format is the following:

more info: https://aws.amazon.com/public-datasets/sentinel-2/

Installation

Requirement

• AWS Account
• Docker
• node + npm

Create package

Creating a python lambda package with some C (or Cython) libraries like Rasterio/GDAL has never been an easy task because you have to compile and build it on the same infrastructure where it's going to be used (Amazon linux AMI). Until recently, to create your package you had to launch an EC2 instance using the official Amazon Linux AMI and create your package on it (see perrygeo blog or Remotepixel blog).

But this was before, Late 2016, the AWS team released the Amazon Linux image on docker, so it's now possible to use it locally to compile C libraries and create complex lambda package (see Dockerfile).

Custom or Simple

Sentinel-2 data are stored in JPEG2000 format. While this has some advantage (mainly file size), this format is not really cloud optimized ref. JPEG2000 started as a proprietary format but recently there has been a massive work by University of Louvain and also Even Rouault to develop an open source driver to decode and encode JPEG2000 files.

Quote form OpenJPEG repo

OpenJPEG is an open-source JPEG 2000 codec written in C language. It has been developed in order to promote the use of JPEG 2000, a still-image compression standard from the Joint Photographic Experts Group (JPEG). Since April 2015, it is officially recognized by ISO/IEC and ITU-T as a JPEG 2000 Reference Software.

That said, OpenJPEG driver is not yet as fast as the other proprietary Jpeg2000 drivers like ECW, MrSID or Kakadu. In this repo you can have the possibility to create the a custom GDAL install with any of the drivers you have the license for.

Update: Even Rouault has made some major improvements in OpenJPEG 2.3.0 blog.

Simple (with rasterio wheels (OpenJPEG 2.3.0))

# Build Amazon linux AMI docker container + Install Python modules + create package
git clone https://github.com/mapbox/sentinel-tiler.git
cd sentinel-tiler/
make wheel

# Deploy
make deploy-wheel
# Or
sls deploy --type wheel

Custom

1. Edit ./Dockerfiles/Custom

2. Create the package:

# Build Amazon linux AMI docker container + Install Python modules + create package
git clone https://github.com/mapbox/sentinel-tiler.git
cd sentinel-tiler/
make custom

# Deploy
make deploy-custom
# Or
sls deploy --type custom

Note: to stay under AWS lambda package sizes limits (100Mb zipped file / 250Mb unzipped archive) we need to use some tricks.

• remove every packages that are already available natively in AWS Lambda (boto3, botocore ...)
• keep only precompiled python code (.pyc) so it lighter and it loads faster

🎉 You should be all set there.

---

Use it: Sentinel-viewer

sentinel-tiler + Mapbox GL + Satellite API

The viewer/ directory contains a UI example to use with your new Lambda Sentinel tiler endpoint. It combine the power of mapbox-gl and the nice developmentseed sat-api to create a simple and fast Sentinel-viewer.

To be able to run it, edit those two lines in viewer/js/app.js

// viewer/js/app.js
3  mapboxgl.accessToken = '{YOUR-MAPBOX-TOKEN}';
4  const sentinel_tiler_url = "{YOUR-API-GATEWAY-URL}";

---

Workflow

1. One AWS ƛ call to get min/max percent cut value for all the bands and bounds

Path: /sentinel/metdata/{sentinel scene id}

Inputs:

• sceneid: Sentinel scene id (S2{A|B}_tile_{YYYYMMDD}_{utm_zone}{latitude_band}{grid_square}_{img_number})

Options:

• pmin: Histogram cut minimum value in percent (default: 2)
• pmax: Histogram cut maximum value in percent (default: 98)

Output: (dict)

• bounds: (minX, minY, maxX, maxY) (list)
• sceneid: scene id (string)
• rgbMinMax: Min/Max DN values for the linear rescaling (dict)

Example: <api-gateway-url>/sentinel/metadata/S2A_tile_20161202_16SDG_0?pmin=5&pmax=95

2. Parallel AWS ƛ calls (one per mercator tile) to retrieve corresponding Sentinel data

Path: /sentinel/tiles/{sentinel scene id}/{z}/{x}/{y}.{ext}

Inputs:

• sceneid: Sentinel scene id (S2{A|B}_tile_{YYYYMMDD}_{utm_zone}{latitude_band}{grid_square}_{img_number})
• x: Mercator tile X index
• y: Mercator tile Y index
• z: Mercator tile ZOOM level
• ext: Image format to return ("jpg" or "png")

Options:

• rgb: Bands index for the RGB combination (default: (04, 03, 02))
• histo: - delimited rgb histogram min/max (default: 0,16000-0,16000-0,16000 )
• tile: Output image size (default: 256)

Output:

• base64 encoded image PNG or JPEG (string)

Example:

• <api-gateway-url>/sentinel/tile/S2A_tile_20161202_16SDG_0/10/262/397.png
• <api-gateway-url>/sentinel/tile/S2A_tile_20161202_16SDG_0/10/262/397.png?rgb=04,03,02&histo=100,3000-130,2700-500,4500&tile=512

---

Infos & links

• rio-tiler rasterio plugin that process Sentinel data hosted on AWS S3.
• Introducing the AWS Lambda Tiler
• Humanitarian OpenStreetMap Team oam-dynamic-tiler

sentinel-tiler's Issues

Output transparent images

I would like to output transparent images.
I don't know if that requires png.
Today it looks black outside of the satelite image, if that black color could be transparent, I could have a map (like OSM) and it would be easier to know where you are.
(And I would get something reasonable for two neigbouring (large) tiles.)

CORS

I am hitting CORS issue that is very similar to the one described for the landast-tiler. However, this time setting a AWS_REGION variable doesn't seem to fix the problem.

After digging around a little in the serverless issues (e.g. #4681) it seems that maybe we need to define Access-Control-Allow-* headers ?

If you want to use CORS with the lambda-proxy integration, remember to include the Access-Control-Allow-* headers in your headers object

As defined in the serverless docs

Here is a screenshot of how it looks on the frontend:

example viewer for sentinel

Live example can be found at http://www.cbroderick.me/sentinel/

Not super fast but will give people the idea.

Band 10 always zero

I've created a sentinel-tiler server on AWS, and all seems to be working really well, with the exception that band 10 always produces zero value pixels.

I'm using a URL structured as follows:
api-gateway-url/sentinel/tile/S2A_tile_20161202_16SDG_0/10/262/397.png?rgb=09,10,11&tile=256

All of the other bands are working just fine. Rearranging the rgb parameter (e.g. '10,11,12' or '9,11,10') doesn't help.

Is this a known issue, or am I likely doing something wrong? Or might there be something wrong with my tile server?

EDIT: I'd thought this was happening for band 11, but it's band 10.

Simple docker image not working after #9

Got the error A defined coordinate reference system is required which appears to be a rasterio error. After rebuilding with the custom image it works fine.

Sorry haven't investigated past that, just thought I'd drop a note here in case anyone else hits this while upgrading.

Histogram Parameters

In histo: - delimited rgb histogram min/max (default: 0,16000-0,16000-0,16000 )

It seems like parameter 1, 2, 3 corresponds to RGB. But what does the parameter 0 do? Modifying doesn't seem to do anything. When you supply a single number, does it specify the max?

code url:
https://github.com/mapbox/sentinel-tiler/blame/master/README.md#L133