developmentseed/titiler

Build your own Raster dynamic map tile services

A modern dynamic tile server built on top of FastAPI and Rasterio/GDAL.

---

Documentation: https://devseed.com/titiler/

Source Code: https://github.com/developmentseed/titiler

---

Titiler, pronounced tee-tiler (ti is the diminutive version of the french petit which means small), is a set of python modules that focus on creating FastAPI application for dynamic tiling.

Note: This project is the descendant of cogeo-tiler and cogeo-mosaic-tiler.

Features

• Built on top of FastAPI
• Cloud Optimized GeoTIFF support
• SpatioTemporal Asset Catalog support
• Multiple projections support (see TileMatrixSets) via morecantile.
• JPEG / JP2 / PNG / WEBP / GTIFF / NumpyTile output format support
• OGC WMTS support
• Automatic OpenAPI documentation (FastAPI builtin)
• Virtual mosaic support (via MosaicJSON)
• Example of AWS Lambda / ECS deployment (via CDK)

Packages

Starting with version 0.3.0, the TiTiler python module has been split into a set of python namespace packages: titiler.{package}.

Package	Version	Description
titiler.core		The Core package contains libraries to help create a dynamic tiler for COG and STAC
titiler.extensions		TiTiler's extensions package. Contains extensions for Tiler Factories.
titiler.mosaic		The mosaic package contains libraries to help create a dynamic tiler for MosaicJSON (adds cogeo-mosaic requirement)
titiler.application		TiTiler's demo package. Contains a FastAPI application with full support of COG, STAC and MosaicJSON

Installation

To install from PyPI and run:

# Make sure you have pip up to date
python -m pip install -U pip

python -m pip  install titiler.{package}
# e.g.,
# python -m pip  install titiler.core
# python -m pip  install titiler.extensions
# python -m pip  install titiler.mosaic
# python -m pip  install titiler.application (also installs core, extensions and mosaic)

# Install uvicorn to run the FastAPI application locally
python -m pip install uvicorn

# Launch application locally
uvicorn titiler.application.main:app

To install from sources and run for development:

git clone https://github.com/developmentseed/titiler.git
cd titiler

python -m pip install -U pip
python -m pip install -e src/titiler/core -e src/titiler/extensions -e src/titiler/mosaic -e src/titiler/application
python -m pip install uvicorn

uvicorn titiler.application.main:app --reload

Docker

Ready to use/deploy images can be found on Github registry.

• https://github.com/developmentseed/titiler/pkgs/container/titiler

docker run --name titiler \
    -p 8000:8000 \
    --env PORT=8000 \
    --env WORKERS_PER_CORE=1 \
    --rm -it ghcr.io/developmentseed/titiler:latest

• Built the docker locally

git clone https://github.com/developmentseed/titiler.git
cd titiler

docker-compose up --build titiler  # or titiler-uvicorn

Some options can be set via environment variables, see: https://github.com/tiangolo/uvicorn-gunicorn-docker#advanced-usage

Project structure

src/titiler/                     - titiler modules.
 ├── application/                - Titiler's `Application` package
 ├── extensions/                 - Titiler's `Extensions` package
 ├── core/                       - Titiler's `Core` package
 └── mosaic/                     - Titiler's `Mosaic` package

Contribution & Development

See CONTRIBUTING.md

License

See LICENSE

Authors

Created by Development Seed

See contributors for a listing of individual contributors.

Changes

See CHANGES.md.

titiler's Issues

titiler deployment without CDN - alternative deployment scenarios

I'm wondering if there are alternative deployment scenarios without CDN for titiler.

We've deployed titiler on a large single machine and the performance are of course not a good as your endpoint.

What would be the alternative deployment scenario for acceptable performance on a private cloud? Our use-cases include using titiler for restricted access data in a private cloud deployment.

Use absolute paths for loading templates

Titiler fails to find location of html templates when installed as a python library because the current template paths are relative and not absolute (ex. https://github.com/developmentseed/titiler/blob/master/titiler/endpoints/stac.py#L35). Using absolute paths to load html templates would fix this issue.

add mypy check

remove mosaicid magic path translation in default PathParams dependency

titiler/titiler/dependencies.py

Lines 98 to 99 in 97a5901

	if parsed.scheme.split("+")[-1] == "mosaicid":
	self.url = f"{settings.DEFAULT_MOSAIC_BACKEND}{settings.DEFAULT_MOSAIC_HOST}/{parsed.netloc}.json.gz"

Having this force to have some custom code which I believe should be left to the user to do (it's really easy to create and pass a PathParams dependency to the factory)

Deleting Github Tags!

We are now really close to the next TiTiler release. 🎉

We have refactored the code to create a proper python module with helper classes like the new titiler.endpoint.factory.TilerFactory which will help users to integrate COG/STAC/MosaicJSON tilers (FastAPI router) directly to thier applications

from titiler.endpoints.factory import TilerFactory
from fastapi import FastAPI

app = FastAPI()

cog = TilerFactory()

print([r.path for r in cog.router.routes])
> [
    '/openapi.json',
    '/docs',
    '/docs/oauth2-redirect',
    '/redoc',
    '/bounds',
    '/info',
    '/metadata',
    '/tiles/{TileMatrixSetId}/{z}/{x}/{y}@{scale}x.{format}',
    '/tiles/{TileMatrixSetId}/{z}/{x}/{y}@{scale}x',
    '/tiles/{TileMatrixSetId}/{z}/{x}/{y}.{format}',
    '/tiles/{TileMatrixSetId}/{z}/{x}/{y}',
    '/tiles/{z}/{x}/{y}@{scale}x.{format}',
    '/tiles/{z}/{x}/{y}@{scale}x',
    '/tiles/{z}/{x}/{y}.{format}',
    '/tiles/{z}/{x}/{y}',
    '/{TileMatrixSetId}/tilejson.json',
    '/tilejson.json',
    '/{TileMatrixSetId}/WMTSCapabilities.xml',
    '/WMTSCapabilities.xml',
    '/point/{lon},{lat}',
    '/preview.{format}',
    '/preview',
]
app.include_router(cog.router, tags=["Cloud Optimized GeoTIFF"])

Because of the PyPi release I think it makes sense to start the release at version 0.0.1 and get rid of old tags

Replace Async code with latest MosaicBackend

For sake of code simplicity we will remove those to block

titiler/titiler/endpoints/mosaic.py

Lines 198 to 216 in b6c8132

	# Rio-tiler provides a helper function (``rio_tiler.reader.multi_point``) for reading a point from multiple assets
	# using an external threadpool. For similar reasons as described below, we will transcribe the rio-tiler code to
	# use the default executor provided by the event loop.
	futures = [
	run_in_threadpool(
	_read_point, asset, lon, lat, indexes=indexes, expression=expression
	)
	for asset in assets
	]
	values = []
	with utils.Timer() as t:
	async for fut in _process_futures(
	futures, concurrency=int(os.getenv("MOSAIC_CONCURRENCY", 10))
	):
	try:
	values.append(await fut)
	except Exception:
	continue

titiler/titiler/endpoints/mosaic.py

Lines 265 to 301 in b6c8132

	# Rio-tiler-mosaic uses an external ThreadPoolExecutor to process multiple assets at once but we want to use the
	# executor provided by the event loop. Instead of calling ``rio_tiler_mosaic.mosaic.mosaic_tiler`` directly we will
	# transcribe the code here and use the executor provided by the event loop. This also means we define this function
	# as a coroutine (even though nothing that is called is a coroutine), since the event loop's executor isn't
	# available in normal ``def`` functions.
	futures = [
	run_in_threadpool(
	_read_tile,
	asset,
	x,
	y,
	z,
	tilesize=tilesize,
	indexes=image_params.indexes,
	expression=image_params.expression,
	nodata=image_params.nodata,
	**image_params.kwargs,
	)
	for asset in assets
	]
	with utils.Timer() as t:
	async for fut in _process_futures(
	futures, concurrency=int(os.getenv("MOSAIC_CONCURRENCY", 10))
	):
	try:
	tile, mask = await fut
	except Exception:
	# Gracefully handle exceptions
	continue
	tile = numpy.ma.array(tile)
	tile.mask = mask == 0
	pixsel.feed(tile)
	if pixsel.is_done:
	break
	timings.append(("Read-tiles", t.elapsed))

The new MosaicBackends has a .tile and .point method that will take care of the data fetching directly. This is not async but we plan to make a AsyncMosaicBackends at one point

[STAC] resolves relative asset path

like in https://github.com/Terradue/stac-resources/blob/master/kompsat-3a/item.json

deployment is slow

When deploying a new Lambda instance, CDK app has to create the package. We currently use python docker API to run the docker container and create the package

titiler/stack/app.py

Lines 87 to 101 in 3233807

	def create_package(self, code_dir: str) -> aws_lambda.Code:
	"""Build docker image and create package."""
	client = docker.from_env()
	client.images.build(
	path=code_dir,
	dockerfile="Dockerfiles/lambda/Dockerfile",
	tag="lambda:latest",
	)
	client.containers.run(
	image="lambda:latest",
	command="/bin/sh -c 'cp /tmp/package.zip /local/package.zip'",
	remove=True,
	volumes={os.path.abspath(code_dir): {"bind": "/local/", "mode": "rw"}},
	user=0,
	)

The process is really slow and caching doesn't seem to be used 🤔

add ECS deployment

add docs

add tests

switch to starlette config and use .env

deploy ready to use docker image on docker hub

add release and changelog

Update for latest cogeo-mosaic / rio-tiler

right now there is a mismatch between rio-tiler version and it's leading the CI to fail!

Create shared file for openapi definitions

Ref #40 (comment)

Memory filing

setting 1 worker per cpu https://github.com/developmentseed/titiler/blob/master/stack/app.py#L64 seems to resolve this

ref tiangolo/fastapi#596

Pin rio-tiler to 2.0a11 or update the code

a new rio-tiler version was released today (2.0b1) but there is a lot of breaking changes. For now we should pin rio-tiler to 2.0a11 while waiting to make other changes

add mosaics

[STAC] return list of assets with `title` and `eo:common_name`

crop/ should return data in WGS84 (input crs)

In

titiler/titiler/api/endpoints/cog.py

Lines 299 to 308 in 725da5f

	data, mask = cog.part(
	[minx, miny, maxx, maxy],
	height=image_params.height,
	width=image_params.width,
	max_size=image_params.max_size,
	indexes=image_params.indexes,
	expression=image_params.expression,
	nodata=image_params.nodata,
	**image_params.kwargs,
	)

we don't define the output CRS (dst_crs) so by default rio-tiler-crs will use the file CRS: https://github.com/cogeotiff/rio-tiler-crs/blob/master/rio_tiler_crs/reader.py#L307-L335

The problem is when passing width and height, we might loose the information about the actual data transform (because the width and height will be relative to the data in the output crs).

We should force the dst_crs to be the same as the bounds_crs (wgs84) by default on the API side!

reduce code duplication

Add COG validation endpoint

create /demo endpoint to host the viewers

earlier I choose to host the viewer in each endpoints (e.g stac.py, cog.py) but now I changed my mind 😬 I think the viewer are a nice feature for DEMO purposes but the end goal of titiler is to be an API.

The main goal is to let user do

from fastapi import APIRouter
from titiler.endpoints import cog

api_router = APIRouter()
api_router.include_router(cog.router, prefix="/cog", tags=["Cloud Optimized GeoTIFF"])

Add exception handling

Currently the app either returns a 200 on successful request or server error (500) on unsuccessful request. I think there is a lot of value in explicitly raising exceptions for error cases. Most important in my opinion is FileNotFound and TileNotFound errors which both should return a 404 status code indicating that the particular file/tile was not found. These status codes, if raised, may be used by the client to infer that there is no imagery available for the particular request and to fall back to another data source or provider.

I think the best way to do this is global exception handlers which return a particular status code whenever a specific exception is raised. This is inline with having standardized exceptions within the app (which is already implemented, although they aren't currently used anywhere).

url in README.md

Hello,

the README.md says:

:endpoint:/v1/{z}/{x}/{y}[@{scale}x][.{ext}]

(...)
- **url**: Cloud Optimized GeoTIFF URL. **REQUIRED**

I don't quite get where and how the url and the rest of the parameters are specified. Could you provide a specific example?

By the way, the tool seems awesome thanks :)

add custom permission to s3 bucket

Remove intermediary api router

The fastapi router defined in titiler.api.api doesn't really add much. I can see a use case for using an intermediary router if that router used a particular custom defined route class etc. but that isn't the case here. It would be better to attach the routes directly to the fastapi app defined in titiler.main, remove titiler.api.api, and maybe rename titiler.main -> titiler.app to be more standard with fastapi conventions.

add isort

Better documentation via mkdoc

we need better docs

Section/topic

• Dynamic tiling
• COG
• STAC
• MosaicJSON
• TileMatrixSet
• Caching
• Settings/Env
• Deployment
• Customization

add STAC Item validation

add generic JSON response class

It looks like most routes which don't implement ImgResponse are manually setting cache-ttl response headers:

All of these routes (I think) are JSON responses, and should implement a similar JSONResponse object which automatically sets cache-control rather than defining within the scope of each request. The JSONResponse object would be very similar to ImgResponse, but of course implementing a different protocol.

make it STAC friendly

make it async

we need a better UI

switch to Morecantile

let's try morecantile to support more projection

add aws lambda using mangum

titiler is built around FastAPI with the idea of using it with ECS, but it seems that with https://github.com/erm/mangum we could also create a lambda handler easily.

Return asset ids as response header

Users often want to see metadata information on the client about the imagery shown. This is especially true for mosaics, but also desirable for single-scene COG viewing.

I propose adding a response header like X-ASSETS that returns a comma-separated or JSON-encoded list of asset ids that were used to create the response tile. For many sources, the asset id includes enough information to extract a date range and other helpful metadata.

better auto scaling

Right now the autoscaling is based on the number of requests per ecs task

titiler/stack/app.py

Lines 74 to 81 in 89d8d64

	# https://github.com/awslabs/aws-rails-provisioner/blob/263782a4250ca1820082bfb059b163a0f2130d02/lib/aws-rails-provisioner/scaling.rb#L343-L387
	scalable_target.scale_on_request_count(
	"RequestScaling",
	requests_per_target=50,
	scale_in_cooldown=core.Duration.seconds(240),
	scale_out_cooldown=core.Duration.seconds(30),
	target_group=fargate_service.target_group,
	)

This should enable fast scaling UP when number of user are loading tiles, but there might be better configuration.

add CloudWatch dashboard

like:

add cache example

https://github.com/argaen/aiocache

add EKS ?

explore deployment using EKS https://aws.amazon.com/fr/eks/

make the tiler OGC-API-tile friendly

https://github.com/opengeospatial/OGC-API-Tiles

route resolution fails when titiler is mounted as a subapp

request.url_for fails to find the correct route when titiler is mounted as a sub application (https://github.com/developmentseed/titiler/blob/nonTMSReader/titiler/endpoints/factory.py#L1013):

from fastapi import FastAPI
from titiler.endpoints.stac import STACTiler

app = FastAPI()

titiler_app = FastAPI()
titiler_app.include_router(STACTiler().router)

app.mount("/titiler", titiler_app)

This is because request.url_for uses the router of the request's current scope which in this case is app and not titiler_app. Because sub applications are independent of the parent app, the titiler routes are not actually registered on app, so starlette can't find the route and raises starlette.routing.NoMatchFound.

A solution that works is replacing calls to request.url_for with self.router.url_path_for which has the same signature (its called under the hood by request.url_for) but only searches the specific router, ensuring that route resolution will always work no matter how the app is mounted.

Pin FastAPI Version to 0.60.1

A recent change in FastAPI/Starlette is breaking some routes (those with escape characters).

I'm going to fix it and pin fastapi version so it doesn't happen again ;-)

use GitHub Actions

at least for CI and Pypi publish (we could do AWS deploy later)

new doc update

I forget to update the docs in #32

use starlette url_for

titiler/titiler/endpoints/cog.py

Line 369 in 24ec8b6

host = request.headers["host"]

Mixing .env in titiler.settings and stack.config

Forgot that @geospatial-jeff warned me about this 🤦

titiler/titiler/settings.py

Lines 6 to 15 in 97a5901

	config = Config(".env")
	BACKEND_CORS_ORIGINS = config("BACKEND_CORS_ORIGINS", cast=str, default="*")
	DEFAULT_CACHECONTROL = config(
	"DEFAULT_CACHECONTROL", cast=str, default="public, max-age=3600"
	)
	DEFAULT_MOSAIC_BACKEND = config("DEFAULT_MOSAIC_BACKEND", cast=str, default="s3://")
	DEFAULT_MOSAIC_HOST = config("DEFAULT_MOSAIC_HOST", cast=str, default="")

titiler/stack/config.py

Line 50 in 97a5901

env_file = ".env"

having the possibility to manage settings with .env is still a nice feature and I think we shoud also move the API config to pydantic.BaseSettings

Use normal def to declare routes if no async code is called

I've run into an interesting race condition using titiler with arturo-stac-api. For my use case, I am trying to mount titiler's stac and demo routers onto my stac application. In the stac application I have an endpoint (/collections/{collectionId}/items/{itemId}/tiles) which, when called with a text/html accept header, sends a redirect request to the titiler's stac viewer. The call chain looks like:

1. GET request to /collections/{collectionId}/items/{itemId}/tiles.
2. REDIRECT request to /stac/viewer
   2a. GET request to /info
   2b. GET request to /tilejson.json
   2c. GET request to /tiles (as tiles render in the browser).

**Each request in the browser (2a/2b/2c) does its own GET request to /collections/{collectionId}/items/{itemId} using the blocking requests library.

The race condition arises because the application is blocked during 2a/2b/2c as these requests require sending a GET request to the application to fetch the stac item which the application cannot process because it is currently blocked.

Switching all of the appropriate routes in titiler's stac router from async def to def fixed this issue, as the blocking call to requests.get to fetch the stac item is no longer blocking the main thread but instead run in an external threadpool.