mapbox/rio-glui

Explore CloudOptimized geotiff on your browser using Mapbox GL JS

rio-glui

Explore and adjust Cloud-optimized geotiffs (COGs) in your browser using rasterio and Mapbox GL JS.

Install

You can install rio-glui using pip

pip install -U pip
pip install rio-glui

or install from source:

git clone https://github.com/mapbox/rio-glui.git
cd rio-glui
pip install -e .

Usage

Usage: rio glui [OPTIONS] PATH

  Rasterio glui cli.

Options:
-b, --bidx BIDX                   Raster band index
--scale INTEGER Min Max           Min and Max data bounds to rescale data from.
--colormap [cfastie|schwarzwald]  Rio-tiler compatible colormap name ('cfastie' or 'schwarzwald')
--tiles-format [png|jpg|webp]     Tile image format (default: png)
--tiles-dimensions INTEGER        Dimension of images being served (default: 512)
--nodata INTEGER                  Force mask creation from a given nodata value
--gl-tile-size INTEGER            mapbox-gl tileSize (default is the same as `tiles-dimensions`)
--port INTEGER                    Webserver port (default: 8080)
--playground                      Launch playground app
--mapbox-token TOKEN              Pass Mapbox token
--help                            Show this message and exit.

Example: explore COG hosted on aws

rio glui https://oin-hotosm.s3.amazonaws.com/5ac626e091b5310010e0d482/0/5ac626e091b5310010e0d483.tif

Playground

The --playground option opens a playground template where you an interact with the data to apply rio-color formula.

Creating Cloud-Optimized Geotiffs

To create rio-glui friendly files (Cloud-Optimized Geotiff) you can use another rasterio plugin: rio-cogeo. Alternately, you can use GDAL tools:

gdal_translate input.tif output.tif -of COG -co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=JPEG

Extras

This plugin also enables raster visualisation in a Jupyter Notebook using mapboxgl-jupyter

Contribution & Development

Issues and pull requests are more than welcome.

Dev install & Pull-Request

git clone https://github.com/mapbox/rio-glui.git
cd rio-glui
pip install -e .[dev]

Python3.6 only

This repo is set to use pre-commit to run flake8, pydocstring and black ("uncompromising Python code formatter") when commiting new code.

$ pre-commit install
$ git add .
$ git commit -m'my change'
black....................................................................Passed
Flake8...................................................................Passed
Verifying PEP257 Compliance..............................................Passed
$ git push origin

rio-glui's Issues

rio-glui crash on scenetif

@pratikyadav and I found a situation where we can consistently crash rio glui. But we don't have a solid idea as to why yet.

We have a workaround, if the problem is in this repo then it is not urgent. However, we're seeing errors that indicate pxm is producing a corrupt tif. Deets below...

Steps to reproduce

• Create a python 3.6 virtual env and pip install numpy cython, pip install "rasterio>=1.0a9" --no-binary rasterio then pip install -e . from this repo's directory.
• Download the file from s3,
• Run rio glui and visit http://127.0.0.1:5000/
• 💥

⭆ rio glui ~/16-19294-24642-nyc_ortho-2016-nyc_ortho_manhattan_980195.tif
Inspecting /Users/mperry/16-19294-24642-nyc_ortho-2016-nyc_ortho_manhattan_980195.tif at http://127.0.0.1:5000/
ERROR:rasterio._gdal:CPLE_IllegalArg in Temporary destination dataset for _reproject(): GDALDataset::GetRasterBand(2) - Illegal band #

ERROR:rasterio._gdal:CPLE_IllegalArg in panDstBands[0] = 2 ... out of range for dataset.
[2017-07-27 06:19:14,606] ERROR in app: Exception on /tiles/1501161554568/gamma b 1, gamma rg 1/11/603/770.png [GET]
Traceback (most recent call last):
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/mperry/work/rio-glui/rio_glui/cli.py", line 101, in get_image
    tilearr = pk.get_tile(z, x, y)
  File "/Users/mperry/work/rio-glui/rio_glui/cli.py", line 66, in get_tile
    resampling=Resampling.bilinear)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/rasterio/env.py", line 270, in wrapper
    return f(*args, **kwds)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/rasterio/warp.py", line 270, in reproject
    init_dest_nodata, **kwargs)
  File "rasterio/_warp.pyx", line 485, in rasterio._warp._reproject (rasterio/_warp.cpp:7877)
  File "rasterio/_err.pyx", line 176, in rasterio._err.exc_wrap_int (rasterio/_err.c:2098)
rasterio._err.CPLE_IllegalArgError: panDstBands[0] = 2 ... out of range for dataset.
WARNING:rasterio._gdal:CPLE_AppDefined in JPEGLib:Corrupt JPEG data: bad Huffman code
ERROR:rasterio._gdal:CPLE_IllegalArg in Temporary destination dataset for _reproject(): GDALDataset::GetRasterBand(2) - Illegal band #

ERROR:rasterio._gdal:CPLE_IllegalArg in panDstBands[0] = 2 ... out of range for dataset.
[2017-07-27 06:19:14,848] ERROR in app: Exception on /tiles/1501161554568/gamma b 1, gamma rg 1/12/1206/1540.png [GET]
Traceback (most recent call last):
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/mperry/work/rio-glui/rio_glui/cli.py", line 101, in get_image
    tilearr = pk.get_tile(z, x, y)
  File "/Users/mperry/work/rio-glui/rio_glui/cli.py", line 66, in get_tile
    resampling=Resampling.bilinear)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/rasterio/env.py", line 270, in wrapper
    return f(*args, **kwds)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/rasterio/warp.py", line 270, in reproject
    init_dest_nodata, **kwargs)
  File "rasterio/_warp.pyx", line 485, in rasterio._warp._reproject (rasterio/_warp.cpp:7877)
  File "rasterio/_err.pyx", line 176, in rasterio._err.exc_wrap_int (rasterio/_err.c:2098)
rasterio._err.CPLE_IllegalArgError: panDstBands[0] = 2 ... out of range for dataset.
ERROR:rasterio._gdal:CPLE_AppDefined in TIFFReadEncodedTile() failed.
ERROR:rasterio._gdal:CPLE_AppDefined in TIFFReadEncodedTile() failed.
ERROR:rasterio._gdal:CPLE_AppDefined in /Users/mperry/16-19294-24642-nyc_ortho-2016-nyc_ortho_manhattan_980195.tif, band 1: IReadBlock failed at X offset 0, Y offset 3: TIFFReadEncodedTile() failed.
ERROR:rasterio._gdal:CPLE_AppDefined in /Users/mperry/16-19294-24642-nyc_ortho-2016-nyc_ortho_manhattan_980195.tif, band 1: IReadBlock failed at X offset 2, Y offset 2: TIFFReadEncodedTile() failed.
python3.6(18345,0x700010290000) malloc: *** error for object 0x1020d9e10: incorrect checksum for freed object - object was probably modified after being freed.
*** set a breakpoint in malloc_error_break to debug
Abort trap: 6

---

There are a number of potentially relevant GDAL/Rasterio errors here. To highlight a few scary ones:

• ERROR:rasterio._gdal:CPLE_AppDefined in TIFFReadEncodedTile() failed.
• rasterio._err.CPLE_IllegalArgError: panDstBands[0] = 2 ... out of range for dataset.
• WARNING:rasterio._gdal:CPLE_AppDefined in JPEGLib:Corrupt JPEG data: bad Huffman code

To me this might be an indication that the scenetif we're using is somehow corrupt? OTOH this scenetif works ok in every other context, only causing rio glui to crash. Hence adding the ticket to this repo. I'd really like to confirm that the problem is isolated to this repo and that pxm is not churning out corrupt tifs.

@sgillies and @dnomadb: I would love to get your thoughts on debugging this. It's not urgent per se but if it's a problem with pxm itself, that might be 😬

Crashes with error "AttributeError: type object 'IOLoop' has no attribute 'initialized'"

Just cloned from github, installed with pip as per README (Python 3.6.4 environment) with no issues, and tried to launch the example command in the README:

$ rio glui https://oin-hotosm.s3.amazonaws.com/5ac626e091b5310010e0d482/0/5ac626e091b5310010e0d483.tif
Inspecting https://oin-hotosm.s3.amazonaws.com/5ac626e091b5310010e0d482/0/5ac626e091b5310010e0d483.tif at http://127.0.0.1:8080/index.html
Traceback (most recent call last):
  File "/opt/anaconda/envs/python3/bin/rio", line 11, in <module>
    sys.exit(main_group())
  File "/opt/anaconda/envs/python3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/opt/anaconda/envs/python3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/opt/anaconda/envs/python3/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/opt/anaconda/envs/python3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/anaconda/envs/python3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/<USER>/python/rio-glui/rio_glui/scripts/cli.py", line 128, in glui
    app.start()
  File "/home/<USER>/python/rio-glui/rio_glui/server.py", line 125, in start
    is_running = IOLoop.initialized()
AttributeError: type object 'IOLoop' has no attribute 'initialized'

I guess this is a Tornado issue. Installed version appears to be 5.0.1 if that's of any use?

rio-glui crashes with manylinux1 wheels from PyPI

This is an upstream issue. I don't see any problems at all with the system curl 7.54 on Ubuntu 18.04, but using the manylinux1 wheels with curl 7.59-7.61 leads to an eventual crash after some amount of browsing around a COG dataset.

Remove Mapbox Token

We need to remove the default Mapbox token hardcoded and add an option to pass it

enable http url as input

Right now rio-glui cli force the use of local files

rio-glui/rio_glui/cli.py

Line 130 in 536916d

@click.argument('srcpath', type=click.Path(exists=True))

IMO there is a real need to force that.

Just noting that it works great with cloud optimized geotiff ;-)

rio glui http://oin-hotosm.s3.amazonaws.com/5a324a622553e6000ce5aaf5/0/3fa181a1-425c-4e53-b49f-ea72884da097.tif

get_playround_url

Noticed a tiny typo in https://github.com/mapbox/rio-glui/blob/master/rio_glui/server.py#L125

Adding other background layers

@dnomadb @virginiayung can we document how to add other background layers; like DG map api?

Crashes with AttributeError: type object 'IOLoop' has no attribute 'initialized'

I can't get this to work with a fresh install on Mac OS.

Version follow:

• Python3:3.6.5
• pip: 10.0.1
• rio: 1.0.0

I tried installing tornado, as suggested elsewhere, but it already is installed. Also installed pymq but that didn't have any affect.

Error follows:

rio glui https://oin-hotosm.s3.amazonaws.com/5ac626e091b5310010e0d482/0/5ac626e091b5310010e0d483.tif
Inspecting https://oin-hotosm.s3.amazonaws.com/5ac626e091b5310010e0d482/0/5ac626e091b5310010e0d483.tif at http://127.0.0.1:8080/index.html
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.6/bin/rio", line 11, in <module>
    sys.exit(main_group())
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rio_glui/scripts/cli.py", line 128, in glui
    app.start()
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/rio_glui/server.py", line 125, in start
    is_running = IOLoop.initialized()
AttributeError: type object 'IOLoop' has no attribute 'initialized'

numpy type error on tile requests

[2017-09-01 14:24:22,888] ERROR in app: Exception on /tiles/1504297435713/gamma b 1, gamma rg 1/15/8873/13739.png [GET]
Traceback (most recent call last):
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/mperry/env/pxm/lib/python3.6/site-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/mperry/work/rio-glui/rio_glui/cli.py", line 117, in get_image
    tilearr = pk.get_tile(z, x, y)
  File "/Users/mperry/work/rio-glui/rio_glui/cli.py", line 79, in get_tile
    out = np.concatenate(dest_arr, alpha)
TypeError: only length-1 arrays can be converted to Python scalars

Access to massive VRT sources

In theory, we should be able to build a GDAL VRT pointing to hundreds of sources on S3 and use this VRT as a rio-glui source, yes?

In practice, I'm not getting great performance with a VRT containing 811 .jp2 files on S3, even with the latest and greatest OpenJPEG in GDAL. It's been a while since I did this kind of thing with VRTs and network files. I could be missing some optimizations. Maybe we should get some consulting help on this.

Zipped VRT file here: hro_columbus.vrt.zip

@vincentsarago @perrygeo

Image in readme 403s

Just came across this project; looks like the image in the readme 403s.

It is "Gluey", not "Gee El You Eye"

fy @jacquestardie

split view for comparing color formulas

I've found the OSX full screen split view with two chrome tabs, each withrio glui windows, to be useful for comparing color formulas.

I'm thinking, something like ^ but in one interface with the view extents synced.

Visualize one band data

This serve a more general propose than our needs but might be 👌

rio-tiler 1.0.0 breaks rio-glui

With rio-tiler v1.0.0, rio glui runs into this error:

$ rio glui --help
Usage: rio glui [OPTIONS]

  Warning: entry point could not be loaded. Contact its author for help.

  Traceback (most recent call last):
    File "/lib/python3.6/site-packages/click_plugins/core.py", line 37, in decorator
      group.add_command(entry_point.load())
    File "/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2408, in load
      return self.resolve()
    File "/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2414, in resolve
      module = __import__(self.module_name, fromlist=['__name__'], level=0)
    File "/Documents/rio-glui/rio_glui/scripts/cli.py", line 7, in <module>
      from rio_glui import server
    File "/Documents/rio-glui/rio_glui/server.py", line 10, in <module>
      from rio_tiler.utils import array_to_img, linear_rescale, get_colormap
  ImportError: cannot import name 'array_to_img'

No error if rio-tiler is 1.0rc3.

V1 - refactoring

I'm planning to refactor a bit the purpose of this plugin.

Why

For mapboxgl-jupyter we need a tile server to plug to RasterTilesViz. The idea is to use rio-glui that will be shipped with this tile server.

Overview

rio-glui will keep its main purpose (gl-ui) but provide a TileServer class.

How

• move from flask to tornado (to be able to link the web server to the one used in Jupyter)

cc @dnomadb @sgillies