ropensci/tiler

Generate geographic and non-geographic map tiles from R

tiler's Issues

Does tiler take a bounding box params?

Can I generate tiles for a specific region?

Thanks.

temp file race conditions when invoking tile() in parallel

Calling tile() in a parallel (i.e., multiple concurrent tile calls) produces the following problems:

1. python 'missing file' errors on the nodes due to disappearance of the g2ttmp directory.

g2t_tmp_dir <- file.path(tempdir(), "g2ttmp")
## and later
unlink(g2t_tmp_dir, recursive = TRUE, force = TRUE)
...

In order to make this work, each nodes needs its own tmpdir. The following change does the trick, provided the rasters are all of the supported type:

g2t_tmp_dir <- tempfile("g2ttmp_")

2. attempts to use the same temp file when rasters are not one of the supported filetypes:

## this fliename is hardcoded to be the same for all nodes :(
file <- file.path(tempdir(), "tmp_raster.tif")

Unfortunately this is a more involved fix because that file name is hardcoded in multiple places outside of tile(), so would need to pass the temp file name through every other function that checks for that file. Otherwise, I'd use tempfile("tmp_raster_", fileext = ".tif") here too.

segfaults on tile generation

In running tiler on the base example, I get the following

sh: line 1: 60605 Segmentation fault      python "/home/jb92b/R/x86_64-pc-linux-gnu-library/3.5/tiler/python/gdal2tilesIMG.py" --leaflet -p raster -z 2-3 -w none --tmpdir "/tmp/RtmpGvaBZ3/g2ttmp" "/home/jb92b/R/x86_64-pc-linux-gnu-library/3.5/tiler/maps/map.png" "/project/tiler/tiles" 2> /dev/null

SessionInfo:

R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Red Hat Enterprise Linux Server release 6.9 (Santiago)

Matrix products: default
BLAS: /share/pkg/R/3.5.1_gcc8.1.0/lib64/R/lib/libRblas.so
LAPACK: /share/pkg/R/3.5.1_gcc8.1.0/lib64/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] stringr_1.3.1 raster_2.8-4  sp_1.3-1      tiler_0.2.1  

loaded via a namespace (and not attached):
[1] compiler_3.5.1   magrittr_1.5     tools_3.5.1      Rcpp_1.0.0      
[5] stringi_1.2.4    codetools_0.2-16 grid_3.5.1       png_0.1-7       
[9] lattice_0.20-38 

Using python 2.7.14.
gdal 2.4.0

I'm not sure what's causing the segfault and why. Any thoughts?

try to automatically detect python path

It would be great if out-of-the-box tiler would attempt to find the correct python version, rather than leaving it to the user to ensure it's set.

I know it gets complicated (especially on Windows) but I think the following check is simple enough to be set as the default option in lieu of the current "python", and has worked well for me on macOS (with brew install of spatial libraries), Windows (with OSgeo4W), and Ubuntu 20.04+:

  pythonPath <- if (nzchar(Sys.which("python3"))) {
    Sys.which("python3")
  } else if (nzchar(Sys.which("python"))) {
    Sys.which("python")
  } else {
    "python" ## use as is, but tests will likely fail
  }
  tiler_options(python = pythonPath)

sidenote: this came about as a response to #16

python-gdal not available on Ubuntu 20.04 (Focal)

It looks like python-gdal is not available for Ubuntu 20.04 (though python3-gdal is), so no tiles are created.

$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04 LTS
Release:	20.04
Codename:	focal
$ sudo apt install python-gdal
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package python-gdal is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However, the following packages replace it:
  gdal-bin

E: Package 'python-gdal' has no installation candidate
$ apt search gdal-bin
Sorting... Done
Full Text Search... Done
gdal-bin/focal,now 3.0.4+dfsg-1build3 amd64 [installed]
  Geospatial Data Abstraction Library - Utility programs

And in R:

> x <- system.file("maps/map_wgs84.tif", package = "tiler")
> tiles <- file.path(tempdir(), "tiles")
> tile(x, tiles, 0)
Reprojecting raster...
Coloring raster...
Preparing for tiling...
[1] "/tmp/RtmpPJEjuj"
Creating tiles. Please wait...
Traceback (most recent call last):
  File "/home/achubaty/Documents/GitHub/tiler/inst/python/gdal2tiles.py", line 52, in <module>
    from osgeo import gdal
ImportError: No module named osgeo
Creating tile viewer...
Complete.
Warning message:
In raster::projectRaster(r, e, method = method) :
  input and ouput crs are the same
> dir(tiles)
character(0)

I don't get the tiles in the right reference system

I am setting up a webapplication and want to convert .grd files to tiles that can be read by Leaflet.

This example can be easily followed by using the files and putting them under a folder called /data, as you can see in the code, that is needed.

here are the needed files: aoa_result_2.zip

I had to set the routes with tiler_options because there where some errors with systemvariables and the paths. But it runs and there is a output. But the tiles are obviously not in the right reference system. This is my code (remember to install and load the package):

tiler_options(osgeo4w = "C:/OSGeo4W64/OSGeo4W.bat", python="C:/Users/**_USERNAME_**/AppData/Local/Programs/Python/Python37/python.exe")

tile(file= 'data/aoa_result_2.grd',
     tiles= 'layers/aoa_result',
     zoom="10-13",
     viewer= TRUE,
)

Even the passed viewer can't display the layer correctly. The XML file shows that the EPSG is 3857 but I guess that is not right for leaflet. Shouldn't it be EPSG:4326 instead?

<?xml version="1.0" encoding="utf-8"?>
    <TileMap version="1.0.0" tilemapservice="http://tms.osgeo.org/1.0.0">
      <Title>tmp_raster.tif</Title>
      <Abstract></Abstract>
      <SRS>EPSG:3857</SRS>
      <BoundingBox minx="7.40000709521776" miny="51.85465878258405" maxx="7.82132047624863" maxy="52.06491340494047"/>
      <Origin x="7.40000709521776" y="51.85465878258405"/>
      <TileFormat width="256" height="256" mime-type="image/png" extension="png"/>
      <TileSets profile="mercator">
        <TileSet href="10" units-per-pixel="152.87405654296876" order="10"/>
        <TileSet href="11" units-per-pixel="76.43702827148438" order="11"/>
        <TileSet href="12" units-per-pixel="38.21851413574219" order="12"/>
        <TileSet href="13" units-per-pixel="19.10925706787109" order="13"/>
      </TileSets>
    </TileMap>

I tried some reprojections but always got the same refsys and can't display it in my leaflet app nor in the viewere.

Thank you for helping

Please remove dependencies on **rgdal**, **rgeos**, and/or **maptools**

This package depends on (depends, imports or suggests) raster and one or more of the retiring packages rgdal, rgeos or maptools (https://r-spatial.org/r/2022/04/12/evolution.html, https://r-spatial.org/r/2022/12/14/evolution2.html). Since raster 3.6.3, all use of external FOSS library functionality has been transferred to terra, making the retiring packages very likely redundant. It would help greatly if you could remove dependencies on the retiring packages as soon as possible.

Finding OSGeo4w resources

Hello,

I installed tiler on my windows 10 desktop and following the instructions outlined in the introduction, including installing 64-bit OSGeo4W from https://trac.osgeo.org/osgeo4w/. No problems there, however, the default installation went directly to C:/OSGeo4W64 and thus there is no C:/Program Files/QGIS 3.0/OSGeo4W.bat. This file is found at C:/OSGeo4W64/OSGeo4W.bat. When I use tiler_options(osgeo4w = 'C:/OSGeo4W64/OSGeo4W.bat'), I have no issues tiling images, however, I have to set this in each R session, even after adding the .bat file to my path.

Any suggestions?

Python-gdal on Travis and Appveyor

GDAL remains unavailable on Travis and Appveyor. While tests and examples using tile can run without creating tiles, tests and examples will fail if they require access to outputs of tile, as they will not exist. I have not yet figured out how to make the required libraries and bindings available in Travis and Appveyor via .travis.yml and appveyor.yml. Help appreciated.

is it possible to support tile projections other than 4326?

My reading of the documentation suggests that, while non-4326 rasters are supported, they are transformed to 4326 as part of the tile function. So, there is not currently a way to generate tiles in other projections.

Since the leaflet package now supports projections other than Web Mercator (via the Proj4Leaflet add-in), it would be beneficial if tiler could create tiles in other projections.

This is of particular interest for maps with data that cross the anti-meridian (180 longitude) or in polar regions which leaftlet (Web Mercator) currently struggles with.

python syntax errors

I see the following when running the tests in test-tile.R:

  File "inst/python/gdal2tilesIMG.py", line 800
  print 'Cache: %s MB' % (gdal.GetCacheMax() / 1024 / 1024)
  ^
    SyntaxError: invalid syntax

It doesn't trigger an error in R - tiler chugs along and the tests pass, but this needs to be resolved.

tiler only partially writing tiles

OK, here's a new take -

Using the current branch (so, system stderr is being returned), I'm working with https://www.dropbox.com/s/3bhry7hwhyo0vrf/merged_falkands_leaflet.tif?dl=0 - when I attempt to tile it 0-7, starting at different points depending on the runs, I don't get complete output to directories. E.g., 0 is complete, 1 is complete, but if I look at, say, 6, it might only have 21 and 22 - not 0-22.

Not sure why this is so.....

I'm reluctant to loop over and run each zoom level separately, only as I'm doing this with a raster that's projected for leaflet, and the first thing that happens each time I run tile is that the raster is reprojected, which takes time, but, well, thoughts?

RStudio console prompt not returning after successful tiling, sometimes

Occasionally the command prompt in the RStudio console does not become available again after successful tiling. All tiles are generated, Python is no longer running, the console log prints Complete, which signifies that the final part of the R function tile has executed, so nothing has halted or hung up to this point.

The caret shows again in the console, but after pressing return, it remains blank and nothing works. The session must be terminated.

This seems to be more likely on longer-running heavier-duty tiling jobs. I have observed it in Windows. It only happens in RStudio. Any instance in which it is observed, does not occur using the standard R GUI.

I don't know what this is about, so I do not have a solution in case anyone else observes this. But you don't need RStudio to make tiles, so try in the regular R console and hopefully that will work fine for you.

tiler won't write tiles

I'm having the oddest issue. I'm running even the demo code to write out to a directory on a server. And yet, while it even will create the directory, it never ever writes files to the directory, and no error or warning messages are given. Not even sure how to debug this one...

No files generated?

Looks like a sweet package and would love to get working on my computer. If you wouldn't mind taking a look and seeing what might be going on that would be much appreciated. Below is the reprex. What I noticed when running is the .html file is blank when viewed online (boundary will appear as layerControl and a point marker in the lower right corner) and no files are generated into either the temp directory or even when trying another directory. I've tried running the gdal2tiles.py on OSgeo Shell but get an error. Don't know if these are related or not... Thanks for your time.

library(tiler)
library(raster)
#> Loading required package: sp
tiler_options(osgeo4w = "C:/OSGeo4W64/OSGeo4W.bat", python = "C:/Python/ArcGIS10.5/python.exe")

tiles <- file.path(tempdir(), "tiles")
x <- system.file("maps/map_wgs84.tif", package = "tiler")

tile(x, tiles, zoom = '2-3')
#> Reprojecting raster...
#> Warning in raster::projectRaster(r, e, method = method): input and ouput crs are
#> the same
#> Coloring raster...
#> Preparing for tiling...
#> [1] "C:\\Users\\JOSHUA~1\\AppData\\Local\\Temp\\RtmpozBdUM"
#> Creating tiles. Please wait...
#> Creating tile viewer...
#> Complete.
view_tiles(file.path(tempdir(), "tiles"))

^{Created on 2020-12-06 by the reprex package (v0.3.0)}

Error from OSGeo4W Shell

Traceback (most recent call last):
  File "C:\Program Files (x86)\GDAL\gdal2tiles.py", line 44, in <module>
    from osgeo import gdal
  File "C:\Python27\ArcGIS10.5\lib\site-packages\osgeo\__init__.py", line 21, in <module>
    _gdal = swig_import_helper()
  File "C:\Python27\ArcGIS10.5\lib\site-packages\osgeo\__init__.py", line 17, in swig_import_helper
    _mod = imp.load_module('_gdal', fp, pathname, description)
ImportError: DLL load failed: %1 is not a valid Win32 application.

lineup with leaflet off for projection that should work...

I'm working with a raster that is

CRS arguments:
+proj=longlat +datum=WGS84 +no_defs +ellps=WGS84 +towgs84=0,0,0

I have a shape file of the area it should overlay in 4326, and plotting them together is fine. The overlay works. However, a simple call to tiler - just the raster file, a write-out directory and zoom of "0-7" as the arguments, yields a tileset that, when I serve it up locally, has this overlay.

Note, this is a tileset that should overlay the Falkland islands in the southern hemisphere, and is significantly smaller than what is shown here.

Thoughts in where things are going awry?

For reference, the leaflet call is

library(leaflet)
#in the relevant directory
#python -m SimpleHTTPServer 8000 
tilesUrl <- "http://localhost:8000/{z}/{x}/{y}.png"

leaflet() %>% 
  addTiles() %>%
  addTiles(urlTemplate = tilesUrl, options = tileOptions(opacity = 0.8))  %>%
  addLegend(colors = col_default,   position = "bottomright",
            labels = as.character(4:13), title = "Users Agreeing") 

[feature request] add* functions for leaflet package

This is to continue the twitter discussion as per https://twitter.com/TimSalabim3/status/1006414006201077761

As you correctly point out in your response to my tweet, our (mapview's) primary interest is in seamless provision of tiled images for rendering on a leaflet map. The crux here is that the map should not be created as a side of the tile generation but that the generated tiles can be added to an existing leaflet object. Hence, the add* functionality. As I mention in my tweet we have just recently implemented such add* functions and along with those had to also implement functionality very similar to what tiler provides, i.e. tile generation. In fact, we simply had to replace one line in the respective functions to reproduce our functionality with that of tiler. This is great, as it means we can drop the heavy python dependency and replace it with a light(er)-weight R dependency. Taking this a little further, I think it would be beneficial for everyone to have those add* functions provided by tiler, rather than mapview. Specifically, I think that a) these functions address a rather special case which is somewhat on the brink of mapview's scope and b) having control over the whole process from tile creation to rendering is desirable in many ways.

This, of course, introduces some questions to be considered for tiler:
First and foremost, it would mean to add a htmlwidgets and leaflet dependency/import. Secondly, it means that some javascript libs along with proper bindings would need to be added to get a smooth behaviour. Most of these are already in mapview and could easily be ported and adapted to work with tiler. I would be more than happy to help with this.
On the other hand, this would open up possibilities to advance tiler functionality towards a fully functional renderer of non-geographic tiles as tiler could also provide functions to render (circle)markers/polylines/polygons on top of tiled images (via leaflet-rastercoords js binding). Again, I would be happy to help with those.

I know that this is a lot to digest, so I'm gonna stop here. In case anything is unclear, I am happy to elaborate. I do have some further suggestions/questions but I'm gonna reserve those for a later date as I think you will have enough to take in and consider for now.