maplibre/martin

Blazing fast and lightweight PostGIS, MBtiles and PMtiles tile server, tile generation, and mbtiles tooling.

Martin is a tile server able to generate and serve vector tiles on the fly from large PostGIS databases, PMTile (local or remote), and MBTile files, allowing multiple tile sources to be dynamically combined into one. Martin optimizes for speed and heavy traffic, and is written in Rust.

Additionally, there are several tools for generating tiles in bulk from any Martin-supported sources (similar to tilelive-copy), copying tiles between MBTiles files, creating deltas (patches) and applying them, and validating MBTiles files.

See Martin book for complete documentation.

Installation

See installation instructions in the Martin book.

Prerequisites: If using Martin with PostgreSQL database, you must install PostGIS with at least v3.0+, v3.1+ recommended.

You can download martin from GitHub releases page.

Platform	AMD-64	ARM-64
Linux	.tar.gz (gnu) .tar.gz (musl) .deb	.tar.gz (musl)
macOS	.tar.gz	.tar.gz
Windows	.zip

If you are using macOS and Homebrew you can install martin and mbtiles using Homebrew tap.

brew tap maplibre/martin
brew install martin

Running Martin Service

See running instructions in the Martin book.

Martin supports any number of PostgreSQL/PostGIS database connections with geospatial-enabled tables and tile-producing SQL functions, as well as PMTile and MBTile files as tile sources.

Martin can auto-discover tables and functions using a connection string. A PG connection string can also be passed via the DATABASE_URL environment variable.

Each tile source will have a TileJSON endpoint.

Examples

# publish all tables and functions from a single database
export DATABASE_URL="postgresql://user:password@host:port/database"
martin

# same as above, but passing connection string via CLI, together with a directory of .mbtiles/.pmtiles files
martin postgresql://user:password@host:port/database path/to/dir

# publish all discovered tables/funcs from two DBs
# and generate config file with all detected sources
martin postgres://... postgres://...  --save-config config.yaml

# use configuration file instead of auto-discovery
martin --config config.yaml

Docker Example

See Docker instructions in the Martin book.

Martin is also available as a Docker image. You could either share a configuration file from the host with the container via the -v param, or you can let Martin auto-discover all sources e.g. by passing DATABASE_URL or specifying the .mbtiles/.pmtiles files.

export PGPASSWORD=postgres  # secret!
docker run -p 3000:3000 \
           -e PGPASSWORD \
           -e DATABASE_URL=postgresql://user@host:port/db \
           -v /path/to/config/dir:/config \
           ghcr.io/maplibre/martin --config /config/config.yaml

API

See API documentation in the Martin book.

Martin data is available via the HTTP GET endpoints:

URL	Description
/	Status text, that will eventually show web UI
/catalog	List of all sources
/{sourceID}	Source TileJSON
/{sourceID}/{z}/{x}/{y}	Map Tiles
/{source1},…,{sourceN}	Composite Source TileJSON
/{source1},…,{sourceN}/{z}/{x}/{y}	Composite Source Tiles
/sprite/{spriteID}[@2x].{json,png}	Sprites (low and high DPI, index/png)
/font/{font}/{start}-{end}	Font source
/font/{font1},…,{fontN}/{start}-{end}	Composite Font source
/health	Martin server health check: returns 200 OK

Re-use Martin as a library

Martin can be used as a standalone server, or as a library in your own Rust application. When used as a library, you can use the following features:

• postgres - enable PostgreSQL/PostGIS tile sources
• pmtiles - enable PMTile tile sources
• mbtiles - enable MBTile tile sources
• fonts - enable font sources
• sprites - enable sprite sources

Documentation

See Martin book for complete documentation.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT) at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.

martin's Issues

Custom SpriteSheets and Vector Styles

Hi There,

Really big fan of what you guys have built and have enjoyed experimenting with martin! I had a question regarding custom style/spritesheets for vector tiles with martin.

Im trying to create a custom map with naval charts derived from open sea map, and partially inspired by this tegola project: https://github.com/go-spatial/tegola-openseamap

https://map.openseamap.org/

But i prefer the tooling that martin provides as to tegola, is there anyway to add custom spritesheets/datasheets to martin? or is that only done at the osm/land data preparation stage.

Thanks!

SSL, HTTP/2 support

So Actix added SSL a while back (marginal benefit really), and recently HTTP/2; I've confirmed that both work behind nginx (or whichever other reverse proxy you care to use), though it would be nice for dev purposes to be able to run it standalone with HTTP/2, given the noticeable queueing effect incurred by HTTP/1.1 (irrespective of load on the database workers).

I'll tinker with it over the coming weekend (should be relatively straightforward), would you mind if I submit a PR to that effect?

RUSTSEC-2020-0056: stdweb is unmaintained

stdweb is unmaintained

Details
Status	unmaintained
Package	stdweb
Version	0.4.20
URL	koute/stdweb#403
Date	2020-05-04

The author of the stdweb crate is unresponsive.

Maintained alternatives:

• wasm-bindgen
• js-sys
• web-sys

See advisory page for additional details.

handle X-Forwarded-Proto

Question: bounding boxes on functions or queries on tables?

Hi I am new to Martin and just diving into the world of map data.
We have Martin running and serving out tiles onto the mapbox map. We'd like to get boundary boxes working to get city boundaries. Your README says to use TileBBox which hasnt been updated in forever.

I tried loading TileBBox and the function in the Readme anyways and loading it gave me a bunch of "psql:postgis-vt-util.sql:609: ERROR: type geometry does not exist" errors. Not sure if thats because my geometry column is named geom or what. Is there a more modern way of doing this? Should I just be doing it with straight up POSTGIS functions. Looks like a lot of vector tile stuff has been added to POSTGIS nowadays.

Also, are there examples for how to query for specific columns when using tables? I am not sure where the rest of the table columns are supposed to be in the response.

Thanks!

RUSTSEC-2020-0049: Use-after-free in Framed due to lack of pinning

Use-after-free in Framed due to lack of pinning

Details
Package	actix-codec
Version	0.2.0
URL	actix/actix-net#91
Date	2020-01-30
Patched versions	>= 0.3.0-beta.1

Affected versions of this crate did not require the buffer wrapped in Framed to be pinned,
but treated it as if it had a fixed location in memory. This may result in a use-after-free.

The flaw was corrected by making the affected functions accept Pin<&mut Self> instead of &mut self.

See advisory page for additional details.

Dynamic update of list of table sources

Is there a way to generate vector tiles from tables that are added to the DB after starting Martin?

On a separate note, the project looks great. Thanks.

add a proper way to handle long-running and interrupted queries

Martin may become unresponsive while PostgreSQL is handling a lot of long-running SQL queries. Some frontends like Nginx may also cancel requests based on timeout, but these requests might still be running as queries in PostgreSQL.

We need a way to set a timeout for such requests, so we can cancel queries in the PostgreSQL.

filtering a layer before tiles are created

If I would like to filter a large layer based on feature properties, is that something I can do before creating a vectorTile for the entire layer, or do I have to rely on the Mapbox JS filter once the entire layer is loaded?

Improve logging in debug mode

Some errors are missing in logs in debug mode

Allow on-disk caching of tiles

Any plans on allowing on-disk caching of tiles ?

So basically:

if(!tile_on_disk.exists) {
  run_sql
  save_to_disk
  return tile
}
else {
  return tile_on_disk
}

can not parse "{z}" to a u32

Hello I just installed in manual mode Martin. But I have this error:

http://185.216.25.216:3000/public.perimeters/{z}/{x}/{y}.pbf
can not parse "{z}" to a u32

Do you have any idea what is blocking. Thank you

question: working example

Hello,

I'm glad I finally found a simple postgis to pbf on the fly server, but I'm having a really hard time getting it to actually integrate with leaflet or openlayers. Would you be able to provide a very simple working example of how it would work?

Support for Nginx Reverse proxy to support base url and domain name

Hi!

First up, thank you for your awesome work on this project! I have evaluated all the alternatives out there for vector tile server and this is the best one that works out of box!

We plan on running martin behind a reverse proxy. This works fine, but martin currently does not support base url and domain names.

For example: We want to reverse proxy example.com/tileserver/ to martin server.
Now while this works, martin does not adjust the URLs that are output in the json responses to reflect the base path tileserver and the domain example.com. The json responses still say that the endpoints are localhost:3000/something.json.

Support for configuration is required so that the json data will reflect the example.com/tileserver/<something>.json

This is potentially solve able by adding support for environment variables like : DOMAIN_NAME BASE_URL which updates the source table json end points to reflect the correct urls.

Hope my point is clear, but please let me know if you need more details!

Do not use sources with SRID equal to 0

handle X-Rewrite-URL

Set id to feature

Is there a way to set id to feature (last param of ST_AsMVT function)?

id from table is not passed to feature id.

Or int value:

How fast is the real-time loading speed?

I have a mount of data(about >=5 million,Up to 100 million).And the area polygon has many points,now I use tilestrata + postgresql to rendering geo data ,but only can zoom to 13level and the speed is not ideal.，I want to know whether martin can solve my problem（real time rendering big geo data）

Forgive my poor English level，hope you can understand me。
Any Help will appreciated！

CORS policy: No 'Access-Control-Allow-Origin' within localhost and "http://0.0.0.0:3000"

Running in some internal issue. I'm running a local docker
docker run \ -p 3000:3000 \ -e DATABASE_URL=postgres://[email protected]/db \ urbica/martin
on my Mac OS. I've added a table with points. I made a website with mapbox GL and followed the directions in the README. The website is running on localhost.

Looks like I am running into a CORS issue and need help to solve.

Looks like the table is loading but when it fetched the "pbf" files is unreachable.

I'll appreciate the help.

Thanks

Can't generate config: database error

Hi there,
I was just playing a little with Urbica/Martin trying to serve MVTs from my postgis image.
I ran your 'docker run --net=host -p 3000:3000 -e DATABASE_URL=postgres://postgres:postgres@localhost/postgres urbica/martin'

However I got:
[2019-11-05T12:45:11Z INFO martin] Starting martin v0.5.0
[2019-11-05T12:45:11Z INFO martin] Config is not set, scanning database
[2019-11-05T12:45:11Z INFO martin] Connected to postgres://postgres:postgres@localhost/postgres
[2019-11-05T12:45:11Z ERROR martin] Can't generate config: database error

Any clue about the source of this error?

Thx!
Rafael

how to set up vector tiles caching with nginx?

hello guys! I am new with both gis and nginx.
I have try a lot with caching vector tiles throut Nginx with Martin. like that! But it didn't work, my /var/cache/ngix folder still empty.
can any body give me a tutorial to cache and reserve proxy with nginx. Thanks in seriously

nginx.conf.zip

Support SSL connections to PostgreSQL

As mentionned here, martin doesn't support SSL connections to PostgreSQL since TlsMode is hardcoded to None here.

Is it possible to make this configurable, either using sslmode variable in the connection string or with a specific setting?

Thank!

Database user and password

How do we specify user and password for database connections?

RUSTSEC-2020-0016: `net2` crate has been deprecated; use `socket2` instead

net2 crate has been deprecated; use socket2 instead

Details
Status	unmaintained
Package	net2
Version	0.2.34
URL	deprecrated/net2-rs@3350e38
Date	2020-05-01

The net2 crate has been deprecated
and users are encouraged to considered socket2 instead.

See advisory page for additional details.

error: http code 500

hello! I am trying start martin with post 3.0. but everytime i try access .pbf endpoint, it returned err 500 like that
nguyenmanhcuong@NGUYENs-MacBook-Pro ~ % curl -i localhost:3000/import.giao-thong/0/0/0.pbf HTTP/1.1 500 Internal Server Error content-length: 0 access-control-allow-origin: * date: Tue, 07 Apr 2020 09:02:55 GMT

I have run martin tin DEBUG mode and this was the result. help me pls:
nguyenmanhcuong@NGUYENs-MacBook-Pro ~ % export RUST_LOG=actix_web=info,martin=debug,postgres=debug martin postgres://nguyenmanhcuong:123456@localhost/gis [2020-04-07T08:38:24Z INFO martin] Starting martin v0.5.0 [2020-04-07T08:38:24Z INFO martin] Config is not set, scanning database [2020-04-07T08:38:24Z INFO martin] Connected to postgres://nguyenmanhcuong:123456@localhost/gis [2020-04-07T08:38:24Z DEBUG postgres] executing query: [2020-04-07T08:38:24Z DEBUG postgres] preparing query with name ``: WITH columns AS ( SELECT ns.nspname AS table_schema, class.relname AS table_name, attr.attname AS column_name, trim(leading '_' from tp.typname) AS type_name FROM pg_attribute attr JOIN pg_catalog.pg_class AS class ON class.oid = attr.attrelid JOIN pg_catalog.pg_namespace AS ns ON ns.oid = class.relnamespace JOIN pg_catalog.pg_type AS tp ON tp.oid = attr.atttypid WHERE NOT attr.attisdropped AND attr.attnum > 0) SELECT f_table_schema, f_table_name, f_geometry_column, srid, type, COALESCE( jsonb_object_agg(columns.column_name, columns.type_name) FILTER (WHERE columns.column_name IS NOT NULL), '{}'::jsonb ) as properties FROM geometry_columns LEFT JOIN columns ON geometry_columns.f_table_schema = columns.table_schema AND geometry_columns.f_table_name = columns.table_name AND geometry_columns.f_geometry_column != columns.column_name GROUP BY f_table_schema, f_table_name, f_geometry_column, srid, type; [2020-04-07T08:38:24Z DEBUG postgres] executing statement with parameters: [] [2020-04-07T08:38:25Z INFO martin::table_source] Found OSM.2017-07-03_vietnam_ho-chi-minh-city table source [2020-04-07T08:38:25Z INFO martin::table_source] Found OSM.2017-07-03_vietnam_ho-chi-minh-city transportation_name table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.10 table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.14 table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.22 table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.28 table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.Added geom info table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.Centroids table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.Joined layer table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.geocoding table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.giao-thong table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.logioiline table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.mepDuong table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.nen table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.pncenter table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.pndc table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.road table source [2020-04-07T08:38:25Z INFO martin::table_source] Found import.xd table source [2020-04-07T08:38:25Z INFO martin::table_source] Found my_topology.edge table source [2020-04-07T08:38:25Z INFO martin::table_source] Found my_topology.edge_data table source [2020-04-07T08:38:25Z INFO martin::table_source] Found my_topology.face table source [2020-04-07T08:38:25Z INFO martin::table_source] Found my_topology.node table source [2020-04-07T08:38:25Z INFO martin::table_source] Found my_topology.p10 table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.beacons table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.beacons_intersect table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.beacons_views table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.bearing_labels table source [2020-04-07T08:38:25Z WARN martin::table_source] public.bearing_labels has SRID 0, skipping [2020-04-07T08:38:25Z INFO martin::table_source] Found public.boundaries table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.boundary_labels table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.derived_boundaries table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.giao-thong table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.hist_beacons table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.ne_10m_admin_0_countries table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.parcel_overlap_matviews table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.parcels table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.parcels_intersect table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.parcels_line_length table source [2020-04-07T08:38:25Z WARN martin::table_source] public.parcels_line_length has SRID 0, skipping [2020-04-07T08:38:25Z INFO martin::table_source] Found public.parcels_lines table source [2020-04-07T08:38:25Z WARN martin::table_source] public.parcels_lines has SRID 0, skipping [2020-04-07T08:38:25Z INFO martin::table_source] Found public.perimeters table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.perimeters_original table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.pndc table source [2020-04-07T08:38:25Z INFO martin::table_source] Found public.reference_view table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.addrfeat table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.bg table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.county table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.cousub table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.edges table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.faces table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.place table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.state table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.tabblock table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.tract table source [2020-04-07T08:38:25Z INFO martin::table_source] Found tiger.zcta5 table source [2020-04-07T08:38:25Z DEBUG postgres] preparing query with name ``: SELECT routines.specific_schema, routines.routine_name FROM information_schema.routines LEFT JOIN information_schema.parameters ON routines.specific_name=parameters.specific_name WHERE routines.data_type = 'bytea' GROUP BY routines.specific_schema, routines.routine_name, routines.data_type HAVING array_agg(array[parameters.parameter_name::text, parameters.data_type::text]) @> array[array['z', 'integer'], array['x', 'integer'], array['y', 'integer'], array['query_params', 'json']]; [2020-04-07T08:38:25Z DEBUG postgres] executing statement with parameters: [] [2020-04-07T08:38:25Z DEBUG postgres] executing query: [2020-04-07T08:38:25Z DEBUG postgres] preparing query with name ``: select (regexp_matches(postgis_lib_version(), '^(\d+\.\d+\.\d+)', 'g'))[1] as postgis_lib_version [2020-04-07T08:38:25Z DEBUG postgres] executing statement with parameters: [] [2020-04-07T08:38:25Z INFO martin] Martin has been started on 0.0.0.0:3000. [2020-04-07T08:39:18Z INFO actix_web::middleware::logger] 127.0.0.1:49385 "GET /index.json HTTP/1.1" 200 29455 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 0.012087 [2020-04-07T08:39:18Z INFO actix_web::middleware::logger] 127.0.0.1:49385 "GET /favicon.ico HTTP/1.1" 404 0 "http://localhost:3000/index.json" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 0.000102 [2020-04-07T08:39:58Z DEBUG postgres] executing query: [2020-04-07T08:39:58Z DEBUG postgres] preparing query with name ``: WITH bounds AS (SELECT ST_MakeEnvelope(11876479.705184326, 1210151.0316424556, 11877091.201410523, 1209539.5354162594, 3857) as mercator, ST_Transform(ST_MakeEnvelope(11876479.705184326, 1210151.0316424556, 11877091.201410523, 1209539.5354162594, 3857), 4326) as original) SELECT ST_AsMVT(tile, 'import.10', 4096, 'geom' ) FROM ( SELECT ST_AsMVTGeom(ST_Transform(geom, 3857), bounds.mercator, 4096, 64, true) AS geom , "10_id","id","length" FROM import.10, bounds WHERE geom && bounds.original ) AS tile WHERE geom IS NOT NULL [2020-04-07T08:39:58Z INFO actix_web::middleware::logger] 127.0.0.1:49418 "GET /import.10/16/52190/30789.pbf HTTP/1.1" 500 0 "-" "PostmanRuntime/7.24.0" 0.033219 [2020-04-07T08:40:00Z DEBUG postgres] executing query: [2020-04-07T08:40:00Z DEBUG postgres] preparing query with name ``: WITH bounds AS (SELECT ST_MakeEnvelope(11876479.705184326, 1210151.0316424556, 11877091.201410523, 1209539.5354162594, 3857) as mercator, ST_Transform(ST_MakeEnvelope(11876479.705184326, 1210151.0316424556, 11877091.201410523, 1209539.5354162594, 3857), 4326) as original) SELECT ST_AsMVT(tile, 'import.10', 4096, 'geom' ) FROM ( SELECT ST_AsMVTGeom(ST_Transform(geom, 3857), bounds.mercator, 4096, 64, true) AS geom , "10_id","id","length" FROM import.10, bounds WHERE geom && bounds.original ) AS tile WHERE geom IS NOT NULL [2020-04-07T08:40:00Z INFO actix_web::middleware::logger] 127.0.0.1:49418 "GET /import.10/16/52190/30789.pbf HTTP/1.1" 500 0 "-" "PostmanRuntime/7.24.0" 0.000931 [2020-04-07T08:40:20Z INFO actix_web::middleware::logger] 127.0.0.1:49418 "GET /index.json HTTP/1.1" 200 29455 "-" "PostmanRuntime/7.24.0" 0.005915 [2020-04-07T08:41:02Z INFO actix_web::middleware::logger] 127.0.0.1:49418 "GET /import.giao-thong.json HTTP/1.1" 200 317 "-" "PostmanRuntime/7.24.0" 0.001562 [2020-04-07T08:41:05Z INFO actix_web::middleware::logger] 127.0.0.1:49442 "GET /import.giao-thong/%7Bz%7D/%7Bx%7D/%7By%7D.pbf HTTP/1.1" 404 28 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 0.003709 [2020-04-07T08:41:22Z DEBUG postgres] executing query: [2020-04-07T08:41:22Z DEBUG postgres] preparing query with name ``: WITH bounds AS (SELECT ST_MakeEnvelope(-20037508.34, 20037508.34, 20037508.34, -20037508.34, 3857) as mercator, ST_Transform(ST_MakeEnvelope(-20037508.34, 20037508.34, 20037508.34, -20037508.34, 3857), 4326) as original) SELECT ST_AsMVT(tile, 'import.giao-thong', 4096, 'geom' ) FROM ( SELECT ST_AsMVTGeom(ST_Transform(geom, 3857), bounds.mercator, 4096, 64, true) AS geom , "name_ja","name_tr","name_th","name_he","name_uk","name_kn","name_sr-la","name_de_1","name_gd","name_ga","name_de","name_hy","name_bs","name_ka","name_lv","name_be","name_sr","name_ja_rm","name_ko_rm","name_mt","name_ru","name_nonla","name_latin","name_sl","name_ro","name_sk","ref","name_az","name_int","name_ca","name_br","name_fi","name_fr","name","name_nl","name_ja_ka","name_bg","name_pl","name_zh","name_sv","ref_length","name_is","name_fy","name_en_1","name_el","name_et","class","name_la","network","name_lt","name_mk","name_ar","name_es","name_cy","name_lb","name_pt","name_it","name_ko","name_cs","name_en","name_rm","name_sq","name_hu","id","mvt_id","name_hr","name_da","name_kk","name_no" FROM import.giao-thong, bounds WHERE geom && bounds.original ) AS tile WHERE geom IS NOT NULL [2020-04-07T08:41:22Z INFO actix_web::middleware::logger] 127.0.0.1:49442 "GET /import.giao-thong/0/0/0.pbf HTTP/1.1" 500 0 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.149 Safari/537.36" 0.006816 [2020-04-07T09:02:55Z DEBUG postgres] executing query: [2020-04-07T09:02:55Z DEBUG postgres] preparing query with name ``: WITH bounds AS (SELECT ST_MakeEnvelope(-20037508.34, 20037508.34, 20037508.34, -20037508.34, 3857) as mercator, ST_Transform(ST_MakeEnvelope(-20037508.34, 20037508.34, 20037508.34, -20037508.34, 3857), 4326) as original) SELECT ST_AsMVT(tile, 'import.giao-thong', 4096, 'geom' ) FROM ( SELECT ST_AsMVTGeom(ST_Transform(geom, 3857), bounds.mercator, 4096, 64, true) AS geom , "name_ja","name_tr","name_th","name_he","name_uk","name_kn","name_sr-la","name_de_1","name_gd","name_ga","name_de","name_hy","name_bs","name_ka","name_lv","name_be","name_sr","name_ja_rm","name_ko_rm","name_mt","name_ru","name_nonla","name_latin","name_sl","name_ro","name_sk","ref","name_az","name_int","name_ca","name_br","name_fi","name_fr","name","name_nl","name_ja_ka","name_bg","name_pl","name_zh","name_sv","ref_length","name_is","name_fy","name_en_1","name_el","name_et","class","name_la","network","name_lt","name_mk","name_ar","name_es","name_cy","name_lb","name_pt","name_it","name_ko","name_cs","name_en","name_rm","name_sq","name_hu","id","mvt_id","name_hr","name_da","name_kk","name_no" FROM import.giao-thong, bounds WHERE geom && bounds.original ) AS tile WHERE geom IS NOT NULL [2020-04-07T09:02:55Z INFO actix_web::middleware::logger] 127.0.0.1:49641 "GET /import.giao-thong/0/0/0.pbf HTTP/1.1" 500 0 "-" "curl/7.64.1" 0.008419

martin config expose only specific schema

Hello ! Thank you for the superbe work !

I am wondering if we could expose only a entire schema instead of exposing one table at a time ?

Thank you

Automate releases build for macOS and Linux

We need to automate releases build for macOS and Linux on Circle CI

Add security to API with JWT

First thing, thanks your project, its awesome.

I am working in a small company (its name is dotGIS ) and we are using Martin for some projects.

We need in the api an authentication method such as JWT because it is a common standard.

We can help your project...personally I want to touch martin code...I found some libraries for JWT in Rust...but I need to check which one is better.

Well. Could I fork Martin and add (as branch) support for JWT?

Regards.

Can I mix config.yml and environment vars?

Hi folks

Thanks for an excellent piece of software :)

I would like to deploy Martin with specific tables and functions available. I can do this with config.yaml just fine. But if I use the --config=config.yaml option when starting martin, the YAML is required to have a connection_string key. I'm hosting on Heroku, which only has $DATABASE_URL available as an environment variable.

Is it possible to interpolate environment variables in the config file? Or can I set the connection_string key in the config file to some value to tell martin to use the environment variable?

Thanks

--listen-address flag doesn't work from CLI

Expected this to work:

martin --listen_addresses="0.0.0.0:3001" postgres://postgres@localhost:5433/db

But I get this instead:

 INFO 2019-09-25T11:52:56Z: martin: Starting martin v0.2.0
 INFO 2019-09-25T11:52:56Z: martin: Connecting to postgres://postgres@localhost:5433/db
 INFO 2019-09-25T11:52:57Z: martin: Connected to postgres: postgres://postgres@localhost:5433/db
 INFO 2019-09-25T11:52:57Z: martin::table_source: public.points table found
 INFO 2019-09-25T11:52:57Z: martin: Martin has been started on 0.0.0.0:3000.
thread 'main' panicked at 'Can't bind to 0.0.0.0:3000: Os { code: 48, kind: AddrInUse, message: "Address already in use" }', libcore/result.rs:1009:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.

RUSTSEC-2018-0007: Stack overflow when parsing malicious DNS packet

Stack overflow when parsing malicious DNS packet

Details
Package	trust-dns-proto
Version	0.18.0-alpha.2
Date	2018-10-09
Patched versions	>=0.4.3,>=0.5.0-alpha.3

There's a stack overflow leading to a crash when Trust-DNS's parses a
malicious DNS packet.

Affected versions of this crate did not properly handle parsing of DNS message
compression (RFC1035 section 4.1.4). The parser could be tricked into infinite
loop when a compression offset pointed back to the same domain name to be
parsed.

This allows an attacker to craft a malicious DNS packet which when consumed
with Trust-DNS could cause stack overflow and crash the affected software.

The flaw was corrected by trust-dns-proto 0.4.3 and upcoming 0.5.0 release.

See advisory page for additional details.

Attribution is null

I just setup martin and in MapboxGL JS everything is fine. Mapbox GL on Android has a problem with the TileJSON though as the "attribution" field is null.

There is an error log on Android which says:

Failed to load source xyz: source attribution must be a string

When using the XYZ tile url directly without using TileJSON Mapbox gl is happy.

Should it be possible to configure these fields?

Usage query_params in Function Source Tiles

How can I pass query_params in Function Source Tiles from tile source url MapBox?

source: { type: 'vector', url: 'http://localhost:3000/rpc/public.function_source.json?param1=param1' // doesn't work for me }

And how do I need to use query_params here? I want to filter my table by condition from query_params.

CREATE OR REPLACE FUNCTION public.function_source_test(z integer, x integer, y integer, query_params json) RETURNS bytea AS ...

Validation/sanitization of query_params

Hi,

I'm wondering if you plan to add some sanitization and validation features for user input via query_params. It doesn't seem that such idea is available in the code right now. If not, do you use any psql functions, library to do that ? Do you have any recommendation ?

Thanks in advance

Wrong scheme? (TMS Shows nothing, XYZ works)

Im trying to use martin with some Dutch public data in EPSG 28992 but when adding it to the map i get no data.
After some troubleshouting i got it to work by changing the scheme in the tilejson from TMS to XYZ.
Am i doing something wrong, or could this actually be different, and if so, should this be configurable?

Support for multi-layer tiles

I would like your thoughts on how to generate multi-layer tiles.

My idea is to allow requests like /{schema_name}.{table_name},{schema_name}.{table_name}/{z}/{x}/{y}.pbf and use binary concatenation of ST_AsMVT statements to produce the tile, see https://postgis.net/docs/ST_AsMVT.html

I believe the only way to do this now is via function source, for example:

CREATE OR REPLACE FUNCTION
  public.assets(z integer, x integer, y integer, query_params json)
RETURNS bytea
AS
$$
-- Prefer z,x,y inputs over z,x,y table columns.
#variable_conflict use_variable
DECLARE
  poles_mvt bytea;
  meters_mvt bytea;
BEGIN
  SELECT INTO poles_mvt
    ST_AsMVT(tile, 'public.poles', 4096, 'geom')
    FROM (
      SELECT
        ST_AsMVTGeom(shape, TileBBox(z, x, y, 3857), 4096, 64, true) AS geom
        ,"pole_number"
      FROM public.switches
      WHERE shape && TileBBox(z, x, y, 3857)
    ) AS tile
    WHERE geom IS NOT NULL;

  SELECT INTO meters_mvt
    ST_AsMVT(tile, 'public.meters', 4096, 'geom')
    FROM (
      SELECT
        ST_AsMVTGeom(shape, TileBBox(z, x, y, 3857), 4096, 64, true) AS geom
        ,"meter_number"
      FROM public.meters
      WHERE shape && TileBBox(z, x, y, 3857)
    ) AS tile
    WHERE geom IS NOT NULL;

  RETURN poles_mvt || meters_mvt;
END
$$
LANGUAGE plpgsql IMMUTABLE STRICT PARALLEL SAFE;

I have never developed in Rust, but Martin's codebase is so clean I could probably make these changes and issue a decent PR. What do you think?

Fails with beta version of Postgis (3.0.0beta1)

Fails on startup with the following error:

Can't check PostGIS version: Extra junk after valid version: beta1

question: Does it support raster data?

Can we store/map raster data using postgis + martin to speed up rendering on the client side?

Edit: And if it does not, how can that be achieved?

Check if PostGIS is installed on start

We should check if a satisfying version of PostGIS is installed when starting martin

Invalid tile URL in tilejson when using X-Rewrite-URL header

Tiles URL in tile json are incorrect when using a proxy.

The use of header X-Rewrite-URL make the tile URL invalid.

Following the NGINX configuration from Readme

        location ~ /tiles/(?<fwd_path>.*) {
            proxy_set_header  X-Rewrite-URL $request_uri;
            proxy_set_header  X-Forwarded-Host $host;
            proxy_set_header  X-Forwarded-Proto $scheme; # or $http_x_forwarded_proto
            proxy_pass        http://martin/$fwd_path$is_args$args;
        }

Output this tilejson

{
	"tilejson": "2.2.0",
	"id": null,
	"name": "public.cell_agg_by_activity_j_tile",
	"description": null,
	"version": "1.0.0",
	"attribution": null,
	"template": null,
	"legend": null,
	"scheme": "xyz",
	"tiles": ["http://localhost/tiles/rpc/public.cell_agg_by_activity_j_tile.json?activity=&count=tracks&time=/{z}/{x}/{y}.pbf?activity=&count=tracks&time="],
	"grids": null,
	"data": null,
	"minzoom": 0,
	"maxzoom": 30,
	"bounds": [-180.0, -90.0, 180.0, 90.0],
	"center": null
}

With invalid tiles field http://localhost/tiles/rpc/public.cell_agg_by_activity_j_tile.json?activity=&count=tracks&time=/{z}/{x}/{y}.pbf?activity=&count=tracks&time=.

While by removing proxy_set_header X-Rewrite-URL $request_uri; for the NGINX configuration we have:

	"tiles": ["http://localhost/rpc/public.cell_agg_by_activity_j_tile/{z}/{x}/{y}.pbf?activity=&count=tracks&time="],

A partial workaround is by not rewriting the URL

        location /rpc {
            proxy_set_header  X-Forwarded-Host $host;
            proxy_set_header  X-Forwarded-Proto $scheme; # or $http_x_forwarded_proto
            proxy_pass        http://martin;
        }

PostGIS 2.3.0 support - Google Cloud SQL

Hello - brand new to Martin and wanting to check it out; the issue is that we are running our Postgres instance on Cloud SQL, which does not support the PostGIS extension past 2.3.0. Will this even be feasible, or is there just a couple gotchas I should be aware of? Any insight would be greatly appreciated. Thank you!

How to specify float in properties file?

Hi, I have a column called ground with values like 1.5
But it always return a String

properties:
id: int4
ground: numeric(5,2)
height: float

Thanks

add etag

How to make HTTPS ?

Now works under the protocol HTTP (http://192.168.0.1:3000/...)
How to add a certificate for a secure connection? (https://192.168.0.1:3000/...)?

how to stop simplify geometry?

first, thanks for making Martin. it is simple and flexible.
i have a question: how to config simplifing geometry? -

thanks in advance

RUSTSEC-2020-0036: failure is officially deprecated/unmaintained

failure is officially deprecated/unmaintained

Details
Status	unmaintained
Package	failure
Version	0.1.8
URL	rust-lang-deprecated/failure#347
Date	2020-05-02

The failure crate is officially end-of-life: it has been marked as deprecated
by the former maintainer, who has announced that there will be no updates or
maintenance work on it going forward.

The following are some suggested actively developed alternatives to switch to:

• anyhow
• eyre
• fehler
• snafu
• thiserror

See advisory page for additional details.

Joins in Function Source

I understand this is less of a Martin question and more SQL, however I'm running into a significant hurdle with marrying data, and I was hoping that you might have some insight.

Currently, I am running a Function Source that filters through various tables based on zoom. This works very well, and the final table (z >12) is a very large table full of CLUs. This table has an id field (primary key) as well as the geometries. There's another table with an id (foreign key) that has various fields that I would like access to for 'on the fly' styling purposes. There are quite a few rows in the foreign table (19mil) but everything is properly indexed. Still though, when I attempt to make the join everything struggles because each bbox query runs about ~10 seconds.

An example of the join:

SELECT id, crop, year, 
ST_AsMVTGeom(ST_Transform(geometry, 3857), tilebbox(z, x, y, 3857), 4096, 64, true) AS geometry
FROM field
INNER JOIN field_prediction
ON field.id = field_prediction.field_id
WHERE geometry && tilebbox(z, x, y, 4326);

I've tried to get creative and move away from the join altogether with something like this:

WITH field_ids AS (SELECT id, ST_AsMVTGeom(ST_Transform(geometry, 3857), tilebbox(13, 1870, 3058, 3857), 4096, 64, true) AS geometry 
FROM field
WHERE geometry && tilebbox(13, 1870, 3058, 4326))
SELECT field_id, crop, year, coverage, yield
FROM field_prediction
WHERE field_id IN (SELECT id FROM field_ids)

Which, of course, is much faster like 500ms but since it's not joining the tables - I no longer have the geometries themselves - which is kind of a dealbreaker.

Any thoughts? I am a SQL novice so I'm hoping there's some easy solution that I am missing. I know that I could just "move" the foreign table into the primary table, but this would be less than ideal (it's a one to many relationship; each CLU for different years, crops, etc) - and if possible I would like to avoid rewriting the data structure into a bunch of arrays.

Sincere thanks on your efforts with the library - I was blown away by how easy it was to get started!

Support for openmaptiles

The openmaptiles project also supports postgis-based tile generation - it can generate giant SQL statement (as either prepared statement, query, or function creation) that produces MVT from z,x,y. Would it be possible to add support for the openmaptiles layer format in martin as well? See the postgis server (simple test server for testing purposes)

RUSTSEC-2020-0048: Use-after-free in BodyStream due to lack of pinning

Use-after-free in BodyStream due to lack of pinning

Details
Package	actix-http
Version	1.0.1
URL	actix/actix-web#1321
Date	2020-01-24
Patched versions	>= 2.0.0-alpha.1

Affected versions of this crate did not require the buffer wrapped in BodyStream to be pinned,
but treated it as if it had a fixed location in memory. This may result in a use-after-free.

The flaw was corrected by making the trait MessageBody require Unpin
and making poll_next() function accept Pin<&mut Self> instead of &mut self.

See advisory page for additional details.

Docker image versioning for improved stability

@stepankuzmin would you please consider pushing versioned container using git release tag such as “martin:0.5.0” to allow users pin down version and to avoid forcing upgrades silently (and potentially break app)? Also, can you please release a new tag from mainline branch since it has newer changes than release 0.5.0?

Thanks in advance for sharing such a powerful app!