mapbox/vt-pbf

Serialize Mapbox Vector Tiles to binary protobufs in javascript.

vt-pbf

Serialize Mapbox vector tiles to binary protobufs in javascript.

Usage

As far as I know, the two places you might get a JS representation of a vector tile are geojson-vt and vector-tile-js. These both use slightly different internal representations, so serializing each looks slightly different:

From vector-tile-js

var vtpbf = require('vt-pbf')
var VectorTile = require('@mapbox/vector-tile').VectorTile
var Protobuf = require('pbf')

var data = fs.readFileSync(__dirname + '/fixtures/rectangle-1.0.0.pbf')
var tile = new VectorTile(new Protobuf(data))
var orig = tile.layers['geojsonLayer'].feature(0).toGeoJSON(0, 0, 1)

var buff = vtpbf(tile)
fs.writeFileSync('my-tile.pbf', buff)

From geojson-vt

var vtpbf = require('vt-pbf')
var geojsonVt = require('geojson-vt')

var orig = JSON.parse(fs.readFileSync(__dirname + '/fixtures/rectangle.geojson'))
var tileindex = geojsonVt(orig)
var tile = tileindex.getTile(1, 0, 0)

// pass in an object mapping layername -> tile object
var buff = vtpbf.fromGeojsonVt({ 'geojsonLayer': tile })
fs.writeFileSync('my-tile.pbf', buff)

vtpbf.fromGeojsonVt takes two arguments:

• layerMap is an object where keys are layer names and values are a geojson-vt tile,
• options is an object (optional argument). There are 2 supported keys: version to define the version of the mvt spec used and extent to define the extent of the tile. version defaults to 1 and extent to 4096.

vt-pbf's Issues

Close polygon geometries with closepath instead of lineto

Problem

Polygon geometries are not closed properly when encoding tiles. This results in tiles with invalid polygon geometries that Mapbox Uploads API does not accept.

Solution

Similar to tile-count, change the writeGeometry https://github.com/mapbox/vt-pbf/blob/master/index.js#L123 function to close polygons with closepath instead of lineto.

cc @ericfischer

Expose vector-tile-pb.js

Id love to explicitly use the functions inside vector-tile-pb.js to write tiles. Currently, this file is not exposed, and I'd like to avoid require('../../node_modules/vt-pbf/vector-tile-pb.js').

Any reason why this file is only internal? Thank ya!

regression in 3.1.0

Sorry I don't have a minimal test case here, and I haven't tested with the fixture from https://github.com/mapbox/vt-pbf#from-vector-tile-js but when I do basically the same thing but using mapbox-streets-v7 0//0/0, and then package it into an mbtiles, the resulting mbtiles in broken in mbview and uploading as a Tileset to mapbox.com.

It's broken when using 3.1.0, but everything works fine when I use 3.0.1

prepareLayer is slow when there are lots of properties

prepareLayer's self time is not great, and it's because of https://github.com/anandthakker/vt-pbf/blob/master/index.js#L63-L80. I guess it's time to look into the performance of objects-as-maps

String feature ids not supported

Please allow strings as feature ids as in several countries zip codes are alphanumeric or need leading zeroes.

Use GeoJsonWrapper

Just found this in the innards of mapbox-gl-js: https://github.com/mapbox/mapbox-gl-js/blob/e523db31c1a5d3355a5d97d6bbada2cd64b6711a/js/source/geojson_wrapper.js

Could use this to simplify the dual code paths.

Encoding issue when using 3.01

The following geojson-vt and vt-pbf versions ouput a .pbf with a non-encoded buffer, :
"geojson-vt": "2.4.0",
"vt-pbf": "3.0.1"

Whereas the following versions ouput properly encoded PBFs:
"geojson-vt": "^2.4.0",
"vt-pbf": "^2.1.2"

Incorrect encoding of LineStrings when using fromGeojsonVt

When using geojsonvt to create tilesets I found that some layers were incorrectly rendered.
Examining what happened I found that the geometry that comes from geojsonvt contains all the LineString geometry on a single array: instead of having [[x0, y0], [x1, y1], ..., [xN, yN]] it's all flattened as [x0, y0, x1, y1, ... xN, yN].
When using fromGeojsonVT this is not correctly interpreted in FeatureWrapper::loadGeometry so the geometry contains an array of Points with undefined coordinates.

Should something be implemented in vt-pbf::fromGeojsonVt to correctly interpret this case?
I was thinking about modifying GeoJSONWrapper to also store the feature type so when geometry is loaded from a LineString it checks wether the geometry comes as a single array or an array of arrays.

Null attributes cannot be decoded by mapnik/Mapbox uploads

@jameslcarpino alerted me to a customer whose recent mbtiles tileset uploads, created with https://github.com/ChrisLoer/supertiler, had stopped working. After some investigation, we determined that, although there were no intentional changes on the customer's side, where their old uploads had the string "null" for an attribute value, the new uploads contain actual null values (a value message containing none of the optional typed values), and that this change in the tile contents was the result of #39.

The tiles being produced are in subtle violation of the current vector tile spec, which states that

Exactly one of these values must be present in a valid message

replacing the "may be present" in versions of the spec prior to 2015.

The unexpected null value causes the uploads to to be rejected by this check in mapnik-vector-tile.

I am calling this a bug because it is in violation not only of Mapbox upload validation but also the letter of the vector tile spec.

cc @mourner

Malformed protobuff after geojson-vt parsing.

Context

I'm trying to serve a dynamic vector tiles routes from MongoDB documents.

I have some data in a mongodb, which — after some complex query — I am aggregating into a GeoJSON feature collection that looks like this :

{
  "type": "FeatureCollection",
  "features": [
    {
      "type": "Feature",
      "properties": {
        "_id": "63c9081b6826b11dce76b781",
        "name": "Walsh - Romaguera"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          4.5988,
          46.0235
        ]
      }
    },
    {
      "type": "Feature",
      "properties": {
        "_id": "63c9081b6826b11dce76b782",
        "name": "McKenzie, Bashirian and Wintheiser"
      },
      "geometry": {
        "type": "Point",
        "coordinates": [
          4.5777,
          45.993
        ]
      }
    }
  ]
}

I'm building server side a tile index with geojson-vt like so :

import geojsonVt from 'geojson-vt'
export type GeoJSONVT = ReturnType<typeof geojsonVt>

export const tileIndex: GeoJSONVT = geojsonVt(myFeatureCollection, {})

Later, in an express handler, I'm serving tiles like so :

import { tileIndex } from './tile-index.ts'
const vtpbf = require('vt-pbf')

const tilesHandler = (req, res) => {
  const { z, x, y } = req.params

  // Getting tile as json
  const jsonTile = tileIndex.getTile(z, x, y)

  // Converting tile to pbf
  const pbfTile = jsonTile != null ? vtpbf.fromGeojsonVt({ geojsonLayer: jsonTile }) : null

  res
    .status(200)
    .type('application/x-protobuf')
    .send()
}

I bound this to a route like so

router.get('/:z/:x/:y.pbf', tilesHandler)

And am calling this in mapbox-gl (with react-map-gl) like so :

      <Source
        id="my-source"
        type="vector"
        tiles={['http://localhost:1337/tiles/{z}/{x}/{y}.pbf']}
      />

---

Issue

MapboxGL is complaining in frontend :

Error {message: 'Unimplemented type: 3'}

Which obviously tells pbf is malformed

Here are some logs :

jsonTile

{
  "features": [
    {
      "geometry": [
        [
          791,
          1360
        ]
      ],
      "type": 1,
      "tags": {
        "_id": "63c92f4b4b38c39b45d8c5ba",
        "name": "Ryan and Sons"
      }
    },
    {
      "geometry": [
        [
          200,
          2166
        ]
      ],
      "type": 1,
      "tags": {
        "_id": "63c92f4b4b38c39b45d8c5bb",
        "name": "Zieme Inc"
      }
    }
  ],
  "numPoints": 2,
  "numSimplified": 2,
  "numFeatures": 2,
  "source": [
    {
      "id": null,
      "type": "Point",
      "geometry": [
        0.5128838888888889,
        0.3557929645239258,
        0
      ],
      "tags": {
        "_id": "63c92f4b4b38c39b45d8c5ba",
        "name": "Ryan and Sons"
      },
      "minX": 0.5128838888888889,
      "minY": 0.3557929645239258,
      "maxX": 0.5128838888888889,
      "maxY": 0.3557929645239258
    },
    {
      "id": null,
      "type": "Point",
      "geometry": [
        0.5127430555555555,
        0.355985194665117,
        0
      ],
      "tags": {
        "_id": "63c92f4b4b38c39b45d8c5bb",
        "name": "Zieme Inc"
      },
      "minX": 0.5127430555555555,
      "minY": 0.355985194665117,
      "maxX": 0.5127430555555555,
      "maxY": 0.355985194665117
    }
  ],
  "x": 525,
  "y": 364,
  "z": 10,
  "transformed": true,
  "minX": 0.5127430555555555,
  "minY": 0.3557929645239258,
  "maxX": 0.5128838888888889,
  "maxY": 0.355985194665117
}

pbfTile

Uint8Array(96) [
   26, 94, 120,   1,  10,  12, 103, 101, 111, 106, 115, 111,
  110, 76,  97, 121, 101, 114,  40, 128,  32,  18,  15,  18,
    4,  0,   0,   1,   1,  24,   1,  34,   5,   9, 220,  24,
  192, 42,  26,   3,  95, 105, 100,  26,   4, 110,  97, 109,
  101, 34,  28,  10,  26,  34,  54,  51,  99,  57,  50, 102,
   52, 98,  52,  98,  51,  56,  99,  51,  57,  98,  52,  53,
  100, 56,  99,  53,  98,  97,  34,  34,  15,  10,  13,  82,
  121, 97, 110,  32,  97, 110, 100,  32,  83, 111, 110, 115
]

Flatten nested properties

Currently if you have an object as a property (e.g. from a GeopJSON-file), it will be stored as [object Object] in the protobuf. It should probably be flattened, e.g. using the dot-notation so that {"name": {"en": "Place"}} would become {"name.en": "Place"}, before storing it.

tile spec v2 support: prepareLayer defaults to version 1 which throws warning in mapbox-gl-js

Right now prepareLayer defaults to version 1 if no version is provided. This causes mapbox-gl-js to throw a warning that the tile source does not use vector tile spec v2.

There doesn't seem to be a way to provide version in fromGeojsonVt that can override this default, but it would be awesome to have tile spec v2 support when converting from GeoJSON.

The feature.id can only be set to number that worked

I try to set feature.id by geojson-vt,but this library can only set type of number,but i can set grid_id which is string on mapbox-gl works well below:

source: {
  'grid': {
    type: 'vector',
    promoteId: 'grid_id',
    tiles: ['http://test.pbf']
  }
};

Does it encoding Polygon/MultiPoint correctly?

It seems that:

• No command ClosePath appeared in code
• No command MoveTo with count great than 1

Any python version?

Hi,
Is there any python version of this library ?

Values of complex types are incorrectly cached (and duplicated)

This is caused by the cache key being computed to Object:[Object object] here instead of using the JSONified value.

Fix invalid multipoint geometry encoding

Multipoint geometries are currently encoded as multiple 1-length MoveTo commands. The vector-tile-js library decodes this fine, but it's incorrect according to the spec:

4.3.4.2. Point Geometry Type
The POINT geometry type encodes a point or multipoint geometry. The geometry command sequence for a point geometry MUST consist of a single MoveTo command with a command count greater than 0.

If the MoveTo command for a POINT geometry has a command count of 1, then the geometry MUST be interpreted as a single point; otherwise the geometry MUST be interpreted as a multipoint geometry, wherein each pair of ParameterIntegers encodes a single point.

https://github.com/mapbox/vector-tile-spec/blob/master/2.1/README.md#4342-point-geometry-type

Bug in writeLayer function

index.js -> line 55

 for (i = 0; i < layer.length; i++) {
    context.feature = layer.feature(i)
    pbf.writeMessage(2, writeFeature, context)
  }

Iteration over layers but accessing index of feature array nested in layer object. This doesn't make sense
Or is this part of new standard ?

Vector tiles served from multipolygon type geojson are not rendered

My tile server is consuming NC zipcode polygon geojson (multipolygon type) using geojson-vt, vt-pbf and zlib. Mapbox-GL is not able to render these tiles and even not accepting responses and crashing the page. I concluded multipolygon array is the problem as rendering is successful when geometry coordinates are changed to a single array.

However the multipolygon geojson works properly when processed as geojson at client. ( so I'm on the assumption that problem is not with geojson-vt )

expected case: representation as a hole.

geojson:
dummyculprit.txt

Please let me know if this is working as designed or is this a bug?

Error 'ERR_CONTENT_DECODING_FAILED' when serving the result

Not sure what I'm doing wrong in this scenario.

  var vtpbf = require('vt-pbf')
  var geojsonVt = require('geojson-vt')
  var tileindex = geojsonVt(geojson)
  var tile = tileindex.getTile(Number(z), Number(x), Number(y))
  var vt = vtpbf.fromGeojsonVt({ 'geojsonLayer': tile });

  res.set({
    'Access-Control-Allow-Origin': '*',
    'Access-Control-Allow-Headers': 'Origin, X-Requested-With, Content-Type, Accept',
    'Content-Type': 'application/x-protobuf',
    'Content-Encoding': 'gzip'
  });

  res.send(vt);

Can be seen here: https://geojson-vt-test-mlrzlkgssx.now.sh/
Full code here: https://geojson-vt-test-mlrzlkgssx.now.sh/_src

Fix zigzag encoding overflow

The current bit-shifting arithmetic for zigzag encoding:

vt-pbf/index.js

Lines 125 to 127 in ad933a6

	function zigzag (num) {
	return (num << 1) ^ (num >> 31)
	}

overflows for 2^31 - 1, because JavaScript bit-shifting yields signed 32-bit integers: https://www.ecma-international.org/ecma-262/5.1/#sec-11.7.1

This is unlikely to matter in practice, given that tile extents are typically 4096 or 8192

Possibility to use other projections ?

Hey there,
I've described my case here and got the suggestion to re-add the issue here. To sum this up:

We have written our own vectortiles service based on vt-pbf, which queries a really large database and transports the tiles in gzipped-geojson (which have to be in 4326 as per the spec, I believe) to our clients. Currently the geometries in our database are still in 4326, but I thought it would be useful to transform those to 3857 in order to save the transformation from 4326 to 3857 in the frontend.

This is probably also relevant.

@anandthakker replied:

@gerbsen if you're using vt-pbf to generate vector tiles from GeoJSON data, then yes, the input data are assumed to be in 4326 per the spec.

Is what you're looking for an option to vt-pbf allowing the input to be already projected into 3857? If so, please open an issue in the vt-pbf repo describing your use case / request: https://github.com/mapbox/vt-pbf. This is something we've done in other MVT-generating tools --
see, for example, mapbox/tippecanoe#217

cheers,
daniel

attn: @abenrob

fromGeojsonVt returns Uint8Array instead of Buffer

In #15 pbf version 3 started returning Uint8Arrays instead of Buffers. This should have been a breaking major change, as it breaks usage eg. with node zlib, which expects a Buffer.

Handle point features

Currently fails on point features.

Enable setting feature ID

The VT specification allows features to have an id specified, and geojson-vt passes those trought after mapbox/geojson-vt#70 was merged. Vt-pbf should probably write those ids to the vector tiles.

Support Geojson with Linestring

This library support geojson which include linestring only?

thank you