cartodb/windshaft

A Node.js map tile library for PostGIS and torque.js, with CartoCSS styling

Windshaft

A Node.js map tile library for PostGIS and torque.js, with CartoCSS styling.

• Can render arbitrary SQL queries
• Generates image and UTFGrid interactivity tiles
• Accepts, stores, serves, and applies map styles written in CartoCSS
• Supports re-projections

Windshaft is a library used by CARTO, a location intelligence and data visualization tool.

Examples

Build

Requirements:

• Node 10.x (npm 6.x)
• PostgreSQL >= 10.0
• PostGIS >= 2.4
• CARTO Postgres Extension >= 0.24.1
• Redis >= 4
• libcairo2-dev, libpango1.0-dev, libjpeg8-dev and libgif-dev for server side canvas support
• C++11 to build internal dependencies. When there's no pre-built binaries for your OS/architecture distribution.

Install

To fetch and build all node-based dependencies, run:

$ npm install

Installing Mapnik

Note: only needed while developing Mapnik and/or node-mapnik.

node-mapnik comes compiled for some platforms and architectures, in case you need it you can always compile, package and install it manually. The recommended option is via binaries, see mason and install Mapnik like:

$ mason install mapnik <version>

Where <version> is the latest released version of Mapnik. You can also use other alternatives:

• Source
• OSX
• Linux

Usage

Probably one of the more advanced uses of Windshaft library can be found at Windshaft-cartodb project.

Test

$ npm test

Coverage

$ npm run cover

Open ./coverage/lcov-report/index.html.

Docker support

We provide docker images just for testing and continuous integration purposes:

• nodejs-xenial-pg1121
• nodejs-xenial-pg101

You can find instructions to install Docker, download, and update images here.

Useful npm scripts

Run test in a docker image with a specific Node.js version:

$ DOCKER_IMAGE=<docker-image-tag> NODE_VERSION=<nodejs-version> npm run test:docker

Where:

• <docker-image-tag>: the tag of required docker image, e.g. carto/nodejs-xenial-pg1121:latest
• <nodejs-version>: the Node.js version, e.g. 10.15.1

In case you need to debug:

$ DOCKER_IMAGE=<docker-image-tag> npm run docker:bash

Troubleshooting

Fonts: Invalid value for text-face-name

You need to install fonts at system level to be able to use them. If you face an issue like Invalid value for text-face-name, the type font is expected. DejaVu Sans Book (of type string) was given. probably you don't have the required fonts, try to install DejaVu fonts or any other font needed.

Contributing

• The issue tracker: Github.
• We love Pull Requests from everyone, see contributing to Open Source on GitHub.
• You'll need to sign a Contributor License Agreement (CLA) before submitting a Pull Request. Learn more here.

Versioning

We follow SemVer for versioning. For available versions, see the tags on this repository.

License

This project is licensed under the BSD 3-clause "New" or "Revised" License. See the LICENSE file for details.

windshaft's Issues

DELETE /style leaves stale entries in redis

By simply adding a duplicated test just with a sligthly perturbated SQL query (not changing the results) ends up leaving stale entries in redis, thus producing failures in the final check.

There might be a missing synchronization in the backend code, returning before the deletion takes place.

Here's a patch adding the new test showing the problem:

diff --git a/test/acceptance/server.js b/test/acceptance/server.js
index 4149310..f154780 100644
--- a/test/acceptance/server.js
+++ b/test/acceptance/server.js
@@ -865,6 +865,19 @@ suite('server', function() {
         });
     });

+    test("get'ing a json with default style and sql should return a constrained grid",  function(done){
+        var sql = querystring.stringify({sql: "SELECT * FROM test_table limit 2 OFFSET 0"})
+        assert.response(server, {
+            url: '/database/windshaft_test/table/test_table/13/4011/3088.grid.json?' + sql,
+            method: 'GET'
+        },{
+            status: 200,
+            headers: { 'Content-Type': 'text/javascript; charset=utf-8; charset=utf-8' }
+        }, function(res){
+            assert.utfgridEqualsFile(res.body, './test/fixtures/test_table_13_4011_3088_limit_2.grid.json', 2, done);
+        });
+    });
+
     ////////////////////////////////////////////////////////////////////
     //
     // DELETE STYLE

make check failure : rendering style

One of the tests in Windshaft b0.4 branch fails by rendering a red boundary for black dots when that boundary isn't expected.
The test in question is: server dynamically set styles in same session and then back to default

I'm inspecting this

dynamic styles attached to the tile GET request

upgrade cluster implementation to use cluster2

will allow upgrade to node 0.6

500 when requested table does not exist

example query:

http://development.localhost.lan:8181/tiles/undefined/4/9/6.png?sql=&style=

it should return 404

EDIT: updated title and test to add 404

Support for node.js > 0.4

Any plans to make Windshaft available for newer version of Node.js (currently 0.6.12 (stable))?
I have this error : "express according to this error : "Not compatible with your version of node/npm: [email protected]" with Node 0.6.12.
It works fine if I rollback to Node 0.4.12.

Unavailability of redis kills the server

When redis server connection can't be performed the server dies. May make more sense to send internal server errors instead, and retry ?

acceptance/server.js test failing

uncaught undefined: Error: Images not equal(18.0864): /tmp/6223843058105558 /home/src/cartodb/Windshaft/test/fixtures/test_table_13_4011_3088.png
at /home/src/cartodb/Windshaft/test/support/assert.js:29:27
at ChildProcess.exithandler (child_process.js:100:7)
at ChildProcess.emit (events.js:67:17)
at ChildProcess.onexit (child_process.js:192:12)

See the image difference here:
http://strk.keybit.net/tmp/acceptance_server_imgdiff.png

Looks like bigger point size is expected.

Posting a style doesn't update the style of maps served to unauthenticated requests

I belive unauthenticated requests will always be served tiles with the default style, no matter what you set when you POST a style. This is because when you POST you must be authenticated and being authenticated changes a different redis key than the one used by non authenticated requests.

This issue, if confirmed, was introduced with the recent security fix preventing unautenticated access to private tiles.

It's a tricky bug because one version of Windshaft has been loosen on grainstore dependency, and it's really up to grainstore to determine which keys are used.

This is an important bug, most likely affecting both the "b0.4" and the "master" branches.

It's critical to add a testcase for this in the server acceptance suite.

Runnign tests: fe_sendauth: no password supplied

uncaught undefined: TypeError: Cannot read property '_uri' of undefined
at /home/src/cartodb/Windshaft/test/unit/render_cache.test.js:39:28
at Function.handleError (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:79:21)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:54:7)
at Function.returnCallback (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:140:17)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at MapnikSource. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:39:13)
at MapnikSource.g (events.js:143:14)
at MapnikSource.emit (events.js:64:17)
at Array. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:93:20)

uncaught undefined: AssertionError: 1 deepEqual 0
at /home/src/cartodb/Windshaft/test/unit/render_cache.test.js:140:16
at Function.handleError (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:79:21)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:54:7)
at Function.returnCallback (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:140:17)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at MapnikSource. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:39:13)
at MapnikSource.g (events.js:143:14)
at MapnikSource.emit (events.js:64:17)
at Array. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:93:20)

uncaught undefined: AssertionError: 2 deepEqual 0
at /home/src/cartodb/Windshaft/test/unit/render_cache.test.js:51:20
at Function.handleError (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:79:21)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:54:7)
at Function.returnCallback (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:140:17)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at MapnikSource. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:39:13)
at MapnikSource.g (events.js:143:14)
at MapnikSource.emit (events.js:64:17)
at Array. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:93:20)

uncaught undefined: AssertionError: 1 deepEqual 0
at /home/src/cartodb/Windshaft/test/unit/render_cache.test.js:63:20
at Function.handleError (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:79:21)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:54:7)
at Function.returnCallback (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:140:17)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at MapnikSource. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:39:13)
at MapnikSource.g (events.js:143:14)
at MapnikSource.emit (events.js:64:17)
at Array. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:93:20)

uncaught undefined: AssertionError: 2 deepEqual 0
at /home/src/cartodb/Windshaft/test/unit/render_cache.test.js:77:20
at Function.handleError (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:79:21)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:54:7)
at Function.returnCallback (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:140:17)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at MapnikSource. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:39:13)
at MapnikSource.g (events.js:143:14)
at MapnikSource.emit (events.js:81:20)
at Array. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:93:20)

uncaught undefined: AssertionError: 3 deepEqual 0
at /home/src/cartodb/Windshaft/test/unit/render_cache.test.js:124:24
at Function.handleError (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:79:21)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:54:7)
at Function.returnCallback (/home/src/cartodb/Windshaft/lib/windshaft/render_cache.js:140:17)
at next (/home/src/cartodb/Windshaft/node_modules/step/lib/step.js:51:23)
at MapnikSource. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:39:13)
at MapnikSource.g (events.js:143:14)
at MapnikSource.emit (events.js:64:17)
at Array. (/home/src/cartodb/Windshaft/node_modules/tilelive-mapnik-cartodb/lib/mapnik.js:93:20)

Failures: 6

Build failed. Also, fist issue!

I'm seeing a strange build failure with @a232ebc that can be reproduced like this:

git clone https://github.com/Vizzuality/Windshaft.git
npm install Windshaft

Here's the full build output with the error:
https://gist.github.com/1225681

Here's my environment:

OS: Linux Ubuntu 2.6.38-11-generic (Natty 11.04)
Node: v0.4.1
npm: 1.0.30
Mapnik: 2.0.0
PostgreSQL: 8.4.8
PostGIS: 1.5.1
Redis: 2.2.13 

It looks like something related to the [email protected] install, but I'm not sure how to debug it.

README install instructions review

The README file says to install with npm install windshaft but that command does nothing. Instead npm install seems to do a lot of things ...

Incorrect grid.json features

We have a Windshaft installation running on our server (not in production yet).
All our .png tiles look fine but some of our UTFGrid seams incorrect.

We are trying to find the cause of this problem.
I'm logging all queries sent to Postgres to find a hint.

I'm asking for 3/2/3.png and the query sent to the server is : SELECT AsBinary("the_geom_webmercator") AS geom from (SELECT * FROM occurrence) as cdbq WHERE "the_geom_webmercator" && SetSRID('BOX3D(-12523442.71424328 -2504688.542848655,2504688.542848655 12523442.71424328)'::box3d,3857).
The same query is used for URL /database/dataportal/table/occurrence/3/2/3.png?cache_buster=0&sql=SELECT+*+FROM+Occurrence and /database/dataportal/table/occurrence/3/2/3.grid.json?cache_buster=0&sql=SELECT+*+FROM+Occurrence

If I translate (-12523442.71424328 -2504688.542848655,2504688.542848655 12523442.71424328) to WSG84 it
gives -21.94, -112.5 and 74.02, 22.5.
I should get something around 40, -90 and 0, -50.

I would like to know what I'm doing wrong and what I'm not understanding correctly since all our png are fine?

Note : This is probably not a bug related to Windshaft or a bug at all but just a behavior that I would like to understand to find another issue.

acceptance/server.js test hanging

I'm trying to understand why the test/acceptance/server.js test is hanging rather than cleanly exiting.
So far I reduced the test to this (call it test/hang.js):

 var   assert        = require('./support/assert')
     , tests         = module.exports = {}
     //, _             = require('underscore')
     //, querystring   = require('querystring')
     , fs            = require('fs')
     , th            = require('./support/test_helper')
     , Windshaft     = require('../lib/windshaft')
     , ServerOptions = require('./support/server_options')
     , http          = require('http');


 var server = new Windshaft.Server(ServerOptions);

 /* tests["sane"] = function(beforeExit, assert){ assert.ok(1); }; */


 tests["hangme"] = function(beforeExit, assert){
     assert.response(server, {
         url: '/database/windshaft_test/table/test_table/13/4011/3088.png',
         method: 'GET',
         encoding: 'binary'
    /* , timeout: 100 // in milliseconds */
     },{
         status: 200,
         headers: { 'Content-Type': 'image/png' }
     }, function(res){
    console.log("Got response, now how to signal completion ?");
     });
 };

It must have to do with the assert.response test...

re-enable metatiling speed optimisation

metatiling is a 20-30% speedup that is disabled in windshaft via the tilelive-mapnik branch dependency (https://github.com/tokumine/tilelive-mapnik/commit/e5b6d0514a6f23f9ad7fa636fc1219965b34f04f).

It is mainly disabled due to my stupid brain not being smart enough to work out how to both enable it and also ensure that we can alter the tiler styles interactively.

If we re-enable the metatiling, we should also ensure we remove the silly timestamp based naming of our render pools here to get the benefit: https://github.com/Vizzuality/Windshaft/blob/master/lib/windshaft/render_cache.js#L104. Will also need to update our branch to the latest tilelive-mapnik (or just go back to using plan old tilelive-mapnik as a dependency)

For more on the metatiling, and possible speedups, see: http://mapbox.com/blog/tilemill-now-twice-fast/

Styles ignored

I updated my Windshaft to the latest version.
My previous styling is now ignored.

grainstore: {datasource: {user:'*******',password:'********', host: '127.0.0.1', port: 5432},
            styles: {point: "#occurrence {marker-fill: #a82400;marker-opacity: 1;marker-width: 2;marker-line-color: white;marker-lne-width: 0.3;marker-line-opacity: 0.5;marker-placement: point;marker-type: ellipse;marker-allow-overlap: true;}"}
        },

Even if I try through the URL:
http://.../7/30/38.png?style=#occurrence{marker-fill:blue}

Something changed or maybe I'm missing something?
thanks

Mapnik 2.1
Postgis 1.5
Postgres 9.1

Support geometry-type based rendering

No geometry should go undenderer, unless the user really specified that's what the intended behavior.
I guess for most cases the user just wants to see what a query returns, whatever it is.

Sometimes it's even not possible to know in advance, for example an ST_Intersection() between polygons may return some polygons, some lines, some points. Generally speaking doing a geometry-type based rendering would also let you support GEOMETRYCOLLECTION types.

It's been reported by a few that "maps is not shown" and the root cause was a wrong-targetted style (style for polygon and point input, for example). Always providing a style for each possible geometry type would avoid these cases.

Do not include stack trace in error responses

Error responses (500) currently include full stack, don't do that, unless (maybe) when running with "development" environment

Renderer cache is shared between authenticated and non-authenticated requests

Renderer cache is shared between authenticated and non-authenticated requests so it can happen that until a cache item is removed you can have access to private data from unauthenticated request or vice-versa NOT have access to private data from authenticated requests.

This is with the relatively short-lived caches (currently 6 minutes)

Puma concolor gone wild

So I have a shapefile of an obscure species that you've probably never heard of before (puma concolor) in PostGIS that looks like this when viewed with QGIS:

When I tile it via WindShaft, it looks like this in the example viewer (at all zoom levels):

My table has a geometry column named the_geom_webmercator that should be projected to webmercator (EPSG:3857). What am I missing? I'm not seeing any errors in the console.

As a sanity check, here's a link to the original shapefile:
http://dl.dropbox.com/u/13724811/puma_concolor.tar.gz

I also pg_dumped the database in case you want a peek:
http://dl.dropbox.com/u/13724811/db.tar.gz

Thanks!

acceptance/server.js test is using invalid syntax

'background-color' is not a valida parameter for non-map context, but as such is used in 3 tests within acceptance/server.js

There are two issues with this:

1. The exact error doesn't come out but all you see is a 500 return status
2. It must be fixed or will not work with "carto" > 0.4.10

The same issue was fixed recently for grainstore: CartoDB/grainstore#13

grid request return wrong error when the table is empty

steps to reproduce:

1.- create a empty table
2.- GET http://development.localhost.lan:8181/tiles/untitled_table_7/4/6/6.grid.json?interactivity=cartodb_id&cache_buster=1351525379925&map_key=085c3759cab4c9d663dedfbf37374d7f6fd8327a

the following error is in the response

{
error: "Cannot call method 'replace' of undefined"
}

it should return an empty grid or something useful description

Send PostgreSQL errors in the response body

When PostgreSQL errors occur they are not transported to the client, but just happen as logs on the server side. This is from "not authorized" accesses to database tables to SQL syntax errors.

The client only receives a 500 status code and an empty JSON.
Initially reported in CartoDB/Windshaft-cartodb#32

We'll want a testcase for this.

Add support for mapnik 2.1.x

Windshaft is currently pinned to mapnik 2.0, see how to make it work for both 2.0 and 2.1

Using SQL to filter tile results

So I added the following line to my Windshaft config.req2params object:

req.params.sql = "SELECT * FROM ranges where binomial = 'Puma yagouaroundi'";

When I tried to access a tile like http://localhost:4000/database/mol/table/ranges/3/2/4.png, I got the following error:

{
stack: "Error: :
ERROR:  syntax error at or near "SELECT"
LINE 1: SELECT * FROM SELECT * FROM ranges where binomial = 'Puma ya...
                      ^
Full sql was: 'SELECT * FROM SELECT * FROM ranges where binomial = 'Puma yagouaroundi' LIMIT 0'
 (encountered during parsing of layer 'ranges' in map '/home/eighty/Dropbox/projects/')"
message: ":
ERROR:  syntax error at or near "SELECT"
LINE 1: SELECT * FROM SELECT * FROM ranges where binomial = 'Puma ya...
                      ^
Full sql was: 'SELECT * FROM SELECT * FROM ranges where binomial = 'Puma yagouaroundi' LIMIT 0'
 (encountered during parsing of layer 'ranges' in map '/home/eighty/Dropbox/projects/')"
}

It looks like Windshaft is just expecting a nested query here? Easy workaround, I had to change the config.req2params object to this:

req.params.sql = "(SELECT * FROM ranges where binomial = 'Puma yagouaroundi') as q";

Then we get our tile!

Also, does it makes sense to support an SQL parameter for tiles? Something like:

/3/2/4.png?sql=(SELECT * FROM ranges where binomial = 'Puma yagouaroundi') as q,

supports multiple layer

What about supporting for multiple layers at rendering? Any advise to improve it.

Add configuration to set base_dir

A map's base_dir is what's used to resolve relative paths used to reference external resources.
When reading the map from standard input the base_dir is also taken as the map's filename, and as such printed togheter with error messages by Mapnik.

Currently it is set to the dirname of the current working directory at server startup time.

It may have implications for CartoDB/grainstore#4

Make running tests easier

Ideally there should be a single command to invoke for running all tests.
The command return code should tell you if the tests were successful or not (0 for success, anything else for failure).

Send CORS headers for all requests

I found out that CORS headers are only sent in response to tile request, not to style settings or metadata etc.

http://en.wikipedia.org/wiki/Cross-origin_resource_sharing

multilinestring not rendering

Given this the_geom:

{"type":"MultiLineString","coordinates":[[[41.697018,49.175652],[41.697033,49.175625],[41.697037,49.175598],[41.697041,49.175571],[41.697037,49.175537],[41.697056,49.175552],[41.697075,49.175571],[41.697094,49.175594],[41.697113,49.175613],[41.697132,49.17564],[41.697155,49.175667],[41.697174,49.175697],[41.697197,49.175724],[41.69722,49.175755],[41.697235,49.175785],[41.69725,49.175797],[41.697273,49.175827],[41.697292,49.175854],[41.697315,49.17588],[41.697338,49.175911],[41.697361,49.175941],[41.697384,49.175968],[41.697411,49.175999],[41.697433,49.176029],[41.697456,49.17606],[41.697479,49.17609],[41.697502,49.176121],[41.697533,49.176151],[41.697559,49.176186],[41.69754,49.176464],[41.697567,49.176498],[41.697598,49.176529],[41.697609,49.176548],[41.697632,49.176582],[41.697659,49.176613],[41.697674,49.176632],[41.697704,49.176666],[41.69772,49.176685],[41.697742,49.176716],[41.697765,49.176743],[41.697788,49.176773],[41.697811,49.1768],[41.697838,49.176823],[41.697865,49.176834],[41.697884,49.176834],[41.697906,49.176815],[41.697929,49.176781],[41.697945,49.176743],[41.69796,49.176708],[41.697971,49.176689],[41.697987,49.17667],[41.697998,49.176636],[41.698017,49.176598],[41.698029,49.176579],[41.698048,49.176544],[41.698059,49.176525],[41.698303,49.176105],[41.698322,49.176075],[41.698345,49.176041],[41.698357,49.17601],[41.698364,49.175976],[41.69836,49.175949]]]}

when you try to render the table the line is not shown.

If i use the following SQL query it renders:

SELECT ST_Buffer(the_geom_webmercator, 10) as the_geom_webmercator FROM table

Support for node 0.8

Node-0.8 support is still not ready.

acceptance/server.js: global leaks detected: result, env, mem, attributes, fail, name

When not ignoring leaks while running tests, the acceptance/server.js file triggers "global leaks" detection.
I understood this meaning that some variables are left defined in the global namespace, preventing the GC from cleaning them up.

mocha-version of test/acceptance/server.js not really testing requests

After committing the port to mocha (see #20) I realized that the server tests aren't really run.
No wonder I couldn't reproduce my failure reported in #21.

I'm using an implementation of assert.response taken from expresso 0.8.1 and copied into test/support/assert.js

Review renderer cache time to leave

The renderer cache serves the purpose of serving multiple requests for the tiles in the same "metatile" as quick as possible.
But when such caching is done at an higher level we don't need to consume memory for that.

Currently the lifetime of a TileLive object (which is in our cache) is 6 minutes, which is pretty high considering the time it could take for serving a full viewport worth of tiles.

Additionally, the TileLive object itself is currently dropping each single tile (from the metatile mosaic) as soon as served so keeping it alive is not necessarely that helpful either.

Make map style part of the renderer caches

I don't see the map style affecting renderer caches, so continuosly changing a style trigger creation of new cache entries.
Beside killing the caches all at once (which may be a good approach) this ticket is to get the style into the cache keys.

Add an header to advertise cache hits

It'd be useful to get renderer cache info into the response header, for profiling purposes.

api request parameter rename

as we agreed in IRC:

version -> style_version
convert -> style_convert

for /style and /tiles endpoints

100% cpu until the process is killed

i'm using current master (29c17ae4d333840984bddc7ed63bacaf0b89d4ce) and when I start to request tiles (whiout styles, simple sql query) the process start to eat all my CPU and only stops when the process is killed.

postgres log says:

LOG: unexpected EOF on client connection

$ node -v
v0.6.19

the dataset used is gadm0, and the queries are like this:

http://development.localhost.lan:8181/tiles/test/4/2/5.png?sql=SELECT%20*%20FROM%20test&style=

make check fail preparing tests

[~/wrk/viz/cartodb/Windshaft(master)]$ make check
npm test

[email protected] test /Users/javi/wrk/viz/cartodb/Windshaft
sh ./run_tests.sh

Starting redis on port 6333
Preparing the database
sed: 1: "1h;1!H;${g;s/.*postgres ...": extra characters at the end of p command
PGUSER: []
PGHOST: []
PGPORT: []

Clearing the renderers cache leaves something slowing down requests

I've been debugging this for two days now so I tought I'd file a ticket before getting mad.

The "master" branch of Windshaft shows a slowdown everytime caches are cleared due to "cache_buster".
I suspect (but didn't verify) that the same slowdown happens when caches are cleared by their natural expiration (one minute by default). Only the use of "cache_buster" makes this more obvious, especially when you send 1000 requests and clear the caches at each and every one.

The visible effect of expiring the caches at every request over 1000 requests, is that each chunk of 1000 requests are served on an average of twice the time it took serving the precedent chunk.

Note that as of 09426f8 it is not enough to include a "cache_buster=1" parameter in order to obtain cache clearing but you actually have to pass a different value on every request. The benchmark.js file needs manual editing to change the cache clearing strategy. By default it clears the cache every 16 requests as of ed6b79b

Add configuration to set target Mapnik version

See CartoDB/grainstore#19

If I recall correctly we're targetting 2.0 in our documentation and testcases, so targetting 2.0.2 would be a good start.
But we could also make it a configuration item.

Note that failing to set the target while using grainstore 0.4+ would automatically target latest version of mapnik

add CORS headers on grid requests

i.e:

http://development.localhost.lan:8181/tiles/tm2/4/6/6.grid.json?sql=SELECT%20*%20FROM%20tm2&style=%23tm2%7B%20polygon-fill%3A%20red%3B%20line-color%3A%20%23c00%20asdasd%7D&interactivity=cartodb_id%2Cname%2Ciso2&cache_buster=1350643785650

OPTIONS and GET methods should return CORS headers.

This is needed because we need to check returning errors in client side

Update to use tilelive-mapnik

Is there any reason why grainstore is requiring "tilelive-mapnik-cartodb" rather than "tilelive-mapnik" ?

Inspecting the npm registry (npm view tilelive-mapnik; npm view tilelive-mapnik-cartodb) it looks like the "-cartodb" suffixed is a maintainance of the 0.2 branch (the original stops at 0.2.1, the -cartodb goes up to 0.2.4).

TypeError: Cannot set property 'maxListeners' of undefined

Just a heads up that on Ubuntu 11.04 with Node 0.4.1, WindShaft throws the following error while serving tiles:

TypeError: Cannot set property 'maxListeners' of undefined

I traced it down to Windshaft/node_modules/tilelive-mapnik/lib/mapnik.js:36 which dispatches to events.js:12. I wasn't able to figure out exactly why this was happening, but the workaround is easy: Just upgrade to Node 0.4.12 and you're golden.

FWIW, if you need to manage multiple Node versions, check out Nave:

https://github.com/isaacs/nave

occasionally opacity settings are not honoured

http://support.cartodb.com/discussions/problems/102-data-dissapears-from-map-after-modifying-opacity

order-dependent test results

It's been reported in ticket #21 that running unit/render_cache.js before acceptance/server.js triggers failures in the latter.
Sounds like something worth researching on (unclean exit from test ?)

Implement SPDY

ERROR: column "id" does not exist

I'm seeing an "undocumented feature" in @74e4203 where the SELECT AsBinary query is expecting an id column in my table:

[TILE RENDER ERROR]
Error: :                                                                                                                                                                                                                
ERROR:  column "id" does not exist                                                                                                                                                                                      
LINE 1: SELECT AsBinary("the_geom_webmercator") AS geom,"id" from ra...                                                                                                                                                 
                                                        ^                                                                                                                                                               
Full sql was: 'SELECT AsBinary("the_geom_webmercator") AS geom,"id" from range WHERE "the_geom_webmercator" && SetSRID('BOX3D(-20005048.4188 -9039211.13765,19907487.2779 17096598.5401)'::box3d, 3857)' 

Adding an id column makes the error go away of course, but since I didn't see anything in the readme about it, I thought I'd ask. :)

Document supported options of Windshaft.Server

The class takes an object as its only argument, containing all parameters.
Each and every option should be documented, best if within the JS itself so that it is easier to maintain.

Why expecting text/javascript Content-Type ??

I noticed our acceptance/server.js file is expecting "Content-Type: text/javascript; charset=utf-8; charset=utf-8" but why the duplicated charset ?

Also, I'm getting 'application/json; charset=utf-8' with an updated tilelive, anything wrong with that ?