Proof of concept: Caching strategy for a production deployment

[iperdomo]

Description

Now that we have a working proof-of-concept (See: #1), we want to move ahead and start describing how a production deployment of this service will look like. In particular how tile caching should be performed.

We found this document (perhaps obsolete) on how works in CartoDB:
https://github.com/CartoDB/cartodb/wiki/Layergroup-Workflow#workflow-diagram

Deliverables

• Document describing the caching strategy
• Proof of concept on how the interaction of various clients requesting the same data will look lile

[iperdomo]

Just a clarification, I'm not advocating to use a CDN (we'll not have that amount of traffic), but just a cache strategy for avoiding unnecessary work in the service.

[strk]

The idea is to advertise fetched tiles as never-expiring, and encode aspect of the tiles in their access URL (the "map token" element of the accessing URL path).

The current tiler properly encodes the map configuration in the "map token" but does not encode the time of last data update, preventing data chances (in database) to effectively change the tile access token (and thus URL).

In order for a server to embed the "data update timestamp" in the URL, some support is required on the database side, as by default this information is not available in a PostgreSQL database. In CartoDB, this "data update timestamp" is taken care of by database triggers attached to every data table, and implemented in the "cartodb-postgresql" extension. In your case you might have this information already stored somewhere else (you mentioned you don't support client-based data update?).

Is it ok, for the POC, to assume data never changes in the database ?

[strk]

One note: I've just seen that the current POC server does not advertise tiles as never expiring, so in its current incarnation not even the browser would be caching those tiles.

[iperdomo]

Is it ok, for the POC, to assume data never changes in the database ?

I would like to get an example where the map token also encodes a last updated timestamp. Perhaps we can just hack the idea in this POC, e.g. add an extra column timestampz DEFAULT NOW() that is used in the encoding of the URL?

[strk]

Ok for the hack. But the information needs to be associated to a table, not to each single record in it. So it could be a "akvo_metadata" table with a `table regclass, last_updated timestamp` composition. Will look at it.

[strk]

Enough javascript for today. @iperdomo do you have an built image of the tiler in which you could run this for me

docker exec -ti akvo-tiler sh -c "cd /tiler; npm ls" > npm-ls-tiler.txt

And then send me the npm-ls-tiler.txt file ?

I suspect an update in some dependency broke my development tiler, enough that upon first request all available RAM is filled and machine is rendered unusable :/

[strk]

Oh, and if that works:

docker exec -ti akvo-tiler sh -c "cd /tiler; npm shrinkwrap"

And then send the /tiler/npm-shrinkwrap.json file found in the running container.

It was an error not to lock exact versions in the repository.. :(

[iperdomo]

• npm-ls-tiler.txt -> https://gist.github.com/iperdomo/ec663883eb184dbc2082f5c9c18bb2e8
• npm-shrinkwrap.json -> https://gist.github.com/iperdomo/0a58836b51ea880b25055ae50a72a6b1

[strk]

Thanks ! The infinite loop is gone after using the shrinkwrapped dependencies. I've pushed the shrinkwrap file so it doesn't happen again.

Now as of bfc0a8b the tiles are served as never-expiring and both the map configuration and the lastUpdated timestamp for the source data are encoded in the access URL.

[strk]

At the same time, as of that commit we have a new "akvo-varnish" container which can be used as an intermediate cache for different browsers. As usual running "make start" from the top-level dir gives you a link to follow to use the varnish endpoint from the viewer.

[strk]

You can see how tile URLs changes by updating the data:

update liberia set last_updated = now();

It'll take a new map initialization from the browser to get updated URLs (ie: hitting the "Go" button from the viewer)

[strk]

I guess it would be useful to add a per-request logging to the tiler server, to show request not getting there at all when URL does not change. Theoretically only the initial POST (to create the map, and/or keep it alive) and the very first tile request should be served by the tiler, no matter how many different clients are used.

[strk]

I've pushed a short document describing the caching strategy, and with the presence of Varnish this ticket could be considered closed. Let me know if you need more here.

NOTE: the Varnish configuration in the image isn't really tweaked so I've to say I don't know if it really caches things as it should (I'm not a Varnish user) -- any HTTP cache should work equally well as long as it caches everything coming in with Cache-Control: max-age=31536000

[iperdomo]

Hi, I was testing the repo at rev e5834de and found that the data points shown are different than my previous test (before the update). See: #1 (comment)

And current map response.

[strk]

Interesting, the SQL query sent to the PostGIS backend is bogus, swappign ST_MakeEnvelope arguments:

WHERE "geom" && ST_MakeEnvelope(-20037508.3,20037508.25881302,-20037508.25881302,20037508.3,3857)

Sounds like a bug in mapnik.
See correct syntax:
http://www.postgis.net/docs/ST_MakeEnvelope.html

The negative numbers (min) should come first, then the second.
Will check how we build the tiler image as it looks like the mapnik version there is not locked either.
Maybe this should be in another ticket though.

[strk]

I take it back, the bogus call is not coming from mapnik, could be a projection issue

[strk]

Confirmed, is a projection problem.
The geometries in the databse are in lat/long but the map expects them in webmercator.
It looks like windshaft is not properly handling the transformation, and also that the data in the repository is not easily transformable as some latitude/longitude values are wild. I don't remember how we fixed this in the past, maybe we were using a windshaft version working better. Do you still have the other docker image around ? For now I'll massage the data to be in webmercator.

[iperdomo]

Nope, i don't have a previous version of the Docker images, perhaps just checkout rev ea209d8 (the last commit from Nov) and test there?

[strk]

Ok I did some more research and found out that the srid value in config/environments/development.js was actually needed. See af9a13a (the commit which removed it).

Adding that back fixes this case, but the real fix would be ensuring windshaft honour the srid option in the MapConfig, which seems not to be working with the currently locked version of Windshaft/Grainstore.

I'm still researching on this issue.

NOTE that we don't want to put the srid in the configuration because it could be different for different tables/columns and because it would not affect the "map token" thus giving cache related headaches

[strk]

Seems to be a bug in windshaft itself: CartoDB/Windshaft#528

[iperdomo]

After updating, rebuilding and clearing the browser cache, I can see the same map as before. Revisiting the page, sending the same request, with Firefox I get HTTP 304 and 200 (from cache) -

It seems that Chromium behaves a bit different and does not send the HTTP request for the tiles and get them directly from disk.

Not sure why Chromium is reporting an error Unexpected token on some json responses.

I checked the Content-Type of those responses and is application/json; charset=utf-8 which should be OK.

$ curl -s "http://172.17.0.6/akvo/layergroup/e820e3d3089a42f628548683fbc35871:1485980949701/0/1/1/1.grid.json?callback=lu0.lu_1_1_1" | json_verify -q && echo "OK"
$ OK

Not sure why the error.

Thanks!

[strk]

As I said, the hack prevents the token from being updated, thus is still possibly suffering from cached responses. Try slightly changing the map definition to see if that helps (the projection was the issue). I'll see how to lock in the image the Windshaft/Grainstore pair that allow to specify the SRID inside the MapConfig, fetching them from my fork.

[strk]

So with #3 we reference the Windshaft version that I fixed to support per-layer projection. It takes a rebuild of the akvo-tiler image (make from top level dir, and then make restart).

I can reproduce the unexpected token with chromium 55.0.2883.87 (Developer Build) but not with Firefox. Will try to understand the warning.

[iperdomo]

My guess of the Unexpected token is a bug in Dev Tools, trying to pretty print some code. I tried enable Pause on exceptions and Pause on caught exceptions and the debugger didn't stop.

[strk]

Actually I just realized the error also shows in Firefox:

SyntaxError: missing ; before statement
	

{"grid":["

	
1.grid....u_1_1_1 (line 1, col 7)

[strk]

I think the response is actually malformed in that it should contain a function call (to the given callback parameter)

[strk]

This may be related:

express deprecated res.send(body, status): Use res.status(status).send(body) instead http/controllers/map.js:173:21

Warning raised by express (http server module of the tiler). It looks like jsonp support is broken..
BTW, we're not using those UTF8 grids, should I just disable them for now ?

[strk]

The deprecation warning was unrelated (fixed with 8a9e113)

[strk]

Figured and fixed with d14bb02

[iperdomo]

Thanks, updated and tested, the JS error is gone. We have an additional issue.

{
  "version": "1.5.0",
  "layers": [
    {
      "type": "mapnik",
      "options": {
        "sql": "select instance, geom, yearcons::integer as yearcons from liberia where yearcons ~ '^\\d{4}$'",
        "geom_column": "geom",
        "srid": 4326,
        "cartocss": "#s { marker-width: 5; marker-fill:#f45; marker-line-color:#813;
  marker-allow-overlap:true; marker-fill-opacity: 0.3;} #s[yearcons>=2009] {marker-fill: #1F78B4; marker-line-color: #0000FF;}",
        "cartocss_version": "2.0.0",
        "interactivity": "instance"
      }
    }
  ]
}

Steps to reproduce:

• Hit the Go button with the above MapConfig
• The map shows up with red and blue markers (No JS error in the Dev Tools console)
• Zoom in in Liberia area
• Hit again the Go button. This makes the same request and now some JS errors show up. Note: the map shows up correctly, but zooming again in Liberia generates more JS errors

[strk]

I can reproduce this. In firefox the error message is "Node not found". These are coming from leaflet. I'll see if updating leaflet to latest helps. As before, these are coming from UTF8 grids, which in this POC are not really used anyway (maybe should be dropped directly)

[strk]

It turns out that the utfgrid layer is already at the latest version (origin is here: https://github.com/danzel/Leaflet.utfgrid). The reported error seems to be a known and still unresolved one: danzel/Leaflet.utfgrid#47

[iperdomo]

Also tried this, closing the issue