dechristopher/lod

LOD is an intelligent map tile caching proxy for the edge.

• Founder, Director of Engineering at Fitchburg Fiber
• Interests include: network/internet engineering, OSP infrastructure, and geospatial system design
• Currently learning Elixir (OTP/BEAM), and Rust

Connect with me on LinkedIn

lod's Issues

[v1.0.0] Implement admin handler to reload instance configuration

GET /admin/reload should run a handler that re-reads instance capabilities configuration from the config file provided at runtime. This will allow for on-demand hot reloading of the config file. In the future, we may want an endpoint available to tweak capabilities on the fly without needing to change the config file.

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Rate-Limited

These updates are currently rate-limited. Click on a checkbox below to force their creation now.

• chore(deps): update alpine docker tag to v3.20
• fix(deps): update module github.com/burntsushi/toml to v1.4.0
• chore(deps): update github artifact actions to v4 (major) (actions/download-artifact, actions/upload-artifact)
• chore(deps): update golangci/golangci-lint-action action to v6
• 🔐 Create all rate-limited PRs at once 🔐

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• fix(deps): update module github.com/gofiber/fiber/v2 to v2.52.1 [security]
• fix(deps): update module github.com/prometheus/client_golang to v1.19.1
• fix(deps): update module github.com/prometheus/client_model to v0.6.1
• fix(deps): update module github.com/twpayne/go-geos to v0.18.0
• fix(deps): update module github.com/valyala/fasthttp to v1.55.0
• fix(deps): update module golang.org/x/sync to v0.7.0
• chore(deps): update actions/checkout action to v4
• chore(deps): update actions/setup-go action to v5
• chore(deps): update dependency ubuntu to v22
• fix(deps): update module github.com/go-redis/redis/v8 to v9
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

dockerfile

Dockerfile

• golang 1.20.4-alpine3.16
• alpine 3.18

github-actions

.github/workflows/build.yml

• actions/checkout v3
• actions/setup-go v4
• ubuntu 20.04

.github/workflows/release.yml

• actions/upload-artifact v3
• actions/checkout v3
• actions/download-artifact v3
• actions/setup-go v4
• actions/upload-artifact v3
• actions/upload-release-asset v1
• actions/checkout v3
• actions/download-artifact v3
• actions/setup-go v4
• actions/upload-artifact v3
• actions/upload-release-asset v1
• actions/checkout v3
• actions/download-artifact v3
• ubuntu 20.04
• ubuntu 20.04
• ubuntu 20.04

.github/workflows/test.yml

• actions/checkout v3
• actions/setup-go v4
• golangci/golangci-lint-action v3
• ubuntu 20.04

gomod

go.mod

• go 1.19
• github.com/BurntSushi/toml v1.3.2
• github.com/allegro/bigcache/v3 v3.1.0
• github.com/go-redis/redis/v8 v8.11.5
• github.com/gofiber/fiber/v2 v2.43.0
• github.com/joho/godotenv v1.5.1
• github.com/pkg/errors v0.9.1
• github.com/prometheus/client_golang v1.16.0
• github.com/prometheus/client_model v0.3.0
• github.com/twpayne/go-geos v0.13.2
• github.com/valyala/fasthttp v1.45.0
• golang.org/x/sync v0.2.0

---

• Check this box to trigger a request for Renovate to run again on this repository

[v1.0.0] bigcache implementation

BigCache will replace ccache for our internal caching library. It does a better job of keeping GC pauses low while allowing us to store massive amounts data on heap. This relies on #1 to merge before we are able to utilize the binary tile packet datastructure.

[v1.0.0] Admin endpoint for flushing a proxy's cache

GET /admin/flush/:name should flush a proxy's cache by name on the instance. In the future, we'll have a cluster-aware endpoint that will flush caches across the cluster, but for now this will do.

[v1.0.0] Implement configurable cache key based on template strings

The cache key for a given proxy configuration should be configurable with a template. Default template tokens are {x}, {y}, and {z} and should be required. Any configured URL query parameters should be available by name.

Examples:

Description	Template	Result
Default template if none provided	{z}/{x}/{y}	10/42/-71
Custom prefix	osm/{z}/{x}/{y}	osm/10/42/-71
With URL param some_id = 4	osm/{z}/{x}/{y}:{some_id}	osm/10/42/-71:4

[v1.0.0] Implement second level Redis cache with configurable TTL

Redis as a second level to the cache will give us cache persistence and stability across restarts without slamming the tileservers. Having a TTL set on Redis keys can ensure we don't blow up the cluster.

[v1.0.0] Build binary data structure for HTTP headers and tile

Since we're migrating to bigcache and want to use Redis soon, we'll need to be able to serialize a tile and its saved HTTP headers into a binary data structure. This will allow us to stray from using JSON or other formats that aren't efficient to process. Luckily, all of our tile data is either binary serialized protocol buffers or raw raster image data so we don't need to worry much about processing anything but the few bytes prepended to the tile data for the HTTP headers. The implementation should support header values of arbitrary, yet sane length.

Global counter	Num bytes, H1 key	H1 key data	Num bytes, H1 value	H1 value data	Tile data ...
1 byte - uint8	2 bytes - uint16	N bytes	2 bytes - uint16	N bytes	N bytes
1	6	Cached	4	miss	...

The deserializer will read n headers based on what's present in the global counter byte. It'll then read the number of bytes to read from the key and value counters for each header from 0...n. The remainder of the binary data trailing the final header's value data will be the tile data with no specific buffer or separator. This is similar to how options work in DHCP packets.

The serializer will work in reverse, prepending bytes to the front of the tile data along with their counters until we place the global counter byte.

[v1.0.0] Add configurable URL query parameters

Proxy configurations should support URL query parameters for dynamic tile requests from the tileserver. This may or may not be useful for fetching dynamic data from tiles, or for busting any cache local to the tileserver.

https://lod/map/1/2/3.pbf?some_id=4 --> https://tileserver/1/2/3.pbf?some_id=4