Add some VCL to fastly so docs can be purged by top or 2nd level folder

[python]

Add some VCL to fastly so docs can be purged by top or 2nd level folder

[ewdurbin]

Some thoughts:

Implementation of Surrogate-Keys

It could be done via headers/conditions in the service configuration, though currently the docs fastly configuration is not created from version control (outside of fastly's own internal versioning), so I'm not sure if that's the best way to approach it.

Regardless this is better accomplished by setting Surrogate-Key header values directly on the responses served via the backend.

Since as far as I am aware, there's no trivial way to manage adding HTTP headers with sphinx, I recommend doing this by using nginx add_header directives in the nginx config rather than with VCL.

Access for purging

Currently purges using Surrogate-Keys are only accessible via the authenticated API, so we'd need some mechanism for issuing them when necessary.

[JulienPalard]

What about using BANs with regexes implemented purely in VCL so we can do it without the authenticated API?

https://varnish-cache.org/docs/7.2/users-guide/purging.html

Something like:

        if (req.method == "BAN") {
                # Same ACL check as above:
                if (!client.ip ~ purge) {
                        return(synth(403, "Not allowed."));
                }
                # Assumes req.url is a regex. This might be a bit too simple
                if (std.ban("obj.http.url ~ " + req.url)) {
                        return(synth(200, "Ban added"));
                } else {
                        # return ban error in 400 response
                        return(synth(400, std.ban_error()));
                }
        }

While we're at it, implementing an IP whitelist for PURGE and BAN should be great to avoid ReDoS attacks.

[ewdurbin]

Surrogate-Key purges aren't exposed via Fastly's config/VCL but through their API, so I don't think that will work.

[JulienPalard]

I almost never used fastly, I just had plain varnish in prod. Is fastly VCL restricted is some way blocking us to play this kind of tricks? :(