apoikos/http-redirector

Debian mirrors HTTP redirector

Intro

This is a work in progress. Please do send patches and provide feedback. Thanks!

The project is similar to mirrorbrain (.org) and fedora's mirrors system. However, it has a few differences (this list is not intended to be complete):

• it is very specific to the way Debian mirrors are constructed. Details regarding architectures and the different mirror types are taken into consideration.
• because of the previous point and considering many mirrors only support http, it does not perform a full mirror scan. Mirrorbrain does. There's a tool to detect inconsistencies between what the mirrors master list claims a mirror contains and what it actually contains.
• it aims to be httpd-independent. Mirrorbrain requires apache.
• IPv6 support
• no DBMS. Although using a DBMS could provide some advantages, at the moment the Storable database seems to be enough. The idea is to keep everything simple.
• easy deployment

Live instance

There's a live instance of this code (but not necessarily the latest and greatest revision) at http://httpredir.debian.org/ (previously, http.debian.net)

There's some more documentation there. It should be imported into the repository, however.

TODO

There are some TODOs and FIXMEs in the source code, additionally, there's the issues tracker at github: https://github.com/rgeissert/http-redirector/issues

Getting started

Required packages:

• moreutils
• libanyevent-perl
• libanyevent-http-perl
• libev-perl (recommended; or another event loop supported by AE)
• libtimedate-perl
• libgeo-ip-perl
• libwww-perl
• liburi-perl
• libplack-perl
• liblinux-inotify2-perl (for plackup -R, for local tests only)
• starman (if using that server for the application)
• twiggy (alternative server)
• wget

Run ./update.sh, it will download the geoip databases, the mirrors list, build the database used by the redirector, and check the mirrors for errors.

Look at the example below for invocation and plackup(1p) and Plack's documentation for running the application under different server modes.

If you just want to simulate a request (like it used to be possible with redir.pl), use local-request.pl. It sends a request to the application without actually starting the server. You can pass request parameters in the first argument to the script. To fake the IP address of the request, set the REMOTE_ADDR env var when calling it. No other CGI-like env var is recognised.

Getting started for development

Required packages (in addition to the ones above):

• libtest-trap-perl

Keeping everything in shape

update.sh should be run at least once a month[1], this allows the changes to the mirror list(s) to be reflected. By default it will run:

• build-main-db.pl
• build-peers-db.pl
• check.pl

check.pl should be run multiple times a day[2]

build-peers-db.pl should be run after every execution of build-main-db.pl or whenever the peers lists are updated[3]

NOTE: update.sh will leave the new database in a file called db.in to be renamed to db if it is the first time it is created. When updating the db of an in-production instance, the new db will be picked up by the next run of check.pl.

NOTE: build-main-db.pl and check.pl do NOT lock the database. You must ensure that no more than one script is running at the same time.

[1] the script rebuilds the database, so any info collected by check.pl regarding the availability of mirrors is lost. check.pl --check-everything should be run after build-main-db.pl, this is done automatically when running update.sh.

[2] it really depends on the kind of setup one wants and the hosts that conform the mirrors network. For Debian's archive, it should be run at least every ten minutes, every five minutes being better.

[3] it only applies when using an AS peers database. The name of the peers database is specific to the mirrors database on which it is based. At present, peers databases for old mirrors databases are not cleaned up automatically.

Real life testing

If using apache, you will want to run the redirector locally and make apache forward the requests (therefore acting as a reverse proxy). For example, if you run the application on port 5000 you can:

ProxyPass /redir http://127.0.0.1:5000/

RewriteEngine On
RewriteRule ^/?(?:(demo)/)?debian-(security|backports|ports)/(.*) /redir/?mirror=$2&url=$3&action=$1 [PT]
RewriteRule ^/?(?:(demo)/)?debian-archive/(.*) /redir/?mirror=old&url=$2&action=$1 [PT]
RewriteRule ^/?(?:(demo)/)?debian/(.*) /redir/?mirror=archive&url=$2&action=$1 [PT]

# mirror:// method support:
RewriteRule ^/?debian-(security|backports|ports)\.list(?:$|\?(.+)) /redir/?mirror=$1.list$2 [QSA,PT]
RewriteRule ^/?debian-archive\.list(?:$|\?(.+)) /redir/?mirror=old.list$1 [QSA,PT]
RewriteRule ^/?debian\.list(?:$|\?(.+)) /redir/?mirror=archive.list$1 [QSA,PT]

You can for example make it listen on 127.0.1.10, setup a vhost, and use the following on your sources.list:

deb http://127.0.1.10/debian/ sid main
deb-src http://127.0.1.10/debian/ sid main

deb http://127.0.1.10/debian-security/ testing/updates main
deb http://security.debian.org/ testing/updates main

Note: accessing the redirector from a local IP address is not ideal and may only work with hacks.

Forcibly disabling mirrors

If necessary, mirrors can forcibly be disabled by passing a file name to check.pl's --disable-sites option (default: sites.disabled)

The format of this file is:

<domain name>[/mirror type]

Whenever the option is passed and the file exists, every mirror matching an entry in the file will be disabled without further checks. If a mirror type is specified, only that mirror type of the given mirror will be disabled.

An empty file name can be specified to override the default and to prevent the parsing of said file. In order to re-enable a mirror an existing file name must be specified.

NOTE: any disabled mirror that is no longer in the list will be re-enabled. E.g. passing --disable-sites /dev/null to check.pl will re-enable all disabled mirrors.

Running the redirector on top of a real mirror

It is possible to run the redirector on a sever that has the files itself. The use case would be: serve some users, send others to a better mirror.

In order to use it in this mode, a few things need to be setup.

Mirror::Redirector:

• Set the subrequest_method variable as appropriate:

• redirect: works on any httpd, but requires another roundtrip
• sendfile: for apache with mod_xsendfile, lighttpd 1.5, cherokee
• sendfile1.4: for lighttpd 1.4
• accelredirect: for nginx

• Set the list of hosts for which the files will be served, in the this_host variable. For example, if your host is my.mirror.tld, that's what you need to add.

NOTE: even though it is possible to list other mirror's host names, care should be taken when doing so. The mirror checker is not aware of this mapping and may lead to erroneous behaviour.

NOTE2: make sure your httpd works as expected. The redirector sends a Content-Location header when the file should be delivered by the server itself. Make sure it is not removed. Also look for issues with Range, Last-Modified, and other features.

Then, setup your httpd so that requests for serve/ are served directly, bypassing the redirector. Finally, make all the usual traffic go through the redirector. Make sure you don't break directory listing when doing so. mod_xsendfile, for example, breaks it because it bypasses mod_autoindex.

A sample configuration for apache follows:

XSendFile On

# Should be possible to do it without an alias, but it makes it a
# bit clearer
Alias /debian/serve/ "/var/www/debian/"

RewriteEngine On
# Exclude /debian/serve/ from mod_rewrite, mod_alias will handle it
RewriteRule ^/?debian/serve/.*$ $0 [PT,L]
# Send all file requests through the redirector
RewriteRule ^/?debian/(.*[^/]$) /redir/?mirror=archive&url=$1 [NS,PT]
# Directory listing requests will pass-through and be handled by
# apache itself

IMPORTANT: do not enable this setup, or run the redirector at all, on a mirror that is part of Debian's mirrors network without consulting the mirroradmin group and WAITING for their APPROVAL.

AS peers database

It is possible to instruct the redirector to serve clients from an originating AS to one or a set of destination AS' where mirrors are located. The database can be built with the build-peers-db.pl script.

Its input are *.lst files in the peers.lst.d directory in the following format:

<client AS> <mirror domain name|mirror AS> [distance [IPv]]

The preferred form of the second value is by domain name. When an AS is specified, it will internally be rewritten to the existing mirrors in the corresponding AS.

Comments may be specified by prefixing them with a # character. Multiple client ASNs can be specified by enclosing them in braces and separating them with commas. E.g. {13138,2210} some.mirror.tld

The distance is currently not used by the redirector, but at some point it might. It defaults to 0. Any positive integer may be specified. It is expected to be read as 0 being the most preferred mirror, 1 being the second, 2 being the third, and so on.

The IPv field, if specified, should be v4, v6, or the two separated by a comma with no specific order. It can be used to indicate that the given peering rule only applies to the version of the IP specified in the field. It defaults to v4.

Mirrors chosen by this database are still subject to geo location restriction. I.e. from the set of candidates, a subset of those that are geographically closer will be created and used.

Caveats: since the database is AS-based, large Autonomous Systems that traverse countries or continents will still be considered, even if not desired. At present, the redirector skips mirrors that are located in a different continent, but that's done to work around another issue and the behaviour is not guaranteed to persist after the other issue is properly addressed.

Understanding the db

The database consists of (mostly inverted) indexes that are supposed to provide fast and cheap lookups.

In order to save space on the database, a few unusual things are done. For example, hash entries with undef as value are valid. undef is smaller in a Storable database than an integer. Any script using the database should therefore test for 'exists' instead of 'defined'.

To better understand what the database looks like, run ./dump-db.pl | pager

Credits

"This product includes GeoLite data created by MaxMind, available from http://maxmind.com/"