A whitelist proxy that uses socat. ๐๐ผ
tl;dr: To workaround moby/moby#36174.
Basically, Docker supports internal networks; but when you use them, you simply cannot open ports from those services, which is not very convenient: you either have full or none isolation.
This proxy allows some whitelist endpoints to have network connectivity. It can be used for:
- Allowing connection only to some APIs, but not to the rest of the WWW.
- Exposing ports from a container while still not letting the container access the WWW.
Use these environment variables:
Required. It's the host name where the incoming connections will be redirected to.
Default: 0
Set to 1
to enable healthcheck with pycurl http requests. This is useful if the target
uses a deployment where the ip of the service gets changed frequently (e.g.
accounts.google.com
) and you are using PRE_RESOLVE
When you enable the http healthcheck the container marks itself as unhealthy but does nothing. (see moby/moby#22719)
If you want to restart your proxies automatically, you can use https://github.com/willfarrell/docker-autoheal.
Default: http://$TARGET/
Url to use in HTTP_HEALTHCHECK
if enabled. $TARGET
gets
replaced inside the url by the configured TARGET
.
Default: 2000
Timeout in milliseconds for http healthcheck. This is used as a timeout for connecting and receiving an answer. You may end up with twice the time spend.
Default: tcp
Set to udp
to proxy in UDP mode.
Default: 100
Limits the maximum number of accepted connections at once per port.
For each port and open connection a subprocess is spawned. Setting a number too high might make your host system unresponsive and prevent you from logging in to it. So be very careful with setting this setting to a large number.
The typical linux system can handle up to 32768 so if you need a lot more parallel open connections make sure to also set the corresponding variables on your host system. See https://stackoverflow.com/questions/6294133/maximum-pid-in-linux for reference. And divide this number by at least the number of ports you are running through docker-whitelist.
docker-whitelist basically starts socat
so the behaviour is the same. In case no more
subprocesses can be forked:
- UDP mode: You won't see a difference on the connecting side. But no more packets are forwarded for new connections until the number of connections for this port is reduced.
- TCP mode: docker-whitelist no longer accepts the connection and your connection will wait until the number of connections for this port is reduced. Your connection may time out.
Default: 208.67.222.222 8.8.8.8 208.67.220.220 8.8.4.4
to use OpenDNS and Google DNS
resolution servers by default.
Only used when pre-resolving is enabled.
Default: 80 443
. If you're proxying HTTP/S services, no need to specify!
The port where this service will listen, and where the target service is expected to be listening on also.
Default: 0
Set to 1
to force using the specified nameservers to resolve the
target before proxying.
This is especially useful when using a network alias to whitelist an external API.
Default: 0
Set to 1
to enable healthcheck with pycurl smtp requests. This is useful if the target
uses a deployment where the ip of the service gets changed frequently (e.g.
smtp.eu.sparkpostmail.com
) and you are using PRE_RESOLVE
see HTTP_HEALTHCHECK
Default: smtp://$TARGET/
Url to use in SMTP_HEALTHCHECK
if enabled. $TARGET
gets
replaced inside the url by the configured TARGET
.
Default: HELP
Enables changing the healthcheck command for servers that do not support HELP
(e.g.
for MailHog you can use QUIT
)
Default: 2000
Timeout in milliseconds for smtp healthcheck. This is used as a timeout for connecting and receiving an answer. You may end up with twice the time spend.
Default: 1
1
means the process will wait for an answer from the server before the forked child
process terminates (until this happens the connection counts towards the connection
limit). Set to 0
if no answers are expected from the server, this prevents
subprocesses waiting for an answer indefinitely.
Setting to 0
is recommended if you are using this to connect to a syslog server like
graylog.
Default: 0
Set to 1
to log all connections.
So say you have a production app called coolapp
that sends and reads emails, and uses
Google Font APIs to render some PDF reports.
It is defined in a docker-compose.yaml
file like this:
# Production deployment
version: "2.0"
services:
app:
image: Tecnativa/coolapp
ports:
- "80:80"
environment:
DB_HOST: db
depends_on:
- db
db:
image: postgres:alpine
volumes:
- dbvol:/var/lib/postgresql/data:z
volumes:
dbvol:
Now you want to set up a staging environment for your QA team, which includes a fresh copy of the production database. To avoid the app to send or read emails, you put all into a safe internal network:
# Staging deployment
version: "2.0"
services:
proxy:
image: traefik
networks:
default:
public:
ports:
- "8080:8080"
volumes:
# Here you redirect incoming connections to the app container
- /etc/traefik/traefik.toml
app:
image: Tecnativa/coolapp
environment:
DB_HOST: db
depends_on:
- db
db:
image: postgres:alpine
networks:
default:
internal: true
public:
Now, it turns out your QA detects font problems. Logic! app
cannot contact
fonts.google.com
. Yikes! What to do? ๐คท
tecnativa/whitelist
to the rescue!! ๐ช๐ค
# Staging deployment
version: "2.0"
services:
fonts_googleapis_proxy:
image: tecnativa/whitelist
environment:
TARGET: fonts.googleapis.com
PRE_RESOLVE: 1 # Otherwise it would resolve to localhost
networks:
# Containers in default restricted network will ask here for fonts
default:
aliases:
- fonts.googleapis.com
# We need public access to "open the door"
public:
fonts_gstatic_proxy:
image: tecnativa/whitelist
networks:
default:
aliases:
- fonts.gstatic.com
public:
environment:
TARGET: fonts.gstatic.com
PRE_RESOLVE: 1
proxy:
image: traefik
networks:
default:
public:
ports:
- "8080:8080"
volumes:
# Here you redirect incoming connections to the app container
- /etc/traefik/traefik.toml
app:
image: Tecnativa/coolapp
environment:
DB_HOST: db
depends_on:
- db
db:
image: postgres:alpine
networks:
default:
internal: true
public:
And voilร ! app
has fonts, but nothing more. โ๐ฎ
All the dependencies you need to develop this project (apart from Docker itself) are managed with poetry.
To set up your development environment, run:
pip install pipx # If you don't have pipx installed
pipx install poetry # Install poetry itself
poetry install # Install the python dependencies and setup the development environment
To run the tests locally, add --prebuild
to autobuild the image before testing:
poetry run pytest --prebuild
By default, the image that the tests use (and optionally prebuild) is named
test:docker-whitelist
. If you prefer, you can build it separately before testing, and
remove the --prebuild
flag, to run the tests with that image you built:
docker image build -t test:docker-whitelist .
poetry run pytest
If you want to use a different image, pass the --image
command line argument with the
name you want:
# To build it automatically
poetry run pytest --prebuild --image my_custom_image
# To prebuild it separately
docker image build -t my_custom_image .
poetry run pytest --image my_custom_image