riotkit-org/riotkit-harbor

Complete compose environment, from development to production with service discovery and health checking. Kubernetes features on Compose scale, with Compose simplicity

RiotKit's Harbor

docker-compose based framework for building production-like environments - developing and testing on your local computer, deploying to your server or cluster from shell or from CI.

Demo: https://asciinema.org/a/348698

Notice

This project will be in the future replaced by "Space Harbor", which is successor of Harbor. Why? We started Harbor project as lightweight and elastic replacement for Kubernetes usage in tiny scale (for smaller services like blogs, forums etc.) but recently we researched usage of a lightweight Kubernetes distribution which fits very well for tiny scale and does not require an extra development of Python code.

We will support bugfixes fixes to Harbor till 01.01.2023, please consider migration to https://github.com/riotkit-org/k3s-space-harbor as soon as it will be announced stable.

Harbor

Is Kubernetes or OKD too big overhead or not suitable for your environment?

Is standard docker-compose too primitive to use in production?

Harbor fits perfectly, while providing some of Kubernetes-like solutions in docker-compose!

Features:

• Service discovery (pins containers into WWW domains by labelling)
• Deployment strategies: compose's standard, recreation, and rolling-updates (zero-downtime updates)
• Automatic Letsencrypt SSL
• Standardized directory structures and design patterns
• Ready to use snippets of code and solutions (one command to install database, Wordpress or other application from our repository)
• Ansible integration to prepare your production/testing server and deploy updates in extremely intuitive way

Roadmap:

Harbor 2.1

• Init containers support (basing on Kubernetes idea) (#5)
• delayed-request update strategy (minimum-downtime deployment with holding all HTTP requests waiting for new application) (#11)
• Webhook handling update daemon to trigger container and git updates (#10)

Read documentation for more: https://environment.docs.riotkit.org/en/latest/?badge=latest

Changelog

Harbor 2.0

• Service upgrade strategies: Compose-native, recreation, rolling-updates
• Rewrite from Makefile to Python
• Containers startup priority
• Maintenance mode is now not only global, but also per domain
• Development mode now supports not only .localhost domains, but also *.xip.io

From authors

We are grassroot activists for social change, so we created this software while we were helping those fantastic initiatives:

• RiotKit (https://riotkit.org)
• International Workers Association (https://iwa-ait.org)
• Anarchistyczne FAQ (http://anarchizm.info) a translation of Anarchist FAQ (https://theanarchistlibrary.org/library/the-anarchist-faq-editorial-collective-an-anarchist-faq)
• Federacja Anarchistyczna (http://federacja-anarchistyczna.pl)
• Związek Syndykalistów Polski (https://zsp.net.pl) (Polish section of IWA-AIT)
• Komitet Obrony Praw Lokatorów (https://lokatorzy.info.pl)
• Solidarity Federation (https://solfed.org.uk)
• Priama Akcia (https://priamaakcia.sk)

Special thanks to Working Class History for very powerful samples that we could use in our unit tests.

riotkit-harbor's Issues

Deployment: Clean up generated files after finished deployment

This is a security issue - the generated files with credentials are kept in the .rkd/deployment directory

Add possibility to enter SSH shell - harbor :deployment:ssh

It should be easy and possible to implement a task to ssh into remote host(s), and very practical. There could be also a switch --print-sudo-pass to print sudo password before entering the shell.

Containers deployment priority

Simply allow to assign priority numbers to containers (no complex dependency management - tool for listing all of services ordered by priority should be enough to handle this easily)

Add centralized logging via rsyslog

Logging to host's rsyslog on docker daemon side is a good idea, it could be used to centralize logging and possibly passing it to external server that has kibana for logs browsing.

Support Vouch Proxy as an opional SSO login gateway

Idea:
To allow hiding administrative services behind a login page.

Technically:
The proxy could be optionally installable via Harbor Snippet Cooperative. But the support must be anyway in the NGINX template.

https://github.com/vouch/vouch-proxy
https://developer.okta.com/blog/2018/08/28/nginx-auth-request

Implement Snippet Cooperative tasks

https://github.com/riotkit-org/riotkit-harbor-snippet-cooperative

Compose files loading priority

This is currently unclear what is the order of loading of YAML files for compose. There must be a mechanism to apply overrides to existing services.

Need to specify order, example (later - better, to work as an override):

• apps/conf
• apps/conf/override
• apps/conf.dev
• apps/conf.prod

Create a network forwarding

Cases:

• Remote host -> container
• Remote host -> host
• Local host -> container

Delayed-request update strategy

Idea:
"delayed-request" strategy can be implemented if possible (delay current requests on gateway - extend timeout until the service is not replaced)

Notice: This is an investigation + implementation task. If the investigation would result in - "cannot do" then this feature should be cancelled.

Effect:
Users would not get 500 error or other weird errors, they would need to wait a few seconds more for page to load

Support for IS_DEBUG_ENVIRONMENT

Generic: Services up/down/rm

No defined --ask-ssh-login for :deployment:files:update

Basic services management

• exec
• logs
• inspect (given service instance - container)
• report (service and it's all instances)

Correctly handle RSA keys generated by latest OpenSSH

Issue: Not all keys begins with -----BEGIN OPENSSH PRIVATE KEY, some begins with -----BEGIN RSA PRIVATE KEY which makes Harbor not to recognize value as a properly formatted private key. The result is that key is pasted into ansible.cfg making a syntax issue.

Resolution: check only first -----BEGIN at the at least beginning of the private key string

Init containers

Like in Kubernetes

Hooks support

Git repositories support

Like in Harbor 1.0

Implement :create:project basing on :rkd:create-structure

:vault:edit, :vault:encrypt, :env:encrypt tasks

Create a task that will encrypt current .env into .env-prod using Ansible Vault, and a task that will allow to directly edit the .env-prod

Ability to resolve naming conflicts in docker-compose

Correctly handle case, when docker-compose says it cannot create containers because the name is already used - at least what we can do is retry the operation which usually works.

   ===> Starting "gateway_letsencrypt" (1 instances)...
[2020-07-11 08:27:34.483][info]: Doing a "rolling" deployment for "gateway_letsencrypt"
[2020-07-11 08:27:34.484][info]: Processing instance #1/1
[2020-07-11 08:27:34.484][info]: Suspending service discovery
[2020-07-11 08:27:34.971][info]: Scaling up to 2
Creating anarchist_project_gateway_letsencrypt_1 ... 
Creating anarchist_project_gateway_letsencrypt_2 ... 
Creating anarchist_project_gateway_proxy_gen_1
Creating anarchist_project_gateway_letsencrypt_1 ... done
Creating anarchist_project_gateway_letsencrypt_1

ERROR: for anarchist_project_gateway_letsencrypt_2  Cannot create container for service gateway_proxy_gen: Conflict. The container name "/anarchist_project_gateway_proxy_gen_1" is already in use by container "c6b73cdcb025091d31db78df79010819388bacf220bb42f9502da1393fc953fa". You have to remove (or rename) that container to be able to reuse that name.

ERROR: for gateway_letsencrypt  Cannot create container for service gateway_proxy_gen: Conflict. The container name "/anarchist_project_gateway_proxy_gen_1" is already in use by container "c6b73cdcb025091d31db78df79010819388bacf220bb42f9502da1393fc953fa". You have to remove (or rename) that container to be able to reuse that name.
Encountered errors while bringing up the project.
[2020-07-11 08:27:37.405][error]: Scaling back to declared state as error happened: Command 'bash' returned non-zero exit status 1.
anarchist_project_gateway_letsencrypt_1 is up-to-date
[2020-07-11 08:27:37.960][info]: Starting service discovery
Starting anarchist_project_gateway_1 ... done
Starting anarchist_project_gateway_proxy_gen_1 ... 
Starting anarchist_project_gateway_proxy_gen_1 ... done
Traceback (most recent call last):
  File "/project/.venv/lib/python3.6/site-packages/rkd/executor.py", line 65, in execute
    env=declaration.get_env()
  File "/project/.venv/lib/python3.6/site-packages/harbor/tasks/base.py", line 136, in execute
    return self.run(context)
  File "/project/.venv/lib/python3.6/site-packages/harbor/tasks/service.py", line 101, in run
    return strategies[strategy]()
  File "/project/.venv/lib/python3.6/site-packages/harbor/tasks/service.py", line 87, in <lambda>
    'rolling': lambda: self.deploy_rolling(service, context),
  File "/project/.venv/lib/python3.6/site-packages/harbor/tasks/service.py", line 170, in deploy_rolling
    raise e
  File "/project/.venv/lib/python3.6/site-packages/harbor/tasks/service.py", line 151, in deploy_rolling
    existing_containers = self.containers(ctx).scale_one_up(service)
  File "/project/.venv/lib/python3.6/site-packages/harbor/driver.py", line 372, in scale_one_up
    raise e
  File "/project/.venv/lib/python3.6/site-packages/harbor/driver.py", line 368, in scale_one_up
    capture=True
  File "/project/.venv/lib/python3.6/site-packages/harbor/driver.py", line 194, in compose
    return self.scope.sh(cmd, capture=capture)
  File "/project/.venv/lib/python3.6/site-packages/rkd/contract.py", line 282, in sh
    cmd=cmd, capture=capture, verbose=verbose, strict=strict, env=env
  File "/project/.venv/lib/python3.6/site-packages/rkd/taskutil.py", line 92, in sh
    return check_output('bash', shell=True, stdin=read).decode('utf-8')
  File "/usr/lib/python3.6/subprocess.py", line 356, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'bash' returned non-zero exit status 1.
Command '#!/bin/bash -eopipefail 
set -euo pipefail; export PYTHONUNBUFFERED=1;  /project/.venv/bin/harbor --no-ui --no-ui :harbor:service:up gateway_letsencrypt  --strategy=auto' returned non-zero exit status 1.Cannot start service "gateway_letsencrypt"

Steps to reproduce:
Deploy to a fresh environment - do a docker prune on containers, networks and volumes.

Security: Implement a possibility to use WAF (example: ModSecurity)

Deployment: Notify about unpushed and uncommited local changes

It is a very common mistake to try to deploy changes that were not commited or pushed to the server yet.

Update daemon

Harbor can optionally be setting up a daemon on host to listen for rolling-update of a service on UNIX socket - UNIX socket can be forwarded to inside of a docker container that takes webhooks (eg. thin-deployer)

This concept is very secure and clear.

Create ./harborw wrapper as of RKD 3.0.0 wrappers are supported

Then merge Wolnosciowiec-Archive/ansible-harbor-role#3

Automatically decrypt ansible-vault encrypted files on the fly during deploment - configuration files and application files

Ansible deployment hangs on facts gathering

Harbor Simple Router support

The Harbor Simple Router could be used to perform a domain redirection, or informational HTML page publication (eg. "coming soon").

Use cases:

• Commit to repo and deploy to production for longer time
• In case of emergency do it on production, quickly replace existing container with a Harbor Simple Router forwarding the traffic somewhere / displaying maintenance page while turning off the original service

[:deployment:apply] Add a switch --service-selector to be able to specify a raw profile string

Often an administrator would like to quickly deploy a single container, without having to add a profile file into git.
This could be done by adding a new switch that could allow to read profile from commandline.

harbor :deployment:apply --service-selector 'name.startswith("matrix")'

:deployment:apply - do not use groups by default, Harbor is single-server by default

Demo Gif is Very Distorted

I'm not sure if it works for other people, but for me the demo Gif on the README is extremely distorted:

If you want, you could try out my cast2gif renderer for Asciinema recordings. But unfortunately I don't have automated builds yet. 😕

[BC] Make templating more elastic

Templates should have a possibility to be rendered anywhere in the project root directory.

Expectations:

• Rendered from ./containers/templates
• Into directory ./

Rolling updates

Make gateway optional

Make gateway, service discovery and SSL optional, installable via Snippet Cooperative.

Support multiple services in :service:up

Built-in deployment to production with Ansible

Migrate from Travis to Github Actions

Travis is veeeeeery sloooow. Github Actions would improve the CI performance and clarity alot.