bpicolo/herd

A library for managing small server clusters.

Herd

Herd is an in-progress, easy-to-use framework for management of small-scale server clusters in pure python. Tools like puppet, chef, ansible seem to target more large-scale systems, and in my experience aren't great for getting some small, dead simple machines and apps going.

In addition, Herd aims to provide a simple framework for creating and handling your nodes across multiple cloud providers, to help limit and manage your server costs. :)

While in theory, Herd can do anything SSH can manage, it aims to provide one good strategy for everything, to eliminate complicated configs.

Getting started

• pip install herd

• Create a basic config file:

    [ssh]
    path = "/path/to/rsaprivatekey"
    password = 'rsa_key_passphrase'
  
    [providers.digitalocean]
    token = "MY_PRIVATE_TOKEN"
  
    [clusters.cluster_name]
    provider = 'digitalocean'
    server_count = 1
    max_monthly_cost = 5
    ssh_keys = ["RSA_KEY_FINGERPRINT"]
    region = 'sfo1'
    image = 'ubuntu-14-04-x64'
  

• herd up cluster_name --config path/to/config (default ./config.toml)

• herd info cluster_name

• herd install cluster_name git

herd's Issues

User support

What do you mean "only supporting root login" is a bad idea.

How does one define a task?

I suspect we should have some reasonable API/strategy for determining whether a task is done (other than just retrying it). This way we could avoid rerunning steps. It would be pretty easy to get pre/post install hooks along with this.

I think it's probably best to avoid just keeping a log, and rather actually check if things are done? A log isn't necessarily in sync with the host. Basically, a task needs a test case I suppose? : )

I hear writing tests is a good plan

Probably reasonable to have a small integration test suite that can test against providers but it sure as crap needs to work hard to clean up after itself. (And of course you'll need to supply API keys to run it).

Could against a docker container for ssh bits in theory... But setting that up on mac is unfun

Unit tests will be a fine first line. I think I remember python-digitalocean having some mock responses already which helps

Support many OSs

I mean, in theory it's easy to install anything the provider gives you access to, but if it doesn't just use default apt, etc, it will be tricky.

Probably easiest to have our own versioning scheme and convert from it to those available ( Use DOs scheme perhaps? It seems sane)

Improved cluster sync

I guess technically we could end up with multiple node of the same name right now (other than the fact that the provider api will break).

Sync should:
Check size.
Keep current nodes alive if they match. (Can they be renamed to match?)
Sync should be able to install programs, etc (basically, whatever config supports, sync should support)
- idempotence
On that note, it should wait for nodes to be available.

To sudo or not to sudo.

Probably not to sudo (as a default)

Unify script running with a python api

Instead of just running python -m bin.herd syncall, for example:

Herd.load_from_config(path)
Herd.sync_all()

There should be a command line script layer that more or less matches the basic python api.

Terms

Defining terms can probably help organize things as I build this out... I aim for a stupid simple, obvious naming scheme.

Cluster
A group of machines. If cloud machines, these probably share an identical configuration for efficient, easy provisioning. Probably tend to do tasks as a group, though I suspect supporting them running different tasks isn't unreasonable.

Command
The basic building blocks for performing tasks. Ideally these should be comprehensively built in.
E.g. install a package, copy a file to a machine, execute a command, etcetc. All tasks should boil down to some combination of commands. In an ideal world, commands are idempotent and we can test if one has been performed.

Task
Some combination of commands and other tasks that represents something you want to do on a machine. Like commands, ideally testable and idempotent.

Role
The mapping of clusters / (maybe individual machines) to tasks they are expected to perform