nixos/nixops

NixOps is a tool for deploying to NixOS machines in a network or cloud.

NixOps

Warning

NixOps is in low-maintenance mode and probably not suited for new projects. Use at your own risks.

Note

An experimental rewrite of NixOps is happening under https://github.com/nixops4/nixops4

NixOps is a tool for deploying to NixOS machines in a network or the cloud. Key features include:

• Declarative: NixOps determines and carries out actions necessary to realise a deployment configuration.
• Testable: Try your deployments on VirtualBox or libvirtd.
• Multi Cloud Support: Currently supports deployments to AWS, Hetzner, and GCE
• Separation of Concerns: Deployment descriptions are divided into logical and physical aspects. This makes it easy to separate parts that say what a machine should do from where they should do it.
• Extensible: NixOps is extensible through a plugin infrastructure which can be used to provide additional backends.

For more information, please refer to the NixOps manual.

Running

NixOps is included in nixpkgs and can be executed in a shell as follows:

$ nix-shell -p nixops

or for a bleeding edge version, including many fixes relative to the 1.7 series,

$ nix-shell -p nixopsUnstable

You may need access to a Nix remote builder if your system does not support the deployment's system builds directly. MacOS users may use a virtual machine with NixOS for this purpose.

It is also possible to use cross-compilation with NixOps, by setting nixpkgs.localSystem and nixpkgs.crossSystem. A mix of remote, emulated and cross builds is also possible; see this writeup on eno.space.

Building And Developing

Building The Nix Package

You can build the Nix package by simply invoking nix-build on the project root:

$ nix-build

Development Shell

shell.nix provides an environment with all dependencies required for working on NixOps. You can use nix-shell to enter a shell suitable for working on NixOps which will contain all Python dependencies specified in pyproject.toml

$ nix-shell

Executing Tests

Inside the development shell the tests can be executed as follows:

$ pytest

Documentation

NixOps' documentation uses reStructuredText. When editing the docs, get a live-reloading, rendered version of the docs:

nixops$ ./live-docs.py
Serving on http://127.0.0.1:5500

and verify its lints before committing:

nixops$ lint-docs

Contributing

Contributions to the project are welcome in the form of GitHub PRs. Please consider the following guidelines before creating PRs:

• Please make sure to format your code using black.
• Please add type signatures using mypy.
• If you are planning to make any considerable changes, you should first present your plans in a GitHub issue so it can be discussed.
• If you are adding features, please also add reasonable tests.

License

Licensed under LGPL-3.0.

nixops's Issues

/nix is aufs of EBS volume and ephemeral

This is likely to fail when a server is stopped and started, especially when part of the nix store is on ephemeral, which is lost on stop/start.

Support encrypted volumes without storing a key

Obviously, it's not desirable to store LUKS encryption keys on the target machine. But not having them there breaks unattended reboots. We should be able support unattended boots like this:

• The LUKS service calls "systemd-ask-password" to get the password. This will block until some systemd password agent provides the required password. Services that depend on the encrypted filesystem will likewise block.
• We add some command "charon send-keys" that connects via SSH to provide the requested keys to systemd. (We may have to write a systemd password agent for this.) "charon send-keys" could be invoked periodically from some monitoring machine (e.g. every 10 minutes from a cronjob). Since SSH performs host key checking, we know we're sending the keys to the right machine and there is no man in the middle attack.
• After receiving the keys, "systemd-ask-password" will return, LUKS will attach the encrypted volume, and the boot will continue.

Add ad hoc expression in "charon deploy"

It would be nice to have a flag like

charon deploy --extra foo.nix

that adds foo.nix to the model for this deployment only. We could even have options on the command line, e.g.

charon deploy --extra-option 'webserver.services.httpd.enable = false'

to disable Apache for this one deployment.

Add missing admin functionalities

e.g. SystemTypes, Create User. Check which others are missing.

Add scp subcommand

Similar to current charon ssh command, it would be nice to have scp feature to easily transfer files to and from a machine.

netcat doesn't accept IPv6 addresses

When trying to connect to an IPv6 address, the nc instruction, which checks whether TCP port 22 accepts connections reports the following:

.Error: Couldn't resolve host "2001:610:685:1:216:3eff:fe01:00fe"
.Error: Couldn't resolve host "2001:610:685:1:216:3eff:fe01:00fe"
.Error: Couldn't resolve host "2001:610:685:1:216:3eff:fe01:00fe"
.Error: Couldn't resolve host "2001:610:685:1:216:3eff:fe01:00fe"

and charon never gets past this phase

Apparently, netcat does not understand IPv6 addresses

EC2: add support for encrypted disks

Encrypted disk support means that we must get the key from somewhere, possibly a remote location. So this should be pluggable.

Of course this becomes extra tricky if the disk is mounted in the initrd. It might be an option to get the key from the EC2 userdata.

Respect mkOverridden values for the 'network' attribute (and its attribtues)

For some cases, it might be desirable to specify network (or some attribute of network, e.g. network.description) in multiple files and use mkOverride to determine which gets priority. Right now a list is created, which doesn't work in all cases.

Add recovery feature

If the state file of a Charon deployment is lost, it would be nice if it could be reconstructed automatically given the UUID of the network.

A related feature would be to enumerate all known machine instances in all backends (along with their network UUIDs).

So you could recover like this:

$ charon discover-machines
ec2 33bced96-5f26-11e1-b9d7-9630d48abec1 foo
ec2 33bced96-5f26-11e1-b9d7-9630d48abec1 bar

$ charon -s ./state.json recover 33bced96-5f26-11e1-b9d7-9630d48abec1

Support building without Perl bindings

The Perl bindings are a problem on some platforms (in particular Cygwin and cross-compiled environments), so it would be nice if there was a configure flag to disable building them.

However, the Perl bindings are now used all over the place (e.g. in download-using-manifests), so we'd need some fallback code to handle the absense of the bindings. For instance, functions such as isValidPath() and queryPathHash() can be emulated slowly by calling nix-store.

Add headless support to the VirtualBox backend

Add distributed rollback feature

Charon could use a Nix profile to store all previous top-level machine configurations. Then we could roll back al or some machines to a previous configuration.

Elastic IP does not get attached when it is stolen by another instance

Elastic IPs can be taken by any instance at any point. If another instance grabs it, charon deploy on the original instance with that elastic ip goes wrong (ssh host key different). charon deploy --check does not fix it. Needed to remove elasticIP from json to fix situation.

Monitor Upstart job failures and possibly take action

If, in the process of activating a new configuration, any Upstart job fails, it should be reported to the user. And maybe an automatic rollback should be started.

Handle bogus EC2 "does not exist" errors on instance creation

EC2 can briefly return "instance ID does not exist" errors after an instance has been created. Charon should handle this properly.

Example:

$ charon -s apache-ec2-multizone.json deploy
creating EC2 instance ‘backend1’ (AMI ‘ami-732c1407’, type ‘m1.small’, region ‘eu-west-1’)...
creating EC2 instance ‘backend2’ (AMI ‘ami-d9409fb0’, type ‘m1.small’, region ‘us-east-1’)...
creating EC2 instance ‘proxy’ (AMI ‘ami-732c1407’, type ‘m1.small’, region ‘eu-west-1’)...
waiting for IP address of ‘backend2’... [pending] error: EC2ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?>
<Response><Errors><Error><Code>InvalidInstanceID.NotFound</Code><Message>The instance ID 'i-e1e51ba9' does not exist</Message></Error></Errors><RequestID>bb416482-01c1-433d-9214-6d24cda529ea</RequestID></Response>

Charon wants to delete detached volume

If volume is detached from original machine, it should not be deleted on termination of the original machine.

EC2: add support for making snapshots of all EBS disks in network

• Create snapshots on demand, for whole network, or part
• Show backups/latest backups

Document network.* options

Charon leaves .lock file arournd

If I run "charon destroy" in a directory without a network.json file, I get the following error
[HEAD_HG batch]$ charon destroy
error: [Errno 2] No such file or directory: '........./network.json'
and it leaves a network.json.lock file laying around.

Generalised encrypted connections between machines

Charon currently sets up VPN connections over SSH for EC2 machines in different regions. It would be very useful if this were possible between arbitrary machines, e.g., EC2 machines in the same region, or non-EC2 ("physical") machines.

To ensure that we don't accidentally make unencrypted connections, Charon should generate VPN IP addresses and use those in /etc/hosts and other places.

Add memory size deployment option for VirtualBox backend

ad-hoc cloud is no longer supported

When specifying deployment.targetEnv = "adhoc"; charon reports this option as invalid

"charon deploy --dry-run" doesn't show what would be done

Currently "charon deploy --dry-run" just runs "nix-build --dry-run", so it shows what derivations would be built. But it should also show what machines/volumes would be created, and so on.

Add machine-readable output

Current output of "charon info" is not useful for automated processing. So some simple ASCII (or JSON / XML) output would be useful.

Support EBS-backed NixOS instances

n/t

Filesystem creation should be done from Upstart

Currently Charon performs LUKS and filesystem creation. That's annoying because it requires the necessary tools to be present in the base image (unless we copy them over beforehand using nix-copy-closure). It would be better to move this to an Upstart task executed before mountall. Then it's part of the system closure, and it simplifies Charon.

Implement --show-trace parameter

Sometimes when you make a mistake in one of the Nix expressions, you can't track its origins which can be very annoying. It would be nice to have a --show-trace option, that can you help you a bit.

Implementing this is not so hard I think. Just pass this parameter to any nix-build invocation.

Add a libvirt or QEMU/KVM backend

VirtualBox is pretty but buggy. A QEMU/KVM backend would be nicer.

Support recursive Nix

Nix builders should be able to call Nix to build things. This is essential if we want to use Nix as a "low-level" build tool (i.e. as a Make replacement), since then we need to support derivations that unpack a source distribution containing a Nix expression to do the rest of the build.

Add support for managing elastic IPs

Might be nice if Charon could automatically allocate/deallocate elastic IPs.

Race in Charon evaluation

It is possible for the Nix expressions defining the Charon network to change between the first and second evaluation, for instance if you edit the expressions while running "charon deploy" in the background. Example scenario (which happened to me):

• Run "charon deploy".
• "charon deploy" performs the first evaluation to get the deployment parameters.
• At the same time edit the network.nix to add a new EBS volume to the definition.
• "charon deploy" performs the second evaluation to build and deploy /etc/fstab, mountall.conf and so on.
• The configuration activated on the EC2 machine now refers to an EBS volume that doesn't exist.

Kinda tricky to prevent this. We could at least check after the second evaluation that the timestamps on the network expressions haven't changed.

Add a manpage for the Charon NixOS config options

Charon's deployment.* options used to be part of NixOS and so included in the configuration.nix(5) manpage, but now they're part of the Charon distribution. So a separate manpage should be generated.

Elastic IP: wait until the instance has started

Attaching an elastic IP while the machine hasn't started yet seems to fail randomly. So wait until it's started.

nix-1.1

Lock the JSON file to prevent races

We already lock the JSON file if we rewrite it, but this doesn't prevent multiple simultaneous invocations of "charon deploy" from messing with each other. So an exclusive lock should be acquired globally.

Don't interrupt activation

When activation fails on one machine, we should still let the other machines continue. Interrupting activation of a configuration can give an inconsistent system.

Support tags for EC2 machines

Add command to "rename" (promote) a machine

It would be useful to have the ability to change the logical name of a deployed machine in the state file. For instance, in a network consisting a primary and secondary (hot standby) database server, if the primary machine fails, we could fail over like this:

• do "charon rename primary primary-failed"
• do "charon rename secondary primary"
• do "charon deploy -k"; this will update the old secondary machine to start acting as the primary, destroy the old primary ("primary-failed"), and start a new secondary.

This is also useful to support machine name changes in the logical model without destroying/creating machines.

Add option for 'persistent' disks

Add option to specify a persistent disk (like EBS), with size and maybe (?) FS type as configuration options.

With --include / --exclude, only build the configs of the selected machines

n/t

Add tests

Charon needs automated tests. Only problem is that testing the VirtualBox and EC2 backends is hard in a Hydra job.

Ask for confirmation on destruction actions

Maybe Charon should (by default) ask for interactive confirmation when it tries to do something destructive (like deleting an EBS volume). Of course, for non-interactive use there should be a flag to force the answer to yes or no.

Support Elastic IPs

n/t

Network state available inside charon spec in nix

It would be nice to be able to refer to instance state, like instance-id / region / etc from within the charon network specification.

Detached EBS disks should be re-attached when deploy

It might happen sometimes that an EC2 EBS volume get manually (outside of charon) detached. It would be nice if charon could recover from this, by re-attaching the volumes that are supposed to be there, assuming they are still available. Perhaps a stop/start is needed to do this cleanly.

Add documentation

Add --force option to perform 'dangerous' stop/start actions automatically

Some actions, like resizing EC2 instance, produce a warning to stop an instance before redeploying. It would be nice if charon can stop the machine for us, and then resize the machine.

Support multiple EC2 accounts

It should be possible to specify the EC2 account to use (i.e. the EC2 access keys) in the deployment model.

Link to charon website in README produces 404 error.

Steps to reproduce:

1. View project README.
2. Click 'Charon website' link
3. See that http://nixos.org/charon returns an Error 404 page.

Option to query NixOS configuration value in network

It would be nice to be able to query what the value is of a NixOS configuration value similar to nixos-option, e.g.

charon option machine1.services.logstash.enable

which would show the value of services.logstash.enable on machine1. Perhaps even one that allows querying it for all machines.