weaveworks/footloose

Container Machines - Containers that look like Virtual Machines

Deprecated

This repository is no longer maintained. For a more up-to-date way to manage microVMs, please take a look at Flintlock.

footloose

footloose creates containers that look like virtual machines. Those containers run systemd as PID 1 and a ssh daemon that can be used to login into the container. Such "machines" behave very much like a VM, it's even possible to run dockerd in them :)

footloose can be used for a variety of tasks, wherever you'd like virtual machines but want fast boot times or need many of them. An easy way to think about it is: Vagrant, but with containers.

footloose in action:

Install

footloose binaries can be downloaded from the release page:

Linux

curl -Lo footloose https://github.com/weaveworks/footloose/releases/download/0.6.3/footloose-0.6.3-linux-x86_64
chmod +x footloose
sudo mv footloose /usr/local/bin/

macOS

On macOS we provide a direct download and a homebrew tap:

curl --silent --location https://github.com/weaveworks/footloose/releases/download/0.6.3/footloose-0.6.3-darwin-x86_64.tar.gz | tar xz
sudo mv footloose /usr/local/bin

or

brew tap weaveworks/tap
brew install weaveworks/tap/footloose

From source

Alternatively, build and install footloose from source. It requires having go >= 1.11 installed:

GO111MODULE=on go get github.com/weaveworks/footloose

Usage

footloose reads a description of the Cluster of Machines to create from a file, by default named footloose.yaml. An alternate name can be specified on the command line with the --config option or through the FOOTLOOSE_CONFIG environment variable.

The config command helps with creating the initial config file:

# Create a footloose.yaml config file. Instruct we want to create 3 machines.
footloose config create --replicas 3

Start the cluster:

$ footloose create
INFO[0000] Pulling image: quay.io/footloose/centos7 ...
INFO[0007] Creating machine: cluster-node0 ...
INFO[0008] Creating machine: cluster-node1 ...
INFO[0008] Creating machine: cluster-node2 ...

It only takes a second to create those machines. The first time create runs, it will pull the docker image used by the footloose containers so it will take a tiny bit longer.

SSH into a machine with:

$ footloose ssh root@node1
[root@1665288855f6 ~]# ps fx
  PID TTY      STAT   TIME COMMAND
    1 ?        Ss     0:00 /sbin/init
   23 ?        Ss     0:00 /usr/lib/systemd/systemd-journald
   58 ?        Ss     0:00 /usr/sbin/sshd -D
   59 ?        Ss     0:00  \_ sshd: root@pts/1
   63 pts/1    Ss     0:00      \_ -bash
   82 pts/1    R+     0:00          \_ ps fx
   62 ?        Ss     0:00 /usr/lib/systemd/systemd-logind

Choosing the OS image to run

footloose will default to running a centos 7 container image. The --image argument of config create can be used to configure the OS image. Valid OS images are:

• quay.io/footloose/centos7
• quay.io/footloose/fedora29
• quay.io/footloose/ubuntu16.04
• quay.io/footloose/ubuntu18.04
• quay.io/footloose/ubuntu20.04
• quay.io/footloose/amazonlinux2
• quay.io/footloose/debian10
• quay.io/footloose/clearlinux

For example:

footloose config create --replicas 3 --image quay.io/footloose/fedora29

Ubuntu images need the --privileged flag:

footloose config create --replicas 1 --image quay.io/footloose/ubuntu16.04 --privileged

footloose.yaml

footloose config create creates a footloose.yaml configuration file that is then used by subsequent commands such as create, delete or ssh. If desired, the configuration file can be named differently and supplied with the -c, --config option.

$ footloose config create --replicas 3
$ cat footloose.yaml
cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 3
  backend: docker
  spec:
    image: quay.io/footloose/centos7
    name: node%d
    portMappings:
    - containerPort: 22

If you want to use Ignite as the backend in order to run real VMs, change to backend: ignite.

cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 3
  backend: ignite
  spec:
    image: weaveworks/ignite-centos:7
    name: node%d
    portMappings:
    - containerPort: 22
  # All Ignite options shown below here are optional and can be omitted.
  # These are the defaults:
  ignite:
    cpus: 2
    memory: 1GB
    diskSize: 4GB
    kernel: weaveworks/ignite-ubuntu:4.19.47

This configuration can naturally be edited by hand. The full list of available parameters are in the reference documentation.

Examples

Interesting things can be done with footloose!

• Customize the OS image
• Run Apache
• Specify which ports on the hosts should be bound to services
• Use Ansible to provision machines
• Run Docker inside footloose machines!
• Isolation and DNS resolution with custom docker networks
• OpenShift with footloose

Under the hood

Under the hood, Container Machines are just containers. They can be inspected with docker:

$ docker ps
CONTAINER ID    IMAGE                        COMMAND         NAMES
04c27967f76e    quay.io/footloose/centos7    "/sbin/init"    cluster-node2
1665288855f6    quay.io/footloose/centos7    "/sbin/init"    cluster-node1
5134f80b733e    quay.io/footloose/centos7    "/sbin/init"    cluster-node0

The container names are derived from cluster.name and cluster.machines[].name.

They run systemd as PID 1, it's even possible to inspect the boot messages:

$ docker logs cluster-node1
systemd 219 running in system mode.
Detected virtualization docker.
Detected architecture x86-64.

Welcome to CentOS Linux 7 (Core)!

Set hostname to <1665288855f6>.
Initializing machine ID from random generator.
Failed to install release agent, ignoring: File exists
[  OK  ] Created slice Root Slice.
[  OK  ] Created slice System Slice.
[  OK  ] Reached target Slices.
[  OK  ] Listening on Journal Socket.
[  OK  ] Reached target Local File Systems.
         Starting Create Volatile Files and Directories...
[  OK  ] Listening on Delayed Shutdown Socket.
[  OK  ] Reached target Swap.
[  OK  ] Reached target Paths.
         Starting Journal Service...
[  OK  ] Started Create Volatile Files and Directories.
[  OK  ] Started Journal Service.
[  OK  ] Reached target System Initialization.
[  OK  ] Started Daily Cleanup of Temporary Directories.
[  OK  ] Reached target Timers.
[  OK  ] Listening on D-Bus System Message Bus Socket.
[  OK  ] Reached target Sockets.
[  OK  ] Reached target Basic System.
         Starting OpenSSH Server Key Generation...
         Starting Cleanup of Temporary Directories...
[  OK  ] Started Cleanup of Temporary Directories.
[  OK  ] Started OpenSSH Server Key Generation.
         Starting OpenSSH server daemon...
[  OK  ] Started OpenSSH server daemon.
[  OK  ] Reached target Multi-User System.

Run real VMs with Ignite

FAQ

Is footloose just like LXD?

In principle yes, but it will also work with Docker container images and on MacOS as well.

Help

We are a very friendly community and love questions, help and feedback.

If you have any questions, feedback, or problems with footloose:

• Check out the examples.
• Join the discussion
  • Invite yourself to the Weave community Slack.
  • Ask a question on the #footloose Slack channel.
  • Join the Weave User Group and get invited to online talks, hands-on training and meetups in your area.
• File an issue.

Weaveworks follows the CNCF Code of Conduct. Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting a Weaveworks project maintainer, or Alexis Richardson ([email protected]).

Your feedback is always welcome!

footloose's Issues

ssh: Validate the machine name

We should check the machine name correspond to the configuration.

Find a way to create a test that runs on all supported base images

While some tests need OS-specific commands to be executed (ie. to install an RPM), a lot of tests are really the same, just changing the container docker image.

https://github.com/dlespiau/footloose/blob/master/tests/test-create-delete-amazonlinux2.cmd
https://github.com/dlespiau/footloose/blob/master/tests/test-create-delete-centos7.cmd
https://github.com/dlespiau/footloose/blob/master/tests/test-create-delete-ubuntu18.04.cmd.

We should find a way to write that test only once, but executed for each image.

Maybe https://github.com/dlespiau/footloose/blob/master/tests/test-create-delete-all.cmd and, we'd detect within that test, %image would be replaced by the image name. We may be able to avoid having a special name ending with -all for the test: if we detect the %image replacements is really an array, we could instantiate tests for each element of the array. That would make it a bit more generic so the e2e tests could be reused elsewhere.

Verbose option

We should have a verbose option that shows everything we do, including the commands being run on the host and in the containers.

Update kind docker package after the Inspect quote fix

I tuned a bit kind's docker.Inpect in kubernetes-sigs/kind@7e8624b, we should update to this version.

Document how to write a new tests

tests/ should have a readme explaining:

• How to run the tests (footloose in path, go test -v)
• How to create a new tests
• Document the various variables available in the .cmd files.

Users should be able to use an environment variable to point at the cluster description

Say export FOOTLOOSE_CLUSTER=$PWD/footloose.yaml

Document what is needed to docker in docker

The main thing is to create a /var/lib/docker volume.

Run provisioning steps at machine creation

Extracted from #76

footloose could run provisioning commands when creating a machine. That means we could run an ansible playbook on machine creation automatically.

Of course there are security concerns and we shouldn't allow to execute arbitrary commands. By default I think we shouldn't allow any command to be run unless the user tells us it's ok:

$ footlose create --allow-provisioning=ansible # <- name(s) of the command(s) the user allows us to exec.

Release binaries on tag

I like the pattern of pushing a tag and get CI to attach binaries to the Github release. Should do just that!

Expose user creation in footloose.yaml

It'd be nice to be able to provision non-root users when creating container machines so we can login as non-root. Maybe something like:

cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 3
  spec:
    image: quay.io/footloose/centos7
    name: node%d
    portMappings:
    - containerPort: 22
    groups:
    - name: damien
      id: 1001
    users:
    - name: damien 
      id: 1001
      gid: 1001
      groups:
      - docker    

This has some overlap with #11. I'm not sure how useful it is to create a user with a fixed (id,gid) compared to a dynamic user that mirrrors the current user inside the container.

Maybe we want both and they should be different features. The issue with implementing #11 with the idea above is that the configuration file would change depending on which machine/user is using the configuration.

This could mean #11 may be implemented in a different way:

cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 3
  spec:
    image: quay.io/footloose/centos7
    name: node%d
    portMappings:
    - containerPort: 22
    mirrorUser: true

mirrorUser (or any better name!) would instruct footloose to mirror the user running footloose inside the CM.

Add ubuntu 16.04 base image

Just like ubuntu 18.04, we should add the 16.04 LTS base image to footloose.

Re-do the asciicinema capture with the new config create command

The capture is a bit old and the CLI has seen two changes since then:

• footloose config create
• footloose ssh root@node0

It'd be nice to keep that capture in sync.

example: footloose + ansible

It'd be nice to have an exemple showing how to set up ansible with footloose machines.

Examples should go in examples/ and should be linked to from the README.

Add a create sub-command to config

config can only be used to create a new config file at the moment, but we probably should also have something that can retrieve config values and something that can change an already existing config file.

A nice to do that would be to add sub-commands to config:

$ footloose config create --replicas 3
$ footloose config get cluster.name
$ footloose config set cluster.machines[0].privileged true

These are separate tasks, this issue is about the first line.

Document how to use custom docker images

It's often useful to pre-seeed the machine image with packages we want installed. This can be easily done by deriving a footloose image and adding a few RUN steps. With then need to indicate the image we want to run in the config file.

We should document the above.

Use SIGRTMIN+3 to cleanly shutdown systemd

We should be able to use the STOPSIGNAL command to the dockerfiles to correctly shutdown systemd.

Source: https://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/

hostPort doesn't work for more than one machine and swaps its value with containerPort

Steps to reproduce the issue

$ footloose config create --replicas 2

### edit footloose.yaml to add hostPort, as per: https://godoc.org/github.com/dlespiau/footloose/pkg/config#PortMapping
$ cat footloose.yaml
cluster:
  name: cluster
  privateKey: cluster-key
machines:
- count: 2
  spec:
    image: quay.io/footloose/centos7
    name: node%d
    portMappings:
    - containerPort: 22
      hostPort: 2222

### check ports specified are free (under macOS):
$ sudo lsof -iTCP:2222 -sTCP:LISTEN
$ sudo lsof -iTCP:22 -sTCP:LISTEN
$ sudo lsof -iTCP:2223 -sTCP:LISTEN

$ footloose create
INFO[0000] Image: quay.io/footloose/centos7 present locally
INFO[0000] Creating machine: cluster-node0 ...
INFO[0001] Creating machine: cluster-node1 ...
ERRO[0001] 222c7f0c606a48beb919b4137a5d492ed985dbdf892de1d43b91ae6e2cf3a144
ERRO[0001] docker: Error response from daemon: driver failed programming external connectivity on endpoint cluster-node1 (b1bd36d39368f0af75b48987f1d970f6bb7b2fbe38e43cf952ef8c94033c8490): Bind for 0.0.0.0:22 failed: port is already allocated.
FATA[0001] exit status 125

$ docker ps
CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS              PORTS                  NAMES
426cf434fa00        quay.io/footloose/centos7   "/sbin/init"        7 seconds ago       Up 6 seconds        0.0.0.0:22->2222/tcp   cluster-node0

Expected behaviour

CONTAINER ID        IMAGE                       COMMAND             CREATED             STATUS              PORTS                  NAMES
deadbeefbabe        quay.io/footloose/centos7   "/sbin/init"        7 seconds ago       Up 6 seconds        0.0.0.0:2222->22/tcp   cluster-node0
deadbeefcafe        quay.io/footloose/centos7   "/sbin/init"        6 seconds ago       Up 5 seconds        0.0.0.0:2223->22/tcp   cluster-node1

Validate the config file

There are a few things that could do with validation, empty fields should default to reasonable values, machine names having to have a %d in them, validate machine names are valid hostnames, that both cluster name and machine names are valid docker container names, ...

Examples: split examples out of the README to their own directory/README

We're planning more examples, can't really cram them all in the README. We should split them into their own directory so we can provide example files and attach screenshots when necessary.

The main README should have the list of examples and link to them for discoverability.

Provide binaries with goreleaser

config: Add a CLI option to configure port mappings

We should be able to create configuration files with new port mappings from the command line, probably using the same syntax as docker for binding an address or supplying a port: https://docs.docker.com/config/containers/container-networking/.

footloose config create --port 8080:80
footloose config create -p 192.168.1.100:8080:80

(yes, -p is for --publish with docker, I like --port a tiny bit better for the option long name 🤷‍♂️ )

Add an Ubuntu option for the container image

Bootstrap Travis

We should start CI with some basic Travis tests: create a couple of CMs, check the containers are running, execute an ssh command.

Display the cluster status/information

We could show the cluster status and various information about the "machines", maybe.

A CRI backend

Using docker directly has been great to bring the tool to a usable state. An idea is to be able to use CRI instead to create containers, that would open the door to very interesting container runtimes out there, including ones that actually create VMs!

Add a version command

Now that we have our first release, we should get a version command! Use git describe --tags and inject the version through a linker flag.

Run the end to end tests in parallel

I took care of isolating tests enough so they should be runnable in parallel. Now we need to test that hypothesis.

Another way to do it would be to shard the tests per OS, have n test runners in CI, each of them selecting an OS.

We probably need to start with a docker pull to retrieve the OS images so they are local when the tests need them to start the machines.

Add a way to set labels on the created containers

It'd be nice to be able to select the containers footloose creates with labels, exposing them in the configuration.

Question: Should we have a default footloose label?
Question: Do they belong to the Cluster or Machine object?

Make the ssh command able to execute a remote command

it's ssh after all, we should be able to have:

$ footloose ssh node1 ls /

Ensure a set of basic commands is available on all base images

Container images tend to be quite stripped down, but for container machines, we want a basic set of tools to be available. Defining the exact criteria to decide if a command should be there or not is going to be tricky so we'll have to play it by ear. eg.

• hostname
• ps
• ifconfig/ip
• netstat?
• vi/vim
• ping
• curl
• wget

Support and document docker in docker

Create a terraform provider

That's quite a bit of work but it'd be nice to have a footloose terraform provider so people can fake terraforming VMs on their development machine.

add a list command

to integrate with external tools, it would be very useful to have a 'list' command that outputs all the machine hostname, and maybe optionally the current host-container port mapping. This can be done with docker but there can be other containers running, we need only the ones created by footloose.

Add a Fedora option for the base image

Make the user explicit in the ssh command

At the moment we hardcode "root@nodeIP" as the ssh argument, we should probably make the user explicit and default to the current user, like the ssh command does, to avoid surprising the user:

$ footloose ssh root@nodes

This should also play better with the plan of letting users login with heir own id/gid as described in #11.

Add a set sub-command to config

Following on #27, we should have a set sub-command:

$ footloose config set cluster.machines[0].privileged true

Set the machine hostname

We should set the machine hostnames to config.machines[i].Name instead of leaving them to their default value of the container ID.

Make create and delete idempotent

At the moment, running create while the cluster is running will result in an error (because the containers already exist):

$ footloose create
INFO[0000] Image: quay.io/footloose/centos7 present locally 
INFO[0000] Creating machine: cluster-node0 ...          
ERRO[0000] docker: Error response from daemon: Conflict. The container name "/cluster-node0" is already in use by container "5134f80b733ecf4561f154c7c43eca2fb5273ae8da0f7c73402b2205ceef1a3e". You have to remove (or rename) that container to be able to reuse that name. 
ERRO[0000] See 'docker run --help'.                     

Same with delete.

We should make them idempotent.

Remove displaying usage on error

It's a bit annoying to have the command usage displayed on error and makes the error not the most prominent thing that appears on the console. Should find a way to not display the usage.

Push docker images from CI

We should push the docker images used in the container machines from the CI run, not me pushing them by hand .

Add a get sub-command to config

Following on #27, we should have a get sub-command:

$ footloose config get cluster.name

Make passing option to ssh command more friendly

At the moment, cobra/pflag will bail out if we do something like:

$ footloose ssh node0 yum install -y iptables
Error: unknown shorthand flag: 'y' in -y

This can be worked around by explicitly stopping option parsing:

$ footloose ssh node0 -- yum install -y iptables

We could probably do better though? tell cobra/pfag to just leave unknown options alone?

Mount a local directory and login as the user

A very frequent test case is to be able to mount a local directory (say a git checkout, or $HOME) to expose parts of the host fs inside the machine.

There's one detail though, we are likely to want that, when writing files to that mount, we want to write them with the id, gid of the user of the host machine.

Provide an Amazon linux 2 base image

It'd be nice to provide Amazon Linux 2 footloose base images. From a quick glance, it looks like a centos/rhel based image and version 2 comes with systemd.

Took the official image for as spin and systemd isn't installed by default but it's only a yum install systemd away

Given the above, building an amazon linux 2 base image should be quite close to what we do for centos7: https://github.com/dlespiau/footloose/blob/master/images/centos7/Dockerfile. We should put it in `

A detail is that currently, only I can push images to quay.io/footloose, this should be automated and pushed from CI (#23). It doesn't prevent people from creating and using a new locally though.

Example: Custom image exposing an http service

We have a simple example of image customization by adding a package, it'd be nice to have a slightly augmented version of this installing apache and exposing port 80 to the host.

Create a 0.1.0 centos image

Releases should have docker images locked down to specific versions so users don't have the OS image changing between two invocations of footloose create.

Move to CircleCI for testing on macOS

Unfortunately, the macOS machines on Travis are VMs with no nested virtualization support. This means docker for mac doesn't work there.

travis-ci/travis-ci#7773
https://hackernoon.com/why-xod-switched-from-travis-to-circle-ci-98b6832f2d96

So we can't test footloose on macOS with travis. CircleCI should work though.

Expose port mapping to the user

We don't currently use port mapping to access the SSH daemon because, on Linux, the container IP is routable from the host and we can just open a connection.

This doesn't work on macOS, we need to map the container port 22 to a localhost port before we can login. This will also be true for any service the user would like to access from the host.

We should expose port mappings, just like we do with volumes and default to mapping port 22 to either a fixed port or a port automatically picked up by docker.

Add it to awesome docker

Hi @dlespiau,

I really like your project, would you consider adding it yourself to https://github.com/veggiemonk/awesome-docker#development-environment ?

If you have a idea for better suited section, let me know in the PR we can have discussion there with the other team members :D

Have a great day,

Julien

config shouldn't override an existing file

Two more desirable behaviours:

• Error out if a file exists
• "Merge" the existing config file with the option(s) given on the command line