clearcontainers/runtime

OCI (Open Containers Initiative) compatible runtime using Virtual Machines

Project no longer maintened

WARNING:

This project is no longer maintained.

All users are encouraged to migrate to Kata Containers.

See the Kata Containers project installation documentation for further details.

To see the old Clear Containers README, click here.

runtime's Issues

Support process.consoleSize

From @jodh-intel on October 18, 2016 8:5

See: https://github.com/opencontainers/runtime-spec/blob/master/config.md#process-configuration

process.consoleSize is specified by docker 1.12.2.

Copied from original issue: intel/cc-oci-runtime#330

Implement the runtime ps command

ps shows all processes running inside a container.

Enable networking PnP tests

We want to be able to run the networking metrics tests on CC3.x like we did for CC2.x

linux.seccomp not supported

From @jodh-intel on August 30, 2016 16:48

See: https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#seccomp

Copied from original issue: intel/cc-oci-runtime#193

Evaluate use of capabilities in executables

We should probably do a review of our executables 'capabilities', and start to remove any that are not needed (to reduce attack surface).
On the host side we should check what we can do for:

• the runtime
• the shim
• the proxy

and on the guest side we probably need to set or remove the capabilities around the workload according to the configuration (from the OCI file for instance) that is passed in/requested of us.

Enable and tune KSM

Problem Statement

All Clear Containers share the same kernel and guest OS image, and thus the hypervisor will allocate anonymous pages for both of those components. For many of them, those pages could be shared and deduplicated through the Kernel Same Page Merging (KSM) technology. Unfortunately, we are currently not taking full advantage of it for the following reasons:

1. Many distribution OSes do not enable KSM by default
2. Enabling KSM is only the first step. We need to also tune KSM to efficiently reduce our per-container memory consumption

Solution

Both our runtime and proxy components can gather pieces of information about the current number of running/started/starting Clear Containers, the number of started containers during the last N seconds, etc...
By combining this data with at least the host available memory and CPU utilization, we should implement a ksm tuning loop that will:

1. Trigger KSM scans, depending on the number of started Clear Containers since the last scan and the number of already running containers.
2. Dynamically set the KSM scan period, depending on the system available memory, the CPU utilization and the rate of Clear Containers start (how many CC do we start per minute). We can even decide to disable periodic scanning (if possible) and control it fully.

Minimizing KSM impact

Running KSM frequently can have a significant impact on the overall system performance. We need to take advantage of KSM memory overhead reduction while at the same time minimizing the number of pages the KSM kernel thread wants and can scan:

• Enhance the KSM kernel interface to trigger it only on specific PIDs.
• Do not systematically marge all QEMU allocated pages as mergeable and be a little smarter about it.

cc @jtkukunas

Kernel 4.9 crashes inside the VM when used with Semaphore CI

For some unknown reasons, we cannot use the latest kernels for Clear Containers guest kernel when using Semaphore CI. Indeed, using some 4.9 kernels (maybe something changed in the config and it is not related to the kernel version), makes the kernel crashing inside the VM, while it works perfectly with kernel 4.5-50. This was the kernel released just before we switch to 4.9.4-53.

Implement the runtime create command

Implement the runtime create command according to the OCI specs.

Include memory consumption and boot time to our CI

Gate runtime, @clearcontainers/proxy, @clearcontainers/shim PR merges by basic PnP runs: Container density and boot time for docker run busybox true.

cc @jtkukunas

Implement the runtime delete command

Implement the runtime delete command according to the OCI specs.

Network support

Implement Clear Containers networking support:

• Create container networking namespace.
• Run networking pre-hooks.
• Discover the created network interfaces.
• Add the related tap interfaces.
• Build the integration bridge.
• Add the pre-hook created interfaces and the tap ones to the integration bridge.

Should this be done from virtcontainers ?

implement "list" command.

Publish the Clear Containers PnP scripts

So that one can easily reproduce and track PnP locally.

'clr-oci-runtime ps CID' shows empty

From @devimc on July 18, 2016 18:53

ps command has not implementation

Copied from original issue: intel/cc-oci-runtime#24

Switch to an initrd image for the guest VM

Compare a full OS image vs an uncompressed intrd/initramfs one for:

• Memory consumption
• Boot time
• CPU utilization

We may be ready to accept slight boot time degradation for significant memory consumption reduction.

cc @jtkukunas

Enable boot time PnP test

We want to be able to run the boot time tests for CC 3.x like we did for CC 2.x.

linux.readonlyPaths not supported

From @jodh-intel on August 30, 2016 16:49

See: https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#readonly-paths

Copied from original issue: intel/cc-oci-runtime#196

Implement the runtime state command

Implement the runtime state command according to the OCI specs.

Add networking benchmarks

With Clear Containers we are pushing all network packets between the container and the host through a Linux bridge, with or without SRIOV.
Having basic networking benchmarks (iperf based ?) for Clear Containers with and without SRIOV would be needed.

cc @jtkukunas

Improve runtime unit tests code coverage

Runtime's unit testing only covers 35% of the runtime code today.
The code will need some refactoring but this needs to be improved.

Use a basic VM as part of our memory baseline

We want to measure the overall Clear Containers (QEMU, guest VM, runtime, shim, proxy) memory overhead, but we should also measure our memory overhead compared to a very basic QEMU instance.
This will give us the runtime+shim+proxy+guest VM overhead.

Our basic QEMU VM should not even boot a kernel, but stop at the firmware level.

namespaces are not created as specified by OCI spec

From @jodh-intel on August 30, 2016 16:35

The following namespaces are not honoured:

• pid (see #143).
• mount
• ipc
• uts
• user
• cgroup

See: https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#namespaces.

Note that the network namespace is created outside the VM - this is mandatory to work with docker and have working networking.

Copied from original issue: intel/cc-oci-runtime#189

Implement the runtime run command

run creates and starts a container

process.capabilities not implemented

From @jodh-intel on August 30, 2016 16:26

See https://github.com/opencontainers/runtime-spec/blob/master/config.md#process-configuration.

Copied from original issue: intel/cc-oci-runtime#184

Add I/O benchmarks

We should add I/O tests for measuring disk/volumes and filesystem throughput.
We know 9pfs is slow and we hope to be able to improve it over time, and thus we need to include benchmarks for it to our PnP suite.

cc @jtkukunas

Modify Travis build to make use of docker

From @jodh-intel on September 29, 2016 9:32

Travis is an excellent facility, but the fact that the newest environment they support is Ubuntu Trusty (14.04) is problematic given the old versions of tooling and libraries.

We should use Travis to trigger a build and test run inside a docker image since that way we can use any environment we wish (even multiple).

Related: #255, #295, #114, #155.

Copied from original issue: intel/cc-oci-runtime#305

linux.sysctl not supported

From @jodh-intel on August 30, 2016 16:47

See: https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#sysctl

Copied from original issue: intel/cc-oci-runtime#192

Define, document and publish reference workloads for our benchmarks

We should define and publish a short set of custom workloads for our benchmarks.

Define

For example:

• A minimal workload: A busybox container that sleeps for one hour
• An I/O specific workload
• A networking workload

Document and publish

Each of those workloads should:

• Have their Dockerfile published here: https://github.com/clearcontainers/dockerfiles
• Be pushed to https://hub.docker.com/u/clearcontainers/
• Be the only used workloads in our benchmarking scripts

Host networking is not supported

From @sameo on January 24, 2017 15:44

We do not support running Clear Containers within the host network (docker --net=host run foo).
Although this sounds like a corner case, it seems that Kubernetes starts at least one such container per node (for kube-proxy) in order to mess with the host iptables and routes.

Copied from original issue: intel/cc-oci-runtime#617

Implement the runtime kill command

Implement the runtime kill command according to the OCI specs.

Add more memory consumption benchmarks

cc @jtkukunas

To be completed with more tests

Memory consumption exponential scaling

Does our memory consumption scale with the number of started containers ? With KSM enabled the average memory consumption per container should decrease with the number of containers.

We should measure the total average memory consumption per container for 1, 10, 100, 500 and 1000 running containers.

Memory density

We want to be able to measure Clear Containers memory density.
For a given machine (Available RAM, distro, etc...), we should measure how many Clear Containers can be started before we crash or before docker refuses to launch more.
This benchmark should be run with at least 2 parameters:

• How many containers do we launch at the same time (container batch) ?
• How long do we wait between each container batch ?

the user and group settings in config.json are not honoured.

From @jodh-intel on July 18, 2016 16:58

Specifically, process.user.uid, process.user.gid and process.additionalGids are ignored.

See:

• clr_oci_create_container_workload() in src/oci.c.
• https://github.com/opencontainers/runtime-spec/blob/master/config.md#user.

Copied from original issue: intel/cc-oci-runtime#20

Missing support for a runtime configuration file

The runtime needs a configuration file for runtime specific settings:

• Hypervisor

  • hypervisor path
  • Guest kernel path
  • Guest OS image path

• Agent

  • Agent configuration (socket paths)

Instrument QEMU for reducing the number of VM exits

In order to improve boot time, we want to instrument QEMU and understand where we can reduce the number of VM exits by e.g. tuning VT-x to skip some of them.

cc @jtkukunas

Enhance PnP memory footprint data

Besides the raw memory consumption in KBytes, we should measure more fine grained data:

• Total shared pages
• Total anonymous (private) pages

We would also need to get how those pages are distributed between the process mappings. For example how many anonymous pages are used by glibc for a given QEMU process. This will allow us to understand where e.g. our QEMU processes are consuming the most pages:

• Per mapping shared memory
• Per mapping private pages

Finally, we need to add huge pages consumption to that data.

Use Clear Linux, Fedora and Ubuntu as host distros

We should use more than one distro for measuring our Clear Containers performance data.
Clear Linux, Fedora and Ubuntu for example.

cc @jtkukunas

Implement default config generation command

We should have a runtime command (runtime config) for generating a default configuration with all options set to reasonable values. For binary paths (shim, proxy), we should use the default Clear Containers values.

Add more boot time benchmarks

cc @jtkukunas

To be completed with more tests

Measure boot time scalability

On a given system running N containers, how does our boot time per container increase when starting a single container? N=[10, 50, 100]

clearcontainers vs clear-containers in paths

We currently use both, sometimes it's /var/run/clearcontainers and other times /var/lib/clear-containers. Pick one and stick with it. A quick git grep suggest we like clear-containers better.

support linux.cgroupsPath and linux.resources (resource constraints)

From @jodh-intel on August 22, 2016 13:27

The OCI config file, config.json, specifies resource constraints in the form of linux.resources JSON objects which map closely to cgroups.

We need to find a way to implement such constraints.

See:

• https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#control-groups

Copied from original issue: intel/cc-oci-runtime#148

CRI-O integration

Integrate with Kubernetes at the CRI level, through the CRI-O project.

OCI runtime CLI implementation

Add support for the OCI CLI spec.
Use an existing Go package for command and options handling like e.g. github.com/urfave/cli.

Enable I/O PnP tests

We'd like to be able to run the I/O (storage) metrics on CC3.x like we did for CC2.x.
We'd like to be able to run for local and remote volumes and rootfs.

Annotations are not exposed to the Clear Container

From @jodh-intel on Jul 18, 2016

See https://github.com/opencontainers/runtime-spec/blob/master/config.md#annotations.

Copied from original issue: intel/cc-oci-runtime#21

Use atime on mounts if appropriate

We should check if we have 'atime' set (or more correctly, noatime not set) on our mount points. Currently we believe it is not, and timestamp updates may be trickling down to layers that really don't need any updates, and the data is lost on container quit anyhow.
@mcastelino , as he was involved in the original discussion - so please add thoughts as appropriate.

/tmp is shared between the workload process and the VM OS

From @dlespiau on August 19, 2016 17:21

We could use PrivateTmp (https://www.freedesktop.org/software/systemd/man/systemd.exec.html#PrivateTmp=) as a quick solution, but might as well take all the other init process/exec related issue and come up with a better solution.

Copied from original issue: intel/cc-oci-runtime#144

Enable memory consumption PnP tests

We want to be able to run the memory consumption metrics for CC3.x like we did for CC2.x

linux.maskedPaths not supported

From @jodh-intel on August 30, 2016 16:49

See: https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#masked-paths.

Copied from original issue: intel/cc-oci-runtime#195

Implement the runtime start command

Implement the runtime start command according to the OCI specs.

Convert an OCI runtime configuration to a virtcontainers PodConfig

We want a generic API for converting an OCI compatible config.json file into a virtcontainers PodConfig.
The runtime will then use the PodConfig to call into virtcontainers API and manage the containers lifecycles.