Stargz Snapshotter

Pulling image is one of the time-consuming steps in the container lifecycle. Research shows that time to take for pull operation accounts for 76% of container startup time[FAST '16]. Stargz Snapshotter is an implementation of snapshotter which aims to solve this problem by lazy pulling leveraging stargz image format by CRFS. The following histogram is the benchmarking result for startup time of several containers measured on Github Actions, using Docker Hub as a registry.

legacy shows the startup performance when we use containerd's default snapshotter (overlayfs) with images copied from docker.io/library without optimization. For this configuration, containerd pulls entire image contents and pull operation takes accordingly. When we use stargz snapshotter with stargz images we are seeing performance improvement on the pull operation because containerd can start the container before the entire image contents locally available and fetches each file on-demand. But at the same time, we see the performance drawback for run operation because each access to files takes extra time for fetching them from the registry. When we use the further optimized version of images(estargz) we can mitigate the performance drawback observed in stargz images. This is because stargz snapshotter prefetches and caches some files which will be most likely accessed during container workload. Stargz snapshotter waits for the first container creation until the prefetch completes so create sometimes takes longer than other types of image. But this wait only occurs just after the pull completion until the prefetch completion and it's shorter than waiting for downloading all files of all layers.

The above histogram is the benchmarking result on the commit 288c338. We are constantly measuring the performance of this snapshotter so you can get the latest one through the badge shown top of this doc. Please note that we sometimes see dispersion among the results because of the NW condition on the internet and the location of the instance in the Github Actions, etc. Our benchmarking method is based on HelloBench.

Stargz Snapshotter is a non-core sub-project of containerd.

Quick Start with Kubernetes

For using stargz snapshotter on kubernetes nodes, you need the following configuration to containerd as well as run stargz snapshotter daemon on the node. We assume that you are using containerd newer than at least commit d8506bf as a CRI runtime.

# Plug stargz snapshotter into containerd
# Containerd recognizes stargz snapshotter through specified socket address.
# The specified address below is the default which stargz snapshotter listen to.
[proxy_plugins]
  [proxy_plugins.stargz]
    type = "snapshot"
    address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"

# Use stargz snapshotter through CRI
[plugins."io.containerd.grpc.v1.cri".containerd]
  snapshotter = "stargz"

This repo contains a Dockerfile as a KinD node image which includes the above configuration. You can use it with KinD like the following,

$ docker build -t stargz-kind-node https://github.com/containerd/stargz-snapshotter.git
$ kind create cluster --name stargz-demo --image stargz-kind-node

Then you can create stargz pods on the cluster. In this example, we create a stargz-converted Node.js pod (stargz/node:13.13-esgz) as a demo.

apiVersion: v1
kind: Pod
metadata:
  name: nodejs
spec:
  containers:
  - name: nodejs-stargz
    image: stargz/node:13.13.0-esgz
    command: ["node"]
    args:
    - -e
    - var http = require('http');
      http.createServer(function(req, res) {
        res.writeHead(200);
        res.end('Hello World!\n');
      }).listen(80);
    ports:
    - containerPort: 80

The following command lazily pulls stargz/node:13.13.0-esgz from Docker Hub and creates the pod so the time to take for it is shorter than the original image library/node:13.13.

$ kubectl --context kind-stargz-demo apply -f stargz-pod.yaml && kubectl get po nodejs -w
$ kubectl --context kind-stargz-demo port-forward nodejs 8080:80 &
$ curl 127.0.0.1:8080
Hello World!

Creating stargz images and further optimization

For lazy pulling images, you need to prepare stargz images first. You can use CRFS-official stargzify command or our ctr-remote command which has further optimization functionality. You can also try our pre-converted images listed in this doc.

In this section, we introduce ctr-remote command for converting images into stargz with further optimization for the performance of reading files. On-demand lazy pulling improves the performance of pull but it has runtime performance penalty because reading files induce remotely downloading contents. For solving this, ctr-remote has workload-based optimization for images. This section shows how to convert and pull an image lazily using ctr-remote command and briefly describes workload-based optimization.

First, prepare the demo environment with the following command (put this repo on ${GOPATH}/src/github.com/containerd/stargz-snapshotter).

$ cd ${GOPATH}/src/github.com/containerd/stargz-snapshotter/script/demo
$ docker-compose build containerd_demo
$ docker-compose up -d
$ docker exec -it containerd_demo /bin/bash
(inside container) # ./script/demo/run.sh

Generally, container images are built with purpose and the workloads are defined in the Dockerfile with some parameters including entrypoint command, environment variables and user. By default, ctr-remote optimizes the performance of reading files that are most likely accessed in the workload defined in the Dockerfile. You can also specify the custom workload using options.

The following example converts the legacy library/ubuntu:18.04 image into stargz. The command also optimizes the image for the workload of executing ls on /bin/bash. The thing actually done is it runs the specified workload in a sandboxed environment and profiles all file accesses. Then these files are marked in the image as likely accessed also in production. Then it pushes the converted image to the local registry (registry2:5000). The converted image is still docker-compatible so you can run it with other runtimes (e.g. Docker).

# ctr-remote image optimize --plain-http --entrypoint='[ "/bin/bash", "-c" ]' --args='[ "ls" ]' ubuntu:18.04 http://registry2:5000/ubuntu:18.04

Finally, the following commands pull the stargz image lazily. Stargz snapshotter prefetches files that are most likely accessed in the optimized workload, which hopefully increases the cache hit rate for that workload and mitigates runtime overheads as shown in the benchmarking result shown top of this doc.

# time ctr-remote images rpull --plain-http registry2:5000/ubuntu:18.04
fetching sha256:728332a6... application/vnd.docker.distribution.manifest.v2+json
fetching sha256:80026893... application/vnd.docker.container.image.v1+json
# ctr-remote run --rm -t --snapshotter=stargz registry2:5000/ubuntu:18.04 test /bin/bash
root@8dab301bd68d:/# ls
bin  boot  dev  etc  home  lib  lib64  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var

Authentication

We support the following methods for private repository authentication.

Using DOCKER_CONFIG or ~/.docker/config.json
Using Kubernetes secrets (type = kubernetes.io/dockerconfigjson)

Following example enables stargz snapshotter to access to private registries using docker login command. Stargz snapshotter gets credentials from DOCKER_CONFIG(or ~/.docker/config.json).

# docker login
(Enter username and password)
# ctr-remote image rpull --user <username>:<password> docker.io/<your-repository>/ubuntu:18.04

Following configuration enables stargz snapshotter to access to private registries using kubernetes secrets (type = kubernetes.io/dockerconfigjson) in the cluster using kubeconfig files. You can specify the path of kubeconfig file to use with kubeconfig_path option. It's no problem that the specified file doesn't exist when this snapshotter starts. In this case, snapsohtter polls the file until actually provided. This is useful for some environments (e.g. single node cluster with containerized apiserver) where stargz snapshotter needs to start before everything, including booting containerd/kubelet/apiserver and configuring users/roles. If no kubeconfig_path is specified, snapshotter searches kubeconfig files from KUBECONFIG or ~/.kube/config.

[kubeconfig_keychain]
enable_keychain = true
kubeconfig_path = "/etc/kubernetes/snapshotter/config.conf"

We don't share credentials with containerd so credentials specified by ctr's --user option in the above example is just for containerd's side. If you have no right to access to the repository with credentials specified to stargz snapshotter, pull operations fall back to the normal one(i.e. overlayfs).

Project details

Stargz Snapshotter is a containerd non-core sub-project, licensed under the Apache 2.0 license. As a containerd non-core sub-project, you will find the:

information in our containerd/project repository.

nghialv / stargz-snapshotter Goto Github PK