How the magic works
This repo utilizes aggressively a number of multi-stage Docker tricks that are worth explaining, because they can be used broadly
We are doing three things in the Dockerfile, denoted by their named stage:
- Building small vpp and vpp-dbg images, including the ability to patch
- generating the govpp 'binapi' for the build vpp image, using the standard 'go:generate' idiom
- Extracting the VPP_VERSION being used so we can utilize it to tag published docker images.
Building small vpp images
Ultra small vpp images are built via three Docker stages:
- vppbuild
- vppinstall
- vpp
The reason for these stages is to cauterize the bloat from each activity.
Building vpp means bloating an image
up with a bunch of build dependencies, build artifacts, etc. Building vpp installation on top of that would
lead to a multi-GB image, which is undesirable. So we isolate that work in the 'vppbuild' stage.
Installing vpp means copying the resulting *.deb packages from the 'vppbuild' stage and installing them. Unfortunately,
because you can't combine a COPY and RUN step, that results in image bloat from the *.deb files themselves, so we isolate
that in the 'vppinstall' image.
Finally, we utilize a trick to trim out the bloat when building the 'vpp' stage.
The 'vppbuild' stage
|
FROM ubuntu:${UBUNTU_VERSION} as vppbuild |
|
ARG VPP_VERSION |
|
RUN apt-get update |
|
RUN DEBIAN_FRONTEND=noninteractive TZ=US/Central apt-get install -y git make python3 sudo asciidoc |
|
RUN git clone -b ${VPP_VERSION} https://github.com/FDio/vpp.git |
|
WORKDIR /vpp |
|
COPY patch/ patch/ |
|
RUN git apply patch/*.patch |
|
RUN DEBIAN_FRONTEND=noninteractive TZ=US/Central UNATTENDED=y make install-dep |
|
RUN make pkg-deb |
|
RUN ./src/scripts/version > /vpp/VPP_VERSION |
is a fairly standard Ubuntu oriented build. It results in a bunch of *.deb files.
The 'vppinstall' stage
|
FROM ubuntu:${UBUNTU_VERSION} as vppinstall |
|
ARG VPP_VERSION |
|
COPY --from=vppbuild /var/lib/apt/lists/* /var/lib/apt/lists/ |
|
COPY --from=vppbuild [ "/vpp/build-root/libvppinfra_*_amd64.deb", "/vpp/build-root/vpp_*_amd64.deb", "/vpp/build-root/vpp-plugin-core_*_amd64.deb", "/vpp/build-root/vpp-plugin-dpdk_*_amd64.deb", "/pkg/"] |
|
ARG VPP_VERSION |
|
RUN VPP_INSTALL_SKIP_SYSCTL=false apt install -f -y --no-install-recommends /pkg/*.deb ca-certificates iputils-ping iproute2 tcpdump; \ |
|
rm -rf /var/lib/apt/lists/*; \ |
|
rm -rf /pkg |
|
|
uses
|
COPY --from=vppbuild /var/lib/apt/lists/* /var/lib/apt/lists/ |
to copy in the indexes in /var/lib/apt/lists/ that result from having run apt-get update in the 'vppbuild' stage,
thus avoiding the cost of redownloading them.
|
COPY --from=vppbuild [ "/vpp/build-root/libvppinfra_*_amd64.deb", "/vpp/build-root/vpp_*_amd64.deb", "/vpp/build-root/vpp-plugin-core_*_amd64.deb", "/vpp/build-root/vpp-plugin-dpdk_*_amd64.deb", "/pkg/"] |
copies the *.deb files we wish to install from where they were built in 'vppbuild'
|
RUN VPP_INSTALL_SKIP_SYSCTL=false apt install -f -y --no-install-recommends /pkg/*.deb ca-certificates iputils-ping iproute2 tcpdump; \ |
installs the *.deb files:
-f causes apt-get to install any missing dependencies.-y causes apt-get to run in an unattended mode where the answer to the questions are y--no-install-recommends causes apt-get to only install required (rather than recommended) dependencies to keep the image size down
|
rm -rf /var/lib/apt/lists/*; \ |
removes the apt indexes from apt-get update
and
removes the *.deb files
There is one problem with this. Because the image still has the layers from
|
COPY --from=vppbuild /var/lib/apt/lists/* /var/lib/apt/lists/ |
|
COPY --from=vppbuild [ "/vpp/build-root/libvppinfra_*_amd64.deb", "/vpp/build-root/vpp_*_amd64.deb", "/vpp/build-root/vpp-plugin-core_*_amd64.deb", "/vpp/build-root/vpp-plugin-dpdk_*_amd64.deb", "/pkg/"] |
the 'vppinstall' stage is still going to be bloated by that amount. We solve this in the 'vpp' stage
The 'vpp' stage
The 'vpp' stage is our final lean runnable. It uses a very simple but slick trick, which has a small caveat to it.
|
FROM ubuntu:${UBUNTU_VERSION} as vpp |
|
COPY --from=vppinstall / / |
Simply starts from 'ubuntu:${UBUNTU_VERSION}' (thus reusing the layers form that standard image) and then
copies the entire '/' directory in from 'vppinstall' (which has removed the apt-get indexes and *.deb files).
Because docker COPY is generally smart enough to only copy in the changed files... the layer resulting from
|
COPY --from=vppinstall / / |
should only contain the deltas between the final result of 'vppinstall' and the starting point of 'ubuntu:${UBUNTU_VERSION}'
Resulting in something that looks like:
docker history ghcr.io/edwarnicke/govpp/vpp:v20.09 ──(Sun,Jan31)─┘
IMAGE CREATED CREATED BY SIZE COMMENT
8b4ea0febd25 42 minutes ago /bin/sh -c #(nop) COPY dir:b8f7abee062c48863… 96.2MB
<missing> 10 days ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 10 days ago /bin/sh -c mkdir -p /run/systemd && echo 'do… 7B
<missing> 10 days ago /bin/sh -c [ -z "$(apt-get indextargets)" ] 0B
<missing> 10 days ago /bin/sh -c set -xe && echo '#!/bin/sh' > /… 811B
<missing> 10 days ago /bin/sh -c #(nop) ADD file:2a90223d9f00d31e3… 72.9MB
The caveat is, there is a bug in docker that will, depending on the storage driver
you are using, copy over all the files, resulting in a much larger image. Fortunately, building in GitHub Actions does not
seem to hit this issue. Unfortunately, building in Docker for Mac does.
To fix the issue in Docker for Mac, follow the instructions for setting Docker Engine options
and set your storage driver to 'overlay'
{
"storage-driver": "overlay"
}Building idiomatic go:generate for 'binapi'
In go, we generally generate code using a //go:generate directive.
In the case of govpp, you need to use the 'binapi-generator' run against the json api files installed by the vpp debs in /usr/share/vpp/api/.
We utilize Docker to make this easy by:
- Building 'binapi-generator' as a Docker stage
- Creating a 'gen' Docker stage that copies in
/usr/share/vpp/api/ from our 'vpp' stage and run's binapi, outputting to /gen
- Add a
//go:generate line to gen.go to run the 'gen' stage to generate the code.
The 'bin-apigenerator' stage
|
FROM golang:1.15.3-alpine3.12 as binapi-generator |
|
ENV GO111MODULE=on |
|
ENV CGO_ENABLED=0 |
|
ENV GOBIN=/bin |
|
ARG GOVPP_VERSION |
|
RUN go get git.fd.io/govpp.git/cmd/binapi-generator@${GOVPP_VERSION} |
Is a pretty standard 'go get to build' stage for 'binapi-generator'
The 'gen' stage
|
FROM alpine:3.12 as gen |
|
COPY --from=vpp /usr/share/vpp/api/ /usr/share/vpp/api/ |
|
COPY --from=binapi-generator /bin/binapi-generator /bin/binapi-generator |
|
COPY --from=vppbuild /vpp/VPP_VERSION /VPP_VERSION |
|
WORKDIR /gen |
|
CMD VPP_VERSION=$(cat /VPP_VERSION) binapi-generator ${PKGPREFIX+-import-prefix ${PKGPREFIX}} |
Actually performs the generation if 'docker run'
|
COPY --from=vpp /usr/share/vpp/api/ /usr/share/vpp/api/ |
copies in the '/usr/share/vpp/api/*' json api files from the 'vpp' stage
|
COPY --from=binapi-generator /bin/binapi-generator /bin/binapi-generator |
copies in the 'binapi-generator' from the 'binapi-generator' stage
|
COPY --from=vppbuild /vpp/VPP_VERSION /VPP_VERSION |
copies in the VPP_VERSION we had stashed in the 'vppbuild' stage.
|
WORKDIR /gen |
|
CMD VPP_VERSION=$(cat /VPP_VERSION) binapi-generator ${PKGPREFIX+-import-prefix ${PKGPREFIX}} |
sets the Workdir to /gen and runs the binapi-generator.
VPP_VERSION=$(cat /VPP_VERSION) - sets the VPP_VERSION env binapi-generator needs from the stashed VPP_VERSION value
${PKGPREFIX+-import-prefix ${PKGPREFIX}} - will output -import-prefix ${PKGPREFIX} if the PKGPREFIX env variable is set, and nothing otherwise.
The //go:generate directive
|
//go:generate bash -c "docker run -e PKGPREFIX=$(go list)/binapi -v $(go list -f '{{ .Dir }}'):/gen $(docker build . -q --build-arg GOVPP_VERSION=$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git))" |
uses 'docker run' to run the 'gen' stage and generate the code
bash bash -c "docker run ... " is used to give us a shell to work with (because we are doing a lot of magic here)-e PKGPREFIX=$(go list)/binapi - sets the env variable PKGPREFIX in the docker container to the value of
$(go list)/binapi. $(go list) is the value of the package in which the gen.go file resides.-v $(go list -f '{{ .Dir }}'):/gen mounts $(go list -f '{{ .Dir }}') from the host into /gen/ in the container.
$(go list -f '{{ .Dir }}') outputs the directory the module containing gen.go is in.
$(docker build . -q --build-arg GOVPP_VERSION=$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git))
-q - 'quiet' - outputs the id of the resulting image built--build-arg GOVPP_VERSION=$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git)) - sets the build-arg GOVPP_VERSION$(go list -m -f '{{ .Version }}' git.fd.io/govpp.git) - outputs the version of git.fd.io/govpp.git in the go.mod file.
Extracting the VPP_VERSION being used in the Dockerfile
In .github/workflows/ci.yaml we want to be able to tag and push images based
on the VPP_VERSION. Because we only wish to specify the VPP_VERSION once in the Dockerfile, we have a job:
|
FROM ubuntu:${UBUNTU_VERSION} as version |
|
ARG VPP_VERSION |
|
ENV VPP_VERSION ${VPP_VERSION} |
|
CMD echo ${VPP_VERSION} |
that can be used to extract that version:
|
VPP_VERSION=$(docker run $(docker build -q . --target version)) |
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent