gardener/etcd-custom-image

Custom etcd container image

etcd-custom-image

This repository is used to produce a Docker image that contains the etcd binary along with pre-installed CLI tools such as bash, curl and wget.

The rationale behind it is that Gardener is not only used to operate shoot and seed clusters in public clouds, but also in private restricted environments.

Since etcd v3.4 release, the etcd image has moved away from an alpine-linux based image to a debian-based base image. The new base image lacks CLI tools like bash and wget out-of-the-box, which are required for communication and coordination between the etcd container and the backup-restore container that is deployed as a sidecar to etcd.

Some restricted environments forbid access to public apt repositories, which need to be contacted in order to install these additional binaries. Hence, this repository defines a CI pipeline that regularly fetches the latest etcd release image, installs the necessary additional binaries and builds a "custom etcd" Docker image from that.

etcd-custom-image's Issues

Fetching etcd configuration from backup-restore is prone to transient errors

What happened:

As part of the bootstrap, a REST endpoint hosted by backup-restore container is invoked:

curl "$BACKUP_ENDPOINT/config" -o $CONFIG_FILE

Backup-Restore container queries the current etcd StatefulSet to figure out the correct initial cluster state. It then modifies the etcd configuration with this state and returns to its caller. It is possible that due to transient issues with the API server, it is unable to get the StatefulSet. Currently the bootstrapping fails if there is a transient error.

What you expected to happen:

Bootstrapping process should be tolerant (to a limited extent) to transient failures that are returned from backup-restore container.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know:

Environment:

Upgrade to etcd v3.4.26 to fix vulnerabilities from go runtime

What would you like to be added:

Hi colleagues, the current etcd image (eu.gcr.io/gardener-project/gardener/etcd:v3.4.13-bootstrap-10) maintained by gardener has around 140+ vulnerabilities (mainly from go runtime).

I check the official etcd v3.4.26 is using the latest go runtime to fix all these vulnerabilities, is it possible to upgrade to this version?

BTW I also saw this issue which slowly upgrade to v3.6.x to avoid vulnerabilities from base image, is there any timeline?

Why is this needed:

fix vulnerabilities from go runtime for security compliance.

Currently lost track of releases

What would you like to be added:
The current version of etcd is v3.5.6 (https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md#v356-2022-11-21). The latest gardener releases version of etcd is v3.5.2

Why is this needed:
Patches,Bugfixes, you name it :)

Transition to Distroless image

What would you like to be added:
The etcd-custom-image should be based on a distroless image. See also etcd-io/etcd#13556.

Why is this needed:

• To reduce the common perception of an extended attack surface of non-distroless images.
• To reduce maintenance effort of updating images and tooling.

This issue depends on #16.

Implement etcd_bootstrap_script.sh in Golang

What would you like to be added:
The current bootstrap script etcd_bootstrap_script.sh should be replaced by a Golang based implementation.

Why is this needed:
The etcd_bootstrap_script.sh has several dependencies to OS utilities like bash, curl, etc. This makes it very hard to change to Distroless container base image in the future. The etcd project moved to Distroless for 3.6.x (etcd-io/etcd#13556 reduce the attack surface and we should do the same. This will also eliminate some tedious maintenance tasks like updating base images and utilities because of reported vulnerabilities.

Tasks

Project Structure

• Create project structure
• Implement bootstrap script logic
• Add signal handler
• Add command and flags
• Add readiness endpoint
• Add TLS support
• Add Makefile
• Add Dockerfile
• Add vendor directory
• Add unit tests
  • app
  • bootstrap
  • brclient
  • util
  • types
    - [x] Use 3.5.6 version of etcd for etcd-wrapper.
• Use 3.4.26 of etcd in etcd-wrapper (See #33). Move to the latest version of etcd in another increment (another PR). Backup-Restore continues to use the current client version.
• Add hack/local-dev scripts to help ease local KIND + skaffold based tests.
• CI/CD
  • Pipeline definition
  • Scripts
• Documentation
  • Ops guide
  • Operator docs
  • Developer docs
• Security requirements:
  • Change base image to distroless for both etcd-wrapper and etcd-backup-restore
  • Add an init container to change the ownership of /var/etcd/data from root to nonroot user (required for existing clusters)
  • Add pod security context to run all containers by default as nonroot.
• Add Dockerfile for ephemeral container which will be used for all diagnostics, this container will also be run as nonroot.
• Adapt etcd-druid and run druid integration and e2e
• Run g/g e2e tests
  • Run additional test where one of the pods is using etcd-wrapper and other two continue to use etcd-custom-image. This is possible during a stuck software update or a partially successful software update of the statefulset. Ensure that both can co-exists together.
  • Run additional test where one tries to roll-back from pods using etcd-wrapper to pods using etcd-custom-image.