gardener/gardener-extension-os-ubuntu

Gardener extension controller for the Ubuntu operating system.

Gardener Extension for Ubuntu OS

This controller operates on the OperatingSystemConfig resource in the extensions.gardener.cloud/v1alpha1 API group. It manages those objects that are requesting Ubuntu OS configuration (.spec.type=ubuntu). An experimental support for Ubuntu Pro is added (.spec.type=ubuntu-pro):

---
apiVersion: extensions.gardener.cloud/v1alpha1
kind: OperatingSystemConfig
metadata:
  name: pool-01-original
  namespace: default
spec:
  type: ubuntu
  units:
    ...
  files:
    ...

Please find a concrete example in the example folder.

After reconciliation the resulting data will be stored in a secret within the same namespace (as the config itself might contain confidential data). The name of the secret will be written into the resource's .status field:

...
status:
  ...
  cloudConfig:
    secretRef:
      name: osc-result-pool-01-original
      namespace: default
  command: /usr/bin/env bash <path>
  units:
  - docker-monitor.service
  - kubelet-monitor.service
  - kubelet.service

The secret has one data key cloud_config that stores the generation.

An example for a ControllerRegistration resource that can be used to register this controller to Gardener can be found here.

Please find more information regarding the extensibility concepts and a detailed proposal here.

---

How to start using or developing this extension controller locally

You can run the controller locally on your machine by executing make start. Please make sure to have the kubeconfig to the cluster you want to connect to ready in the ./dev/kubeconfig file. Static code checks and tests can be executed by running make verify. We are using Go modules for Golang package dependency management and Ginkgo/Gomega for testing.

Feedback and Support

Feedback and contributions are always welcome. Please report bugs or suggestions as GitHub issues or join our Slack channel #gardener (please invite yourself to the Kubernetes workspace here).

Learn more!

Please find further resources about out project here:

• Our landing page gardener.cloud
• "Gardener, the Kubernetes Botanist" blog on kubernetes.io
• "Gardener Project Update" blog on kubernetes.io
• Gardener Extensions Golang library
• GEP-1 (Gardener Enhancement Proposal) on extensibility
• Extensibility API documentation

gardener-extension-os-ubuntu's Issues

Run all `make` commands in container.

From https://github.com/gardener/gardener-extensions/issues/122

All make commands require various dev resources to be available:

• golang
• helm
• various gnu utils such as tar, gzip, tr, base64, cut, find, grep, xargs.
• gingo
• linters.

I think that the most reasonable way to get reproducible builds and things such as make generate is to run those commands in a pre-build container image.

Docker & Kubelet are using cgroupfs as Cgroup Driver.

Directly from the the source

When systemd is chosen as the init system for a Linux distribution, the init process generates and consumes a root control group (cgroup) and acts as a cgroup manager. Systemd has a tight integration with cgroups and will allocate cgroups per process. It’s possible to configure your container runtime and the kubelet to use cgroupfs. Using cgroupfs alongside systemd means that there will then be two different cgroup managers.

Control groups are used to constrain resources that are allocated to processes. A single cgroup manager will simplify the view of what resources are being allocated and will by default have a more consistent view of the available and in-use resources. When we have two managers we end up with two views of those resources. We have seen cases in the field where nodes that are configured to use cgroupfs for the kubelet and Docker, and systemd for the rest of the processes running on the node becomes unstable under resource pressure.

Changing the settings such that your container runtime and kubelet use systemd as the cgroup driver stabilized the system.

We meet the same problem with Ubuntu. Both docker and kubelet are using cgroupfs.
Docker can be easily set to use systemd as Cgroup Driver vie the Ubuntu Controller Extension but the configuration of the kubelet is comes from the gardener via OperationSystemConfig CRD and is common for all OS distributions.

Incorrect DNS configuration when using containerd in conjunction with ubuntu

How to categorize this issue?

/area os
/kind bug
/priority 3
/os ubuntu

What happened:
If containerd is chosen as container runtime and ubuntu as operating system the DNS configuration on the node is in a state that cannot be used in containers. In such an environment /etc/resolv.conf is defined as followed:

# ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Jan 31  2020 /etc/resolv.conf -> ../run/systemd/resolve/stub-resolv.conf

In ordinary configurations with docker as container runtime the same looks like this:

# ls -l /etc/resolv.conf
lrwxrwxrwx 1 root root 39 Jan 31  2020 /etc/resolv.conf -> /run/systemd/resolve/resolv.conf

/run/systemd/resolve/stub-resolv.conf contains the systemd-resolved address 127.0.0.53, which works fine on the node, but does not work in a network namespace with a different loopback interface.
/run/systemd/resolve/resolv.conf contains the infrastructure specific DNS server, which should actually be used for containers with DNS policy default.

What you expected to happen:
/etc/resolv.conf should contain (or link to a file that contains) the infrastructure specific DNS server.

How to reproduce it (as minimally and precisely as possible):
Create a shoot cluster with gardener with ubuntu as operating system and containerd as container runtime.
Please note that it may not reconcile with ReversedVPN enabled as the vpn-shoot pod will fail to resolve the control plane address. You can use ssh to get to the node, though.
It is also possible to simply add one node with this configuration (ubuntu/containerd) to an existing cluster.

Anything else we need to know?:
https://github.com/gardener/gardener-extension-os-ubuntu/blob/master/pkg/generator/templates/cloud-init-ubuntu.template seems to suggest that a symbolic link should have been created, but I do not see the "correct" link in my cluster.

Environment:

• Gardener version (if relevant):
• Extension version:
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• Others:

Enhance this controller to support also Alicloud

How to categorize this issue?

/area os dev-productivity
/kind enhancement
/priority 3
/os ubuntu

What would you like to be added:
Now when this extension is using bash script instead of cloud init, it should be able to bootstrap ubuntu nodes also on Alicloud.
The alicloud version has some configurations that are missing in this one. They are looking general enough and might be needed also for this extension.

The alicloud version is also setting PROVIDER_ID environment variable, but since some time it is properly injected by the provider-alicloud and does not need to be migrated.

Why is this needed:
To maintain only one os ubuntu extension.

Manage /etc/hosts via cloud-init

What would you like to be added:
On some cloud providers, the /etc/hosts file might be missing which then prevents kubelet to properly start the containers of the pods, including system pods like kube-proxy and calico-node.

Warning  Failed     2m1s (x12 over 4m10s)  kubelet            Error: open /etc/hosts: no such file or directory

To prevent such situation, we can let cloud init to manage this file by https://cloudinit.readthedocs.io/en/latest/topics/modules.html#update-etc-hosts, e.g.

#cloud-config
...
manage_etc_hosts: true
...

Why is this needed:
To make nodes more resilient to the underplaying VM image changes.

Thanks to @kayrus for investigating and providing the above suggestion.

Switch from cloud-config to bash scripts

How to categorize this issue?

/area os
/kind enhancement
/priority normal
/os ubuntu

What would you like to be added:
To migrate from cloud-config to bash scripts as they can be used on all hyperscalers. This will also let us to archive https://github.com/gardener/gardener-extension-os-ubuntu-alicloud.

Why is this needed:

Failing to populate the command, cloudConfig and units in the OSC status

How to categorize this issue?

/area os
/kind bug
/priority 3
/os ubuntu

What happened:
Shoot creation is failing with

Error while waiting for OperatingSystemConfig shoot--foo--bar/cloud-config-worker-1-a93d4-downloader to become ready: no cloud config information provided in status.

For some reason, the OSC status is not updated properly here

gardener-extension-os-ubuntu/vendor/github.com/gardener/gardener/extensions/pkg/controller/operatingsystemconfig/reconciler.go

Line 146 in 168ece1

setOSCStatus(osc, secret, command, units)

even though the resource is successfully reconciled by the extension. There is no error also in the extension's logs.

What you expected to happen:
The osc.status.cloudConfig to be properly populated.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Gardener version (if relevant): master
• Extension version: 168ece1
• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration: Observed on Alicloud and OpenStack, but most probably is not cloud specific.
• Others: