rancher/elemental-toolkit

:snowflake: The toolkit to build, ship and maintain cloud-init driven Linux derivatives based on container images

Elemental Toolkit

Elemental-toolkit is a toolkit to build, ship and maintain cloud-init driven Linux derivatives based on container images with a common featureset - allows container images to be bootable in VMs, baremetals, embedded devices, and much more.

It is designed to reduce the maintenance surface, with a flexible approach to provide upgrades from container registries. It is cloud-init driven and also designed to be adaptive-first, allowing easily to build changes on top.

Documentation is available at https://rancher.github.io/elemental-toolkit/docs

Design goals

• A Manifest for container-based OS. It contains just the common bits to make a container image bootable and to be upgraded from, with little customization on top
• Immutable-first, but with a flexible layout
• Cloud-init driven
• Based on systemd
• Built and upgraded from containers
• OTA updates
• Easy to customize
• Cryptographically verified

Quick start

Check out our getting-started section in the documentation.

License

Copyright (c) 2020-2024 SUSE, LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

elemental-toolkit's Issues

cos-reset

We should have a command, cos-reset that resets entirely COS_STATE (recreating images, installing grub) and optionally COS_OEM and COS_PERSISTENT.

Relates to #23

OEM customizations

A mechanism to extend cloud-init configuration mechanism ( see #4 ) for OEM customizations. This includes also executing cloud-init yaml on boot from a persistent folder, like /oem or /usr/local/oem for example.

The installation process should allow optional cloud-init config to be configured as persistent as well.

Updates delivered as docker images

To ease out this, we should deliver mudler/luet#169 first which addresses "final images" for each package. In this way an external component (or luet itself) can upgrade a system from a given docker image/package.

Needs #28 and #8 first

Kernel should create unique symlink

https://github.com/mudler/cOS/blob/85e764da2ab6698d9c3d216de0e55834a8ee25d7/packages/kernels/build.yaml#L9

Instead of symlinking with the explicit minimal kernel name, there should be a unique one. So the installer doesn't need to guess the name when creating the GRUB configuration, like in: https://github.com/mudler/cOS/blob/85e764da2ab6698d9c3d216de0e55834a8ee25d7/packages/installer/installer.sh#L137

Action items:

• Drop explicit symlink, and pick a unique one based from the package name
• Make other kernels, if presents, to conflict between each other (optional, not strictly required at the moment)
• Change the installer to use that one
• Verify installer still works

Multiple cloud-init files breaks boot

See #37 (comment)

While testing custom oem config files, seems that having multiple yamls loaded at the initramfs stage breaks boot up.

Splitting into a separate card as it seems completely unrelated to the oem mechanism itself

Switchable base os

This is about adding the relative fields to https://github.com/mudler/cOS/blob/master/values/amd64.yaml to generate different repositories with a different base os.

Action items:

• Identify where tumbleweed is hardcoded
• Replace all above occurences to use a templated field e.g. {{.Values.base_os}}. Mind to set up defaults properly in definition.yaml files and https://github.com/mudler/cOS/blob/master/values/amd64.yaml as needed
• Rename packages as necessary
• QA Boot/Install

Kernel packages

We need kernel packages from SUSE, and in case we don't use dracut (See #1) , also a -minimal variant used to build initramfs for dev. purposes

Unattended installation

Installable Systemd + base image package

LiveCDs, and specs for customization

Spike: build ARM/arm64 images with GRUB

This card is a spike on how to build images for ARM, but most importantly to find out if we can re-use our GRUB+dracut approach currently used in amd64-land in ARM devices. (maybe chainloading?)

Create issues as necessary as we find more informations and we find a way to achieve ARM images.

Additional context

• For building ARM images, have a look at https://github.com/Luet-lab/luet-embedded/blob/master/scripts/arm_image.sh for inspiration, we might want an extension for it, similarly on how we build ISOs
• https://media.ccc.de/v/946-uefi-grub2-on-raspberry-pi
• https://forum.odroid.com/viewtopic.php?t=26894
• https://www.hellion.org.uk/blog/posts/grub-on-uboot-on-qemu/
• https://packages.debian.org/jessie/grub-uboot
• http://rglinuxtech.com/?p=2595
• https://github.com/pftf/RPi4

Switch the cOS repo to a Docker repository type

Since luet 0.10.2 it's possible to have "Docker" repository types, completely hosted in a registry.

This issue is to switch from the current zstd format to the Docker native one. It also involves changing the current GHA pipelines to push to dockerhub instead of the gh-pages branch https://github.com/mudler/cOS/blob/1656cbdab0212d1ad7273bd722689c4740dae39c/.github/workflows/build.yaml#L61

To note, to push a luet repository to dockerhub, for now it's sufficient to:

luet create-repo --type docker --output "dockerimage"

And as a client, to supply the docker image in the urls field, denoting the repository type as docker

Documentation: docs website

"landing page" documentation

• a tag line
• how to installbuild
• a description of the target audience
• some use cases and link to sample repositories
• link to github docs directory

Minimal busybox

We need a minimal busybox from suse repo, which is containing all the tools required by init. This would reduce the final ISO size quite a lot. Alternativetly, we can shrink the tw image with just the minimum necessary.

While experimenting, I've noticed that everything is almost there, except for switch_root.

GRUB setup

Setup automatic bump-bot with GHA pipelines

As GHA supports cron tasks, we could setup an automatic bump bot with the luet autobumper extension running directly over GHA.

If we find a way to trigger it externally (with GHA) on releases even better.

A/B seamless Dual copy updates

Upgrades shouldn't run on the real system, either be transactional with rollbacks or just apply the upgrade on a transient rootfs (a/b) that finally gets promoted once final checks succeeds.

The rootfs partition will be "image based", that means:

• single version for whole OS, that gets upgraded from 1 package only, or either directly from a Docker Image with a custom component (doesn't have to be luet)
• update entire OS and reboot
• rollback on old partition if boot into updated system is not successful
• ships a configuration for the upgrade channel, that can be override with a cloud-init approach

Action items

• Create release-channel file, which is a luet config pointing to the DockerHub repository we use to release final-images ( https://github.com/mudler/cOS/blob/master/Makefile#L17 ). The configuration will be either constructed in the OEM cloud-init config, or in the system/cos package
• Such file has to be editable also via cloud-init, if not supplied by it
• Create an upgrade script, which sets up the new partition and flag it properly where the system will boot next - and install the docker image with luet or a different tool. Once the installation is over, boot into the new partition
• If the system fails to boot the new partition, fallback to the old one
• Make sure the above steps are idempotent, and can be executed if things either goes good or bad. Ensure that the partition 'flip' works as expected
• Make sure the above steps works for other release channels too

Needed by #21
Depends on #29, Related to #5

Shrink image ISO size

Mostly coming from the kernel modules, and from the initramfs fs which is shipping part of TW.
If for the time being we aren't using dracut, we need to at least have a look at #20 to reduce the size of it.

# cbd0cf3430f9:/luetbuild # rpm -qa --queryformat '%10{size} - %-25{name} \t %{version} \t %{os} \n' | sort -rh | head -25 | awk '{print $1/1024/1024, $2, $3, $4}' 
# 180.077 - kernel-default 5.10.5
# 12.4307 - glibc-locale-base 2.32
# 11.0003 - systemd 246.9
# 7.60744 - udev 246.9
# 7.49606 - file-magic 5.39
# 7.49081 - libzypp 17.25.5
# 7.10046 - zypper 1.14.41
# 5.95417 - glibc 2.32
# 5.9036 - coreutils 8.32
# 5.74755 - libsolv-tools 0.7.16
# 5.5852 - perl-base 5.32.0
# 5.58036 - gpg2 2.2.25
# 4.06195 - util-linux 2.35.1
# 4.05395 - kbd 2.3.0
# 3.5594 - libopenssl1_1 1.1.1h
# 3.39311 - gawk 5.1.0
# 3.01739 - shadow 4.8.1
# 2.68816 - rpm 4.15.1
# 2.6414 - elfutils 0.182
# 1.92351 - krb5 1.18.3
# 1.8625 - libstdc++6 10.2.1+git1030
# 1.71149 - libp11-kit0 0.23.20
# 1.69769 - libxml2-2 2.9.10
# 1.67791 - libglib-2_0-0 2.66.4
# 1.61761 - pam 1.5.1

Generate ISOs with no host dependencies

Currently the build iso process is unfriendly - it requires deps from the host system, and moreover the dependency on luet-extensions make it rather complex to follow.

Keep in mind that the building process might require "external" artifacts as the EFI image, and those could be provided separately, or as part of the system that we are trying to boot from the ISO.

We should explore if it's feasable to replace the process with a single golang binary which uses https://github.com/diskfs/go-diskfs/

https://github.com/diskfs/go-diskfs/blob/master/examples/bootable_iso.go

Update: a small PoC it's here: https://github.com/mudler/luet-makeiso

RW rootfs for maintenance mode

Recovery partition to reset state

We should have a recovery partition which is not COS_STATE that can be used to recovery and reset COS_STATE and grub. It should also be possible to reset entirely by pruning /oem and /usr/local and start from scratch

Related to #23

SELinux support

We need to setup SELinux policy for openSUSE/SLE to be able to provide SELinux capabilities for the underlying kubernetes engine. See https://github.com/rancher/rke2-selinux as an example

File system layout

By default, only the relevant dir for the container runtime should be writable.

We should have then:

• read only /usr - / if feasable
• ephemeral /etc that gets wiped out each time (tmpfs mount?)
• persistency only on /usr/local and /oem folder

Mind also to switch back https://github.com/mudler/cOS/blob/31749456b54f3eb60ad203cd3f212c6717b29320/packages/initramfs/loader.sh#L225

Persistency partitions and /etc/fstab configuration can be driven by cloud-init config files, currently in https://github.com/mudler/cOS/blob/31749456b54f3eb60ad203cd3f212c6717b29320/packages/cloud-config/yipfiles/01_defaults.yaml#L12 and from the installation phase
Every user-side or oem configuration, will be handled by #4 and #24

Action items:

• Investigate layout from other projects on how they feased problematics regarding this topic
• Configure the layout in the init, and cloud-config files
• Modify installer to create a persistent partition (that occupy all the space left)
• Tweak init to mount such partition
• QA

Depends on #4 Merged

Enable fedora QEMU/Vbox/OVA on CI

It's currently disabled due to failures that have to be yet investigated. Currently we just build docker images and ISO.

ISO Persistent sector for /oem

Ideally as user, If I want to adapt cOS to my needs, I need to carry the yaml files or in some other medium (labeled with COS_OEM) or by providing inputs to cOS in different ways: cos.setup from boot cmdline, /oem and /usr/local/cloud-config locally, etc.

If the ISO had a persistent sector, then it would be possible to customize the ISOs more easily ( customizing becomes dropping a file into an ISO ) to the installation need besides carrying over persistent changes to liveCDs.

This issue is about exploring a way to load from Live boot the /oem partition from a persistent ISO sector, and install the files present in there as part of the installation process too (when copying them to the OEM folder)

Replace isoscan with a custom dracut script

The isoscan script is really simple, we should be capable to provide our own. We could even avoid the "scan" functionality which looks for the iso image at any available device, there is no need for that, we know the device that holds the image

Originally posted by @davidcassany in #58 (comment)

while isoscan works, it is better to provide a script that doesn't scan at all, as we already know were the img files are (COS_STATE).
Also, isoscan forces cos-upgrade to remount the COS_STATE partition rw as it is already mounted in /run/initramfs/isoscan after boot.

Unify the system/tumbleweed image to be a single-package only

Currently the installer and the iso generation phase are taking the needed files to be able to complete install, for example the generated initramfs. While this is fine for development, in long-term we want a single image containing all the required files, including the initramfs that boots livecd and the real system.

The packages that should be bundled within system/tumbleweed (like luet, which is already bundled):
https://github.com/mudler/cOS/blob/master/iso/cOS-local.yaml#L2

Taking care of course that the installer keeps working as expected, with the required changes in the installer script.

This likely depends on #28 - as the package will probably exceeds the limits of gh-pages (to check) done

Sign final packages images

Once we deliver mudler/luet#169, we will be able to consume final docker images in place of http/https served artifacts.

It should be trivial then to come up with a mechanism to sign those images, either during build (a luet plugin might fit, too), or just afterwards.

Create github action pipeline to build and release ISO whenever a new tag is added

There should be a Github Action pipeline which builds and release the ISO as github artifact whenever we add a new tag. It should generate the artifacts that are currently generated in master runs. It should behave more or less the scheduled pipeline, which just builds artifact from published repositories.

Workflow:

• Have two branches: one for development ( current master, for example ) which pushes signed images to a registry and a image reference and one for stable releases which pushes to a separate image reference
• The release channels in the images have to point to the stable repository
• Before tagging, we merge from master in the release branch, the merging action should copy the images (caches + final images) to the stable release channel
• We tag, the tag produces artifact from the published channels and uploads them over standard Github releases.

cOS variants

It is possible already to specify a different base image in https://github.com/mudler/cOS/blob/master/packages/base/definition.yaml#L6 but it's not sufficient to build cOS on top of a foo distribution.

We have already a mechanism for templating globally, that can be exploited to provides the values in the specific templates ( e.g. https://github.com/mudler/cOS/tree/master/values ) .

The Issue is about creating a new cOS variant (ubuntu, debian, etc) , and attaching to the CIand build it aside the standard cOS.

The values files in values corresponds to a specific variant which is being targeted while running make build. For now, we are sticking with a generic "amd64" which is empty only because the mechanism can be used also to provide multi-arch builds, but we can reuse the same mechanism to provide different cOS flavors.

See also: https://luet-lab.github.io/docs/docs/concepts/packages/templates/#interpolating-globally

Make cloud-init style file less verbose

There are several things that can be eased out on the cloud-init supported syntax:

• First-class support for systemd commands (systemctl, etc.)
• // for sysctl commands
• // setup of users, and authorized_keys for ssh login
• // hostname
• // modules to modprobe
• ntp_server (?)
• wifi/password(?)
• users/password ( see e.g. https://github.com/rancher/k3os#k3ospassword )
• Ability to load github/gitlab repository's yaml files as would be with local directories

For inspiration, take a look at https://github.com/rancher/k3os#sample-configyaml

We can switch to a different format, or either change also the current implementation. For reference, at the moment we are using https://github.com/mudler/yip - which can be also tweaked towards that purpose as needed.

Reports on used licenses

We need to scan and collect all the licenses used by the go code - we could use https://github.com/google/go-licenses to generate a licenses folder that can be automatically updated and tracked in git

Configuration driven from cloud-init style yaml files

/etc should be configured on boot driven by extensible cloud-init style yaml files.

Similarly, users can drop config files into the /oem ( or /usr/local/oem ) or /usr/local/setup folder to achieve persistent configurations. The cloud-init format should allow extensions too.

A default config should be among the lines of : https://github.com/rancher/k3os/blob/1688b718da51b39d48c17e2a20d453f37f3d58c7/overlay/libexec/k3os/boot

This issue is about finding a tool that accomplishes that, and embed it into the various stages of the OS (at least): initramfs, boot, network. Replacing also the current skel package that does handle /etc setup.

Ideally, the tool that mimic cloud-init shouldn't have dependency, and be portable (possibly written in Go) see for e.g.:

• https://github.com/rancher/k3os/tree/master/pkg
• https://github.com/coreos/ignition
• https://github.com/mudler/yip
• .. ?

See also https://github.com/mocaccinoOS/os-commons/tree/master/packages/systemd/yip-systemd for a possible integration with systemd boot

Permanent custom cloud-config during install

Currently the installation process copies over the squashfs content, we should provide a way to persist also a custom cloud-config during install.

• Modify the installer to accept a custom cloud-config
• Copy the cloud-config over the persistent partition
• QA

Needs #5

Simplify makefile

Besides some general refactoring which is needed, we should have one command that :

• generates all packages (e.g. already present make rebuild-all)
• the ISO (e.g. already present make local-iso)
• Finally a target to run qemu with it (to write make run-qemu)

Consider also adapting current GHA pipelines to test the new process too (without run-qemu, if that's hard to test)

read /proc/cmdline without stdin

https://github.com/mudler/cOS/blob/a4a27eefb0024ee23abce352e9eff89ba438c5b4/packages/cos-setup/cos-setup#L33

This made the boot fail 1 out of 4. Let's make yip read /proc/cmdline directly instead of piping, as requires /dev/stdin and we have no guarantee that's there yet

Dracut based initramfs

Now that the initramfs is split into a separate file, it's more easy to provide the logic of the initramfs package with Dracut too.

Handling image mirroring to different registries

• mudler/luet#158
• mudler/luet#162
• mudler/luet#161
• mudler/luet#185

Build QEMU, VBox, AWS, etc. ready-to-use images

Have a look also at packer, and if it makes sense using it for the purpose. Consider that packer can create bootable images from ISOs too, that means we could reuse the current generated artifacts.

See also:

• https://github.com/OSInside/kiwi-descriptions
• https://github.com/rancher/k3os/tree/master/package/packer
• https://github.com/Sabayon/packer-templates (images directly from LiveCD ISOs)

On packer:

• Amazon: http://work.haufegroup.io/automate-ami-with-packer/
• GCE: https://www.packer.io/docs/post-processors/googlecompute-import#qemu-builder-example (from QEMU image to gce)
• Azure: https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/virtual-machines/windows/build-image-with-packer.md- a use case at hand: https://github.com/garvincasimir/Azure-ISO-To-VHD

Sub Issue:

• #174
• #208
• #217

Split live-boot

https://github.com/rancher-sandbox/cOS-toolkit/tree/master/packages/livecd/live-boot - 65f7f87

Possibly we got duplicated files during first dev stage of the package. Need to analyze if those are all really needed, and avoid unintentional dups that slipped in

Live boot setup

We need UEFI/not-EFI boot support for ISOs, that means that atm we could slightly re-adapt what's in https://github.com/Luet-lab/livecd-specs for the time being until we find a better solution.

Packages to add to the rootfs image

For the ones coming from zypper, should be as simple as adding them here: https://github.com/mudler/cOS/blob/85e764da2ab6698d9c3d216de0e55834a8ee25d7/packages/tumbleweed/definition.yaml#L5

• conntrack-tools
• curl
• dosfstools
• e2fsprogs
• findutils
• haveged
• iproute2
• iptables
• lsscsi
• lvm2
• mdadm
• multipath-tools
• nfs-utils
• open-iscsi
• open-vm-tools
• python-azure-agent*
• qemu-guest-agent
• rng-tools
• rancher-agent

Grub install without chroot in the installer

https://github.com/mudler/cOS/blob/85e764da2ab6698d9c3d216de0e55834a8ee25d7/packages/installer/installer.sh#L173

We are chrooting, while it shouldn't be necessary. Let's have a closer look and get rid of the chroot call

Automatically fallback in case of boot failure

This is a follow-up card from #8 . Seems the mechanism isn't correctly working and in place for GRUB.

We should by default fallback to the COS_PASSIVE grub entry, and if that fails, enter to the recovery partition #63 .

Also, we should configure https://github.com/systemd/systemd/blob/a2689fa5fcbfa31eca51db1b2b4a808835429332/docs/AUTOMATIC_BOOT_ASSESSMENT.md automatic boot assesment too if possible.

Action items

• Setup grub for fallback in case of boot failure to the COS_PASSIVE boot entry. If it possible, in case of failure, switch automatically to recovery.
• Setup if possible systemd Automatic Boot assessment features

Initramfs generation

Need to handle initramfs generation - both for LiveCDs and for system installs. We currently are using dev packages to achieve that to ease out and discover what we really need, but it should be enhanced to be extensible.

This card is also about generating an initramfs that could be used directly for #29

Alternatives:

• https://github.com/u-root/u-root
• dracut

WebUI to drive OS customization. Applies changes in RW mode if necessary

Not a prio atm.

Sleek and simple UI for customization. It might accept optionally cloud-init style yaml files (see #4) but eventually UI should convert "clicks" to yaml.

Initramfs then will read those changes and react accordingly, within the same upgrade system (which should do sanity checks).

Be able to reset state

We should have a mechanism in place that makes possible to reset the Distro state as it was after installation.

Documentation: cloud-init git-ops configuration

Enhance debugging experience

Debugging experience is a large topic, it goes from the building process to cOS usage, and derivative debugging too. This epic is a ticket trying to collect all the Sub/issues and enhancement around enhancing the debugging experience in the project areas:

Building:

• mudler/luet#162
• Print informative message on build failure in https://github.com/mudler/luet to give directions on how to debug with the above extension
• mudler/luet#164

Usage:

• #388
• #389

SSO via PAM and 3rd party providers

Follow-up for #13