lxc/distrobuilder

System container image builder for LXC and Incus

distrobuilder

System container and VM image builder for Incus and LXC.

Status

Type	Service	Status
CI	GitHub
Project status	CII Best Practices

Command line options

The following are the command line options of distrobuilder. You can use distrobuilder to create container images for both Incus and LXC.

$ distrobuilder
System container and VM image builder for Incus and LXC

Usage:
  distrobuilder [command]

Available Commands:
  build-dir      Build plain rootfs
  build-incus    Build Incus image from scratch
  build-lxc      Build LXC image from scratch
  help           Help about any command
  pack-incus     Create Incus image from existing rootfs
  pack-lxc       Create LXC image from existing rootfs
  repack-windows Repack Windows ISO with drivers included

Flags:
      --cache-dir         Cache directory
      --cleanup           Clean up cache directory (default true)
      --debug             Enable debug output
      --disable-overlay   Disable the use of filesystem overlays
  -h, --help              help for distrobuilder
  -o, --options           Override options (list of key=value)
  -t, --timeout           Timeout in seconds
      --version           Print version number

Use "distrobuilder [command] --help" for more information about a command.

Installing from package

distrobuilder is available from the Snap Store.

sudo snap install distrobuilder --classic

Installing from source

To compile distrobuilder from source, first install the Go programming language, and some other dependencies.

• Debian-based:

  sudo apt update
  sudo apt install -y golang-go debootstrap rsync gpg squashfs-tools git make
  

• ArchLinux-based:

  sudo pacman -Syu
  sudo pacman -S go debootstrap rsync gnupg squashfs-tools git make --needed
  

• Red Hat-based:

  sudo dnf check-update
  sudo dnf install golang debootstrap rsync gnupg2 squashfs-tools git make
  

NOTE: Distrobuilder requires Go 1.21 or higher, if your distribution doesn't have a recent enough version available, get it from upstream.

Second, download the source code of the distrobuilder repository (this repository).

mkdir -p $HOME/go/src/github.com/lxc/
cd $HOME/go/src/github.com/lxc/
git clone https://github.com/lxc/distrobuilder

Third, enter the directory with the source code of distrobuilder and run make to compile the source code. This will generate the executable program distrobuilder, and it will be located at $HOME/go/bin/distrobuilder.

cd ./distrobuilder
make

Finally, you can run distrobuilder as follows.

$HOME/go/bin/distrobuilder

You may also add the directory $HOME/go/bin/ to your $PATH so that you do not need to run the command with the full path.

How to use

See How to use distrobuilder for instructions.

Troubleshooting

See Troubleshoot distrobuilder.

distrobuilder's Issues

Command line does not include --version

CLIs usually have a way to present the user with their version.
We should add a --version and -v options to report the installed version of distrobuilder.

Add support for compat level overrides in LXC

LXC has something called compat levels for images, it's effectively an ever increasing number which is used by LXC to look up version-specific overrides inside the images.

This is supported for the following files:

• config
• config.user

For example if I want to override the user config for compat level 5, I'd use config.user.5.
Unfortunately if I want something to apply to everything before 5, I'd have to set it for level 1, 2, 3 and 4 which makes things rather repetitive, instead we probably can simply things using something like this:

In the yaml, this would likely look like:

targets:
  lxc:
    create-message: |
        You just created an Ubuntu 16.04 LTS container (release=bionic, arch=amd64, variant=default)

        To enable sshd, run: apt-get install openssh-server

        For security reason, container images ship without user accounts
        and without a root password.

        Use lxc-attach or chroot directly into the rootfs to set a root password
        or create user accounts.

    config:
      - type: all
        before: 5
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/ubuntu.common.conf

      - type: user
        before: 5
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/ubuntu.userns.conf

      - type: all
        after: 4
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/common.conf

      - type: user
        after: 4
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/userns.conf

      - type: all
        content: |-
          lxc.arch = x86_64

distrobuilder needs to know what the current compat level is in LXC (currently 5) and will generate a version of "config" and "config.user" for each of those (from 1 to 5 right now). The un-suffixed file should be that of the maximum compat level. We may at some point try to be clever and de-dup the configs using symlinks, but for now we can be lazy and just generate them all.

The individual snippets should be appended in the order they're provided, so, "config.user" would then be:

lxc.include = LXC_TEMPLATE_CONFIG/common.conf
lxc.include = LXC_TEMPLATE_CONFIG/userns.conf
lxc.arch = x86_64

While "config.4" would be:

lxc.include = LXC_TEMPLATE_CONFIG/ubuntu.common.conf
lxc.arch = x86_64

This will generate all those files:

• config (compat=5)
• config.1 (compat=1)
• config.2 (same as config.1)
• config.3 (same as config.1)
• config.4 (same as config.1)
• config.user (compat=5)
• config.user.1 (compat=1)
• config.user.2 (same as config.user.1)
• config.user.3 (same as config.user.1)
• config.user.4 (same as config.user.1)

Templating for create-message, config and config-user

I think those 3 keys need to have templating enabled. I'd recommend using pongo2 (flosch/pongo2) as we do in LXD, exporting the "image" struct to it so that I can do:

lxc:
  create-message: |-
    You just created an {% image.Description %} container (release={% image.Release %}, arch={% image.Arch %}, variant={% image.Variant %})
    ...
    config: |
        lxc.include = LXC_TEMPLATE_CONFIG/ubuntu.common.conf
        lxc.arch = {% image.Arch %}
    config-user: |
        lxc.include = LXC_TEMPLATE_CONFIG/ubuntu.common.conf
        lxc.include = LXC_TEMPLATE_CONFIG/ubuntu.userns.conf
        lxc.arch = {% image.Arch %}

No install location outside GOPATH

On Fedora Core 30 the installation (make) failes at the line "go install -v ./..." with:
go install: no install location for directory XXX outside GOPATH

Solution is to execute this before:
mkdir $GOPATH/bin && export GOBIN=$GOPATH/bin

Add timeout argument

In production we pretty often see build hangs, so far that mostly affects Alpine getting stuck somewhere in apk.

Could we get a --timeout argument where we can feed a value (probably in seconds) after which distrobuilder kills whatever it's doing and exits?

For us, I'd probably then pass 30min for the build-dir stage and no timeout for the rest, that would take care of such issues and allow Jenkins to automatically retry things.

CentOS image missing some packages to boot successfully

Hi,

two days ago, I have followed the howto to build distrobuilder and make my own centos 7.5 image.
I needed 7.5, because the application I want to install is not yet compatible to 7.6...

Building the image was successfully, but it was not able to start. I got this error:

lxc info --show-log local:fschmid-centos-lxd-v1
Name: fschmid-centos-lxd-v1
Remote: unix://
Architecture: x86_64
Created: 2019/01/16 15:46 UTC
Status: Stopped
Type: persistent
Profiles: default

Log:

lxc fschmid-centos-lxd-v1 20190116154702.966 WARN     conf - conf.c:lxc_setup_devpts:1616 - Invalid argument - Failed to unmount old devpts instance
lxc fschmid-centos-lxd-v1 20190116154702.967 ERROR    start - start.c:start:2028 - No such file or directory - Failed to exec "/sbin/init"
lxc fschmid-centos-lxd-v1 20190116154702.967 ERROR    sync - sync.c:__sync_wait:62 - An error occurred in another process (expected sequence number 7)
lxc fschmid-centos-lxd-v1 20190116154702.967 WARN     network - network.c:lxc_delete_network_priv:2589 - Operation not permitted - Failed to remove interface "eth0" with index 20
lxc fschmid-centos-lxd-v1 20190116154702.967 ERROR    lxccontainer - lxccontainer.c:wait_on_daemonized_start:842 - Received container state "ABORTING" instead of "RUNNING"
lxc fschmid-centos-lxd-v1 20190116154702.967 ERROR    start - start.c:__lxc_start:1939 - Failed to spawn container "fschmid-centos-lxd-v1"
lxc 20190116154702.972 WARN     commands - commands.c:lxc_cmd_rsp_recv:132 - Connection reset by peer - Failed to receive response for command "get_state"

My second problem:
After fixing first issue, I saw, that the image is missing two important templates:

• ifcfg-eth0.lxd.tpl
• network.lxd.tpl

Here is my centos.yaml file:

image:
  distribution: centos
  release: 7
  variant: Minimal
  description: CentOS {{ image.release }}
  expiry: 3000d
  architecture: x86_64

source:
  downloader: centos-http
  url: http://ownrepo/prod/centos/
  keys:
    - 24C6A8A7F4A80EB5
    - 0946FCA2C105B9DE
  variant: Minimal
#  variant: default
  skip_verification: true

targets:
  lxc:
    create-message: |
        You just created a CentOS container (release={{ image.release }}, arch={{ image.architecture }})

    config:
      - type: all
        before: 5
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/centos.common.conf

      - type: user
        before: 5
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/centos.userns.conf

      - type: all
        after: 4
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/common.conf

      - type: user
        after: 4
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/userns.conf

      - type: all
        content: |-
          lxc.arch = {{ image.architecture_kernel }}

files:
 - name: hostname
   path: /etc/hostname
   generator: hostname

 - name: hosts
   path: /etc/hosts
   generator: hosts

packages:
    manager: yum

    update: false
    cleanup: true

My fix for the first problem (image is not bootable):

• I have added some missing packages to sources/centos-http.go

197 yum --installroot=/rootfs --disablerepo=* --enablerepo=cdrom -y --releasever=%s install basesystem centos-release yum systemd binutils crontabs dhclient glibc-common iputils json logrotate openssh-clients openssh-server passwd rpm-python rsyslog tzdata vim-minimal policycoreutils json-c audit authconfig file postfix gettext net-tools rootfiles

Maybe not all of them are necessary, but most of them are. At least systemd was missing

My fix for the second problem (no networing inside contianer):

• I have used the templates which came with your CentOS 7.6 image and repacked them into my lxd.tar.xz

With this, I also have networking inside the container.

What I have seen, is, that there have been quite a lot changes in the last two weeks to these files:

• distrobuilder/doc/examples/centos
• distrobuilder/sources/centos-http.go

BR Florian Schmid

Multiple instances of distrobuilder

I just tried running multiple instances of distrobuilder at the same time and ran into issues because of /var/cache/distrobuilder. We should make it so that this doesn't happen. It should be possible to build multiple images at the same time.

Replace use of kpartx for ubuntu-core distrobuilder

Because our production image building runs inside a container (privileged though), we can't use kpartx as it attempts to mess with device-mapper which is very very unpleasant to use inside containers.

We do however have working loop devices, so using losetup would be fine.

So we need to change distrobuilder to scan the partition table of the .img file, figure out the offset and size of the partition we want and then map that using losetup, then mount it.

keyserver and keys mandatory

For now, when the keyserver is empty in the yaml, definition.go set a default value "hkps.pool.sks-keyservers.net" for keyserver, but the keys argument is not sanitized anymore since 6c4c7ca, allowing this field to be empty if the value is not specified by the yaml used as argument of distrobuilder.

In my system (Fedora 27), gpg fails if keys isn't informed, and so the distrobuild returns an error.

My question is: Should keyserver also be permitted to be empty, and VerifyFile should return true if both keyserver and keys are empty? Or somehow these two parameters are mandatory, and it should be enforced by definition.go?

One way or another, I'm available to send a patch if this behavior needs to change.

distrobuilder build errors

Building distrobuilder from the master branch on Archlinux gives me:

[mrakotomandimby@arch-00 distrobuilder]$ make
gofmt -s -w .
go get -t -v -d ./...
go install -v ./...
go install: no install location for directory /home/mrakotomandimby/distrobuilder/distrobuilder outside GOPATH
	For more details see: 'go help gopath'
go install: no install location for directory /home/mrakotomandimby/distrobuilder/generators outside GOPATH
	For more details see: 'go help gopath'
go install: no install location for directory /home/mrakotomandimby/distrobuilder/image outside GOPATH
	For more details see: 'go help gopath'
go install: no install location for directory /home/mrakotomandimby/distrobuilder/managers outside GOPATH
	For more details see: 'go help gopath'
go install: no install location for directory /home/mrakotomandimby/distrobuilder/shared outside GOPATH
	For more details see: 'go help gopath'
go install: no install location for directory /home/mrakotomandimby/distrobuilder/sources outside GOPATH
	For more details see: 'go help gopath'
make: *** [Makefile:6: default] Error 1

Go version

[mrakotomandimby@arch-00 distrobuilder]$ go version
go version go1.10.1 linux/amd64
[mrakotomandimby@arch-00 distrobuilder]$ 

How to specify package groups for CentOS (or other)?

I am looking at the centos example config, and I don't see anything to tell me how to install package groups. If it is possible, can we have an instructive comment in the example config? If it is not possible, can we have a comment stating that in the example config, README.md, or some other discoverable place?

Implement the actions

Actions aren't implemented yet and are likely to be needed for some of our supported images.

Add support for Oracle Linux

We seem to be missing support for building Oracle Linux containers, it should be very similar to the CentOS case and is next on my list of distros to switch over to distrobuilder.

Cannot build Arch Linux image

Arch Linux fails to build, using following YAML:

image:
  distribution: archlinux
  release: 2019.07.01
  description: Arch Linux
  expiry: 30d
  architecture: x86_64

source:
  downloader: archlinux-http
  url: https://archive.archlinux.org/iso
  keys:
    - 4AA4767BBC9C4B1D18AE28B77F2D434B9741E8AC

targets:
  lxc:
    create-message: |
      You just created an Arch Linux container (release={{ image.release }}, arch={{ image.architecture }})

    config:
      - type: all
        before: 5
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/archlinux.common.conf

      - type: user
        before: 5
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/archlinux.userns.conf

      - type: all
        after: 4
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/common.conf

      - type: user
        after: 4
        content: |-
          lxc.include = LXC_TEMPLATE_CONFIG/userns.conf

      - type: all
        content: |-
          lxc.arch = {{ image.architecture_kernel }}

files:
  - name: hostname
    path: /etc/hostname
    generator: hostname

  - name: hosts
    path: /etc/hosts
    generator: hosts

packages:
  manager: pacman

  update: true
  cleanup: true

  sets:
    - packages:
        - systemd-sysvcompat
        - neovim
      action: install

$ sudo distrobuilder build-lxd archlinux.yaml
/tmp/archlinux-2019.07.01-x86_64/archlinux-bootstrap-2019.07.01-x86_64.tar.gz: 100% (6.21MB/s)
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
gpg: Generating pacman keyring master key...
gpg: key D4298857ED8932B9 marked as ultimately trusted
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/492044C0453E653D69D37B71D4298857ED8932B9.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
==> Appending keys from archlinux.gpg...
==> Locally signing trusted keys in keyring...
  -> Locally signing key D8AFDDA07A5B6EDFA7D8CCDAD6D055F927843F1C...
  -> Locally signing key DDB867B92AA789C165EEFA799B729B06A680C281...
  -> Locally signing key 684148BB25B49E986A4944C55184252D824B18E8...
  -> Locally signing key 91FFE0700E80619CEB73235CA88E23E377514E00...
  -> Locally signing key 0E8B644079F599DFC1DDC3973348882F6AC6A4C2...
  -> Locally signing key AB19265E5D7D20687D303246BA1DFB64FFF979E7...
==> Importing owner trust values...
gpg: inserting ownertrust of 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
gpg: setting ownertrust to 4
==> Disabling revoked keys in keyring...
  -> Disabling key 8F76BEEA0289F9E1D3E229C05F946DED983D4366...
  -> Disabling key 63F395DE2D6398BBE458F281F2DBB4931985A992...
  -> Disabling key 50F33E2E5B0C3D900424ABE89BDCF497A4BBCC7F...
  -> Disabling key 27FFC4769E19F096D41D9265A04F9397CDFD6BB0...
  -> Disabling key 39F880E50E49A4D11341E8F939E4F17F295AFBF4...
  -> Disabling key 8840BD07FC24CB7CE394A07CCF7037A4F27FB7DA...
  -> Disabling key 5559BC1A32B8F76B3FCCD9555FA5E5544F010D48...
  -> Disabling key 0B20CA1931F5DA3A70D0F8D2EA6836E1AB441196...
  -> Disabling key 07DFD3A0BC213FA12EDC217559B3122E2FA915EC...
  -> Disabling key 4FCF887689C41B09506BE8D5F3E1D5C5D30DB0AD...
  -> Disabling key 5A2257D19FF7E1E0E415968CE62F853100F0D0F0...
  -> Disabling key 7FA647CD89891DEDC060287BB9113D1ED21E1A55...
  -> Disabling key BC1FBE4D2826A0B51E47ED62E2539214C6C11350...
  -> Disabling key 4A8B17E20B88ACA61860009B5CED81B7C2E5C0D2...
  -> Disabling key 5696C003B0854206450C8E5BE613C09CB4440678...
  -> Disabling key 8CF934E339CAD8ABF342E822E711306E3C4F88BC...
  -> Disabling key F5A361A3A13554B85E57DDDAAF7EF7873CFD4BB6...
  -> Disabling key 5E7585ADFF106BFFBBA319DC654B877A0864983E...
  -> Disabling key 65EEFE022108E2B708CBFCF7F9E712E59AF5F22A...
  -> Disabling key 40440DC037C05620984379A6761FAD69BA06C6A9...
  -> Disabling key 34C5D94FE7E7913E86DC427E7FB1A3800C84C0A5...
  -> Disabling key 81D7F8241DB38BC759C80FCE3A726C6170E80477...
  -> Disabling key E7210A59715F6940CF9A4E36A001876699AD6E84...
  -> Disabling key 5357F3B111688D88C1D88119FCF2CB179205AC90...
  -> Disabling key FB871F0131FEA4FB5A9192B4C8880A6406361833...
  -> Disabling key 66BD74A036D522F51DD70A3C7F2A16726521E06D...
  -> Disabling key B1F2C889CB2CCB2ADA36D963097D629E437520BD...
  -> Disabling key 9515D8A8EAB88E49BB65EDBCE6B456CAF15447D5...
  -> Disabling key 40776A5221EF5AD468A4906D42A1DB15EC133BAD...
  -> Disabling key D4DE5ABDE2A7287644EAC7E36D1A9E70E19DAA50...
  -> Disabling key 44D4A033AC140143927397D47EFD567D4C7EA887...
==> Updating trust database...
gpg: key 1EB2638FF56C0C53: no user ID for key signature packet of class 10
gpg: key 1EB2638FF56C0C53: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: key 786C63F330D7CB92: no user ID for key signature packet of class 10
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   6  trust: 0-, 0q, 0n, 0m, 0f, 1u
gpg: depth: 1  valid:   6  signed:  75  trust: 0-, 0q, 0n, 6m, 0f, 0u
gpg: depth: 2  valid:  75  signed:   7  trust: 75-, 0q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2019-10-03
:: Synchronizing package databases...
error: failed retrieving file 'core.db' from mirrors.kernel.org : Could not resolve host: mirrors.kernel.org
error: failed to update core (invalid url for server)
error: failed retrieving file 'extra.db' from mirrors.kernel.org : Could not resolve host: mirrors.kernel.org
error: failed to update extra (invalid url for server)
error: failed retrieving file 'community.db' from mirrors.kernel.org : Could not resolve host: mirrors.kernel.org
error: failed to update community (invalid url for server)
error: failed to synchronize all databases
Error: Failed to manage packages: exit status 1

There is not much documentation available, i.e. how to pass files to distrobuilder, etc etc.

Add Fedora support

This came up on the lxc-users mailing list. We should add Fedora support or encourage the community to work on Fedora support. :)

Write documentation on running LXC created containers

The README mentions how to create. But there is no indication on how to use the created result: How to use it as LXC container?
I'm currently trying and if I succed, I will obviusly report here.

make release

• add page to linuxcontainers.org
• release official tarball

schedule to be discussed

go get ... failed with "no Go files in ..."

# go get -d -v github.com/lxc/distrobuilder
github.com/lxc/distrobuilder (download)
created GOPATH=/root/go; see 'go help gopath'
package github.com/lxc/distrobuilder: no Go files in /root/go/src/github.com/lxc/distrobuilder

Feature request: run distrobuilder as a regular user

Users currently need to run distrobuilder with sudo for building images, I guess for having the right owner/permissions in the root fs. This makes it very unconvenient for integrating distrobuilder in automated environments or with other tools like Terraform.

I was wondering if it would be posssible to drop this requirement, maybe by unshareing some namespaces before starting the build...

gernator to ADD/COPY local files/dir like Dockerfile

We need a mechanism to seamlessly create and publish lxc containers . COPY/ADD generator to copy files/folders from local repository inside container rootfs. This will allow publishing lxc apps from git as part of CI/CD run.

Make actions more flexible

I think we should switch actions to be closer to what we do for files:

actions:
  - trigger: post-update
    action: |-
      #!/bin/sh
      rm -rf /run/*

 - trigger: post-unpack
   action: |-
     #!/bin/sh
     sed -i "s/foo/bar/g" /etc/hosts
   releases:
     - precise
     - trusty

Download template delivers armhf binaries although armel requested

I have just tried to build a container for Debian stretch armel:

lxc-create -n testlxcusr -f testlxcusr.conf -t download -- --dist debian --arch armel --release stretch

And I have been given armhf binaries, which is puzzling:

You just created a Debian stretch armhf (20190617_05:24) container.

And it would seem to be so:

readelf -h testlxcusr/rootfs/bin/tar|grep Flags
  Flags:                             0x5000400, Version5 EABI, hard-float ABI

Here's a trace from the executing template which suggests that it was looking in the right place for armel binaries:

wget_wrapper -T 30 -q https://images.linuxcontainers.org//images/debian/stretch/armel/default/20190617_05:24//meta.tar.xz.asc -O /tmp/tmp.uKsle76i5a/meta.tar.xz.asc

So - is there something amiss with the build process ? Or should I have done something differently to get hold of the armel build ?

LXC version is 2.0.7.

Top section of the generated config:

# Template used to create this container: /usr/share/lxc/templates/lxc-download
# Parameters passed to the template: -d debian -a armel -r stretch
# Template script checksum (SHA-1): 740c51206e35463362b735e68b867876048a8baf
# For additional config options, please look at lxc.container.conf(5)

# Uncomment the following line to support nesting containers:
#lxc.include = /usr/share/lxc/config/nesting.conf
# (Be aware this has security implications)


# Distribution configuration
lxc.include = /usr/share/lxc/config/debian.common.conf
lxc.include = /usr/share/lxc/config/debian.userns.conf
lxc.arch = linux32

The config file I gave it:

lxc.network.type = veth
#lxc.network.flags = up
lxc.network.link = br0
lxc.network.hwaddr = fe:06:ae:7f:c6:38

lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536

Build error

Distrobuilder build is currently broken due to function signature change in lxd dependency.
Modified function signature here

Error log :

gofmt -s -w .
go get -t -v -d ./...
go install -v ./...
github.com/lxc/distrobuilder/sources
# github.com/lxc/distrobuilder/sources
sources/alpine-http.go:91:18: too many arguments in call to "github.com/lxc/lxd/shared".Unpack
	have (string, string, bool, bool, nil)
	want (string, string, bool, bool)
sources/archlinux-http.go:78:18: too many arguments in call to "github.com/lxc/lxd/shared".Unpack
	have (string, string, bool, bool, nil)
	want (string, string, bool, bool)
sources/fedora-http.go:51:18: too many arguments in call to "github.com/lxc/lxd/shared".Unpack
	have (string, string, bool, bool, nil)
	want (string, string, bool, bool)
sources/fedora-http.go:93:21: too many arguments in call to "github.com/lxc/lxd/shared".Unpack
	have (string, string, bool, bool, nil)
	want (string, string, bool, bool)
sources/gentoo.go:91:18: too many arguments in call to "github.com/lxc/lxd/shared".Unpack
	have (string, string, bool, bool, nil)
	want (string, string, bool, bool)
sources/sabayon.go:44:18: too many arguments in call to "github.com/lxc/lxd/shared".Unpack
	have (string, string, bool, bool, nil)
	want (string, string, bool, bool)
sources/ubuntu-http.go:116:19: too many arguments in call to "github.com/lxc/lxd/shared".Unpack
	have (string, string, bool, bool, nil)
	want (string, string, bool, bool)
make: *** [Makefile:6: default] Error 2

Debootstrap source must generate a new sources.list

We need to extend the yaml to include a new template key:

source:
  downloader: debootstrap
  url: http://us.archive.ubuntu.com/ubuntu
  keys:
    - 0x790BC7277767219C42C86F933B4FE6ACC0B21F32
  apt_sources: |-
    deb http://{{ source.url }} {{ image.release }} main restricted universe multiverse
    deb http://{{ source.url }} {{ image.release }}-updates main restricted universe multiverse
    deb http://security.ubuntu.com/ubuntu {{ image.release }}-security main restricted universe multiverse

This should then be templated onto "/etc/apt/sources.list" inside the post-debootstrap chroot.
We also need to make sure that "apt-get update" is executed inside the chroot prior to any other package management action happening after templating.

Have archlinux source select latest ISO by default

We don't want to have to update our .yaml every month or so, it'd be good if distrobuilder could select the most recent ISO by default when source.release is empty.

RFC: command line changes

This is really just a question whether replacing:

Available Commands:
  build-dir   Build plain rootfs
  build-lxc   Build LXC image from scratch
  build-lxd   Build LXD image from scratch
  help        Help about any command
  pack-lxc    Create LXC image from existing rootfs
  pack-lxd    Create LXD image from existing rootfs

should be

Available Commands:
  build dir   Build plain rootfs
  build lxc   Build LXC image from scratch
  build lxd   Build LXD image from scratch
  help        Help about any command
  pack lxc    Create LXC image from existing rootfs
  pack lxd    Create LXD image from existing rootfs

It seems cleaner to me but feel free to close this without discussing it if you think that it doesn't make sense. :)

Make ArchLinux source understand archlinuxarm

We should be able to use http://il.us.mirror.archlinuxarm.org/os/ as the url for archlinux-http with 68B3537F39A313B3E574D06777193F152BDBE6A6 as GPG key and have it pick one of those two based on architectures:

• ArchLinuxARM-aarch64-latest.tar.gz
• ArchLinuxARM-armv7-latest.tar.gz

We'll likely also need an architecture map for Arch which supports:

• x86_64 (ARCH_64BIT_INTEL_X86)
• aarch64 (ARCH_64BIT_ARMV8_LITTLE_ENDIAN)
• armv7 (ARCH_32BIT_ARMV7_LITTLE_ENDIAN)

import cycle not allowed

Hi folks

When building the package, the following shows up:

import cycle not allowed

root@srv1:~/go/src/distrobuilder# make
gofmt -s -w .
go get -t -v -d ./...
import cycle not allowed
package distrobuilder/distrobuilder
imports bufio
imports bytes
imports errors
imports runtime
imports internal/bytealg
imports internal/cpu
imports runtime
Makefile:4: recipe for target 'default' failed
make: *** [default] Error 1

Provide cloud-init with images

Hello:
Is it possible to provide cloud-init packages with images at images.linuxcontainers.org?

Maybe this is too difficult to achieve or people use other methods to provision systems other than Ubuntu from cloud-images.ubuntu.com.

Thanks.

Add support for Plamo Linux

We seem to be missing support for building Plamo Linux containers and it's one of the few remaining distros to switch over.

Failure to import GPG keys

Hi,

Since #192 , our GitLab CI is failing to build images. strace-ing distrobuilder, the keys are correctly fetched, but they are not being picked up. This step fails for Ubuntu-based images as well as Alpine-based ones.

Steps

Create a repository on GitLab (gitlab.com suffices), put an image template of your choice and the following .gitlab-ci.yml file in it. Commit, push and start the CI.

# .gitlab-ci.yml
image: ubuntu:18.04

before_script:
    - apt-get update
    - apt-get -y install git-core golang-go debootstrap rsync gpg squashfs-tools
    - 'go get -d -v github.com/lxc/distrobuilder || :'
    - cd ~/go/src/github.com/lxc/distrobuilder
    - make
    - cd "${CI_PROJECT_DIR}"

image:
    stage: build
    script:
        - ~/go/bin/distrobuilder build-lxd image-of-your-choice.yml

Output

Here are relevant trace output lines for an Alpine-based image:

• [pid 6739] <... read resumed> "gpg: key 293ACD0907D9495A: 2 duplicate signatures removed\ngpg: key 293ACD0907D9495A: 11 signatures not checked due to missing keys\ngpg: /tmp/distrobuilder.071160561/trustdb.gpg: trustdb created\ngpg: key 293ACD0907D9495A: public key \"Natanael Copa <[email protected]>\" imported\n", 1473) = 281
• [pid 6743] read(9, "gpg: key 293ACD0907D9495A: 2 duplicate signatures removed\ngpg: key 293ACD0907D9495A: 11 signatures not checked due to missing keys\ngpg: key 293ACD0907D9495A: \"Natanael Copa <[email protected]>\" not changed\ngpg: Total number processed: 1\ngpg: unchanged: 1\n", 512) = 272
• [pid 6750] write(2, "Error: Error while downloading source: Failed to import keys: 0482D84022F52DF1C4E7CD43293ACD0907D9495A\n", 103Error: Error while downloading source: Failed to import keys: 0482D84022F52DF1C4E7CD43293ACD0907D9495A ) = 103

This matches an execution where the key is correctly fetched and ignored two (2) times.

Fix CentOS support

I just did a quick test of distrobuilder with CentOS, a few things that should get fixed:

• Mount call for the iso should pass the read-only flag (will silence a warning)
• Add support for AltArch (maybe it just works if I set /altarch in the URL)
• Make sure that Centos 6 works too, URLs don't seem to match there

And maybe more critically, running distrobuilder on Ubuntu 18.04, I'm getting this when running the example:

mount: /tmp/distrobuilder/iso: WARNING: device write-protected, mounted read-only.
error: open of rpm-4.11.3-25.el7.x86_64.rpm failed: No such file or directory
error: open of yum-3.4.3-154.el7.centos.noarch.rpm failed: No such file or directory
Traceback (most recent call last):
  File "/bin/yum", line 28, in <module>
    import yummain
ImportError: No module named yummain
Traceback (most recent call last):
  File "/bin/yum", line 28, in <module>
    import yummain
ImportError: No module named yummain
Traceback (most recent call last):
  File "/bin/yum", line 28, in <module>
    import yummain
ImportError: No module named yummain
Error: Failed to manage packages: exit status 1

git executable file not found when building distrobuilder

Following documentation to build distrobuilder but failed with error messages below:

github.com/lxc/distrobuilder (download)
created GOPATH=/root/go; see 'go help gopath'
go: missing Git command. See https://golang.org/s/gogetcmd
package github.com/lxc/distrobuilder: exec: "git": executable file not found in $PATH

debootstrap source

Some notes as I'm playing with it:

• We should allow for a different variant to be passed to debootstrap than the one listed under image (the name of the LXC variant is unlikely to lineup with the debootrstrap variant name)
• We shouldn't need to provide a GPG key id when using it (and it's in fact unused in the code), but if one is provided, then that should be fed to debootstrap properly)
• We're going to need an option to set a "same-as" release option in the debootstrap source as new releases of Ubuntu and Debian often aren't supported by older deboostraps until months after we start publishing images. The "same-as" trick is a bit ugly but we'd effectively be creating the missing symlink under /usr/share/debootstrap/scripts".

Add support for OpenSUSE

We seem to be missing support for building OpenSUSE containers and it's one of the few remaining distros to switch over.

Add some new special generators

I'd like two new file "generators":

• "remove": Simply removes whatever path you give it from the rootfs, if a directory, recursively remove it.
• "dump": Takes an extra "content" key in yaml and dumps what's in there (templated) and puts it into the given path

This should let us easily take care of a good number of simple modifications to the rootfs (mostly /etc files) while avoiding the use of actions for those things.

GPG signature check: no valid OpenGPG data found

Hi, I tried to build the centos LXC container and it turns out I can not manage to verify the sources using GPG and proceed to the build of the container without using skip_verification: true.

I built distrobuilder on a clean CentOS 7 vm and it seems gpg can hang sometimes and fail:

$ sudo /usr/local/bin/distrobuilder build-lxc centos
Error: Error while downloading source: Failed to verify: gpg: no valid OpenPGP data found.
gpg: the signature could not be verified.
Please remember that the signature file (.sig or .asc)
should be the first file given on the command line.

If I interrupt distrobuilder while it waits for gpg (before I get any error message), the temporary folder is not destroyed and I have these files:

$ sudo ls /tmp/distrobuilder.157743850/
pubring.gpg  secring.gpg

Is not distrobuilder.gpg and trustdb.gpg supposed to be in this directory?

I manage to make it work by typing gpg commands myself, but I suspect an issue.

The following produces the expected result:

mkdir -p test && sudo chown root test && sudo chmod 0700 test && sudo gpg --homedir ./test/ --keyserver hkps://keyserver.ubuntu.com --recv-keys '0946FCA2C105B9DE'
sudo gpg --homedir ./test/ --export --output test/distrobuilder.gpg

Make the mapped architecture available in pongo2

For the Ubuntu images at least, the expectation is that the "description" and "architecture" properties in the LXD image include the architecture name as called by the distribution (so amd64 instead of x86_64), but right now I see now way to access this from pongo2.

how do I download distrobuilder?

hey

would like to start using distrobuilder, how do i begin?

Implement yaml override on the command line

For distrobuilder to be practical with the way we build images, I'd like to be able to have a single yaml that I can use for all images of a given distro, covering all releases and all architectures.

I think the easiest way to do so is to avoid for key=value type override of yaml keys on the command line.
I'd therefore be able to do:

distrobuilder build-dir ubuntu.yaml new -o image.release=bionic -o image.description="Ubuntu 18.04 LTS" -o image.architecture=armhf -o source.url=http://us.ports.ubuntu.com/ubuntu-ports

Which would then give me an Ubuntu 18.04 version of the armhf image, using the proper archive mirror for the architecture.

I wrote code that does the right type guessing to go from a key=value string to a struct member in the past, you can find it here: https://github.com/nsec/askgod/blob/master/cmd/askgod/utils.go#L12

The release field shouldn't be required

ArchLinux, Gentoo and Sabayon at least don't have a release.

Add Support for Void Linux

https://voidlinux.org/

Void is a general purpose operating system, based on the monolithic Linux® kernel. Its package system allows you to quickly install, update and remove software; software is provided in binary packages or can be built directly from sources with the help of the XBPS source packages collection.

It is available for the Intel x86®, ARM® and MIPS® processor architectures; Software packages can be built natively or cross compiling through the XBPS source packages collection.

See https://github.com/lxc/lxc-templates/blob/master/templates/lxc-voidlinux.in
Small hint: That template seems to incorrectly link to voidlinux.eu instead of voidlinux.org (domain shift).

Fedora-build fails at mounting filesystems

On Fedora Core 30 the lxd-build fails at mounting filesystems.

# mkdir -p $HOME/ContainerImages/fedora/
# cd $HOME/ContainerImages/fedora/
# cp $HOME/go/src/github.com/lxc/distrobuilder/doc/examples/fedora fedora.yaml
# sed -i 's/release: 28/release: 30/' fedora.yaml
# $HOME/go/bin/distrobuilder build-lxd fedora.yaml

Fedora Modular 30 - x86_64                      1.2 MB/s | 2.3 MB     00:01
Fedora Modular 30 - x86_64 - Updates            131  B/s | 257  B     00:01
Fedora Modular 30 - x86_64 - Test Updates       703 kB/s | 3.1 MB     00:04
Fedora 30 - x86_64 - Test Updates               2.3 MB/s |  14 MB     00:05
Fedora 30 - x86_64 - Updates                    386  B/s | 257  B     00:00
Fedora 30 - x86_64                              4.3 MB/s |  54 MB     00:12
Metadata cache created.
Failed to set locale, defaulting to C
Last metadata expiration check: 0:00:58 ago on Sun Apr 14 20:49:20 2019.
Dependencies resolved.
================================================================================
 Package              Arch     Version                  Repository         Size
================================================================================
Upgrading:
 dbus-broker          x86_64   20-2.fc30                updates-testing   160 k
 langpacks-en         noarch   1.0-17.fc30              updates-testing   8.2 k
 systemd              x86_64   241-6.gita2eaa1c.fc30    updates-testing   3.6 M
 systemd-libs         x86_64   241-6.gita2eaa1c.fc30    updates-testing   473 k
 systemd-pam          x86_64   241-6.gita2eaa1c.fc30    updates-testing   150 k
 systemd-rpm-macros   noarch   241-6.gita2eaa1c.fc30    updates-testing    22 k
Installing dependencies:
 xkeyboard-config     noarch   2.24-5.fc30              fedora            822 k
Installing weak dependencies:
 libxkbcommon         x86_64   0.8.3-1.fc30             fedora            108 k

Transaction Summary
================================================================================
Install  2 Packages
Upgrade  6 Packages

Total download size: 5.3 M
Downloading Packages:
(1/8): systemd-libs-241-5.git3d835d0.fc30_241-6  88 kB/s |  29 kB     00:00
(2/8): systemd-pam-241-5.git3d835d0.fc30_241-6.  48 kB/s |  19 kB     00:00
(3/8): libxkbcommon-0.8.3-1.fc30.x86_64.rpm     724 kB/s | 108 kB     00:00
(4/8): xkeyboard-config-2.24-5.fc30.noarch.rpm  6.3 MB/s | 822 kB     00:00
(5/8): langpacks-en-1.0-17.fc30.noarch.rpm       75 kB/s | 8.2 kB     00:00
(6/8): systemd-rpm-macros-241-6.gita2eaa1c.fc30 113 kB/s |  22 kB     00:00
(7/8): dbus-broker-20-2.fc30.x86_64.rpm         342 kB/s | 160 kB     00:00
(8/8): systemd-241-5.git3d835d0.fc30_241-6.gita 693 kB/s | 838 kB     00:01
[DRPM 1/3] systemd-pam-241-5.git3d835d0.fc30_241-6.gita2eaa1c.fc30.x86_64.drpm: done
/usr/share/man/man8/libnss_myhostname.so.2.8.gz: No such file or directory
cannot reconstruct rpm from disk files
/usr/share/doc/systemd/NEWS: No such file or directory
cannot reconstruct rpm from disk files
Some packages were not downloaded. Retrying.
(1/2): systemd-libs-241-6.gita2eaa1c.fc30.x86_6 415 kB/s | 473 kB     00:01
(2/2): systemd-241-6.gita2eaa1c.fc30.x86_64.rpm 1.6 MB/s | 3.6 MB     00:02
--------------------------------------------------------------------------------
Total                                           1.2 MB/s | 6.0 MB     00:04
Failed Delta RPMs increased 5.3 MB of updates to 6.0 MB (-13.1% wasted)
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1
  Running scriptlet: systemd-libs-241-6.gita2eaa1c.fc30.x86_64              1/1
  Upgrading        : systemd-libs-241-6.gita2eaa1c.fc30.x86_64             1/14
  Running scriptlet: systemd-libs-241-6.gita2eaa1c.fc30.x86_64             1/14
  Upgrading        : systemd-rpm-macros-241-6.gita2eaa1c.fc30.noarch       2/14
  Installing       : xkeyboard-config-2.24-5.fc30.noarch                   3/14
  Installing       : libxkbcommon-0.8.3-1.fc30.x86_64                      4/14
  Upgrading        : systemd-pam-241-6.gita2eaa1c.fc30.x86_64              5/14
  Running scriptlet: systemd-241-6.gita2eaa1c.fc30.x86_64                  6/14
  Upgrading        : systemd-241-6.gita2eaa1c.fc30.x86_64                  6/14
  Running scriptlet: systemd-241-6.gita2eaa1c.fc30.x86_64                  6/14
  Running scriptlet: dbus-broker-20-2.fc30.x86_64                          7/14
  Upgrading        : dbus-broker-20-2.fc30.x86_64                          7/14
  Running scriptlet: dbus-broker-20-2.fc30.x86_64                          7/14
  Upgrading        : langpacks-en-1.0-17.fc30.noarch                       8/14
  Running scriptlet: dbus-broker-19-2.fc30.x86_64                          9/14
  Cleanup          : dbus-broker-19-2.fc30.x86_64                          9/14
  Running scriptlet: dbus-broker-19-2.fc30.x86_64                          9/14
  Cleanup          : langpacks-en-1.0-16.fc30.noarch                      10/14
  Running scriptlet: systemd-241-5.git3d835d0.fc30.x86_64                 11/14
  Cleanup          : systemd-241-5.git3d835d0.fc30.x86_64                 11/14
  Cleanup          : systemd-rpm-macros-241-5.git3d835d0.fc30.noarch      12/14
  Cleanup          : systemd-libs-241-5.git3d835d0.fc30.x86_64            13/14
  Cleanup          : systemd-pam-241-5.git3d835d0.fc30.x86_64             14/14
  Running scriptlet: systemd-pam-241-5.git3d835d0.fc30.x86_64             14/14
  Running scriptlet: systemd-241-6.gita2eaa1c.fc30.x86_64                 14/14
  Verifying        : libxkbcommon-0.8.3-1.fc30.x86_64                      1/14
  Verifying        : xkeyboard-config-2.24-5.fc30.noarch                   2/14
  Verifying        : dbus-broker-20-2.fc30.x86_64                          3/14
  Verifying        : dbus-broker-19-2.fc30.x86_64                          4/14
  Verifying        : langpacks-en-1.0-17.fc30.noarch                       5/14
  Verifying        : langpacks-en-1.0-16.fc30.noarch                       6/14
  Verifying        : systemd-241-6.gita2eaa1c.fc30.x86_64                  7/14
  Verifying        : systemd-241-5.git3d835d0.fc30.x86_64                  8/14
  Verifying        : systemd-libs-241-6.gita2eaa1c.fc30.x86_64             9/14
  Verifying        : systemd-libs-241-5.git3d835d0.fc30.x86_64            10/14
  Verifying        : systemd-pam-241-6.gita2eaa1c.fc30.x86_64             11/14
  Verifying        : systemd-pam-241-5.git3d835d0.fc30.x86_64             12/14
  Verifying        : systemd-rpm-macros-241-6.gita2eaa1c.fc30.noarch      13/14
  Verifying        : systemd-rpm-macros-241-5.git3d835d0.fc30.noarch      14/14

Upgraded:
  dbus-broker-20-2.fc30.x86_64
  langpacks-en-1.0-17.fc30.noarch
  systemd-241-6.gita2eaa1c.fc30.x86_64
  systemd-libs-241-6.gita2eaa1c.fc30.x86_64
  systemd-pam-241-6.gita2eaa1c.fc30.x86_64
  systemd-rpm-macros-241-6.gita2eaa1c.fc30.noarch

Installed:
  libxkbcommon-0.8.3-1.fc30.x86_64      xkeyboard-config-2.24-5.fc30.noarch

Complete!
Failed to set locale, defaulting to C
Last metadata expiration check: 0:01:24 ago on Sun Apr 14 20:49:20 2019.
Package systemd-241-6.gita2eaa1c.fc30.x86_64 is already installed.
Dependencies resolved.
================================================================================
 Package           Arch       Version                 Repository           Size
================================================================================
Installing:
 neovim            x86_64     0.3.4-1.fc30            updates-testing     5.0 M
Installing dependencies:
 libuv             x86_64     1:1.27.0-1.fc30         updates-testing     127 k
 libX11            x86_64     1.6.7-1.fc30            fedora              593 k
 libX11-common     noarch     1.6.7-1.fc30            fedora              155 k
 libXau            x86_64     1.0.9-1.fc30            fedora               30 k
 libtermkey        x86_64     0.20-6.fc30             fedora               25 k
 libvterm          x86_64     0-0.8.bzr681.fc30       fedora               35 k
 libxcb            x86_64     1.13.1-2.fc30           fedora              202 k
 luajit            x86_64     2.1.0-0.9beta3.fc30     fedora              327 k
 msgpack           x86_64     3.1.0-2.fc30            fedora               29 k
 unibilium         x86_64     2.0.0-4.fc30            fedora               32 k
Installing weak dependencies:
 xsel              x86_64     1.2.0-24.fc30           fedora               26 k

Transaction Summary
================================================================================
Install  12 Packages

Total download size: 6.6 M
Installed size: 25 M
Downloading Packages:
(1/12): libuv-1.27.0-1.fc30.x86_64.rpm          787 kB/s | 127 kB     00:00
(2/12): libX11-common-1.6.7-1.fc30.noarch.rpm   233 kB/s | 155 kB     00:00
(3/12): libX11-1.6.7-1.fc30.x86_64.rpm          683 kB/s | 593 kB     00:00
(4/12): libXau-1.0.9-1.fc30.x86_64.rpm          314 kB/s |  30 kB     00:00
(5/12): neovim-0.3.4-1.fc30.x86_64.rpm          5.0 MB/s | 5.0 MB     00:01
(6/12): libtermkey-0.20-6.fc30.x86_64.rpm       187 kB/s |  25 kB     00:00
(7/12): libvterm-0-0.8.bzr681.fc30.x86_64.rpm   374 kB/s |  35 kB     00:00
(8/12): libxcb-1.13.1-2.fc30.x86_64.rpm         1.7 MB/s | 202 kB     00:00
(9/12): luajit-2.1.0-0.9beta3.fc30.x86_64.rpm   2.1 MB/s | 327 kB     00:00
(10/12): msgpack-3.1.0-2.fc30.x86_64.rpm        203 kB/s |  29 kB     00:00
(11/12): unibilium-2.0.0-4.fc30.x86_64.rpm      347 kB/s |  32 kB     00:00
(12/12): xsel-1.2.0-24.fc30.x86_64.rpm          288 kB/s |  26 kB     00:00
--------------------------------------------------------------------------------
Total                                           3.5 MB/s | 6.6 MB     00:01
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                        1/1
  Installing       : unibilium-2.0.0-4.fc30.x86_64                         1/12
  Installing       : libtermkey-0.20-6.fc30.x86_64                         2/12
  Installing       : msgpack-3.1.0-2.fc30.x86_64                           3/12
  Installing       : luajit-2.1.0-0.9beta3.fc30.x86_64                     4/12
  Installing       : libvterm-0-0.8.bzr681.fc30.x86_64                     5/12
  Installing       : libXau-1.0.9-1.fc30.x86_64                            6/12
  Installing       : libxcb-1.13.1-2.fc30.x86_64                           7/12
  Installing       : libX11-common-1.6.7-1.fc30.noarch                     8/12
  Installing       : libX11-1.6.7-1.fc30.x86_64                            9/12
  Installing       : xsel-1.2.0-24.fc30.x86_64                            10/12
  Installing       : libuv-1:1.27.0-1.fc30.x86_64                         11/12
  Installing       : neovim-0.3.4-1.fc30.x86_64                           12/12
  Running scriptlet: neovim-0.3.4-1.fc30.x86_64                           12/12
  Verifying        : libuv-1:1.27.0-1.fc30.x86_64                          1/12
  Verifying        : neovim-0.3.4-1.fc30.x86_64                            2/12
  Verifying        : libX11-1.6.7-1.fc30.x86_64                            3/12
  Verifying        : libX11-common-1.6.7-1.fc30.noarch                     4/12
  Verifying        : libXau-1.0.9-1.fc30.x86_64                            5/12
  Verifying        : libtermkey-0.20-6.fc30.x86_64                         6/12
  Verifying        : libvterm-0-0.8.bzr681.fc30.x86_64                     7/12
  Verifying        : libxcb-1.13.1-2.fc30.x86_64                           8/12
  Verifying        : luajit-2.1.0-0.9beta3.fc30.x86_64                     9/12
  Verifying        : msgpack-3.1.0-2.fc30.x86_64                          10/12
  Verifying        : unibilium-2.0.0-4.fc30.x86_64                        11/12
  Verifying        : xsel-1.2.0-24.fc30.x86_64                            12/12

Installed:
  neovim-0.3.4-1.fc30.x86_64             xsel-1.2.0-24.fc30.x86_64
  libuv-1:1.27.0-1.fc30.x86_64           libX11-1.6.7-1.fc30.x86_64
  libX11-common-1.6.7-1.fc30.noarch      libXau-1.0.9-1.fc30.x86_64
  libtermkey-0.20-6.fc30.x86_64          libvterm-0-0.8.bzr681.fc30.x86_64
  libxcb-1.13.1-2.fc30.x86_64            luajit-2.1.0-0.9beta3.fc30.x86_64
  msgpack-3.1.0-2.fc30.x86_64            unibilium-2.0.0-4.fc30.x86_64

Complete!
Failed to set locale, defaulting to C
47 files removed
Failed to set locale, defaulting to C
Error: Failed to mount filesystems: mkdir /var/cache/distrobuilder.175713338/rootfs/.distrobuilder/0: file exists

Document or remove code in "init()"

It's unclear to me why we must change the environment of distrobuilder during init.

I can see how having a fixed value for PATH, SHELL, TERM, ... may be useful when calling the package managers, but then that should probably be done through an exec wrapper rather than modify distrobuilder's own environment.

Get latest ArchLinux release from mirror instead of website

Right now we grab the latest ArchLinux release by parsing https://www.archlinux.org/download/ but this isn't guaranteed to match what's on the mirror we're using, leading to download failures.

We should instead pick whatever's the highest timestamp on the download mirror itself.

Add support for Ubuntu Core

We seem to be missing support for building Ubuntu Core containers and it's one of the few remaining distros to switch over.

This one will be slightly tricky as we need to inject our fake bootloader and some other data and I still need to figure out what's needed for core 18, so I'll be taking this one myself.

debian-cloud-init example lost the cloud-init part of the example

with d29bfc5 the cloud init example for Debian is replaced with an example without the cloud-init support in the container.

is this intentional that the documentation is lost, or just an oversight in updating the documentation ?

CLI behavior

It took me a while to figure out what was going on when first trying to run distrobuilder as "--help" doesn't show you anything and running it without arguments just hangs as it tries to read from stdout.

I think it'd be best to always require some kind of input, using "-" (as already supported) to denote reading from stdin.

I'd expect the following to work at least:

• distrobuilder => Print help
• distrobuilder --help => Print help
• distrobuilder --version => Print version number

Looking at the current cli structure, I suspect the easiest would be to use sub-commands for:

• lxc
• lxd
• plain (maybe rename to "directory"?)

The following would be LXD specific options:

• compression
• type (unified or split)

So it'd look something like:

distrobuilder [--help|--version]
              build-lxc <filename|-> [target dir] [KEY=VALUE...]
              build-lxd <filename|-> [target dir] [--type=TYPE] [--compression=COMPRESSION] [KEY=VALUE...]
              build-dir <filename|-> <target dir> [KEY=VALUE...]

              pack-lxc <filename|-> <source dir> [target dir]
              pack-lxd <filename|-> <source dir> [target dir] [--type=TYPE] [--compression=COMPRESSION]

The expectation there as far directory handling is that for lxc and lxd, we use a temporary directory to create stuff (in /tmp) and don't leave anything behind.

In the case where we want to publish both a LXC and LXD image, I'd do:

distrobuilder build-dir ubuntu.yaml my-image/
distrobuilder pack-lxc ubuntu.yaml my-image/
distrobuilder pack-lxd ubuntu.yaml my-image/

As a reminder, the KEY=VALUE stuff is so that I can override some individual fields like:

distrobuilder build-lxd ubuntu.yaml image.release=bionic image.arch=armv7l

Add missing tests for lxc/lxd image generation and file templating

Title says it all, we need tests for lxc/lxd image generation and for the file templating code.

OpenSUSE leap 15 download pattern change

Looks like Leap 15 URLs have changed as they're all failing now.

Example of a now broken one:

Error: Error while downloading source: Error downloading openSUSE image: Unable to fetch https://download.opensuse.org/repositories/Virtualization:/containers:/images:/openSUSE-Leap-15.0/containers//opensuse-leap-15.0-image.x86_64-lxc.tar.xz: 404 Not Found

Tumbleweed appears to be fine