go-debos/debos

Debian OS builder

debos - Debian OS image builder

Synopsis

debos [options] <recipe file in YAML>
debos [--help]

Application Options:

  -b, --fakemachine-backend=   Fakemachine backend to use (default: auto)
      --artifactdir=           Directory for packed archives and ostree repositories (default: current directory)
  -t, --template-var=          Template variables (use -t VARIABLE:VALUE syntax)
      --debug-shell            Fall into interactive shell on error
  -s, --shell=                 Redefine interactive shell binary (default: bash) (default: /bin/bash)
      --scratchsize=           Size of disk backed scratch space
  -c, --cpus=                  Number of CPUs to use for build VM (default: 2)
  -m, --memory=                Amount of memory for build VM (default: 2048MB)
      --show-boot              Show boot/console messages from the fake machine
  -e, --environ-var=           Environment variables (use -e VARIABLE:VALUE syntax)
  -v, --verbose                Verbose output
      --print-recipe           Print final recipe
      --dry-run                Compose final recipe to build but without any real work started
      --disable-fakemachine    Do not use fakemachine.

Description

debos is a tool to make the creation of various Debian-based OS images simpler. While most other tools focus on specific use-cases, debos is designed to be a toolchain making common actions trivial while providing enough rope to do whatever tweaking which might be required behind the scenes.

debos expects a YAML file as input and runs the actions listed in the file sequentially. These actions should be self-contained and independent of each other.

Some of the actions provided by debos to customise and produce images are:

• apt: install packages and their dependencies with 'apt'
• debootstrap: construct the target rootfs with debootstrap
• download: download a single file from the internet
• filesystem-deploy: deploy a root filesystem to an image previously created
• image-partition: create an image file, make partitions and format them
• ostree-commit: create an OSTree commit from rootfs
• ostree-deploy: deploy an OSTree branch to the image
• overlay: do a recursive copy of directories or files to the target filesystem
• pack: create a tarball with the target filesystem
• pacman: install packages and their dependencies with pacman
• pacstrap: construct the target rootfs with pacstrap
• raw: directly write a file to the output image at a given offset
• recipe: includes the recipe actions at the given path
• run: allows to run a command or script in the filesystem or in the host
• unpack: unpack files from archive in the filesystem

A full syntax description of all the debos actions can be found at: https://godoc.org/github.com/go-debos/debos/actions

Installation (Docker container)

An official debos container is available:

docker pull godebos/debos

See docker/README.md for usage.

Installation from source (under Debian)

sudo apt install golang git libglib2.0-dev libostree-dev qemu-system-x86 \
     qemu-user-static debootstrap systemd-container
export GOPATH=/opt/src/gocode # or whatever suits your needs
go install -v github.com/go-debos/debos/cmd/debos@latest
/opt/src/gocode/bin/debos --help

Simple example

The following example will create an arm64 image, install several packages in it, change the file /etc/hostname to "debian" and finally make a tarball of the complete system.

{{- $image := or .image "debian.tgz" -}}

architecture: arm64

actions:
  - action: debootstrap
    suite: bookworm
    components:
      - main
      - non-free-firmware
    mirror: https://deb.debian.org/debian
    variant: minbase

  - action: apt
    packages: [ sudo, openssh-server, adduser, systemd-sysv, firmware-linux ]

  - action: run
    chroot: true
    command: echo debian > /etc/hostname

  - action: pack
    file: {{ $image }}
    compression: gz

To run it, create a file named example.yaml and run:

debos example.yaml

The final tarball will be named "debian.tgz" if you would like to modify the fileame, you can provided a different name for the variable image like this:

debos -t image:"debian-arm64.tgz" example.yaml

Other examples

Example recipes are collected in a separate repository:

https://github.com/go-debos/debos-recipes

Environment variables

debos reads a predefined list of environment variables from the host and propagates them to the fakemachine build environemnt. The set of environment variables is defined by environ_vars in cmd/debos/debos.go. Currently the list of environment variables includes the proxy environment variables documented at:

https://wiki.archlinux.org/index.php/proxy_settings

The list of environment variables currently exported to fakemachine is:

http_proxy, https_proxy, ftp_proxy, rsync_proxy, all_proxy, no_proxy

While the elements of environ_vars are in lower case, for each element both lower and upper case variants are probed on the host and if found propagated to fakemachine. So if the host has the environment variables HTTP_PROXY and no_proxy defined, both will be propagated to fakemachine respecting the case.

The command line options --environ-var and -e can be used to specify, overwrite and unset environment variables for fakemachine with the syntax:

$ debos -e ENVIRONVAR:VALUE ...

To unset an environment variable, or in other words, to prevent an environment variable being propagated to fakemachine, use the same syntax without a value. debos accepts multiple -e simultaneously.

Proxy configuration

While the proxy related environment variables are exported from the host to fakemachine, there are two known sources of issues:

• Using localhost will not work from fakemachine. Use an address which is valid on your network. debos will warn if the environment variables contain localhost.

• In case you are running applications and/or scripts inside fakemachine you may need to check which are the proxy environment variables they use. Different apps are known to use different environment variable names and different case for environment variable names.

Fakemachine Backend

debos (unless running debos with the --disable-fakemachine argument) creates and spawns a virtual machine using fakemachine and executes the actions defined by the recipe inside the virtual machine. This helps ensure recipes are reproducible no matter the host environment.

Fakemachine can use different virtualisation backends to spawn the virtual machine, for more information see the fakemachine documentation.

By default the backend will automatically be selected based on what is supported by the host machine, but this can be overridden using the --fakemachine-backend / -b option. If no backends are supported, debos reverts to running the recipe on the host without creating a fakemachine.

Performance of the backends is roughly as follows: kvm is faster than uml is faster than qemu. Using --disable-fakemachine is slightly faster than kvm, but requires root permissions.

Benchmark times for running pine-a64-plus/debian.yaml on an Intel Pentium G4560T with SSD:

Backend	Wall Time	Prerequisites
--disable-fakemachine	8 min	root permissions
-b kvm	9 min	access to /dev/kvm
-b uml	18 min	package user-mode-linux installed
-b qemu	166 min	none

debos's Issues

Make the --merged-usr option for debootstrap configurable

Some old packages (libpng12-0_1.2.54-1ubuntu1, I'm looking at you) return puzzling errors with a merged /usr.

To make debos usable in those cases it should be possible to disable passing the --merged-user option to deboostrap.

Ability to unpack '.deb' packages to use files without package install

Recipes validator is needed

Need a recipes validator able to warn user about incorrect type passed to avoid issues similar to #74

Future documentation ideas

From some discussions outside github, some ideas for a future documentation:

• move to rst/md all the non developer information about debos
• document the need to have rw permission to use /dev/kvm by the user
• document how to install debos and/or use public docker image ?
• explain that recipes are run though go templating

Print YAML after template expansion when failing to parse it

The YAML parser returns line numbers when encountering errors, but that's useless as they don't necessarily match the original line numbers in the source file due to template expansion.

Ideally it would be great to print a few lines around the error but the yaml module does not provide the line number information in its error type (it's in the string, but let's not go that way). Missing that, I think the best viable option would be to just dump the whole expanded YAML on errors.

action: overlay: add verification either 'source' or 'origin' has been set

Source or origin must be set to prevent copying of recipedir

Add live-build-like match-based overlay folders

Live-build has a fancy match-based overlay system, where it looks for the folders matching the current suite, arch and platform, even partially.

It basically expands the (suite,arch,platform) tuple and automatically overlays the contents of any existing folder that matches the expansion:

$ bash -c 'SUITE=buster; ARCH=amd64; PLATFORM=uefi; echo {suite,$SUITE}-{arch,$ARCH}-{platform,$PLATFORM}'
suite-arch-platform suite-arch-uefi suite-amd64-platform suite-amd64-uefi buster-arch-platform buster-arch-uefi buster-amd64-platform buster-amd64-uefi

Having some kind of folder matching for overlays in debos could be useful, doing that through templates becomes quite verbose very quickly.

action: raw: allow to select partition

Sometimes it is needed to write directly to partition instead of the disk image.
Need additional parameter for Raw action for pointing to correct partition on image.

Fall into interactive debug shell if action fails

[Idea] Add support for pip from yaml

Add option to install things with pip in yaml, I can imagine something like:

- action: pip
  version: 3
  packages:
      - pyserial
      - sarlanga

Open a shell in the fakemachine on error

When something breaks nothing beats opening a shell and poking things.

Ideally all the exec'ed programs should be (optionally) run as "$PROGRAM || sh". Of course that's a wild simplification as we want $PROGRAM output to be mangled by debos while the shell need access to the user's stdin/stdout, we need to print a nice header before firing up the shell, the return value should be the original error, etc. etc., but hopefully you get the point. :)

Note that to be useful the shell should be fired in the same fakemachine as the command that failed, otherwise developers won't be able to properly inspect what went wrong.

Allow to use disk image on /scratch instead of tmpfs-based

Currently have 'Out of memory' for medium and large sized OS.

Be able to download .deb files from an apt repository

#8 adds the ability to unpack .deb files which is quite useful, but to download a .deb on demand you'd have to know the exact filename. In the case wehre you want to get the .deb because you want to raw write the bootloader into an image file this is a bit annoying as the url will change on every package update. If there were an action similar to download but takes an apt repository as parameters that could nicely be avoided

RFE: pass http_proxy through to commands

For those of us not adjacent to a mirror, apt-cacher-ng in proxy mode is a useful way to speed up the test/debug/fix cycle. Setting http_proxy is probably the best way: that has the benefit that it can work for both debootstrap and apt, and the generated filesystem is left with a canonical mirror like deb.debian.org hard-coded into it (as opposed to having the proxy or a local mirror end up in /etc in the output).

I think this should be a command-line option or environment variable instead of part of the recipe, because it's really a fact about the machine/LAN where you're running debos rather than a fact about the recipe itself, and doesn't make a whole lot of sense to version-control.

no_proxy might also be useful to pass through, for the case where locally-served packages can't or shouldn't go through the proxy.

https_proxy and ftp_proxy are perhaps less useful but work the same way as http_proxy, so wouldn't need be significantly more code.

OStree integration using systemd-nspawn

This is not a bug, more a feature request.
We used systemd-nspawn in a script called from debos to use target specific tools.
While they could have been in debos itself, they were already available in the target rootfs and were usable by script.
On arm, we needed mkimage, and on amd64 bootctl. mkimage was used thanks to qemu-arm-static.
It would be useful to have bind as option to bind the required folders.
To use with OStree, it is needed to get the OStree commit id from the image.

cat debootstrap.log if debootstrap fails

If debootstrap fails, it often doesn't print full details, just "see debootstrap.log", which is not a whole lot of use if the sysroot you're bootstrapping is in a transient virtual machine's /scratch filesystem.

In other projects I've done things like this (argv is a debootstrap command-line):

            try:
                self.root_worker.check_call(argv)
            except:
                with suppress(Exception):
                    self.root_worker.check_call([
                        'cat',
                        '{}/debootstrap/debootstrap.log'.format(base_chroot),
                    ])
                raise

debootstrap versus /debootstrap/debootstrap

Hi,

In https://github.com/go-debos/debos/blob/master/actions/debootstrap_action.go there is
for Run
cmdline := []string{"debootstrap", ... }
and for RunSecondStage
cmdline := []string{"/debootstrap/debootstrap", ... }

I would expect that both paths to the debootstrap executable would be the same.

Why isn't the path the same?

Cheers
Geert Stappers

actions: image-partition: remove hardcoded path to disk image

Currently '/dev/vda' is hardcoded. So adding any other disk image prior to this action breaks normal work sequence.

Not able to correctly use downloaded file with in action ''raw"

In case if a single file is needed the origin is set to the file name.
It is not possible to use the pair origin + filename in other actions.
Example:

  - action: download
    name: uboot
    url: http://linode.boundarydevices.com/u-boot-images/u-boot.nitrogen6q

Ths action sets origin named uboot to downloaded file /scratch/u-boot.nitrogen6q

• Expected raw action

  - action: raw
    description: Install U-Boot
    origin: uboot
    source: u-boot.nitrogen6q
    offset: {{ sector 2 }}

is trying to write file named /scratch/u-boot.nitrogen6q/u-boot.nitrogen6q

• Workaround

  - action: raw
    description: Install U-Boot
    origin: uboot
    source: ../u-boot.nitrogen6q
    offset: {{ sector 2 }}

Expose the --include option of deboostrap in the configuration

When deboostrapping HTTPS-based repositories the apt-transport-https and ca-certificates should be installed right from the start, and that's usually handled with the --include option.

In my specific case their availability is probably better ensured through a custom debootstrap script (see #16), but I wonder if there's value in exposing more deboostrap options.

Preseed support

Need a preseed mechanism support.

Please merge recepies here

https://github.com/go-debos/debos-recipes should be merged in here so examples, as essential part of documentation, are shipped together with Debos.

usable end-to-end examples?

It would be nice to have some usable end-to-end examples of how to make an image using Debos and how to run the image. Preferably for some popular architectures like amd64, arm, etc.

I've successfully built an image using Debos but I can not boot it...

how to INSTALL

Hi,

How to install debos?

Yes, I'm new to the go-lang world.

In the GCC world there is the good tradition of providing an INSTALL file.

Please provide an eqivalent of the

./configure
make
sudo make install

dance, preferable in INSTALL :-)

See also https://en.wikipedia.org/wiki/README#As_a_generic_term

Make the debootstrap script argument configurable

debos currently hardcodes /usr/share/debootstrap/scripts/unstable as the debootstrap script, which of course is not always appropriate.

It should be customizable, possibly with some magic to make local files available in the fakemachine even without installing the script in the host's /usr (it would be handy to ship the script in the same repository as the YAML for debos and directly use that).

commands: systemd-nspawn changes resolv.conf in target environment

During call of any command/script in chroot environment 'systemd-nspawn' copies resolv.conf from host's environment.

Clarify that debos recipes are pre-processed using go templating before yaml parsing

Consider the following snippet in a recipe:

{{ $kver := "4.9.0" }}
#{{ $kver := "4.16.0" }}

Currently the value of kver will be set to 4.16.0, since the Golang templating engine is not aware of the YAML syntax. But this is definitely not what an user expects, comments should be ignored.

Support having devices as image files

Ability to create OS directly on block devices.

Install debug packages automatically for each installed package

Prop: produce a folder with package sources

Useful to run autopkgtests

actions: filesystem-deploy: add option for kernel command parameters

Add the same option 'append-kernel-cmdline' as we have for 'ostree-deploy' action.

actions: image-partition: add support for Partition Type GUIDs

Partition Type GUIDs are used to identify GPT partitions.

They are at the core of the Discoverable Partition Spec, which is implemented by systemd-auto-gpt-generator. This generator can't work without it.

Right now I set these Partition Type GUIDs through a post-process step such as:

  # EFI System Partition
  - action: run
    postprocess: true
    command: /sbin/sfdisk --part-type image.img 1 c12a7328-f81f-11d2-ba4b-00a0c93ec93b

It would be nice to have it supported in the image-partition action though, with something like:

  partitions:
    - name: label
    - part-type: c12a7328-f81f-11d2-ba4b-00a0c93ec93b
      ...

Is it something that you would like to support in debos?

Update installation steps for ubuntu 16.04

I don't see any caveats that this tool wont work on 16.04 (xenial) however the installation steps in the README do not work on 16.04. Is there:

1. any reason this won't work on older ubuntu distributions
2. are there alternate installation steps for those distros (I can update the README if so).

Print commandline for commands to be executed

Sometimes knowing exactly what is getting executed is extremely helpful when debugging an error. Printing the commandline before executing a program provides a good hint to reproduce it (just an hint because there's fakemachine and what not to make things more complex, but often that's enough). :)

Should explicitly disable merged /usr if disabled in YAML

In debootstrap between 1.0.85 (inclusive) and 1.0.87 (exclusive), and in debootstrap 1.0.102+, merged /usr is the default. debos correctly uses the --merged-usr option when merged-usr: true is in the YAML, but does not use --no-merged-usr when merged-usr: false is in the YAML.

This particularly matters when bootstrapping distributions that are so old they are not mentioned in debootstrap's own merged-usr blacklist, like Ubuntu 12.04 'precise'.

Or possibly merged-usr should be some sort of tristate: true, false, or use the debootstrap default?

Running debos in a chroot

Hi,

Debos is now in Debian and I have it installed in a chroot.

Testdriving debos with the rpi3 recipe reveals that some test on the host system are performed.
I don't understand why /lib/modules and Kernel are needed in early execution stage.

Screen shot:

root@yanur:/src/githubclones/debos-recipes/debian/arm64/image-rpi3# debos debimage-rpi3.yaml 
open /lib/modules: no such file or directory
root@yanur:/src/githubclones/debos-recipes/debian/arm64/image-rpi3# mkdir /lib/modules
root@yanur:/src/githubclones/debos-recipes/debian/arm64/image-rpi3# debos debimage-rpi3.yaml 
No kernel found
root@yanur:/src/githubclones/debos-recipes/debian/arm64/image-rpi3# 

Why the search for a kernel?

should have a way to turn off metadata_csum for mkfs.ext*

When using a recent host distro (Debian >= stretch) to build images based on an older distro (Debian <= jessie), mkfs.ext4 will enable the metadata_csum feature by default, but jessie's e2fsck doesn't understand that feature. I'm not sure whether jessie kernels understand it either (I'm currently working with a jessie derivative that happens to have a newer kernel).

vmdebootstrap automatically disables metadata_csum for jessie, but that's probably too much magic for debos' design. It would be useful to have a way to turn off that flag, even if it's just adding ^metadata_csum or -O ^metadata_csum to the YAML somewhere.

Example in README.md fails to build

Just discovered debos, and I love it! I copied the example out of the readme to get started:

{{- $image := or .image "debian.tgz" -}}

architecture: arm64

actions:
  - action: debootstrap
    suite: "buster"
    components:
      - main
      - non-free
    mirror: https://deb.debian.org/debian
    variant: minbase

  - action: apt
    packages: [ sudo, openssh-server, adduser, systemd-sysv, firmware-linux ]

  - action: run
    chroot: true
    command: echo debian > /etc/hostname

  - action: pack
    file: {{ $image }}
    compression: gz

When I run debos example.yaml, the scratch VM boots and starts building, but debootstrap fails near the end of base system installation:

<most debootstrap spam snipped>
2018/08/17 19:28:33 Debootstrap (stage 2) | I: Configuring the base system...
2018/08/17 19:28:33 Debootstrap (stage 2) | I: Configuring apt-transport-https...
2018/08/17 19:28:33 Debootstrap (stage 2) | I: Configuring libssl1.1:arm64...
2018/08/17 19:28:34 Debootstrap (stage 2) | I: Configuring openssl...
2018/08/17 19:28:34 Debootstrap (stage 2) | I: Configuring ca-certificates...
2018/08/17 19:29:09 Debootstrap (stage 2) | I: Configuring libc-bin...
2018/08/17 19:29:09 Debootstrap (stage 2) | I: Configuring ca-certificates...
2018/08/17 19:29:18 Debootstrap (stage 2) | I: Base system installed successfully.
2018/08/17 19:29:19 Action `debootstrap` failed at stage Run, error: exec: "systemd-nspawn": executable file not found in $PATH
Powering off.

I've only dug a tiny bit, but it looks like systemd-nspawn maybe got split out into its own package, systemd-container, and debootstrap isn't aware of this yet? Is there anything debos can do to work around this?

vfat formatting failing on first run

I'm hitting an issue when running the following recipe:

architecture: amd64

actions:

  - action: image-partition
    imagename: test.img
    imagesize: 20G
    partitiontype: gpt

    mountpoints:
      - mountpoint: /
        partition: system
        flags: [ boot ]
      - mountpoint: /boot/efi
        partition: EFI

    partitions:
      - name: EFI
        fs: vfat
        start: 6176s
        end: 256M
        flags: [ boot ]
      - name: system
        fs: btrfs
        start: 256M
        end: 100%

This fails on first run with a mkfs.vfat error (as shown below) but will succeed on subsequent attempts. Whether it succeeds or fails appears to be linked to the use of /dev/loop0:

user@apertis:~$ losetup
user@apertis:~$ sudo debos test.yaml
2018/05/23 14:00:56 fakemachine not supported, running on the host!
2018/05/23 14:00:56 ==== image-partition ====
2018/05/23 14:00:56 Formatting partition 1 | mkfs.vfat: Will not try to make filesystem on full-disk device '/dev/loop0p1' (use -I if wanted)
2018/05/23 14:00:56 Formatting partition 1 | mkfs.vfat 2.11 (12 Mar 2005)
2018/05/23 14:00:56 Action `image-partition` failed at stage Run, error: exit status 1
user@apertis:~$ losetup
user@apertis:~$ sudo debos test.yaml
2018/05/23 14:01:03 fakemachine not supported, running on the host!
2018/05/23 14:01:03 ==== image-partition ====
2018/05/23 14:01:03 Formatting partition 1 | mkfs.vfat 2.11 (12 Mar 2005)
2018/05/23 14:01:03 Formatting partition 1 | unable to get drive geometry, using default 255/63
2018/05/23 14:01:03 Formatting partition 2 | btrfs-progs v4.4
2018/05/23 14:01:03 Formatting partition 2 | See http://btrfs.wiki.kernel.org for more information.
2018/05/23 14:01:03 Formatting partition 2 | 
2018/05/23 14:01:03 Formatting partition 2 | Performing full device TRIM (18.39GiB) ...
2018/05/23 14:01:03 Formatting partition 2 | Label:              system
2018/05/23 14:01:03 Formatting partition 2 | UUID:               29114924-0aa2-47da-aaaf-3e59343371b0
2018/05/23 14:01:03 Formatting partition 2 | Node size:          16384
2018/05/23 14:01:03 Formatting partition 2 | Sector size:        4096
2018/05/23 14:01:03 Formatting partition 2 | Filesystem size:    18.39GiB
2018/05/23 14:01:03 Formatting partition 2 | Block group profiles:
2018/05/23 14:01:03 Formatting partition 2 |   Data:             single            8.00MiB
2018/05/23 14:01:03 Formatting partition 2 |   Metadata:         DUP               1.01GiB
2018/05/23 14:01:03 Formatting partition 2 |   System:           DUP              12.00MiB
2018/05/23 14:01:03 Formatting partition 2 | SSD detected:       no
2018/05/23 14:01:03 Formatting partition 2 | Incompat features:  extref, skinny-metadata
2018/05/23 14:01:03 Formatting partition 2 | Number of devices:  1
2018/05/23 14:01:03 Formatting partition 2 | Devices:
2018/05/23 14:01:03 Formatting partition 2 |    ID        SIZE  PATH
2018/05/23 14:01:03 Formatting partition 2 |     1    18.39GiB  /dev/loop1p2
2018/05/23 14:01:03 Formatting partition 2 | 
2018/05/23 14:01:04 ==== Recipe done ====
user@apertis:~$ losetup
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         0  0 /home/user/test.img
/dev/loop1         0      0         0  0 /home/user/test.img

It also appears that debos is not clearing up loopback devices that it has created.

Tested on commit 8f2bc2a

Create a standard docker image with debos to push to docker hub

RFE: pack, `ostree commit`, or run commands in a subdirectory

debos knows how to pack up the entire $ROOTDIR into a tarball, but it can't pack up (for example) just /usr into a tarball.

Similarly, it can commit the entire $ROOTDIR to an ostree repository, but can't commit something other than the root. This would be useful for Flatpak runtimes, which are meant to contain:

• /files: a copy of a merged /usr
• /metadata: a metadata file

Workaround: use action: run, chroot: false to run tar or ostree yourself. The way I'm currently doing this is to create ostree/main and ostree/sources in the $ROOTDIR, move usr to ostree/main/files, fetch corresponding source code into ostree/sources, and pack up ostree/sources and ostree/main as though they were my root directory.

If/when I split out time zones and locales into Flatpak extensions, I'll need to commit those separately: for instance, a timezone info extension would contain /files (a copy of /usr/share/zoneinfo) and /metadata.

Similarly, at the moment I'm using a debos recipe for the Platform runtime and a separate debos recipe for the Sdk runtime, and duplicating the common preparation used for both. Ideally my process would go something like this:

• populate $ROOTDIR with a complete Debian system that is slightly larger than the Platform
• do the /usr merge and other common preparation
• copy /usr, /etc and /var twice, for example into /platform and /sdk (hard-link or reflink to avoid duplicating large files)
• chroot into /platform and delete Essential packages that are pointless in an immutable, non-bootable runtime, like login, init, apt and dpkg, then commit/pack
• chroot into /sdk, install extra -dev packages, and download corresponding source code for the union of Platform and Sdk, then commit/pack

Workaround: use chroot: false and make the scripts enter the appropriate chroot themselves, using chroot or systemd-nspawn.

Overlap with vmdb2

debos looks very similar to https://github.com/larswirzenius/vmdb2/ :)

Perhaps you could collaborate or merge the two projects.

cc @larswirzenius @d4s

actions: debootstrap: remove hardcoded '--no-check-gpg'

Add correct property for debootstrap action.

actions: image-partition: Add build-time mount points

Add mountpoint which are mounted only in build time, e.g. do not need to add them into fstab and need to remove mount points on rootfs after umounting.

Occasional build failure

I'm using go-debos and I am getting build failure.
It does not always occur. I'm on ubuntu and have a couple patches.

fredo@X250:~/Documents/bos0009/apertis-image-recipes$ ../go/src/github.com/go-debos/debos/debos -t architecture:$ARCH --debug-shell apertis-image-$ARCH.yaml
2017/11/17 11:36:47 Ignored link: /etc/ld.so.conf.d//etc/ld.so.conf.d/i386-linux-gnu_GL.conf
2017/11/17 11:36:47 Ignored link: /etc/ld.so.conf.d//etc/ld.so.conf.d/x86_64-linux-gnu_EGL.conf
2017/11/17 11:36:47 Ignored link: /etc/ld.so.conf.d//etc/ld.so.conf.d/x86_64-linux-gnu_GL.conf
2017/11/17 11:36:47 Getting module from host kernel/drivers/virtio/virtio.ko
2017/11/17 11:36:47 Ignoring kernel/drivers/virtio/virtio.ko
2017/11/17 11:36:47 Getting module from host kernel/drivers/virtio/virtio_pci.ko
2017/11/17 11:36:47 Ignoring kernel/drivers/virtio/virtio_pci.ko
2017/11/17 11:36:47 Getting module from host kernel/net/9p/9pnet.ko
2017/11/17 11:36:47 Getting module from host kernel/drivers/virtio/virtio_ring.ko
2017/11/17 11:36:47 Ignoring kernel/drivers/virtio/virtio_ring.ko
2017/11/17 11:36:47 Getting module from host kernel/fs/9p/9p.ko
2017/11/17 11:36:47 Getting module from host kernel/net/9p/9pnet_virtio.ko
2017/11/17 11:36:47 Getting module from host kernel/fs/fscache/fscache.ko
2017/11/17 11:36:47 Getting module from host modules.order
2017/11/17 11:36:47 Getting module from host modules.builtin
2017/11/17 11:36:47 Getting module from host modules.dep
2017/11/17 11:36:47 Getting module from host modules.dep.bin
2017/11/17 11:36:47 Getting module from host modules.alias
2017/11/17 11:36:47 Getting module from host modules.alias.bin
2017/11/17 11:36:47 Getting module from host modules.softdep
2017/11/17 11:36:47 Getting module from host modules.symbols
2017/11/17 11:36:47 Getting module from host modules.symbols.bin
2017/11/17 11:36:47 Getting module from host modules.builtin.bin
2017/11/17 11:36:47 Getting module from host modules.devname
[    1.341739] systemd-fstab-generator[137]: Failed to create mount unit file /run/systemd/generator/home-fredo-Documents-bos0009-apertis\x2dimage\x2drecipes.mount, as it already exists. Duplicate entry in /etc/fstab?
[    1.835466] Error: Driver 'pcspkr' is already registered, aborting...
Running /debos --artifactdir /home/fredo/Documents/bos0009/apertis-image-recipes --template-var architecture:"amd64" /home/fredo/Documents/bos0009/apertis-image-recipes/apertis-image-amd64.yaml --debug-shell --shell /bin/bash --internal-image /dev/disk/by-id/virtio-fakedisk-0
2017/11/17 10:36:51 ==== Unpack ospack_17.12-amd64-minimal_00000000.0.tar.gz ====
2017/11/17 10:36:54 ==== image-partition ====
2017/11/17 10:36:55 Formatting partition 1 | mkfs.fat 4.0 (2016-05-06)
2017/11/17 10:36:56 Formatting partition 2 | mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
2017/11/17 10:36:56 Formatting partition 2 | mkfs.fat 4.0 (2016-05-06)
2017/11/17 10:36:56 Formatting partition 4 | mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
2017/11/17 10:36:56 Formatting partition 4 | mkfs.fat 4.0 (2016-05-06)
2017/11/17 10:36:57 Formatting partition 5 | mkfs.fat: warning - lowercase labels might not work properly with DOS or Windows
2017/11/17 10:36:57 Formatting partition 5 | mkfs.fat 4.0 (2016-05-06)
2017/11/17 10:36:58 Formatting partition 6 | btrfs-progs v4.9.1
2017/11/17 10:36:58 Formatting partition 6 | See http://btrfs.wiki.kernel.org for more information.
2017/11/17 10:36:58 Formatting partition 6 | 
2017/11/17 10:36:58 Formatting partition 6 | Label:              system
2017/11/17 10:36:58 Formatting partition 6 | UUID:               61db6be5-758c-4153-8d28-12ad22f7048d
2017/11/17 10:36:58 Formatting partition 6 | Node size:          16384
2017/11/17 10:36:58 Formatting partition 6 | Sector size:        4096
2017/11/17 10:36:58 Formatting partition 6 | Filesystem size:    3.81GiB
2017/11/17 10:36:58 Formatting partition 6 | Block group profiles:
2017/11/17 10:36:58 Formatting partition 6 |   Data:             single            8.00MiB
2017/11/17 10:36:58 Formatting partition 6 |   Metadata:         DUP             195.31MiB
2017/11/17 10:36:58 Formatting partition 6 |   System:           DUP               8.00MiB
2017/11/17 10:36:58 Formatting partition 6 | SSD detected:       no
2017/11/17 10:36:58 Formatting partition 6 | Incompat features:  extref, skinny-metadata
2017/11/17 10:36:58 Formatting partition 6 | Number of devices:  1
2017/11/17 10:36:58 Formatting partition 6 | Devices:
2017/11/17 10:36:58 Formatting partition 6 |    ID        SIZE  PATH
2017/11/17 10:36:58 Formatting partition 6 |     1     3.81GiB  /dev/disk/by-id/virtio-fakedisk-0-part6
2017/11/17 10:36:58 Formatting partition 6 | 
2017/11/17 10:36:58 Formatting partition 7 | btrfs-progs v4.9.1
2017/11/17 10:36:58 Formatting partition 7 | See http://btrfs.wiki.kernel.org for more information.
2017/11/17 10:36:58 Formatting partition 7 | 
2017/11/17 10:36:58 Formatting partition 7 | Label:              general_storage
2017/11/17 10:36:58 Formatting partition 7 | UUID:               3d24d634-dbd0-4779-b17e-51925beea726
2017/11/17 10:36:58 Formatting partition 7 | Node size:          16384
2017/11/17 10:36:58 Formatting partition 7 | Sector size:        4096
2017/11/17 10:36:58 Formatting partition 7 | Filesystem size:    2.16GiB
2017/11/17 10:36:58 Formatting partition 7 | Block group profiles:
2017/11/17 10:36:58 Formatting partition 7 |   Data:             single            8.00MiB
2017/11/17 10:36:58 Formatting partition 7 |   Metadata:         DUP             110.81MiB
2017/11/17 10:36:58 Formatting partition 7 |   System:           DUP               8.00MiB
2017/11/17 10:36:58 Formatting partition 7 | SSD detected:       no
2017/11/17 10:36:58 Formatting partition 7 | Incompat features:  extref, skinny-metadata
2017/11/17 10:36:58 Formatting partition 7 | Number of devices:  1
2017/11/17 10:36:58 Formatting partition 7 | Devices:
2017/11/17 10:36:58 Formatting partition 7 |    ID        SIZE  PATH
2017/11/17 10:36:58 Formatting partition 7 |     1     2.16GiB  /dev/disk/by-id/virtio-fakedisk-0-part7
2017/11/17 10:36:58 Formatting partition 7 | 
2017/11/17 10:36:58 Action `image-partition` failed at stage Run, error: system mount failed: no such file or directory
2017/11/17 10:36:58 >>> Starting a debug shell

Documentation for actions

Description of actions and properties.

Option for partition mountpoint does not appeared in `/etc/fstab`

Option for mountpoint is ignored. For example:

  - action: image-partition
    imagename: {{ $image }}.img
    mountpoints:
      - mountpoint: /
        partition: system
        flags: [ boot ]
        options: ro

Not using local DNS; Unable to use local APT proxy

Debootstrap | E: Failed getting release file http://apt-cacher-ng:3142/debian/dists/unstable/Release

Apparently using a mirror URL without a port (e.g. http://deb.debian.org/debian) works...

commands: systemd-nspawn produce races for disk partitions usage

Udevd have no enough time to start and create all needed partition symlinks for commands using direct access to device:

2017/11/20 09:29:33 ==== Prepare uInitrd ====
2017/11/20 09:29:33 DEBUG: [systemd-nspawn -q -D /scratch/mnt sh -c dd if=/dev/zero of=/dev/disk/by-id/virtio-fakedisk-0-part3 bs=512 count=1]
2017/11/20 09:29:33 dd if=/dev/zero of=/dev/disk/by-id/virtio-fakedisk-0-part3 bs=512 count=1 | host's /etc/localtime is not a symlink, not updating container timezone.
2017/11/20 09:29:33 dd if=/dev/zero of=/dev/disk/by-id/virtio-fakedisk-0-part3 bs=512 count=1 | /bin/dd: opening `/dev/disk/by-id/virtio-fakedisk-0-part3': No such file or 

Improve support for local mirrors via FS as opposed to network

I have a local mirror, which I was attempting to run debos against it passing file:///srv/mirror as mirror, of course that was failing as debootstrap stage happens inside virtual fakemachine.