andsens/build-debian-cloud

This project has been superseeded by andsens/bootstrap-vz and is no longer maintained - Script to create Debian Squeeze & Wheezy Amazon Machine Images (AMIs) and Google Compute Engine images

build-debian-cloud's Issues

Blacklist pcspkr

The pcspkr driver gives error messages on boot, disable it like it is done in joshkraemer/ec2debian-build-ami@2e26bb400b765f13a9f19775da6c8caa0584b3f9

Implement logfile

The logging functions can log to a file, it is not utilized currenty.
Implement the remaining stuff.

Find workaround for hostname.sh

hostname.sh has a bug. It falsely relies on hostname to return nothing, when the hostname is not set.
The errors are:

hostname: the specified hostname is invalid
startpar: service(s) returned failure: hostname.sh ... 

Append `; exit $?` to subshells

A lot of subshells that are spawned do not get checked for error codes.
Since the script runs with the -e option we can simply append ; exit 1 to these commands and force the script to exit if a subshell command fails.

spin function hides exit status codes >0

Non zero exit codes are ignored when piping into the spinning function.
That can be dangerous when apt-get is unable to fix a broken dependency etc.
It does however help when bootstrapping. deboostrap exits with non zero error codes for whatever reason, this needs to be taken care of simultaneously.

Drivers missing

Seems some XEN drivers are missing:

[13509379.040346] XENBUS: Device with no driver: device/vbd/2049
[13509379.040351] XENBUS: Device with no driver: device/vif/0
[13509379.040356] XENBUS: Device with no driver: device/console/0

(suggestions please!) Write plugin to include standard packages

The packages installed right now are at a bare minimum, which is fine.
Most installations share some common denominators however, a plugin which adds those packages, would be nice.
Any input as to which packages that should be would be very welcome.

EDIT: The current list can be seen here

must have iam role

Tasks 10 and 11 if an IAM role is not set on the ec2 instance launching the ec2debian-build-ami run.

Run task: 07-host-information
Run task: 10-create-volume
Unable to create volume.

because the output of CMD is
$ aws ec2 create-volume --region "ap-southeast-2" --availability-zone "ap-southeast-2a" --size "1"
'NoneType' object has no attribute 'access_key'

because
aws/aws-cli#12

and the code relies on
wget http://169.254.169.254/latest/meta-data/iam/

but
wget http://169.254.169.254/latest/meta-data/iam/
--2013-01-29 02:14:03-- http://169.254.169.254/latest/meta-data/iam/
Connecting to 169.254.169.254:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2013-01-29 02:14:03 ERROR 404: Not Found.

Change UUID of root volume on first boot

When debugging an instance, mounting the volume on another instance is the best way to go. This is a problem when the instance used for debugging and the instance to debug use the same AMI.
The system will complain that the UUID of the root volume and the one of the buggy instance is the same. This is an unnecessary problem people have to fix by googling how to override that check.

By changing the UUID on first boot, this problem can be avoided.

Add --version param

To reproduce specific AMIs, the user needs to be able to give a version parameter to the tool.
Figure out how to pass this on to debootstrap.

Use grub to generate menu.lst

The menu.lst is hardcoded. To autogenerate it, grub needs to be installed.

cloud-init

The ec2 init script are quite primitive, maybe cloud-init can be ported to debian?

Mounting local filesystem boot error

There is an error when "Mounting local filesystem", in the bootmessage

Mounting local filesystems...
    [31mfailed.
    [39;49m

Create alternate user plugin

Add --help option

Add an option listing all the possible options (+env vars).
Maybe just steal a lot from the readme.

commit 521cda1 Breaks Networking For Me

Hello,

Thank you for maintaining and working on this project. Today, re-pulled from master and tried to make an AMI. The AMI booted, but I was unable to get a network interface. I took a look at task 39 and I noticed you changed it to append instead of overwrite.

This seems to be break networking, and the instance no longer gets assigned an IP address. Only the local interface comes up.

I changed it back to overwrite and it started working again. Not sure what the motivation was behind making it append, but it doesn't work for me.

I'm using debian wheezy though, so if it's just a problem for that distribution, I can live with it.

Thanks

Ec2 regions issue

Script doesn't work at all when using e.g. EBS in EU location even if parameter "--region" is used because default is always US location so API commands will fail. It would be better to work in one location:

diff --git a/ec2debian-build-ami b/ec2debian-build-ami
index 6e49d2d..07623ac 100755
--- a/ec2debian-build-ami
+++ b/ec2debian-build-ami
@@ -208,6 +208,8 @@ if [ ! -f $EC2_PRIVATE_KEY ]; then
die "$0: AWS private key file not found: $EC2_PRIVATE_KEY"
fi

+export EC2_URL="https://$region.ec2.amazonaws.com"
+
if [ -z "$ebs" ]; then

mkfs.xfs - force overwrite

When EBS contains any fs you need the -f option to force overwrite:

diff --git a/tasks/create-volume b/tasks/create-volume
index bbd7bcd..e23b9ae 100755
--- a/tasks/create-volume
+++ b/tasks/create-volume
@@ -41,7 +41,7 @@ do
attachment_status=ec2-describe-volumes | grep "ATTACHMENT $volume_id" | awk '{print $5}'
done
echo "Formatting the device $device_path with $filesystem"
-mkfs.$filesystem $device_path
+mkfs.$filesystem -f $device_path
mount $device_path $builddir
echo "The EBS volume has been formatted and mounted"

Dirty metadata buffer

Somethings amiss on bootup. dmesg says:

JBD: Spotted dirty metadata buffer (dev = xvda1, blocknr = 1). There's a risk of filesystem corruption in case of system crash.

Unable to open rtc device

Error on bootup:

[13509379.040376] /build/buildd-linux-2.6_2.6.32-46-amd64-_ApuPc/linux-2.6-2.6.32/debian/build/source_amd64_xen/drivers/rtc/hctosys.c: unable to open rtc device (rtc0)

Seems to be something regarding the system clock.

Use debootstrap --include --exclude

debootstrap has --include and --exclude options for packages. Maybe they can be used instead of manually installing some packages and manually removing isc-dhcp.
If so, maybe we don't need to disable demons either.
There would actually be a lot of stuff that could be removed.

Architecture depends on host system

The --arch param does not really work since it depends upon the architecture of the host system

Don't delete root password

root doesn't have a password to begin with, so why chroot $imagedir passwd -d root in 38-security.

networking task is overly complex

Do we really need all the stuff that is in 40-networking

Create "dont-delete-volume" plugin

When developing, the volume deletion task makes it hard to inspect the whole thing after creation.
We need a plugin that removes the volume deletion task and remounts/reattaches it after the ami has been registered.

Hostname matches instance that created the image

Hello,
Awesome script.
I created an AMI on a host with hostname ip-10-130-29-xxx. I then launched this AMI, and the created images had /etc/hostname set to this hostname from the instance that created the AMI. Also, the motd referenced the host that created the image. Does this need a patch to ./tasks/70-cleanup to tidy up?
James
[email protected]

aptitude complains about locales

aptitude says:
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory

"dpkg-reconfigure locales" does the trick, but the problem should not exist in the AMI in the first place.

Remove file package when using no-ec2-run-user-data

The file package is included because ec2-run-user-data needs it. When using the plugin no-ec2-run-user-data, that packages should be removed.

ebs volume is hardcoded to /dev/sdf

The ebs volume attachment is hardcoded to /dev/sdf, it should instead use whatever device binding is available.

ebs-upload only specifying eu kernels

When using --ebs, the original kernel selection is needlessly circumvented and the european kernels are hardcoded.

Ec2 API commands not working correctly

Ec2 API commands not working correctly because some variables were not exported:

diff --git a/ec2debian-build-ami b/ec2debian-build-ami
index 6e49d2d..adcd551 100755
--- a/ec2debian-build-ami
+++ b/ec2debian-build-ami
@@ -204,10 +204,15 @@ true ${AWS_USER_ID:?} ${AWS_ACCESS_KEY_ID:?} ${AWS_SECRET_ACCESS_KEY:?}
if [ ! -f $EC2_CERT ]; then
die "$0: AWS certificate file not found: $EC2_CERT"
fi
+
+export EC2_CERT
+
if [ ! -f $EC2_PRIVATE_KEY ]; then
die "$0: AWS private key file not found: $EC2_PRIVATE_KEY"
fi

+export EC2_PRIVATE_KEY
+
if [ -z "$ebs" ]; then

Test dist-upgrade to wheezy

Test if upgrading from squeeze to wheezy works or if something breaks.

ec2-run-user-data is not working

The plugin ec2-run-user-data references a nonexistent task filename.
Also, the task to add the init.d script uses $plugindir, which is not available at the time of running a task.
This is perhaps something that should be fixed in general.

plugin admin-user not working for wheezy

The admin-user plugin is not working in wheezy.
When using sudo a password is required.
This is probably just a change in the sudoers file layout.

Fix error about sysfs

On boot we get:

Mounting local filesystems...mount: none already mounted or /sys busy
mount: according to mtab, sysfs is already mounted on /sys

This needs to be fixed.

Bootstrap tarball not being reused

The bootstrap tarball is not being reused between bootstraps.
This is probably because of the latest changes to $imagedir, which now depends on the volume-id that is mounted.

apt sources should depend on region of host system

Since AMIs cannot be moved between regions anyways we might as well set the apt sources to be point at mirrors close to the region (e.g. eu-west-1: ftp.ie.debian.org)

Check if ec2-mkdir-tmp is really necessary

Make ec2-ssh-host-key-gen better

I am pretty sure we don't need the ami_id to create a host key, which means we do not need to wait for the network.
We also don't need to create a been run file.

publish-* plugins should output info

The publishing plugins should output information that can be used to create a volume/launch an AMI.
To this end a task could be created: 97-output-info. It should print out the entries of a list, which plugins and tasks have added information to throughout the bootstrapping process.

`depmod -a` might not be necessary

The depmod -a part in the kernel task might not be necessary since aptitude maybe already does that for us.

Disable all gettys (?)

The commitmsg in f88a843 was actually false. There is an advantage to disabling TTYs: less processes.
I have tried disabling the TTYs 2-6 and also all of them (1-6, except 7/X). Debian seems to be running just fine without them.
I disabled them in /etc/inittab though instead of removing the files (messy).
I can however not foresee which consequences, if any, this will have, so any input will be highly appreciated.

Don't attach ephemeral volumes

The new AWS GUI allows users to attach ephemeral volumes themselves, they do not need to be attached by the bootstrapper in the AMI any longer.

LVM for root volume

It should be possible to make the root volume part of a logical volume, allowing easy snapshotting etc.
This should be written as a plugin.

Please use /etc/sudoers.d to give superuser rights to the default user.

Hi Anders,

Could you change plugins/admin-user-tasks/create-user to drop a configuration file in /etc/sudoers.d instead of modifying /etc/sudoers directly ? This will make smoother upgrades each time /etc/sudoers is modified by upgrade of the sudo package. Here is an example from Ubuntu, where the default account is "ubuntu".

root@ip-10-148-5-107:/etc# cat sudoers.d/90-cloudimg-ubuntu 
# ubuntu user is default user in cloud-images.
# It needs passwordless sudo functionality.
ubuntu ALL=(ALL) NOPASSWD:ALL

Cheers,

Charles Plessy
Tsurumi, Kanagawa, Japan

Required update initrd after grub is configured

Required update initrd after grub is configured otherwise instance will not boot:

diff --git a/tasks/grub b/tasks/grub
index 452ce66..9df14c4 100755
--- a/tasks/grub
+++ b/tasks/grub
@@ -10,3 +10,5 @@ root (hd0)
kernel /boot/vmlinuz-2.6.32-5-xen-amd64 root=/dev/xvda1 ro
initrd /boot/initrd.img-2.6.32-5-xen-amd64
EOF
+
+update-initramfs -u -k 2.6.32-5-xen-amd64

regression: EBS deletion on termination

The EBS volume is not set to delete when the instance terminates any longer.

Wrong name of package "xfsprogs"

Entered wrong name of package "xfsprogs":

diff --git a/tasks/host-dependencies b/tasks/host-dependencies
index 2a4e206..a8be1c4 100755
--- a/tasks/host-dependencies
+++ b/tasks/host-dependencies
@@ -1,6 +1,6 @@
#!/bin/bash

Install packages on the builing system to make the rest of this script run smoothly.

-apt-get install -y bzip2 binutils unzip debootstrap openssl curl ca-certificates default-jre-headless xfs
+apt-get install -y bzip2 binutils unzip debootstrap openssl curl ca-certificates default-jre-headless xfsprogs

ephemeral storage is not mapped

The ephemeral storage is not mapped to a device, this has to be done when starting an instance. It is possible howver to do it when registering the image. sda2 and sda3 seem to be good fits for that.

No error on duplicate AMI name

When registering an AMI with a name that already exists, the bootstrapper does not fail with a heplful message.