linutronix/elbe

Embedded Linux Build Environment

elbe's Issues

only one build in parallel is possible

% ./elbe initvm submit examples/x86_64-pc-hdimg-gnome3.xml --keep-files --skip-build-bin --skip-build-sources
elbe control create_project failed.
Server returned error:

project in directory /var/cache/elbe/8c3888f4-4652-4ae8-82ed-6f98c71a3545 of user root is currently busy and cannot be closed

Giving up

This is because the soap daemon holds a single project-manager instance. I think we need to extend the project manager handling mulitple projects.

Fix "Similar code" issue in elbepack/db.py

Similar code found in 4 other locations

https://codeclimate.com/github/Linutronix/elbe/elbepack/db.py#issue_568e29e2b1957d0001889365

there is no sane errorhandling if target image is too small for content

running cmd +cp -a "/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/filesystems/rfs"/* "/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt"+

cp: error writing '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/cache/apt/srcpkgcache.bin': No space left on device
cp: failed to extend '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/cache/apt/srcpkgcache.bin': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/cache/apt/archives': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/cache/debconf': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/cache/ldconfig': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/cache/man': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/cache/elbe': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/tmp': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/spool': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/log': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/local': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/backups': No space left on device
cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/opt': No space left on device

cp: cannot create directory '/var/cache/elbe/c2235b79-a071-4857-9d75-13167b6bb48c/imagemnt/var/mail': No space left on device

elbe hdimg with grub broken

When creating an image with , elbe creates a grub configuration
that passes 'root=/dev/poop1' on the kernel command line.

Funny? Yes.
Useful? No.

Thanks,
Richard

elbe chg_archive deletes xml comments

Running 'elbe chg_archive' does not preserve xml comments but rather deletes
them without any kind of warning. This command really should be polite and leave
the comments alone when updating the section. After all, somebody's
important notes or other critical information may be in the comment blocks.

Thanks,
Richard

Fix "Similar code" issue in elbepack/projectmanager.py

Similar code found in 3 other locations

https://codeclimate.com/github/Linutronix/elbe/elbepack/projectmanager.py#issue_568e29e397e18c000119d6b0

Copy a file from rfs to the build directory

It would be nice to be able to copy a file from the rfs to the build directory.

change bash prompt if 'elbe chroot' was called

The bash prompt is not modified if 'elbe chroot' is called to enter the target or buildimg RFS.

It would be very helpful to generate a prompt including the architecture and projectname/version of the target.

problems downloading kinitrd from cdrom iso unpacked onto httpd

files-to-extract is no longer generated

elbe init / make / relies on the files-to-extract file, however this file is no longer generated.

error while submitting large XML file (88M)

large XML file (88M) causes error on elbe initvm submit:
$ elbe initvm submit castellum.xml
elbe control create_project failed.
Traceback (most recent call last):
File "/usr/bin/elbe", line 61, in
cmdmod.run_command( sys.argv[2:] )
File "/usr/lib/python2.7/dist-packages/elbepack/commands/control.py", line 102, in run_command
action.execute (control, opt, args[1:])
File "/usr/lib/python2.7/dist-packages/elbepack/soapclient.py", line 171, in execute
print (client.service.create_project ( xml_base64 ))
File "/usr/lib/python2.7/dist-packages/suds/client.py", line 542, in call
return client.invoke(args, kwargs)
File "/usr/lib/python2.7/dist-packages/suds/client.py", line 602, in invoke
result = self.send(soapenv)
File "/usr/lib/python2.7/dist-packages/suds/client.py", line 641, in send
reply = transport.send(request)
File "/usr/lib/python2.7/dist-packages/suds/transport/https.py", line 64, in send
return HttpTransport.send(self, request)
File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 77, in send
fp = self.u2open(u2request)
File "/usr/lib/python2.7/dist-packages/suds/transport/http.py", line 118, in u2open
return url.open(u2request, timeout=tm)
File "/usr/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib/python2.7/urllib2.py", line 1227, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
urllib2.URLError: <urlopen error [Errno 104] Connection reset by peer>

Giving up
$

README file is old

elbe github frontpage talks about elbe init

elbe-initvm manpage needs update for --skip-build-source and --skip-build-bin option

crash if elbe-bootstrap can't be downloaded

% elbe initvm create --directory=/scratch/elbe (15:16 - 15-06-23)
/scratch/elbe
Reading package lists... Done
Building dependency tree... Done
No keyring installed in /tmp/tmpS0o6XI/etc/apt/trusted.gpg.d/.No keyring installed in /tmp/tmpS0Reading package lists... Done
Building dependency tree... Done
No keyring installed in /tmp/tmpS0o6XI/etc/apt/trusted.gpg.d/.No keyring installed in /tmp/tmpS0o6XI/etc/apt/trusted.gpg.d/.Traceback (most recent call last):
File "/usr/bin/elbe", line 61, in
cmdmod.run_command( sys.argv[2:] )
File "/usr/lib/python2.7/dist-packages/elbepack/commands/init.py", line 148, in run_command
copy_kinitrd(xml.node("/initvm"), out_path, defs, arch="amd64")
File "/usr/lib/python2.7/dist-packages/elbepack/pkgutils.py", line 156, in copy_kinitrd
uri = get_initrd_uri(prj, defs, arch)
File "/usr/lib/python2.7/dist-packages/elbepack/pkgutils.py", line 115, in get_initrd_uri
uri = x.archive_uri(r.filename)
AttributeError: 'NoneType' object has no attribute 'archive_uri'
'elbe init' Failed
Giving up

Fix "Similar code" issue in elbepack/projectmanager.py

Similar code found in 4 other locations

https://codeclimate.com/github/Linutronix/elbe/elbepack/projectmanager.py#issue_568e29e20afd3300012c8236

elbe-daemon sys.exit() on error

There are many places where the elbe-daemon will simply print() an error and then sys.exit(). For example, if an invalid buildtype is specified in the XML. This is a problem because:

1. The elbe-daemon is no longer running on the initvm.
2. No useful information about the error is communicated over the SOAP interface to the user.

target without serial getty is cumbersome

for systemd its ok to do
etc/systemd/system/getty.target.wants/[email protected]/rm>
in finetuning

but for sysv-init its more complicated

/dev not in ubi-image

/dev is in use during generation of ubi-image. So it ould not be moved to a new directory and is therefore not included in the ubi-image.

Fix "Similar code" issue in elbepack/db.py

Similar code found in 3 other locations

https://codeclimate.com/github/Linutronix/elbe/elbepack/db.py#issue_568e29e397e18c000119d6b2

Fix "Similar code" issue in elbepack/efilesystem.py

Similar code found in 3 other locations

https://codeclimate.com/github/Linutronix/elbe/elbepack/efilesystem.py#issue_568e29e2dae62c000119f3d8

rpcaptcache: project in undefined status if connection to repositories fails

if the connection to a repository gets broken during a build, rpcaptcache throws an exception that isn't
caught anywhere, so elbe buildchroot terminates with a backtrace. If it was an inital build, /etc/elbe_version was not written, but the system may be debootstraped. Another call to 'elbe buildchroot' is not able to detect this status correctly and so the project is in an unusable state.

elbe initvm create bin-cdrom.iso misses some packages

elbe initvm create examples/bbb-jessie.xml
....
elbe initvm create elbe-build-*/bin-cdrom.iso
...

• package libmnl0 is not installed
• package libnetfilter-acct1 is not installed
• package libsigc++-2.0-0c2a is not installed
• package nfacct is not installed

elbe buildchroot --build-sources is broken

if the attached xml file is used with 'elbe buildchroot -o bla.log -t bla --build-sources --skip-cdrom bla.xml' no source cdrom is generated. However if afterwards 'elbe mkcdrom bla' is called the cdrom
is generated correctly.

---

<ns0:RootFileSystem xmlns:ns0="https://www.linutronix.de/projects/Elbe" xmlns:xs
i="http://www.w3.org/2001/XMLSchema-instance" created="2009-05-20T08:50:56" revi
sion="6" xsi:schemaLocation="https://www.linutronix.de/projects/Elbe dbsfed.xsd"

    <project>
            <name>src-test</name>
            <version>1</version>
            <description>
                    bla
            </description>
            <buildtype>i386</buildtype>
            <mirror>
                    <primary_host>ftp.de.debian.org</primary_host>
                    <primary_path>/debian</primary_path>
                    <primary_proto>http</primary_proto>
                    <url-list>
                            <url>
                                    <source>
                                            http://ftp.de.debian.org/debian  wheezy main
                                    </source>
                            </url>
                    </url-list>
            </mirror>
            <noauth/>
            <suite>wheezy</suite>
    </project>
    <target>
            <hostname>lx-training</hostname>
            <domain>tec.linutronix.de</domain>
            <passwd>root</passwd>
            <console>ttyS0,115200</console>
            <pkg-list>
                    <pkg>bash</pkg>
            </pkg-list>
    </target>

/ns0:RootFileSystem

rebuild from cdrom is broken in elbe 1.9.x

Select and install software ... 10%... 27%... 31%... 40%... 50%... 61%... 70%... 80%... 90%... 100%
Installing GRUB boot loader ... 16%... 33%... 50%... 66%... 83%... 100%
The system is going down NOW!.. 12%... 20%... 33%... 41%... 50%... 62%... 70%... 83%... 91%
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot
[ 499.033275] reboot: Restarting system
mkdir -p .stamps
touch .stamps/stamp-install-initial-image

---

Uploading CDROM. This might take a while
Upload finished
Fri Feb 5 08:19:26 2016-- project still busy, waiting

Build finished !

Server returned error:

OSError: [Errno 2] No such file or directory: '/var/cache/elbe/2d9bf523-8378-4731-a79d-36047416eb3e/validation.txt'
Project failed to generate validation.txt
Getting log.txt
Enqueueing project for build
Build started

ELBE Report for Project SinA33

report timestamp: 20160205-071925
running cmd +mkdir -p "/var/cache/elbe/2d9bf523-8378-4731-a79d-36047416eb3e/chroot"+

debootstrap log

getting output from cmd +dpkg --print-architecture+
running cmd +mkdir -p "/var/cache/elbe/2d9bf523-8378-4731-a79d-36047416eb3e/chroot/cdrom"+

running cmd +mount -o loop "/var/cache/elbe/2d9bf523-8378-4731-a79d-36047416eb3e/uploaded_cdrom.iso" "/var/cache/elbe/2d9bf523-8378-4731-a79d-36047416eb3e/chroot/cdrom"+

mount: /dev/loop0 is write-protected, mounting read-only

running cmd +debootstrap --foreign --arch=armhf "jessie" "/var/cache/elbe/2d9bf523-8378-4731-a79d-36047416eb3e/chroot" "file:///var/cache/elbe/2d9bf523-8378-4731-a79d-36047416eb3e/chroot/cdrom/targetrepo"+

I: Retrieving Release
I: Retrieving Release.gpg

E: Failed getting release signature file file:///var/cache/elbe/2d9bf523-8378-4731-a79d-36047416eb3e/chroot/cdrom/targetrepo/dists/jessie/Release.gpg

Command failed with errorcode 1
running cmd +umount "/var/cache/elbe/2d9bf523-8378-4731-a79d-36047416eb3e/chroot/cdrom"+
Build failed !!!

Debootstrap failed to install the base rootfilesystem.
Probable cause might be:

• Problems with Internet Connection
• Broken mirrors

contrib/linux doesn't work anymore with linux > 4.3

don't care on ordering of lists in the XML file

the code is flexible enough, just reworking the XML schema is needed

update-grub is called without "chroot $imgmount" so initvm grub is installed

Fix "Similar code" issue in elbepack/soapclient.py

Similar code found in 3 other locations

https://codeclimate.com/github/Linutronix/elbe/elbepack/soapclient.py#issue_568e29e2dae62c000119f3d3

using mirror without source packages ends in cryptic error

If a mirror is used, that doesn't contain the source packages, the source CDROM can't be generated.
However the resulting error messages are very confusing.

Please improve validation that runs before the buld, that it checks if the source mirrors are availbable
if --skip-build-sources wasn't specified.

---

Installing GRUB boot loader ... 16%... 33%... 50%... 66%... 83%... 100%
The system is going down NOW!.. 13%... 22%... 31%... 40%... 50%... 63%... 72%... 81%... 90%
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot
[ 610.013885] reboot: Restarting system
mkdir -p .stamps
touch .stamps/stamp-install-initial-image

---

XML file is inavlid: Repository http://10.0.2.2//debian/dists/jessie/Release can not be validated

termi@Ubuntu12:~/elbe-v1$ ./elbe initvm submit ./testlinux.xml
Build started, waiting till it finishes
Tue Mar 8 15:03:31 2016-- project still busy, waiting
Tue Mar 8 15:03:36 2016-- project still busy, waiting
Tue Mar 8 15:03:41 2016-- project still busy, waiting
...
Build finished !

Server returned error:

OSError: [Errno 2] No such file or directory: '/var/cache/elbe/59969b01-165f-400d-8940-3bb0006401e0/validation.txt'
Project failed to generate validation.txt
Getting log.txt
Enqueueing project for build
Build started

ELBE Report for Project elbe-gnome3

report timestamp: 20160308-140329
running cmd +mkdir -p "/var/cache/elbe/59969b01-165f-400d-8940-3bb0006401e0/chroot"+

debootstrap log

getting output from cmd +dpkg --print-architecture+

running cmd +debootstrap --arch=amd64 "jessie" "/var/cache/elbe/59969b01-165f-400d-8940-3bb0006401e0/chroot" "http://10.0.2.2//debian"+

I: Retrieving Release
I: Retrieving Release.gpg
...
I: Configuring systemd...

I: Base system installed successfully.

running cmd +chroot /var/cache/elbe/59969b01-165f-400d-8940-3bb0006401e0/chroot debconf-set-selections < /var/cache/elbe/59969b01-165f-400d-8940-3bb0006401e0/chroot/var/cache/elbe/preseed.txt+
update cache failed
Build failed !!!

Failed to build the Apt Cache.
Not all Mirror Index Files could be downloaded.
Probable cause might be:

• Problems with Internet Connection
• Broken mirrors

not able to run created Debian-Image in graphical mode

the steps i have done:

1. elbe init ./elbe-init.xml
2. make
3. make run

i tried these steps already on three diffrent Systems:

1. Debian Wheezy (Guest OS in VMWare)
2. Fedora 20 (Guest OS in VMWare)
3. Linux Mint (nativ Linux-Machine)

the results were always the same. it was not possible to run the image-file in graphical mode.
Grub and linux booted well. but Init hangs in its last step!!

declaring <suite>stable</suite> failed to build

Trying to build an image with stable failed silently:

...
Package List validation
-----------------------

No Errors found

Getting generated Files

Saving generated Files to elbe-build-20160202-181513
source.xml      (Current source.xml of the project)
cdrom.cpio      (Image)
licence.txt     (License file)
validation.txt  (Package list validation result)
elbe-report.txt         (Report)
log.txt         (Log file)
$

Looking into log.txt shows a build failure:

------------------------------------------------------------------------------

Build failed
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/elbepack/asyncworker.py", line 86, in execute
    self.project.build(skip_pkglist=True, build_bin=self.build_bin, build_sources=self.build_src)
  File "/usr/lib/python2.7/dist-packages/elbepack/elbeproject.py", line 260, in build
    cdrom_size=cdrom_size )
  File "/usr/lib/python2.7/dist-packages/elbepack/cdroms.py", line 93, in mk_binary_cdrom
    repo_path, log, cdrom_size, mirror)
  File "/usr/lib/python2.7/dist-packages/elbepack/repomanager.py", line 213, in __init__
    maxsize )
  File "/usr/lib/python2.7/dist-packages/elbepack/repomanager.py", line 73, in __init__
    self.gen_repo_conf()
  File "/usr/lib/python2.7/dist-packages/elbepack/repomanager.py", line 96, in gen_repo_conf
    fp.write( "Suite: " + codename2suite[ att.codename ] + "\n" )
KeyError: 'stable'

powerpc builds are broken

this can't be fixed in elbe, it is caused by a bug in qemu-ppc-static:

running cmd +cp /usr/bin/qemu-ppc-static /var/cache/elbe/d1995287-5d24-426f-8bd7-057a5d09d7f9/chroot/usr/bin+

running cmd +chroot /var/cache/elbe/d1995287-5d24-426f-8bd7-057a5d09d7f9/chroot /debootstrap/debootstrap --second-stage+

qemu-ppc-static: /build/qemu-1000/qemu-1.1.2+dfsg/linux-user/signal.c:4341: setup_frame: Assertion `({ unsigned long __guest = (unsigned long)(ka->_sa_handler) - guest_base; (__guest < (1ul << 32)) && (!reserved_va || (__guest < reserved_va)); })' failed.
qemu: uncaught target signal 6 (Aborted) - core dumped

Aborted

Command failed with errorcode 134
Build failed !!!

elbe buildchroot with a xml file including the <setsel/> tag is broken

The applied file "var/cache/elbe/pkg-selections" is not generated before
(see elbepack/filesystems.py:250)

Also the mount/umount stuff there should be reworked. Currently the dpkg --set-selections quits with
an exceptions and then the mounts are not umounted.

pbuilder/arm: might fail if initvm SMP != 1

please validate that, before pbuilder is started and give at least a warning

cdrom generation

cdrom images don't include the packages needed to recreate the initvm.

toolchainextract has no man page

gen_update is not available on host pc

we need a SOAP interface for the update generation

rework contrib/linux

at the moment make deb-pkg generates a debian directory and builds the kernel.
Let's generate the debianization in a separate make target

elbe chroot alters files

elbe chroot into an existing chroot or target directory alters files, e.g. /etc/apt/sources.list

This is caused by

    self.initialize_dirs (build_sources=build_sources)                      

in the BuildEnv constructor.

I'm afraid of moving this function without testing. It might affect pbuilder, etc

unwanted files in target

on incremental buildchroot calls temporary buildfiles are stored in the target image:

/var/cache/elbe/binaries/*
/var/cache/elbe/sources/*

Fix "Similar code" issue in elbepack/db.py

Similar code found in 4 other locations

https://codeclimate.com/github/Linutronix/elbe/elbepack/db.py#issue_568e29e2b1957d0001889365

elbe-report.txt list files from archive as 'postinst generated'

Test ticket from Code Climate

missing qemu dependency

When running elbe under debian wheezy, the 'make' command fails when
starting qemu. The error message is:

kvm: -device virtio-rng-pci: Parameter 'driver' expects device type

Apparently wheezy's qemu (1.1.2+dfsg-6a+deb7u6) is either too old or was
compiled without a critical option. Qemu from wheezy-backports works fine.

Elbe depends on qemu-utils and qemu-system, but without any version
restriction. However it should depend on the minimal qemu version needed
in order to run properly.

Thanks,
Richard

Fix "Identical code" issue in elbepack/daemons/soap/faults.py

Similar code found in 5 other locations

https://codeclimate.com/github/Linutronix/elbe/elbepack/daemons/soap/faults.py#issue_568e29e397e18c000119d6ac

portforwarding tags are unused

either allow them in initvm or remove them completely

processing large XML file (66M) failed on second run

Processing a large XML file failed on a second run:
$ elbe initvm submit castellum.xml
.... (build OK)
$ elbe initvm submit castellum.xml
elbe control create_project failed.
Server returned error:

Elbe XML Validation Error
XML Parse error
Memory allocation failed : growing buffer

Giving up
$

After restarting the initvm the elbe initvm submit works as expected for the next run.
After increasing the memory of the initvm from 1024M to 2048M the project could be build several times.

elbe setsel is broken

the packages are placed on the wrong position in the XML file

debootstrapping jessie systems is broken

due to https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=766459

If you need to do this, using debootstrap from wheezy-backports is a solution.

Fix "Similar code" issue in elbepack/asyncworker.py

Similar code found in 3 other locations

https://codeclimate.com/github/Linutronix/elbe/elbepack/asyncworker.py#issue_568e29e3a66af7000119b661