gkernelci/gbuildbot

Gentoo kernel automated

GKernelCI

Automated build and testing for Gentoo Kernel packages and eclasses

quick start

GKernelCI is currently working with docker-compose

Clone GKernelCI_docker repository
git clone --recursive https://github.com/GKernelCI/Gdocker
Change the docker-compose.yml and start docker-compose
docker-compose up -d

Gentoo kernel stabilization

how Gentoo kernel stabilization work

Kernel ebuilds referenced in the Handbook have certain exemptions from the usual stabilization policy, so stabilization requests are normally only filed for the first version in a long term stable branch (subsequent versions can be stabilized at the discretion of the maintainer).

First, test all available kernel options:
user $cd /usr/src/example-sources-1.2.3
user $make allyesconfig
user $make # add '-j' as appropriate for the hardware

If that succeeds, build with a normal kernel configuration:
user $make distclean
user $make menuconfig
user $make
user $make modules_install # if you use modules

After reboot, check dmesg for anything strange and use the system as normal, trying to get a bit of uptime.

If stabilizing a special feature variant, try to test relevant features. 

Code

Any contribute is welcome

Please check the issues for contributing

gbuildbot's Issues

GitPoller uses constant workdir by default leading to crash with other GitPoller instances when it is not overridden

opened issue to buildbot buildbot/buildbot#5890

Intermediate stage failures do not terminate build

As evidenced in #66 amongst other places, failure of an intermediate step in the build process does not cause buildbot to abort the build (as you might expect).
This is mostly a note to @mjeveritt to implement this change in our master.cfg .

Document arm testing enviroment installation procedure

kernel2 disk increase

Clean generated vmlinux and installed files

Document connecting lava machines to GkernelCI

This would allow non-developer Gentoo users to contribute CPU time similar to BOINC, or any other grid computing effort.

gentoo qcow2 image is locked and cannot be used at same time

qemu-system-x86_64: -hda /tmp/gentoo.qcow2: Failed to get "write" lock
Is another process using the image?

http://kernel1.amd64.dev.gentoo.org:8010/#/builders/3/builds/49/steps/13/logs/stdio

buildbot worker script dependency checker

Looks like we need some dependency installer for make buildbot worker work, maybe we can make in the future a script for checking for unsatisfied dependency and install it automatically.
The dependency are also different from python 2 and python 3

Documentation for test workflow

We need documentation

• how to reproduce the environment
• how to connect a worker to the main buildbot server

Don't use the qemu check with package different from *-sources

Don't use the qemu check with package different from *-sources
http://kernel1.amd64.dev.gentoo.org:8010/#/builders/16/builds/11

ebuilds for kernel branches are the same package

Pull request should be checked/ build-tested on each branch.

This code grabs a list of packages to test:

https://github.com/gentoo/Gentoo_kernelCI/blob/376010acdd4188debb1f9338d987608a1f4e6636/files/stabilize-packages.py#L10-L13

Then later, the kernel CI bot checks to see that the
most recent commit for each package is working:

https://github.com/gentoo/Gentoo_kernelCI/blob/376010acdd4188debb1f9338d987608a1f4e6636/files/stabilize-packages.py#L29

I believe this part is correct, because it's doing what
it was written to do: perform a build test on each package ...

---

The head of the branch (the git branch) for a PR is
normally the correct state to test. For a non-kernel PR,
testing the most recent commit in a PR is enough.

hypothetical failure test case / example:

sys-kernel/foo-sources-4.4.131.ebuild {commit #1}
sys-kernel/foo-sources-4.15.2.ebuild {commit #2}

The current state of the kernel CI bot treats these commits
the same package, and performs one test of the package.

both of these are part of sys-kernel/foo-sources package,
but each ebuild is on a different branch, and needs tested.

I don't understand how to change this buildbot code:

https://github.com/gentoo/Gentoo_kernelCI/blob/376010acdd4188debb1f9338d987608a1f4e6636/files/stabilize-packages.py#L10-L13

.

this is not python code:

---

.
.
pseudocode / algorithm:

branch_list = new object_collection[] // key-value err... python dictionary?
commit_list = this.bot(input_data).commits
for each commit in commit_list
  {
    if !contains_ebuild(commit) then
      continue // no ebuild, check next iteration

    chk_vers = ebuild_getbranch(commit) // which branch is this?
    if branch_list.has_key(chk_vers) then
      branch_list(chk_vers) = commit // not testing obsolete revision anyway (overwrite)
    else
      branch_list(chk_vers) = commit // is this how python adds a NEW key/value pair?
  }

.
Then update the iterator itself to check for branches:

https://github.com/gentoo/Gentoo_kernelCI/blob/376010acdd4188debb1f9338d987608a1f4e6636/files/stabilize-packages.py#L29

should now be something like:

for branch in branch_list: 

---

At this time, requiring one PR per bumped branch would be a workaround.

I don't write python.

qemu: could not load kernel '/boot/vmlinuz-sys-kernel/ck-sources/ck-sources-4.13.13': No such file or directory

WARNING: Image format was not specified for '/tmp/gentoo.qcow2' and probing guessed raw.
Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
qemu: could not load kernel '/boot/vmlinuz-sys-kernel/ck-sources/ck-sources-4.13.13': No such file or directory
vmimage present: /tmp/gentoo.qcow2

ARM kernel config missing CONFIG_NET

Patch_kernel and build_kernel both contain warnings about missing dependencies on CONFIG_NET. This results in a failure to build the kernel image and the resulting trickle-down failures.
I suggest we create a patched _defconfig which is used by the patch_kernel script.

kernel2 update to cross-armv7a-hardfloat-linux-gnueabi/gcc-7.3.0-r3

gcc 7.3 is now keyword stabilized for arm architecture.

After I've finished testing (locally) I'd like to bump this crossdev target on

kernel2.amd64.dev.gentoo.org

(which is still on gcc-6.4.0-r1)

I don't see any reason to run CI / QA for older version of toolchain components than the latest keyword-stabilized version.

I have time in the next few days to perform the bump, but if anyone's if anyone's workflow will be disrupted by this change, please let me know before thursday (July 26th).

gitpoller: exceptions must be classes or instances deriving from BaseException, not type

buildbot_1               | 2021-02-13 09:28:45+0000 [-] trying to poll branch refs/heads/4.1 of https://github.com/GKernelCI/linux-patches.git
buildbot_1               |      Traceback (most recent call last):
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
buildbot_1               |          current.result = callback(current.result, *args, **kw)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 1475, in gotResult
buildbot_1               |          _inlineCallbacks(r, g, status)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
buildbot_1               |          result = result.throwExceptionIntoGenerator(g)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
buildbot_1               |          return g.throw(self.type, self.value, self.tb)
buildbot_1               |      --- <exception caught here> ---
buildbot_1               |        File "/usr/lib/python3.8/site-packages/buildbot/changes/gitpoller.py", line 220, in poll
buildbot_1               |          yield self._process_changes(revs[branch], branch)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
buildbot_1               |          result = result.throwExceptionIntoGenerator(g)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
buildbot_1               |          return g.throw(self.type, self.value, self.tb)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/buildbot/changes/gitpoller.py", line 311, in _process_changes
buildbot_1               |          results = yield self._dovccmd('log', revListArgs, path=self.workdir)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
buildbot_1               |          result = result.throwExceptionIntoGenerator(g)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
buildbot_1               |          return g.throw(self.type, self.value, self.tb)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/buildbot/changes/gitpoller.py", line 405, in _dovccmd
buildbot_1               |          stdout = yield self._dovccmdImpl(command, args, path, None)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
buildbot_1               |          result = result.throwExceptionIntoGenerator(g)
buildbot_1               |        File "/usr/lib/python3.8/site-packages/twisted/python/failure.py", line 512, in throwExceptionIntoGenerator
buildbot_1               |          return g.throw(self.type, self.value, self.tb)
buildbot_1               |      builtins.TypeError: exceptions must be classes or instances deriving from BaseException, not type

Use a more reliable mail server

feature parity for arm VS amd64 testing

current state of amd64:

1. kernel source is compiled
2. kernel image is passed to a wrapper
3. qemu is used to boot a disk image
   (gentoo stable profile / a real OS)

rather than a minimalist test, arm should boot a
stable profile / realistic OS (more than busybox)

stale version during stabilize-packages.py (hardcoded?)

This has been an undocumented / ongoing issue since 2017

"Why is this still trying to download genpatches-4.9-53"

---

"picture is worth a thousand words"

ValueError: unsupported pickle protocol: 3

<kuzetsa> stdio log from buildbot job 899

build log

add email notification

add cross build enviroment in kernel2

we need to install
armv7a-hardfloat-linux-gnueabi-gcc
in kernel2.amd64.dev.gentoo.org

Adding architecture support for ARM

Hi,

Buildbot should support the most common popular architectures used in gentoo, these are amd64, arm and arm64. As a first step support for ARM architecture would be great:

• Create a dedicated builder for ARM (kernel 4.x as a first step, then gentoo_sources and other_sources)
• Cross build linux kernel for ARM (use an armv7a hardfloat toolchain)
• Test with qemu-system-arm for the vexpress a9 target (vexpress or versatile are usually the most tested and ready to use targets for ARM on qemu )

https://kernel-ci.emjay-embedded.co.uk/ is down!

please check what's wrong in you mirror

Fix badge URL for Travis CI

Filter out removed files from ChangeSet

Files removed from the repo still appear in the 'test' list, causing an error when they're missing (duh!).
Check which git operation took place on the files listed, and ignore those removed.
(probably relevant in the appropriate ChangeFilter function in master.cfg)
See https://kernel-ci.emjay-embedded.co.uk/#/builders/8/builds/681 for example.

Add kselftest to Gentoo KernelCI

https://kselftest.wiki.kernel.org/start

worker dosen't work without FEATURE="-userpriv"

make menuconfig or make oldconfig still need to be run before build-kernel.sh, .config not detected

The way I'm checking this is by downloading a somewhat recent kernel and its corresponding patch from kernel.org, putting the patch suffixed by .patch to work with the build-kernel.sh script into a one-higher level directory "linux-patches" as required, unpacking the kernel source, and putting the scripts inside the kernel source folder.

I run "bash build-kernel.sh" which blows up needing make menuconfig/make xconfig/make oldconfig first.

Second, I created a .config via make menuconfig and then ran the build-kernel.sh script which still claims a .config doesn't exist even after the make menuconfig step, following by saving to a .config.

Add some hardened configs to test

https://www.openwall.com/lists/kernel-hardening/2018/07/18/1 has some useful defconfigs making use of well-known existing hardening features/options which we could easily deploy in kernel testing.
These could be potentially added under a 'hardened' USE flag with reference to the upstream inspirations in ewarn/einfo messages.

loosing commit for identical `buildrequest_claims_brids` on different commits

from the database

ERROR:  duplicate key value violates unique constraint "sourcestamps_ss_hash_key"
ERROR:  duplicate key value violates unique constraint "buildrequest_claims_brids"

Issue with 4.12-rc3 AttributeError: 'NoneType' object has no attribute 'split'

http://kernel1.amd64.dev.gentoo.org:8010/#/builders/8/builds/3

Traceback (most recent call last):
File "check-kernelpage.py", line 94, in
new_version_split = new_version_revision.split('.',2)
AttributeError: 'NoneType' object has no attribute 'split'

add name to build factory steps

database errors

 ~/Gdocker $ docker-compose logs |grep -ie error
db_1        | 2022-12-16 16:12:35.945 UTC [32] ERROR:  relation "version" does not exist at character 15
db_1        | 2022-12-16 16:12:35.946 UTC [32] ERROR:  relation "migrate_version" does not exist at character 15

DNS Resolution failure results in false negative (build marked as failed)

Upon failure to resolve github.com the CI bot seems to have prematurely marked the build as a failure, and not retried the fetch or performed a build test.

References:

1. recent ck-sources build test by Gentoo_kernelCI bot
2. new commit, showing same DNS-type error as above

/bin/sh: qemu-system-arm: command not found

for making the Gentoo kernel ci compatible with arm testing we need to install qemu-system-arm
in the kernel2 machine.