clangbuiltlinux / continuous-integration2 Goto Github PK

View Code? Open in Web Editor NEW

15.0 48.0 8.0 4.53 MB

The definitive edition (for real this time, until next year, electric boogaloo)

License: Apache License 2.0

Python 98.75% Shell 1.25%

continuous-integration2's People

Contributors

Stargazers

Watchers

Forkers

nathanchance vishalbhoj kees ivoire a125f phongt

continuous-integration2's Issues

Wire up Android clang builds?

TuxMake and TuxSuite are gaining support for AOSP's clang.

Should we wire this up into CI? If so, what should we build with it?

Android branches probably are not of the highest priority given that is what it is being tested against right now. -next, mainline, and maybe latest LTS?

We should add testing for @samitolvanen 's LTO/CFI tree. We can build both the mainline and tip based branches. TuxSuite has the ability to specify the additional configurations needed on top of a configuration as a URL or in the tree under kernel/configs/: https://gitlab.com/Linaro/tuxsuite#kconfig-examples

Sami, if you wanted to, you could add an arm64 and x86_64 configuration fragment in that folder with the options you want us to test.

arm64 defconfig + full LTO failure

https://github.com/ClangBuiltLinux/continuous-integration2/runs/2494932409?check_suite_focus=true

I am trying to reproduce locally so that I can see if this is something caused by LLVM or Linux (I suspect the former because LLVM 12 and older is fine).

Add all{mod,yes}config for certain architectures

Now that we have the build power, it is worth considering this. Main issue right now is that all{mod,yes}config for arm64 and x86_64 are broken on mainline:

ClangBuiltLinux/linux#1212
ClangBuiltLinux/linux#1213
ClangBuiltLinux/linux#256
ClangBuiltLinux/linux#1243

All of these can currently be worked around though (disabling CONFIG_UBSAN_UNSIGNED_OVERFLOW or CONFIG_UBSAN_TRAP).

32-bit x86 should be using i386 for tuxbuild

Using x86_64 for the target_arch value with i386_defconfig as the kconfig argument results in an x86_64 kernel. Testing locally:

$ tuxmake --runtime podman --target-arch x86_64 --toolchain clang-nightly --kconfig i386_defconfig
...

$ rg 64BIT ~/.cache/tuxmake/builds/10/config
250:CONFIG_64BIT=y
807:CONFIG_PHYS_ADDR_T_64BIT=y
4406:CONFIG_ARCH_DMA_ADDR_T_64BIT=y

If i386 is used for target_arch, we at least run into ClangBuiltLinux/linux#1210, which shows us that it works:

$ tuxmake --runtime podman --target-arch i386 --toolchain clang-nightly --kconfig defconfig
...
Unsupported relocation type: R_386_PLT32 (4)
...

$ rg 64BIT ~/.cache/tuxmake/builds/12/config

Testing with clang-11 for the toolchain, we get a bootable kernel.

Ultimately, generator.yml should be updated to use i386 for the target_arch.

Build failures with LLVM 13.0.0-++20210330111113+6d2fb3cefba6-1~exp1~20210330091825.3874

Example: https://github.com/ClangBuiltLinux/continuous-integration2/runs/2234582121?check_suite_focus=true

However, if I check out LLVM at 6d2fb3cefba6 and try to reproduce outside of TuxSuite / tuxmake, it works fine :(

$ git checkout 6d2fb3cefba6
Note: switching to '6d2fb3cefba6'.                                                                                                                                                                                                                                                                                        
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 6d2fb3cefba6 [clangd] Perform merging for stale symbols in MergeIndex

$ cmake \
-B build \
-G Ninja \
-Wno-dev \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_C_COMPILER=clang \
-DCMAKE_CXX_COMPILER=clang++ \
-DLLVM_CCACHE_BUILD=ON \
-DLLVM_ENABLE_ASSERTIONS=ON \
-DLLVM_ENABLE_PROJECTS="clang;lld" \
-DLLVM_TARGETS_TO_BUILD=X86 \
-DLLVM_USE_LINKER=lld \
llvm
...

$ ninja -C build check-clang check-lld check-llvm check-llvm-unit
...

$ export PATH=${PWD}/build/bin:${PATH}

$ make -C "${SRC_FOLDER}"/linux -skj"$(nproc)" LLVM=1 LLVM_IAS=1 O=build/x86_64 distclean defconfig bzImage
...

$ echo ${?}
0

@sylvestre is this something in Debian's patches?

linux-4.19.y arm64 build failure with LLVM_IAS=1

https://github.com/ClangBuiltLinux/continuous-integration2/runs/1766081167

/builds/1naHcoAbfI9xTt3y7LGe10d6xyC/arch/arm64/kernel/vdso/gettimeofday.S:230:24: error: too many positional arguments
 clock_gettime_return, shift=1
                       ^
/builds/1naHcoAbfI9xTt3y7LGe10d6xyC/arch/arm64/kernel/vdso/gettimeofday.S:253:24: error: too many positional arguments
 clock_gettime_return, shift=1
                       ^
/builds/1naHcoAbfI9xTt3y7LGe10d6xyC/arch/arm64/kernel/vdso/gettimeofday.S:274:24: error: too many positional arguments
 clock_gettime_return, shift=1
                       ^
make[2]: *** [/builds/1naHcoAbfI9xTt3y7LGe10d6xyC/arch/arm64/kernel/vdso/Makefile:54: arch/arm64/kernel/vdso/gettimeofday.o] Error 1

This is "fixed" by https://git.kernel.org/linus/28b1a824a4f44da46983cd2c3249f910bd4b797b, which is obviously not suitable for stable. We should disable LLVM_IAS=1 for linux-4.19.y arm64. Android does not have this problem as that patch was backported: https://android.googlesource.com/kernel/common/+/6545d64a18f7e1686813801109d8e255649e40a5

incomplete logs?

cc @danrue @terceiro

it looks like some android12-5.10 kernel builds failed last night: https://github.com/ClangBuiltLinux/continuous-integration2/runs/2061883868?check_suite_focus=true

From one of the logs:

# to reproduce this build locally: tuxmake --target-arch=arm64 --kconfig=gki_defconfig --toolchain=clang-nightly --wrapper=none --runtime=podman --image=public.ecr.aws/tuxmake/arm64_clang-nightly LLVM=1 LLVM_IAS=1 config default kernel modules
make --silent --keep-going --jobs=8 O=/home/tuxbuild/.cache/tuxmake/builds/1/tmp LLVM=1 LLVM_IAS=1 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- HOSTCC=clang CC=clang gki_defconfig
make --silent --keep-going --jobs=8 O=/home/tuxbuild/.cache/tuxmake/builds/1/tmp LLVM=1 LLVM_IAS=1 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- HOSTCC=clang CC=clang
Killed
make[1]: *** [/builds/1pUri8YWwVI6nOm0AIUlTG0sPWS/Makefile:1241: vmlinux] Error 137
make[1]: Target '__all' not remade because of errors.
make: *** [Makefile:185: __sub-make] Error 2
make: Target '__all' not remade because of errors.

was there more info to this log? Is this an artifact of make with -s?

The tuxsuite client is reporting 0 errors, so the bot thinks it should fetch the kernel image for a boot test, which 404s, failing the test. The tuxsuite host might need to update a regex or two based on the currently observed error?

TypeError: 'usedForSecurity' is an invalid keyword argument for new()

On Arch Linux, I get the following traceback:

$ ./generate_workflow.py < generator.yml mainline
Traceback (most recent call last):
  File "/home/nathan/cbl/github/ci/./generate_workflow.py", line 200, in <module>
    print_builds(config, args.tree)
  File "/home/nathan/cbl/github/ci/./generate_workflow.py", line 177, in print_builds
    check_logs_defconfigs.update(get_steps(build, "defconfigs"))
  File "/home/nathan/cbl/github/ci/./generate_workflow.py", line 133, in get_steps
    sanitize_job_name(name): {
  File "/home/nathan/cbl/github/ci/./generate_workflow.py", line 95, in sanitize_job_name
    h = hashlib.new("md5", name.encode("utf-8"), usedForSecurity=False)
  File "/usr/lib/python3.9/hashlib.py", line 160, in __hash_new
    return _hashlib.new(name, data, **kwargs)
TypeError: 'usedForSecurity' is an invalid keyword argument for new()

@nickdesaulniers do we need this option or can we just drop it?

Unexpected argument for toolchain, 'llvm-nightly'

cc @danrue

I suspect this changed recently in tuxbuild?

via: https://github.com/ClangBuiltLinux/continuous-integration2/runs/1671529350?check_suite_focus=true

Error: Unexpected argument for toolchain, 'llvm-nightly' needs to be one of ('gcc-8', 'gcc-9', 'gcc-10', 'clang-10', 'clang-11', 'clang-nightly')

Issues with full LTO

I have not done any triage yet but I believe the switch from ThinLTO to full LTO in android-5.10 has caused all of our builds to start failing:

https://android-review.googlesource.com/c/kernel/common/+/1621945

https://github.com/ClangBuiltLinux/continuous-integration2/actions/runs/634065920

Full LTO is single threaded and requires way more memory, which appears to be the cause of the failure because there is just a "Killed" message in the logs. If we want to continue testing android-5.10, I am guessing we will need to switch back over to ThinLTO or seeing if there is any way to up the memory limits on the TuxSuite side. I will try to build these configs within the next couple days to see what the maximum memory use is.

Enable PowerPC builds

This is blocked on tuxmake upstream:

https://gitlab.com/Linaro/tuxmake/-/issues/108

linux-4.14.y x86_64 boot timeout

https://github.com/ClangBuiltLinux/continuous-integration2/runs/1766193534

Fixed by https://git.kernel.org/linus/09e43968db40, which it looks like never made it back to 4.14 (or beyond). I will send backports to stable tomorrow.

RISC-V needs a newer version of QEMU

From https://github.com/nathanchance/continuous-integration2/runs/1834110909, we run into a crash when shutting down the machine. As it turns out, I actually fixed this in upstream QEMU but it seems like that patch never got pulled into Ubuntu's version. I think the easiest solution is to do what we did for s390x and package our own qemu-system-riscv64.

Handle llvm-* in addition to clang-*

Currently, tuxmake/tuxmake set LLVM=1 when using the llvm-* toolchain option, rather than clang-*. generate_tuxbuild.py hardcodes llvm-nightly, which has this as a TODO above it :)

continuous-integration2/generate_tuxbuild.py

Lines 53 to 54 in 7adb649

    
           # TODO: llvm-11, llvm-10, etc. 
        
           toolchain = "llvm-nightly"

enable llvm_full for 32b allyesconfig

Pretty sure we can start using llvm_full for 32b ARM allyesconfig

Enable S390 booting

This is blocked on getting a newer copy of QEMU that has https://lore.kernel.org/qemu-devel/[email protected]/

We could potentially start distributing a qemu-system-s390 binary in the boot-utils repo.

job_fragment.yml hardcodes --tux-config argument

is causing the failure for the -next workflow.

distro configs

@nathanchance suggests fetching distros' configs and building those

job_fragment.yml hardcodes URL

via: nickdesaulniers/github_actions_playground#1

https://github.com/nickdesaulniers/github_actions_playground/blob/a5cb81a33be488e2e0021a0e39937dcbf76eeb07/job_fragment.yml#L21

This will need to be retrieved from the generator.yml (repo/ref).

The fragment can be modified via the https://github.com/nickdesaulniers/github_actions_playground/blob/main/generate_workflow.py similar to what is done for the workflow name: https://github.com/nickdesaulniers/github_actions_playground/blob/a5cb81a33be488e2e0021a0e39937dcbf76eeb07/generate_workflow.py#L72

Handle LLVM 12 to LLVM 13 transition

Wait for LLVM 12 binaries on apt.llvm.org
Report availability of LLVM 12 binaries on apt.llvm.org to TuxMake issue tracker: https://gitlab.com/Linaro/tuxmake/-/issues/119
Wait for TuxMake to provide clang-12 images: https://gitlab.com/Linaro/tuxmake/-/merge_requests/103
Wait for TuxSuite to allow clang-12 to be specified: https://gitlab.com/Linaro/tuxsuite/-/issues/98
Update generator.yml to provide llvm_11 anchor, shift llvm_latest to 12, and shift llvm_tot to 13

Just a heads up @danrue

Cron schedule

@nickdesaulniers had suggested adding a cron option in the configurator.yml so that different trees can be run at different frequencies via the GitHub Actions cron: option: https://docs.github.com/en/free-pro-team@latest/actions/reference/events-that-trigger-workflows#scheduled-events

mainline and the android-* trees see fairly regular development so doing these every six hours or so would probably be a good starting point.

next is updated only once a day so that should be done daily.

stable updates do not happen more than once a week usually but keeping them daily is still a good idea to try and catch compiler regressions.

Error: make variable 'LLVM' value invalid '0'

Looks like something might have changed on the TuxSuite side of things: https://github.com/ClangBuiltLinux/continuous-integration2/runs/1745918086?check_suite_focus=true

Error: make variable 'LLVM' value invalid '0'

However, I think that they did us a favor. LLVM=0 and LLVM=1 are technically equivalent: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Makefile?h=v5.11-rc4#n428

I think that the way that we generate make_variables in the tuxsuite files from generator.yml needs to change. Currently, we rely on the llvm and llvm_ias values but I think that this is too rigid. We have three general levels of support:

CC=clang (make_variables is empty/non-existent)
LLVM=1 (make_variables: {LLVM: 1})
LLVM=1 LLVM_IAS=1 (make_variables: {LLVM: 1, LLVM_IAS: 1})

but there are also instances where we need to add another variable, like LD for riscv. I am just not sure what the best way to do this is though.

set CROSS_COMPILE_COMPAT=arm-linux-gnueabi- for arm64 builds

we should be testing the compat vdsos on ARCH=arm64 by setting that env var

clang-13 not getting upgraded?

so 5.10 (and mainline) are failing for arm32 allnoconfig:
https://github.com/ClangBuiltLinux/continuous-integration2/runs/1996403133?check_suite_focus=true

This looks like ClangBuiltLinux/linux#1296 which has been fixed for a few days.

CI1 had a check for the clang version being too old (older than 5 days). I think we need to port that check to CI2.

enable hexagon

@androm3da has hexagon working in git://git.kernel.org/pub/scm/linux/kernel/git/bcain/linux for-linus branch; we should wire this up to CI builds ASAP.

regen all

as #74 requiring an explicit list of targets is error prone. A special all target, or separate script, or flag that rebuilds all possible configs would be nice.

Enable clang-nightly for certain targets

Currently blocked on ClangBuiltLinux/linux#1261.

shiny button in readme uses the wrong URL

it should point to this github repo, not the one I was messing around in.

Enable arm64 big endian for AOSP LLVM

https://github.com/ClangBuiltLinux/continuous-integration2/runs/2655483207?check_suite_focus=true

LLVM 11 builds are not added to the workflow

Due to name collisions, LLVM 11 builds are not added to the matrix. Current solution:

git dh generate_workflow.py
diff --git a/generate_workflow.py b/generate_workflow.py
index 7898942..26787fb 100755
--- a/generate_workflow.py
+++ b/generate_workflow.py
@@ -29,7 +29,7 @@ def get_repo_ref(config, tree_name):
 def get_job_name(build):
     return "ARCH=" + (build["ARCH"] if "ARCH" in build else "x86_64") \
     + " LLVM=" + str(int(build["llvm"])) + " LLVM_IAS=" + str(int(build["llvm_ias"])) \
-    + " BOOT=" + str(int(build["boot"]))
+    + " BOOT=" + str(int(build["boot"])) + " LLVM " + str(int(build["llvm_version"]))


 def sanitize_job_name(name):

which will generate something like

  ARCH_arm_LLVM_1_LLVM_IAS_1_BOOT_1_LLVM_11_multi_v5_defconfig:
    runs-on: ubuntu-20.04
    needs: kick_tuxbuild
    name: ARCH=arm LLVM=1 LLVM_IAS=1 BOOT=1 LLVM 11

and

  ARCH_arm_LLVM_1_LLVM_IAS_1_BOOT_1_LLVM_12_multi_v5_defconfig:
    runs-on: ubuntu-20.04
    needs: kick_tuxbuild
    name: ARCH=arm LLVM=1 LLVM_IAS=1 BOOT=1 LLVM 12

although @nickdesaulniers had talked about just making the job values a UUID because they do not show up in GitHub's UI (only the name: value does).

socat-external[597025] E connect(5, AF=2 127.0.0.1:4226, 16): Connection refused

quite a few of our mainline builds are failing with thousands of errors like:

2021/04/13 13:19:23 socat-external[597018] E connect(5, AF=2 127.0.0.1:4226, 16): Connection refused
error: failed to execute compile
caused by: Failed to send data to or receive data from server
caused by: Failed to read response header
caused by: failed to fill whole buffer

this looks like an issue with tuxsuite? cc @terceiro @ivoire

s/tuxbuild/tuxsuite/

tuxbuild build-set --set-name foo-bar --json-out builds.json --tux-config mainline.tux.yml
WARNING: `tuxbuild` is deprecated; please use `tuxsuite` instead.
WARNING: ~/.config/tuxbuild/config.ini is deprecated; please rename it to ~/.config/tuxsuite/config.ini.

Enable clang-11 for linux-next

We have committed to supporting clang-10 and newer so we should ensure that people are not breaking us with newer code.

Add clang-10 and enable it for next and mainline

We have committed to supporting clang-10 and newer so we should ensure that people are not breaking us with newer code.

We need LD=riscv64-linux-gnu-ld because ld.lld does not support linker relaxation (ClangBuiltLinux/linux#1020).

Additionally, we will need to kconfig: [defconfig, "CONFIG_EFI=n"] to workaround ClangBuiltLinux/linux#1143.

Enable MIPS builds

This is blocked on tuxmake upstream: https://gitlab.com/Linaro/tuxmake/-/issues/110

properly escape cron string

pretty sure that's why the github action isn't running overnight; the string needs to be quoted. Nothing I can come up with in job_fragment.yml seems to quote the string properly..

ppc32 builds not booting on next

something in next seems to have caused 32b PPC to stop booting
https://github.com/ClangBuiltLinux/continuous-integration2/runs/2388901919?check_suite_focus=true

check_logs.py does not download the right binary for LLVM 11

See https://github.com/ClangBuiltLinux/continuous-integration2/runs/1693329903.

ARCH=arm LLVM=1 LLVM_IAS=1 BOOT=1 LLVM 11 multi_v5_defconfig:

Linux version 5.11.0-rc3 (tuxmake@767e6ac67d3d) (Debian clang version 12.0.0-++20210109111114+9e4eadeb135d-1~exp1~20210109101808.3712, LLD 12.0.0) #2 PREEMPT Wed Jan 13 06:47:37 UTC 2021

I assume the toolchain version is not being taken into account but I have not looked.

spin down LTO tree?

now that LTO has been merged into mainline

do we need to keep testing Sami's LTO trees? I suppose so for CFI? Let's keep an eye out for any new breakages for all{yes|mod}config breakages that may result. cc @samitolvanen @nathanchance

Add testing for CONFIG_THUMB2_KERNEL

Talking with @nickdesaulniers, we should add testing of CONFIG_THUMB2_KERNEL.

multi_v7_defconfig + CONFIG_THUMB2_KERNEL should be a good starting point.

Error: make variable 'LLVM' value invalid '0'

our first scheduled run failed with the above output

looks like tuxmake client is validating inputs?

we can not emit these