Comments (8)
@123ahaha https://osresearch.net/general-building/#generic should address the confusion linked to nix buildstack change and prevent other users encountering initrd corruption and rom non-reproducibility.
Feel free to tag @tlaurion if you disagree with this issue closure.
from heads.
update : so not reproducible on my side, just build the same rom on the same computer, just flashed it externally and the nitropad-nv41 booted my debian again
Here is the only proof I will have of this bug
from heads.
Repro.
Look at build log last lines at https://app.circleci.com/pipelines/github/linuxboot/heads/767/workflows/0b1c3842-40cd-444c-b64c-e5fb2d5a2114/jobs/16361/parallel-runs/0/steps/0-102:
2024-05-10 20:53:32+00:00 INSTALL build/x86/coreboot-nitrokey/nitropad-nv41/coreboot.rom => build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
if cmp --quiet "/heads/build/x86/coreboot-nitrokey/nitropad-nv41/coreboot.rom" "/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom" ; then echo "`date --rfc-3339=seconds` UNCHANGED build/x86/coreboot-nitrokey/nitropad-nv41/coreboot.rom" ; fi ; cp -a --remove-destination "/heads/build/x86/coreboot-nitrokey/nitropad-nv41/coreboot.rom" "/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom" ;
29bd879c005bc5968b8f2d67a2f14f07d63ff5397624e0199440d1bbf0e50373 build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
33554432:build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
rm -rf "/heads/build/x86/nitropad-nv41/update_pkg"
mkdir -p "/heads/build/x86/nitropad-nv41/update_pkg"
cp "/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom" "/heads/build/x86/nitropad-nv41/update_pkg/"
cd "/heads/build/x86/nitropad-nv41/update_pkg" && sha256sum "heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom" >sha256sum.txt
cd "/heads/build/x86/nitropad-nv41/update_pkg" && zip -9 "/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.zip" "heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom" sha256sum.txt
adding: heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom (deflated 62%)
adding: sha256sum.txt (deflated 14%)
29bd879c005bc5968b8f2d67a2f14f07d63ff5397624e0199440d1bbf0e50373 /heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
33554432:/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
Rebuilt locally:
2024-05-11 13:14:43+00:00 INSTALL build/x86/coreboot-nitrokey/nitropad-nv41/coreboot.rom => build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
29bd879c005bc5968b8f2d67a2f14f07d63ff5397624e0199440d1bbf0e50373 build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
33554432:build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
rm -rf "/home/user/heads/build/x86/nitropad-nv41/update_pkg"
mkdir -p "/home/user/heads/build/x86/nitropad-nv41/update_pkg"
cp "/home/user/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom" "/home/user/heads/build/x86/nitropad-nv41/update_pkg/"
cd "/home/user/heads/build/x86/nitropad-nv41/update_pkg" && sha256sum "heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom" >sha256sum.txt
cd "/home/user/heads/build/x86/nitropad-nv41/update_pkg" && zip -9 "/home/user/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.zip" "heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom" sha256sum.txt
adding: heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom (deflated 62%)
adding: sha256sum.txt (deflated 14%)
29bd879c005bc5968b8f2d67a2f14f07d63ff5397624e0199440d1bbf0e50373 /home/user/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
33554432:/home/user/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
@123ahaha : Maybe you didn't follow change of buildsystem to nix that happened yesterday through that exact commit?
- rom name:
heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
- commit embedded into rom name:
1e583e0
- link to commit (BOM): 1e583e0
- which links to PR merged: #1661
from heads.
@123ahaha
EDIT : I was using the .zip file containing the rom and the hash
The buildsystem embeds, solely the calculated rom hash through sha256sum.txt, into the zip file so the user doesn't have to validate it manually as previously. This is why zip files exist for internal firmware upgrade since #1526 (November 17th 2023)
That is, solely, to guarantee that the zip file which is now used as firmware upgrade package matches the rom hash from build time. But that hash doesn't imply it was reproducible.
What changed yesterday by merging #1661 is that CircleCI will produce the same roms then built locally if local is built clean, which CircleCI does.
I'm sorry I cannot reproduce your issue. That shows that for some reason, your initrd.cpio.xz file, containing the init script which couldn't be found, being part of heads.cio (scripts, security policies) got corrupted somehow. Why/how? We cannot know unless you still had the rom image lying around.
Let's go practical in the goal of understanding what is packed under rom, shall we.
Check https://output.circle-artifacts.com/output/job/6916d09c-5784-4292-bb44-d559923d9d17/artifacts/0/build/x86/nitropad-nv41/hashes.txt and bear with me for a minute.
If you take a look at hashes.txt produced alongside of your rom, you will see inside of that file the hashes of each file (last step of building them) in which cpio (outputted in the file when cpio is created), prior of the initramfs (initrd.cpio.xz) and kernel (bzimage) that is stiched reproducibly inside of the rom by the final coreboot building phase of the rom (stitching them as coreboot payload).
So if the rom hash is reproducible, therefore all other parts are reproducible:
heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom
, containing:bzImage
(compiled in kernel)initrd.cpio.xz
, containing:heads.cpio
(scripts, security policies)tools.cpio
: (libraries, binaries)modules.cpio
(kernel modules to be loaded on demand inside of heads)
So again, if rom is having same hash output locally than over CircleCI, then we have reproducible builds.
Do you happen to still have the old rom image on your usb thumb drive, which caused the initrd.cpio.xz to be corrupted in your local build?
A reminder that build instructions at https://github.com/tlaurion/heads/blob/ecbfdbc57b23ef0b884b394e1ad97491b8d2f8b6/README.md#build-docker-from-nix-develop-layer-locally are to be followed.
@JonathonHall-Purism if this kind of issues happen more then one other time in the future, I think I will move the local docker build creation steps in the devel section of heads-wiki and force users to download docker image only, so that nothing can come in between what is done by CI and what users can do.
@123ahaha can you shed some lights on what happened or better, send me your old rom you flashed from USB Thumb drive?
If no repro, it didn't happen, unfortunately. Goal here now is to understand why you happened to be able to produce a rom that was different somehow without it being marked "dirty" from 1e583e0
from heads.
Maybe you didn't follow change of buildsystem to nix that happened yesterday through that exact commit
I did, but I thought we could still use the old method, "make BOARD=nitropad-nv41", I was on a debian-11 system so didn't need the nix build system ...
send me your old rom you flashed from USB Thumb drive?
Yeah ... I'm sorry, I'm still not used to production environments and bug reports, so I thought about it last night, but this morning I just rm -rf the repo git heads and I started from 0, and to start again on something clean I cleaned my USB key ... so no I no longer have the corrupted rom ...
If no repro, it didn't happen, unfortunately
I think it's going to end up like this, I can't supply the rom and the only things I can help with are the steps I took to get to flash using the heads GUI tool.
The only proof is the photo of the screen taken. (which doesn't prove much if you want to go all the way, and doesn't help much).
I'm going to describe once again the steps I took to reach the kernel panic:
- boot a debian 11 (clean, specific for build head, bare OS not a Qubes vm)
- go to the git repo, do a git pull to get the new commits
- make BOARD=nitropad-nv41
- plug in my USB key, cp /home/user/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom /media/user/4GO
- boot nitropad-nv41, go to update utility
- update with freshly copied .zip file
- reboot
-> nitropad reboot, displays bootsplash and kernel panic.
from heads.
Maybe you didn't follow change of buildsystem to nix that happened yesterday through that exact commit
I did, but I thought we could still use the old method, "make BOARD=nitropad-nv41", I was on a debian-11 system so didn't need the nix build system ...
send me your old rom you flashed from USB Thumb drive?
Yeah ... I'm sorry, I'm still not used to production environments and bug reports, so I thought about it last night, but this morning I just rm -rf the repo git heads and I started from 0, and to start again on something clean I cleaned my USB key ... so no I no longer have the corrupted rom ...
If no repro, it didn't happen, unfortunately
I think it's going to end up like this, I can't supply the rom and the only things I can help with are the steps I took to get to flash using the heads GUI tool. The only proof is the photo of the screen taken. (which doesn't prove much if you want to go all the way, and doesn't help much).
I'm going to describe once again the steps I took to reach the kernel panic:
- boot a debian 11 (clean, specific for build head, bare OS not a Qubes vm)
- go to the git repo, do a git pull to get the new commits
- make BOARD=nitropad-nv41
- plug in my USB key, cp /home/user/heads/build/x86/nitropad-nv41/heads-nitropad-nv41-v0.2.0-2147-g1e583e0.rom /media/user/4GO
- boot nitropad-nv41, go to update utility
- update with freshly copied .zip file
- reboot
-> nitropad reboot, displays bootsplash and kernel panic.
Same thing here, I can only go with logic, but the point I don't get is how you got a "dirty" rom not being tagged dirty, building per old instructions.
I will try to repro using my debian-12 machine and build qemu rom the way thingswere done prior of #1661 being merged and see if there needs to be a bit gfat warning in case initrd gets corrupted.
@123ahaha thanks for the report, I just don't quite know what to do with it yet.
from heads.
Nope. Couldn't reproduce
I guess you rebuilt on clean checkout when building clean reproducible rom the second time where not the first time.
Only thing to report is that the target/qemu.mk target for qemu boards was modified since docker image builds as root from its container, so I had to do a sudo call to launch the make run part
Previous call was prior of #1661 working:
cd ~/heads && make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=~/pubkey.asc inject_gpg
make BOARD=qemu-coreboot-fbwhiptail-tpm2 USB_TOKEN=Nitrokey3NFC PUBKEY_ASC=~/pubkey.asc ROOT_DISK_IMG=~/qemu-disks/debian-9.cow2 run`
But now produces:
----------------------------------------------------------------------
!!!!!! BUILD SYSTEM INFO !!!!!!
System CPUS: 12
System Available Memory: 7538 GB
System Load Average: 2.43
----------------------------------------------------------------------
Used **CPUS**: 12
Used **LOADAVG**: 18
Used **AVAILABLE_MEM_GB**: 7538 GB
----------------------------------------------------------------------
**MAKE_JOBS**: -j12 --load-average=18
Variables available for override (use 'make VAR_NAME=value'):
**CPUS** (default: number of processors, e.g., 'make CPUS=4')
**LOADAVG** (default: 1.5 times CPUS, e.g., 'make LOADAVG=54')
**AVAILABLE_MEM_GB** (default: memory available on the system in GB, e.g., 'make AVAILABLE_MEM_GB=4')
**MEM_PER_JOB_GB** (default: 1GB per job, e.g., 'make MEM_PER_JOB_GB=2')
----------------------------------------------------------------------
!!!!!! Build starts !!!!!!
mkdir -p "/home/user/heads/build/x86/qemu-coreboot-fbwhiptail-tpm2/vtpm"
swtpm_setup --create-config-files root skip-if-exist
File /home/user/.config/swtpm-localca.conf already exists. Refusing to overwrite.
make: *** [targets/qemu.mk:31: /home/user/heads/build/x86/qemu-coreboot-fbwhiptail-tpm2/vtpm/.manufacture] Error 1
Which would now need to now be (UNSUPPORTED ANYWAY POST #1661):
cd ~/heads && make BOARD=qemu-coreboot-fbwhiptail-tpm2 PUBKEY_ASC=~/pubkey.asc inject_gpg
sudo make BOARD=qemu-coreboot-fbwhiptail-tpm2 USB_TOKEN=Nitrokey3NFC PUBKEY_ASC=~/pubkey.asc ROOT_DISK_IMG=~/qemu-disks/debian-9.cow2 run
from heads.
This could be plausibly explained by a bad flash, IMO. I've had exactly this happen for a bad flash - the initrd follows the kernel, so it is possible for a flash to be interrupted at a point where the kernel loads but the initrd is corrupt, producing this result. Hardware flash would then solve it, which is consistent with the observed behavior.
There was a rebuild in between (not sure whether it was a clean + build or just 'make' again), but that too does not point to a problem in the build enviroment.
If the rebuild actually did change the ROM (no way to know now), a more likely explanation IMO is a deficiency in our Makefile that didn't rebuild something that was needed. I'm pretty sure there are still some examples of this (e.g. if you commit, it doesn't necessarily rebuild everything that depends on the version, or if you check out a new commit and module installed files have changed, old files can be left in install/x86/, etc.).
In the future, for a postmortem of a nonbooting flash, it'd be really helpful to dump the ROM contents before hardware flashing for postmortem analysis, if possible 🙂
@JonathonHall-Purism if this kind of issues happen more then one other time in the future, I think I will move the local docker build creation steps in the devel section of heads-wiki and force users to download docker image only, so that nothing can come in between what is done by CI and what users can do.
Probably best for a larger discussion if you really thinking about making such a change - but IMO I think it is worthwhile to keep the docker image build steps where they are.
- Like I said above I don't really think the evidence here points to a problem in the build environment 🤔
- The goal of reproducible builds is to distribute trust, rather than centralizing trust in whoever produced that particular build. If the build can only be reproduced by downloading a nonreproducible binary artifact produced by a single party, it defeats the purpose of reproducible builds.
- Reproducing by rebuilding the build environment with Nix actually works today 🤩 I generally think Nix is precise enough that it will always produce a functionally-equivalent build environment from the information in the repo.
from heads.
Related Issues (20)
- Talos II - GPG prompts wait for input on BMC and timeouts, preventing signing from main console HOT 1
- Switch Haswell boards to NRI (Native Ram Initialization) when ready upstream HOT 9
- librem_14 ROM misfunction, no display or freeze then CPU hard LOCKUP HOT 34
- heads fail to take the right USB dongle when 2 are connected HOT 9
- Create CONTRIBUTING.md HOT 3
- Add more detailed maintainer notes under README.md (oldconfig->defconfig->oldconfig) HOT 2
- What to do after a Firmware flash?OEM Factory, Boot from usb stick or Clean Boot detected? HOT 5
- Why i have 6 Admins/User PINs counters after flashing the heads.rom (Connected device reported status in secure-shell) HOT 7
- Nlnet past funded work placeholder for Accessible Security project (2019) HOT 2
- New build regression with coreboot git forks >= 24.02.01 (purism + dasharo git forks impacted since 2024-07-16 upstream changes) HOT 3
- nix buildstack usage not clear enough for end users to use HOT 14
- Port Optiplex 7010/9010 based on Dasharo coreboot fork HOT 1
- Regression: Some peripherals are not discovered since 4.22.01 -> 24.02.01 Coreboot Version bump (upstream: bogus top down allocator) HOT 16
- Nlnet past funded work placeholder for Authenticated Heads project (2022-ongoing)
- Report: The x230-maximized image is working now on x230t without the backspace keyboard error HOT 9
- Create Makefile target/helper to build Heads payload seperately from coreboot
- flashrom patch for kpge-d16 missing: internal flashing probably broken HOT 1
- remove note for t530 being untested under BOARD_TESTERS.md
- Switch p256/rsa3072 to rsa4096 in oem-factory-reset
- Porting heads to Galago Pro 5, built ROM fails to boot HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from heads.