System upgrade is stuck on boot, renders whole PC unusable

[rpm-software-management]

System upgrade is stuck on boot, renders whole PC unusable

[wgwoods]

First, to fix your system: boot with rd.break, and when you get to the shell, rm /sysroot/system-update. Exit the shell and your system should boot normally.

Second: the problem is that dnf-system-upgrade.service never started - and dnf-system-upgrade.service is what is responsible for deleting that link, normally.

This is a known flaw in the design of systemd's Offline Updates facility - if the updater doesn't delete that link, the user is stuck in a reboot loop. packagekit-offline-updates.service has the same problem.

Upstream has been talking about ways to mitigate that in systemd, but the immediate fix is to ensure that dnf-system-upgrade.service actually runs when it's supposed to.

So: does /lib/systemd/system/system-update.target.wants/dnf-system-upgrade.service exist? (It should be a symlink to ../dnf-system-upgrade.service) If not, that's a packaging problem; if so, we need to figure out why it didn't start...

[kparal]

First, to fix your system: boot with rd.break, and when you get to the shell, rm /sysroot/system-update. Exit the shell and your system should boot normally.

That worked.

This is a known flaw in the design of systemd's Offline Updates facility - if the updater doesn't delete that link, the user is stuck in a reboot loop. packagekit-offline-updates.service has the same problem.

Hmm, that's scary. One broken update and...

Maybe dnf/fedup/PackageKit could create a new boot entry defining a specific keyword on the kernel cmdline and the upgrade service would be set to run only with that keyword on cmdline, otherwise it would be skipped and the system would boot normally? Just an idea, I'm sure the relevant people will come up with even better ways to solve this. It's good to know that it is being worked on.

Do you think the change will be ready before F23?

So: does /lib/systemd/system/system-update.target.wants/dnf-system-upgrade.service exist? (It should be a symlink to ../dnf-system-upgrade.service) If not, that's a packaging problem; if so, we need to figure out why it didn't start...

Yes, however, it also wants package-offline-update.service, which I masked according to the wiki instructions (and it worked with the older copr version). I wonder if that was the problem.

[kparal@localhost ~]$ ls -l /lib/systemd/system/system-update.target.wants/
total 0
lrwxrwxrwx. 1 root root 29 Aug 13 13:59 dnf-system-upgrade.service -> ../dnf-system-upgrade.service
lrwxrwxrwx. 1 root root 36 Jun  8 16:20 packagekit-offline-update.service -> ../packagekit-offline-update.service

I unmasked the services and ran dnf system-upgrade reboot once again. This time the system booted into an upgrade process, however, the upgrade was way too fast, just about a minute. According to the journal afterwards, PK took over instead of fedup and did a standard update with just a few packages instead of full system upgrade. At 100% completion, the system hang with just a blinking cursor. Ctrl+Alt+Delete rebooted it.

I tried once again, this time again masking the two services and also removing /lib/systemd/system/system-update.target.wants/packagekit-offline-update.service. That didn't help, the system was again stuck on boot. I booted it with systemd.log_level=debug and after 10 minutes of waiting I rebooted it and collected the journal, here it is:
https://gist.githubusercontent.com/kparal/b311b987815c26cc334c/raw/4970a5a26a8b1991701cbfb2970f7e1a5e9aefea/upgrade.journal
(search for Activating special unit ctrl-alt-del.target to see the reboot point)

[kparal]

I can reproduce this issue on a completely clean F22 installation:

1. Install F22 Workstation netinst (in order to have up to date system)
2. git clone https://github.com/rpm-software-management/dnf-plugin-system-upgrade.git
3. sudo make install
4. sudo systemctl mask packagekit-offline-update.service fwupd-offline-update.service
5. sudo dnf system-upgrade download --releasever=23
6. sudo dnf system-upgrade reboot

Again gets stuck during boot.

[wgwoods]

Maybe dnf/fedup/PackageKit could create a new boot entry defining a specific keyword on the kernel cmdline and the upgrade service would be set to run only with that keyword on cmdline

That is never going to happen. That's how fedup originally tried to do things, and it caused a huge number of bugs, because it turns out that boot configurations are actually very fragile. The SystemUpdates spec is specifically designed to handle this case without modifying the boot configuration, which makes things much, much more reliable - so long as the spec is implemented properly.

If you want to help figure out how to make that happen, talk to upstream systemd about that spec.

According to the journal afterwards, PK took over instead of fedup and did a standard update

So yeah, to avoid these collisions, upstream recommends that update tools should:

1. refuse to set up an update if another updater has already done so, and
2. exit quietly if the contents of /system-update were prepared by some other tool

dnf system-upgrade implements both these checks already, PackageKit should at least do the second one (as of PackageKit/PackageKit@1381e1d), and I don't know anything about fwupd-offline-update.service.

So - known problem with PackageKit, fixed upstream. But I don't think that fix has landed yet, so probably the wiki should still recommend disabling other update services.

I rebooted it and collected the journal, here it is:

503 error from server? Weird.

Well, I'm back from Rochester now, so I've finally got time to try to reproduce the problem myself. Will let you know what happens.

[kparal]

The spec recommends having OnFailure=reboot.target in the unit file, while you have FailureAction=reboot in it (and I can't find FailureAction in man systemd.unit). However, even if I changed the line to OnFailure=reboot.target, the system still hangs (I hoped at least for an automatic reboot).

503 error from server? Weird.

What do you mean, you can't open the link? Seems to work well here, even when not logged in.

[wgwoods]

The spec recommends having OnFailure=reboot.target in the unit file

@keszybz was going to change the spec, but I'm not sure that's happened yet? The email where upstream said "let's change the spec" is here.

even if I changed the line to OnFailure=reboot.target, the system still hangs

Yeah, that's a different bug - a problem with "do-upgrade" command, which is fixed by the referenced commit (fb3c3ed).

What do you mean, you can't open the link? Seems to work well here, even when not logged in.

Works fine now. But I also have a test system where I reproduced the problem, so no worries!