Support for UDM SE

[fabianishere]

Support for UDM SE

[fabianishere]

You are correct, the UDM SE and UDR are currently unsupported since they were released after this project was developed.

Porting this project to those devices requires building the kexec-mod project for the kernels running on these devices. However, I have no way of testing this since I don’t own a UDM SE.

Once the UDM (Pro) is on 2.x, I will start porting this project to the new firmware.

[fabianishere]

This is still on my ToDo list, but I don't have access to a UDM Pro at the moment. I'll probably start working on this start of next year.

[foxcris]

Ok i am just building the kernel. How can i test it manually? Just by coping the kernel, kexex-module and redir.so to the udm and then run LD_PRELOAD=/root/redir.so kexec -l /path/to/custom/kernel --reuse-cmdline?

[ppkhoa]

UDM/Pro is being updated to the same 3.x branch with UDM Pro SE 3.x, migration is already started for early access so this issue will probably apply to UDM/Pro soon once they go GA

[gtrabanco]

If you only want wireguard support this project works great on UDM-SE:

https://github.com/tusc/wireguard-kmod

[simkin]

Hi @fabianishere wondering if this is still on your radar? :) (We now have moved to Unifi OS 3.x)

[foxcris]

@fabianishere: I have got an udm se and am interested to help getting this project running. Could you point out what we have to achive to get it running? Some advise how to start?

[fabianishere]

@fabianishere: I have got an udm se and am interested to help getting this project running. Could you point out what we have to achive to get it running? Some advise how to start?

You'll have to retrieve the kernel sources from Ubiquiti and verify whether you can build kexec-mod for the kernel.

Then it is a matter of updating the udm-bootctl command to not make use of ssh-proxy, but invoke the commands on the system directly.

[foxcris]

Is there an specific adress at unifi to get the sources? Who did you turn to to get the sources? Am 21.01.2023 um 21:06 schrieb Fabian Mastenbroek:

…

@fabianishere <https://github.com/fabianishere>: I have got an udm se and am interested to help getting this project running. Could you point out what we have to achive to get it running? Some advise how to start? You'll have to retrieve the kernel sources from Ubiquiti and verify whether you can build kexec-mod <https://github.com/fabianishere/kexec-mod> for the kernel. Then it is a matter of updating the udm-bootctl <https://github.com/fabianishere/udm-kernel-tools/blob/master/udm-bootctl> command to not make use of |ssh-proxy|, but invoke the commands on the system directly. — Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYGU3HAXJI6U6Y7RVSLELWTQ6VPANCNFSM6AAAAAARCMKEGA>. You are receiving this because you commented.Message ID: ***@***.***>

[fabianishere]

You can mail Ubiquiti at [email protected] to request the GPL sources for the UDM SE.

[foxcris]

I have got the kernel sources and will take a look wether i can build kexec-mod.

[foxcris]

ok i get the following error:

make KDIR=../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine
make -C ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine M=/mnt/d/github/kexec-mod/kernel modules
make[1]: Entering directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
mkdir -p /mnt/d/github/kexec-mod/kernel/orig
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec.c /mnt/d/github/kexec-mod/kernel/orig/kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_core.c /mnt/d/github/kexec-mod/kernel/orig/kexec_core.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_internal.h /mnt/d/github/kexec-mod/kernel/orig/kexec_internal.h
scripts/Makefile.build:45: /mnt/d/github/kexec-mod/kernel/arch/x86/Makefile: No such file or directory
make[3]: *** No rule to make target '/mnt/d/github/kexec-mod/kernel/arch/x86/Makefile'.  Stop.
make[2]: *** [scripts/Makefile.build:544: /mnt/d/github/kexec-mod/kernel/arch/x86] Error 2
make[1]: *** [Makefile:1550: _module_/mnt/d/github/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
make: *** [Makefile:15: module] Error 2

Did you build kexec-mod on the udm? I tried using the windows subsystem for linux. I will try using a plain linux later.

[fabianishere]

ok i get the following error:

make KDIR=../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine
make -C ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine M=/mnt/d/github/kexec-mod/kernel modules
make[1]: Entering directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
mkdir -p /mnt/d/github/kexec-mod/kernel/orig
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec.c /mnt/d/github/kexec-mod/kernel/orig/kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_core.c /mnt/d/github/kexec-mod/kernel/orig/kexec_core.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_internal.h /mnt/d/github/kexec-mod/kernel/orig/kexec_internal.h
scripts/Makefile.build:45: /mnt/d/github/kexec-mod/kernel/arch/x86/Makefile: No such file or directory
make[3]: *** No rule to make target '/mnt/d/github/kexec-mod/kernel/arch/x86/Makefile'.  Stop.
make[2]: *** [scripts/Makefile.build:544: /mnt/d/github/kexec-mod/kernel/arch/x86] Error 2
make[1]: *** [Makefile:1550: _module_/mnt/d/github/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
make: *** [Makefile:15: module] Error 2

Did you build kexec-mod on the udm? I tried using the windows subsystem for linux. I will try using a plain linux later.

You need to cross-compile to ARM64, by setting ARCH=arm64 and CROSS_COMPILE=aarch64-linux-gnu- (make sure you have gcc-aarch64-linux-gnu installed)

[boostchicken]

I cam here to share I got sources but I have been defeated!

Well done! Its quite telling the UDMSE sources came quick but UDMPRO, nada!

[foxcris]

With cross compile i get a different error. Any hint for this?

export ARCH=arm64; export CROSS_COMPILE=aarch64-linux-gnu-; make KDIR=../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine
make -C ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine M=/mnt/d/github/kexec-mod/kernel modules
make[1]: Entering directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec.c /mnt/d/github/kexec-mod/kernel/orig/kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_core.c /mnt/d/github/kexec-mod/kernel/orig/kexec_core.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/kernel/kexec_internal.h /mnt/d/github/kexec-mod/kernel/orig/kexec_internal.h
mkdir -p /mnt/d/github/kexec-mod/kernel/arch/arm64/orig
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/arch/arm64/kernel/machine_kexec.c /mnt/d/github/kexec-mod/kernel/arch/arm64/orig/machine_kexec.c
ln -sf ../UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine/arch/arm64/kernel/relocate_kernel.S /mnt/d/github/kexec-mod/kernel/arch/arm64/orig/relocate_kernel.S
  CC [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec_drv.o
  CC [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec_compat.o
  CC [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/idmap.o
  AS [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/cpu-reset.o
  AS [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/hyp-shim.o
  CC [M]  /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec.o
/mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec.c:2:10: fatal error: orig/machine_kexec.c: No such file or directory
    2 | #include "orig/machine_kexec.c"
      |          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[3]: *** [scripts/Makefile.build:304: /mnt/d/github/kexec-mod/kernel/arch/arm64/machine_kexec.o] Error 1
make[2]: *** [scripts/Makefile.build:544: /mnt/d/github/kexec-mod/kernel/arch/arm64] Error 2
make[1]: *** [Makefile:1550: _module_/mnt/d/github/kexec-mod/kernel] Error 2
make[1]: Leaving directory '/mnt/d/github/kexec-mod/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine.tar/linux-arm64-dream-4.19.152-ui-alpine'
make: *** [Makefile:15: module] Error 2

[foxcris]

Ok got it working by using the absolut path for KDIR not a relative one. Build finished. Make of the User-space helper was also successfull. What are the next steps?

[boostchicken]

Build a custom kerneland make sure to add the configs for BPF needed for podman. Then you can follow the docs and boot into that kernel.

[jimmy-ungerman]

@foxcris just wanted to reach out and see if you were able to successfully boot into this kernel on your SE and run containers?

[foxcris]

Today i had some time to go on with kexec an a custom kernel. Currently i still didn't manage to boot a custom kernel. My problem is that i cannot load the kexec_mod module. First isse was the wrong version magic

kexec_mod: version magic '4.19.152+ SMP mod_unload aarch64' should be '4.19.152-ui-alpine SMP mod_unload aarch64'

I figured out how to fix this by setting the EXTRAVERSION. However it took me a while to "remove" the "+" sign. Wasn't really obvious and caused by git (you have to set LOCALVERSION manually during build).
Now the version magic missmatch is gone but i get different errors when i try to load the module with modprobe.

kexec_mod: Unknown symbol machine_kexec_prepare (err -2)
kexec_mod: Unknown symbol machine_crash_shutdown (err -2)
kexec_mod: Unknown symbol machine_kexec (err -2)
kexec_mod: Unknown symbol machine_kexec_cleanup (err -2)

@fabianishere: Any hints how to go on now? As soon as i can manually boot the custom kernel i will try to build/adapt your packages.

[foxcris]

Oh i just missed to load "kexec_mod_arm64.ko" first.
Kexec_mod is now loaded. I will try to boot a custom kernel as a next step.

[foxcris]

Kexec is still compaining about miss /proc/kcore.

root@UDM-SE:/ssd1/custom_data# LD_PRELOAD=/ssd1/custom_data/user/redir.so kexec -l /ssd1/custom_data/UDMSE-3.0.13-GPL/linux-arm64-dream-4.19.152-ui-alpine/vmlinux --reuse-cmdline
ERROR: ld.so: object '/ssd1/custom_data/user/redir.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
Can't open (/proc/kcore).
Can't open (/proc/kcore).
kexec_load failed: Function not implemented
entry       = 0xcac690 flags = 0xb70000
nr_segments = 4
segment[0].buf   = 0x7f82b5a010
segment[0].bufsz = 0x8415f0
segment[0].mem   = 0x80000
segment[0].memsz = 0x842000
segment[1].buf   = 0x7f833aa010
segment[1].bufsz = 0x380200
segment[1].mem   = 0x8d0000
segment[1].memsz = 0x3cf000
segment[2].buf   = 0x556552b060
segment[2].bufsz = 0x68e7
segment[2].mem   = 0xca5000
segment[2].memsz = 0x7000
segment[3].buf   = 0x55655327a0
segment[3].bufsz = 0x3460
segment[3].mem   = 0xcac000
segment[3].memsz = 0x4000

[foxcris]

Ok found the error. I used the wrong path to redir.so. Now the kernel is loaded. Will try to boot it with kexec -e.

[foxcris]

After running

kexec -e

they system is hanning. The display on the router tells something about plugging in the wan cable (but its still plugged in). I needed to reboot it manually by switching it off and on again.
With

systemctl kexec

the system directly restarts and uses its own kernel again.

[jimmy-ungerman]

Curious if you've made any progress on this? Or if there's a different kernel that will work on the SE so that I can finally get multicast routing working?

[fabianishere]

Multicast routing should already be supported in the stock UDM SE kernels.

[jimmy-ungerman]

Doesn't seem to be working for me and all I saw was a bunch of posts pointing to here about Multicast routing...is there something I need to enable that I'm missing? Multicast DNS seems to work, but not routing

[jimmy-ungerman]

For example here:

I have a new BGP multicast route between my two nodes of 10.45.0.12, but it times out any time I try to connect

[fabianishere]

@jimmy-ungerman You can verify it on your device as follows:

$ zgrep "MROUTE" /proc/config.gz
CONFIG_IP_MROUTE_COMMON=y
CONFIG_IP_MROUTE=y
# CONFIG_IP_MROUTE_MULTIPLE_TABLES is not set
# CONFIG_IPV6_MROUTE is not set

Make sure you actually have the multicast routes installed into the kernel.

[jimmy-ungerman]

@fabianishere I'm sorry I realized I've been saying "Multicast" when I meant "MultiPATH". Are you aware if this is enabled or not?

[fabianishere]

@jimmy-ungerman No, multipath routing is not supported by the stock kernel. You’ll need to use a custom kernel for that (if you search in the repository, you’ll find others also using it for that feature).

[jimmy-ungerman]

Yeah that's what I thought. That's why I'm following this issue so closely, hoping to be able to get my UDM-Pro SE updated as soon as there is support

[cftechwiz]

Any progress on this? I am really in need of Multipath support also. Looks like some progress was made, any chance we have something for the SE anytime soon?

[EddieRingle]

After running

kexec -e

they system is hanning. The display on the router tells something about plugging in the wan cable (but its still plugged in). I needed to reboot it manually by switching it off and on again. With

systemctl kexec

the system directly restarts and uses its own kernel again.

Just wanted to add that I'm currently hitting this wall as well, on 3.0.13 (I've requested but still haven't received the GPL sources associated with 3.0.18). Working off of the v2.x branch, among other things, I had to update the udm-bootctl script to run kexec -e directly rather than systemctl kexec, because systemctl makes an extra syscall that causes the preloaded redir.so to abort. Even after all of that, the device seems to hang (routing connections stops, at least, so my SSH sessions hang) but eventually restarts itself using the stock kernel.

If there's a particular log I can check to get a better read on what's happening, please let me know. I'm also trying to simply enable multi-path routing, and it's increasingly looking like just overwriting the kernel in the boot image is the more feasible path than trying to get kexec hacks to work. (And as a side note, if anyone has resources handy for quickly modifying the U-Boot image specifically for UDM SE, that'd be appreciated.)

[fabianishere]

@EddieRingle Check the files in /sys/fs/pstore/. The kernel dumps the (partial) logs of the previous boot there (ramoops).

[jimmy-ungerman]

Are we safe to assume that work has stopped on this?