Regression in commits from dec 24

[danderson]

Regression in commits from dec 24

[danderson]

Damnit! Thanks for the report, looking. At first glance, there's no obvious changes to that phase of boot, that section is just grabbing a static TFTP file. The MAC address is normal, but it's basically only for data continuity in the logs, so you can track a MAC through each stage of boot. The important part is the /0, which should serve an ipxe for the x86 arch...

[danderson]

Okay, looking at your pcap and screenshot, something is weird: in the pcap, Pixiecore is running on 192.168.1.192, but in your screenshot, the virtualbox firmware is trying to download the next boot stage from 192.168.1.174

I've seen this before, when there are multiple competing PXE and ProxyDHCP servers on the network. The PXE spec doesn't specify how to merge incoming responses for ProxyDHCP, so you can end up using a server IP from one response, but a filename from another... Unfortunately, I don't see 192.168.1.174 in the pcap, so I have no idea how this machine is somehow convincing your virtualbox to boot from it...

Does this make sense to you? Is there maybe a second pixiecore/other PXE server running?

[cdata]

Hi, I'm new to both Pixiecore and PXE booting in general. As a minor anecdote that I think might be related, I was having trouble booting a small device yesterday and happened upon your conversation as it was unfolding. I tried using several of the "quick" options just to rule out any mistakes I might have made invoking Pixiecore, but the results were the same. Sure enough, rolling back to October 17th in pixiecore enabled my device to load everything and boot.

This may be a completely different issue, but given the title of the issue I figured I would post it in here first since i-dont-know-what-im-doing.jpg.

Attached is a pcap I took using the debug command I found in pixiecore. The device would reboot immediately upon failure to boot, but I was able to glean a reference to this error in the output: http://ipxe.org/3c092083

boot.pcap.zip

I'm not sure how else to be most helpful, but here is a basic description of my setup:

My host machine is a Macbook Pro with VirtualBox. On my host there is a VM with a bridged network adapter, and this VM runs Pixiecore. On my local network there is a small device capable of booting over the network.

[danderson]

Okay, so that's two reports now that rolling back fixes things. I need to reproduce this locally somehow so I can bisect...

[danderson]

Okay! I can reproduce this in virtualbox with the architecture:

• Virtualbox VM with no hard drive, bridged network, configured to boot from network.
• Pixiecore running on the virtualbox host machine

I see the same symptoms, where the TFTP file inexplicably doesn't get served.

In parallel, I'm able to netboot a standard KVM machine, so it's something specific to virtualbox that's broken.

Bisecting...

[danderson]

Debugging a bit before I bisect... It looks like the virtualbox copy of ipxe never tries to speak TFTP at all, network capture only shows the DHCP exchange (which completes correctly afaict), and then the virtualbox firmware just fails without even trying to speak TFTP...

[danderson]

Verified again by comparing with a KVM VM, and I can definitely see DHCP+TFTP happening with that one.

So, a recent change made virtualbox's strange version of ipxe unhappy. Digging...

[danderson]

Found one solution: you need to install the virtualbox extension pack at https://www.virtualbox.org/wiki/Downloads . According to various internet sources, PXE boot from internal iPXE when using an emulated intel NIC is known-broken, and the "fix" is to install the extension pack, which provides the standard intel PXE ROM. This ROM can boot correctly, no problem.

I'm not sure I can fix virtualbox's builtin borked ipxe :( Their iPXE claims that it supports TFTP, but when I try and make it speak TFTP, it just fails without ever connecting to anything, with an error that makes no sense. So, that's not encouraging... I'll keep digging, but it's possible the "solution" will just be "don't use virtualbox, or if you have to use virtualbox, install the extension pack"

@cdata I suspect your own problem is something different, based on the fact that your setup is reversed (running pixiecore on virtualbox, booting a physical system). I'll investigate your report as soon as I close this one with a documentation fix.

[faddat]

@danderson for us it was not only VirtualBox that was broken. We had the same issue (mac address of client showing up as a filename) when booting using bare metal, too.

Reverting to Oct 17 solved issues on bare metal, and VirtualBox showed the same difficulties shown in #51. Provided screenshots of VirtualBox simply because it was showing the same issues as metal, and because it's hard to screencap our bare metal system during boot.

[danderson]

The MAC address in the filename is normal, we tell the firmware to boot //, so that the TFTP server knows which machine is requesting a boot file. You should see that in all boot attempts, and it shouldn't be a problem. Are you seeing problems with booting bare metal as well (no VirtualBox anywhere in the setup) ?

[danderson]

Okay, I think I've found what makes the embedded virtualbox iPXE unhappy. For some reason, it doesn't like the standard PXE format of specifying next-server+filename as a way of saying "you should boot with tftp://<next-server>/<filename>. Instead, it tries to interpret the filename <mac>/<arch> as a boot URL, and fails because it's not a URL.

I verified manually that if I give it a "filename" of tftp://<next-server>/<mac>/<arch>, virtualbox's ipxe is capable of chainloading to Pixiecore's ipxe, and from there we can successfully boot.

So, I think the change we need is:

• Always chainload to Pixiecore's iPXE, so we have a known set of features.
• When we want to chainload, detect if the PXE firmware is iPXE or non-iPXE. If iPXE, set the filename to tftp://<next-server>/<mac>/<arch>. Otherwise, set the filename to <mac>/<arch>.

This will require some refactoring of the boot logic, so it'll take a bit of time to make the change. I'll report back once it's done.

[danderson]

Okay, new problem... I implemented this, but now after chainloading to Pixiecore's iPXE, it cannot configure the network any more... iPXE's website says this can be a problem when you chainload undionly.kpxe (which is what we chainload), due to buggy firmwares :(

[danderson]

So, recapping, the failure you see here is actually a result of both of my attempts to fix #51. The obvious way to fix the problem triggered a bunch of bugs in my understanding of ipxe, and ipxe itself. I've updated #51 with the plan to fix Virtualbox booting.

Unclear if @cdata's issue is the same. The error he's getting suggests that it is, and if so, it's possible my virtualbox fix will also fix that. I'm filing a new bug to investigate that.