make impl failed at u280 fpga 100g project

[corundum]

make impl failed at u280 fpga 100g project

[csunclechen]

have you met this issue before?

[alexforencich]

The CMAC core interface changed in Vivado 2020.1. Previously all of the serdes pins were separate ports, but in 2020.1 this was changed to be one set of ports with the channels concatenated. So, either upgrade to 2020.1 or patch the CMAC instances in fpga.v.

[csunclechen]

thanks for your tips. I can successfully make AU280/fpga_100g and make program now. After I insmod mqnic.ko to test the project, I lsmod to make sure the driver mqnic exits. But I see nothing about the driver mqnic when I dmesg | grep mqnic. I was wondering how the driver test the project and what is the correct test result that is shown by the driver.

[alexforencich]

The machine must be rebooted after programming the FPGA to re-enumerate the PCIe bus due to the PCIe BAR configuration on the card changing (this is an annoying limitation of PCIe, the bus can only be fully enumerated on boot and afterwards certain changes cannot be made, unless provisions are made to reserve resources beforehand). Once you do that, you should be able to load the driver. Now, if you simply load a new revision of corundum onto the card after it has successfully enumerated as corundum, you can usually get away with a PCIe hot reset instead of a full reboot so long as you don't change the BAR configuration. There is a script to do that: fpga/lib/pcie/scripts/pcie_hot_reset.sh. This functionality is also included in mqnic-fw, but that currently has a chicken-and-egg problem of refusing to connect to the card if it needs to be reset, so for the meantime you'll need to use pcie_hot_reset.sh. Also note that if you have the card installed in a server, you may need to run pcie_disable_fatal_err.sh to prevent the IPMI controller from rebooting the server.

[csunclechen]

I rebooted and the driver can monitor the pcie device successfully. Here is the result. Does it work well? The host eth0 and eth1 didn't connect to the card. only pcie connected. Will it effect the whole work?

[csunclechen]

I am trying to run corundum to find out if corundum can support the 100g full duplex transmission between two fpga cards like AU280 through the qsfp28 eth port and network switch. If corundum can't, How hard I can change your project to meet the demand?

[alexforencich]

That looks correct. At that point, you should be able to put the interfaces up, add IP addresses, and ping, run iperf, etc. The 100G interfaces are generally rather robust; I have had issues with 10G interfaces sometimes not coming up without resetting the design.

It should be able to get close to 100 Gbps, especially if you use jumbo frames. The limit is primarily the linux kernel networking stack, although there may be some driver optimizations that could help with throughput, especially for smaller packet sizes. I have had issues with running both high rate RX and TX at the same time, but it's unclear if the bottleneck is in the NIC design, PCIe hard IP core, host system, driver, OS, or application. I have been working to incrementally improve the performance under various conditions, but it's not a simple problem to solve.

[csunclechen]

I try to put the interfaces up. So I ifconfig -a to find the down-state eth. But I can't see it. Then I lspci | grep eth and find the device. af:00.0 can correspond the driver mqnic. Does the driver have some problems that make the host can't identify the device as NIC?

[alexforencich]

Try

ip link

To list the devices. It does appear in lspci, that last entry is the FPGA. And the driver would not show anything in dmesg if it wasn't detected. The driver then registers the network interfaces with the OS. You can see references to eth0 at the bottom of the dmesg output.

To bring the link up, try something like

ip link set dev eth0 up

And then

ip addr add 10.0.0.1/24 dev eth0