How to check whether HW is working correctly?

[corundum]

How to check whether HW is working correctly?

[alexforencich]

Yes, this is the network namespace helper script that I wrote: https://github.com/corundum/corundum/blob/master/fpga/lib/eth/scripts/dev-netns-shell.sh

[alexforencich]

The light comes on when the link is established. Simply connecting the cable is not enough, the cable must carry a valid 100G Ethernet signal that the CMAC can lock on to. The rx_status output goes high when the CMAC achieves RS-FEC lock, alignment marker lock on all 20 virtual lanes, and several other things. Take a look at the CMAC documentation from Xilinx for more details. But yes, if you connect the ports to each other with a QSFP28 DAC, then both link lights should come on shortly after connecting the cable, and both will go out when unplugging one end or the other.

[alexforencich]

Unfortunately I don't have any counters implemented inside the pipeline right now, it's on the to-do list. You did set the ports up, right? (ip link set dev up) And on the U250, I should have link lights working, at least for 100G links. If you're using the 100G design, do you see both of those lights come on? Try disconnecting and reconnecting the cable and see if there are two LEDs that follow the cable connection state - there should be 3 LEDs on the card, one blinks at 1 PPS, the other two should be the link lights. Also, are you using a DAC or an AOC to connect the ports? DACs pretty much "just work", but AOCs can be more sensitive to line rate settings and control pin levels.

[wtao0221]

so do you recommend using AOCs?

[alexforencich]

Depends on the length of cable you need. Under 2 or 3 m, use a DAC, it's cheaper. More than 3m, get AOCs, and whatever length you think you need, double it.

[wtao0221]

Got it. BTW, do we have link status LED on U250? Where are them?

[alexforencich]

Yeah, there should be three LEDs on the board visible through the back of the card - one red, one green, one yellow. I cannot recall which is which off the top of my head, but on the 100G design two are driven by the rx_status pins on the CMACs and so will indicate that the link is working and the other one is driven by the 1 pps output of the PTP hardware clock module. On the 10G design, only the 1 pps LED is connected, the other two are not currently used.

[wtao0221]

Thanks. will check this.

BTW, do you think my setting makes sense here for basic testing? I just connect those two QSFP+28 ports via a cable (will check its type). And tcpreply -i eth0 some.pcap to see whether I can observe traffic or data from the other side eth1.

[alexforencich]

That seems reasonable. I have had issues attempting to loop a host back to itself when attempting to use ping or iperf due to how linux routes packets, but I think there should be no issues with tcpreplay.

[Winters123]

Hi Alex,

Thanks for the clearance. btw, you mentioned the loopback issues you met with ping or iperf. I think you might want to try linux network namespace. So what you want to do is to put one of the interfaces (eth1 or etc.) into a different namespace (other than the default). With such a setting, you can use both ping or iperf for your lookback test.

you might want to check this for these cmds.

[Winters123]

oh, didn't notice it. Thanks for it!

[wtao0221]

BTW, when I inserted mqnic kernel module. It seems that I can not read the MAC address from EEPROM, as dmesg said "Bad MAC in EEPROM; using random MAC".

Is that a normal issue and how can I fix this? I do not want the MAC always being random.

[alexforencich]

On the Alveo boards, the MAC address must be read out of the BMC. I have not yet implemented this. However, on the AU200 and AU250, there is a user-accessible EEPROM that the driver can talk to, so for now what you can do is write a MAC address to the beginning of the EEPROM and the driver will load it. The EEPROM should appear in sysfs under the PCIe device, IIRC.

[wtao0221]

could you please point me to any material of these?

Thanks in advance!

[alexforencich]

Sure, I will test that on my U200 and tell you exactly what to do. But the jist of it is to use a hex editor or dd to write the base MAC address over the first 6 bytes of the EEPROM, and if you dig around in /sys/ with the mqnic driver loaded you should be able to find the entries in question that you can write to with dd. Try something like /sys/class/net/ethN/device/i2c-NN/54-something/eeprom. Possibly with an extra level in there somewhere for the I2C mux.

[alexforencich]

On my machine, the eeprom shows up as

/sys/class/net/eth0/device/i2c-4/i2c-7/7-0054/eeprom

I can read the eeprom by running

xxd /sys/class/net/eth0/device/i2c-4/i2c-7/7-0054/eeprom

I can set the base MAC by running:

echo 02 aa bb 00 00 00 | xxd -r -p - | dd bs=1 count=6 of=/sys/class/net/eth0/device/i2c-4/i2c-7/7-0054/eeprom

After reloading the driver, the card shows up as:

14: enp1s0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:aa:bb:00:00:00 brd ff:ff:ff:ff:ff:ff
15: enp1s0d1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 02:aa:bb:00:00:01 brd ff:ff:ff:ff:ff:ff

I recommend programming the smallest MAC address that's on the sticker on the board. Once the BMC interface is working, the MAC on the sticker will be used instead of whatever is in the EEPROM.

[wtao0221]

Thanks!

[alexforencich]

I also added a wiki page about writing to the EEPROM: https://github.com/corundum/corundum/wiki/Persistent-MAC-Addresses

[TimmonSha]

hello, I want to ask whether there are link lights in 100G design on U200? I connected two ports directly and tcpreplayed some packets on one port, but nothing was received on the other port. I want to make sure the link is up. Thanks a lot.

[alexforencich]

Yes, the 100G designs for the U200, U250, and VCU1525 should all have one link light per port that's connected to the rx_status outputs of the CMAC cores.

[TimmonSha]

Thanks. So when I plug the cable into one port, the corresponding light should be on, and when I unplug the cable the light should be off. Do I understand correctly?