Mellanox Driver Init Failing: PMD: mlx5_rxq.c:976: rxq_ctrl_setup(): too many SGEs (32) needed to handle requested maximum packet size 65536

[juniper]

Mellanox Driver Init Failing: PMD: mlx5_rxq.c:976: rxq_ctrl_setup(): too many SGEs (32) needed to handle requested maximum packet size 65536

[davvore33]

Hi @SSam009 could you try to use this fix? We don't have any mellanox NIC to test

https://github.com/Juniper/warp17/tree/dev/iss67_mlxdriver

submit eventually updates please

[SSam009]

Failing with different error now " ERROR: We don't support net_mlx5 since it has no "Maximum configurable length of RX pkt"!"

EAL: Detected 56 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:05:00.0 on NUMA socket 0
EAL: Device is blacklisted, not initializing
EAL: PCI device 0000:05:00.1 on NUMA socket 0
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:08:00.0 on NUMA socket 0
EAL: probe driver: 15b3:1013 net_mlx5
PMD: mlx5.c:421: mlx5_pci_probe(): PCI information matches, using device "mlx5_0" (SR-IOV: false, MPS: false)
PMD: mlx5.c:444: mlx5_pci_probe(): 1 port(s) detected
PMD: mlx5.c:592: mlx5_pci_probe(): port 1 MAC address is 24:8a:07:a4:ff:c8
EAL: PCI device 0000:82:00.0 on NUMA socket 1
EAL: probe driver: 15b3:1013 net_mlx5
PMD: mlx5.c:421: mlx5_pci_probe(): PCI information matches, using device "mlx5_1" (SR-IOV: false, MPS: false)
PMD: mlx5.c:444: mlx5_pci_probe(): 1 port(s) detected
PMD: mlx5.c:592: mlx5_pci_probe(): port 1 MAC address is 24:8a:07:a4:ff:98
warp17> USER1: Creating mempool global_cfg_mbuf-1 on core 1(size: 1024, obj_size: 2304, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf-3 on core 3(size: 393216, obj_size: 2304, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_tx_hdr-3 on core 3(size: 262144, obj_size: 326, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_clone-3 on core 3(size: 393216, obj_size: 128, priv: 4 cache: 512)
USER1: Creating mempool tcb_pool-3 on core 3(size: 524288, obj_size: 312, priv: 0 cache: 0)
USER1: Creating mempool ucb_pool-3 on core 3(size: 5242879, obj_size: 200, priv: 0 cache: 0)
USER1: Creating mempool global_pkt_mbuf-4 on core 4(size: 393216, obj_size: 2304, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_tx_hdr-4 on core 4(size: 262144, obj_size: 326, priv: 4 cache: 512)
USER1: Creating mempool global_pkt_mbuf_clone-4 on core 4(size: 393216, obj_size: 128, priv: 4 cache: 512)
USER1: Creating mempool tcb_pool-4 on core 4(size: 524288, obj_size: 312, priv: 0 cache: 0)
USER1: Creating mempool ucb_pool-4 on core 4(size: 5242879, obj_size: 200, priv: 0 cache: 0)
USER1: Found 2 Ethernet ports to start.
EAL: Error - exiting with code: 1
Cause: ERROR: We don't support net_mlx5 since it has no "Maximum configurable length of RX pkt"!

[SSam009]

I am not sure it's correct fix, but by looking at the working code (warp17-1.4 ) I have modified following line, now I don't see any errors for port initilization.
//port_dev_info[port].pi_dev_info.max_rx_pktlen = 0;
port_dev_info[port].pi_dev_info.max_rx_pktlen = PORT_MAX_MTU;

[davvore33]

Could you test this last commit for us? Thank you @SSam009

[SSam009]

@davvore33, getting the same error again

USER1: Found 2 Ethernet ports to start.
USER1: [port_setup_port()] Initializing Ethernet port 0.
PMD: mlx5_ethdev.c:547: dev_configure(): 0x960300: TX queues number update: 0 -> 27
PMD: mlx5_ethdev.c:557: dev_configure(): 0x960300: RX queues number update: 0 -> 27
USER1: WARNING: Cores handling port 0 are on different sockets! This will affect performance!
PMD: mlx5_rxq.c:976: rxq_ctrl_setup(): 0x960300: too many SGEs (16) needed to handle requested maximum packet size 18396
USER1: ERROR: Failed rte_eth_rx_queue_setup(0, 0, ...), returned Value too large for defined data type(75)!
EAL: Error - exiting with code: 1
Cause: ERROR: Failed initializing the Ethernets ports!

[SSam009]

@davvore33 , tested the latest code, now it's failing with "Failed to allocate TCB." error. I tried multiple configurations so far no success.

PMD: mlx5_ethdev.c:547: dev_configure(): 0x964380: TX queues number update: 0 -> 1
PMD: mlx5_ethdev.c:557: dev_configure(): 0x964380: RX queues number update: 0 -> 1
USER1: WARNING: Cores handling port 1 are on a different socket than the port itself! This will affect performance!
USER1: Ethernet port 1 initialized with MAC address 24:8A:07:A4:FF:98
USER1: Started 2 Ethernet ports.
USER1: Starting test management loop on lcore 2, core index 1
USER1: Starting receive packet loop on lcore 4, core index 3
USER1: Starting receive packet loop on lcore 3, core index 2
#PORT 0 - client port

PORT 1 - server port

add tests l3_intf port 0 ip 100.64.18.23 mask 255.255.255.0
L3 interface successfully added.
add tests l3_gw port 0 gw 100.64.18.22
Default gateway successfully added.

add tests client tcp port 0 test-case-id 0 src 140.1.1.10 140.1.255.255 sport 1000 20000 dest 100.64.200.3 100.64.200.3 dport 80 80
Test case 0 successfully added to port 0!

set tests timeouts port 0 test-case-id 0 init 0
Port 0, Test Case 0 updated!
set tests timeouts port 0 test-case-id 0 uptime 5
Port 0, Test Case 0 updated!
set tests timeouts port 0 test-case-id 0 downtime 1
Port 0, Test Case 0 updated!

set tests rate port 0 test-case-id 0 open 1000
Port 0, Test Case 0 updated!
set tests rate port 0 test-case-id 0 send 900
Port 0, Test Case 0 updated!

set tests criteria port 0 test-case-id 0 run-time 400
Port 0, Test Case 0 updated!
set tests client http port 0 test-case-id 0 GET google.com /index.html req-size 64
Port 0, Test Case 0 updated!

Server configuration

add tests l3_intf port 1 ip 100.64.18.25 mask 255.255.255.0
L3 interface successfully added.
add tests l3_gw port 1 gw 100.64.18.24
Default gateway successfully added.

add tests server tcp port 1 test-case-id 0 src 141.0.0.1 141.0.0.1 sport 80 80
Test case 0 successfully added to port 1!
set tests server http port 1 test-case-id 0 200-OK resp-size 20
Port 1, Test Case 0 updated!

start tests port 1
USER1: Test Case 0 started on port 1
Tests started on port 1
start tests port 0
USER1: [2:test_case_start_tcp_client()] Failed to allocate TCB.
USER1: [2:test_case_start_tcp_client()] Failed to allocate TCB.
USER1: [2:test_case_start_tcp_client()] Failed to allocate TCB.
USER1: [2:test_case_start_tcp_client()] Failed to allocate TCB.
USER1: [2:test_case_start_tcp_client()] Failed to allocate TCB.
USER1: [2:test_case_start_tcp_client()] Failed to allocate TCB.

[davvore33]

Hi @SSam009 , can you post your command line? can be a matter of arguments as well

[SSam009]

@davvore33
warp17 -c 0x1E -n 4 -m 32768 -b 05:00.0 05:00.1 -- --qmap-default max-c --tcb-pool-sz 1024 --cmd-file test_dns_service.cfg
same command working with older version of warp17.

[davvore33]

@SSam009 with those resource binded to warp17 you won't have >1 billion of sessions (140.1.255.255 - 140.1.1.10 = 65258IPs * 19Kports = ~ 1Billion of sessions), Instead of

add tests client tcp port 0 test-case-id 0 src 140.1.1.10 140.1.255.255 sport 1000 20000 dest 100.64.200.3 100.64.200.3 dport 80 80

you could try with something like

add tests client tcp port 0 test-case-id 0 src 140.1.1.10 140.1.1.255 sport 1000 3000 dest 100.64.200.3 100.64.200.3 dport 80 80

[SSam009]

Thanks, @davvore33, your config change suggestions are working good.

[davvore33]

Here is the commit on dev/common
Commit: 8912d00
I'm going to close the Issue now