Unable to run `./bessctl run up4`

[diwakargoel]

Unable to run `./bessctl run up4`

[amarsri28]

noiommu could mean that virtualization may not be enabled in bios. pls check

[diwakargoel]

Hi @amarsri28 , thanks for your response.

Checked again and do see virtualization enabled in the BIOS.

Here are the flags enabled on my host linux machine, which includes svm

flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca

cpuinfo.txt

I also came across @krsna1729 's comment here: #304 (comment) which mentions that VM would require vfio_noiommu.

What else could I be missing?

Thanks!

[diwakargoel]

Hello - checking in to see if there are other suggestions for configuration I could check on my side?

I did attach the logs for running the dpdk ./examples/dpdk-helloworld example which seemed to succeed.

Would any other example from here https://doc.dpdk.org/guides/sample_app_ug/index.html be useful to run to validate my dpdk or scripts/docker_setup.sh and conf/up4.bess configuration?

Thank you!

[amarsri28]

@diwakargoel Thanks for cpu confg. I understand you are running under vm on amd host . below are few pointers
--> Bess uses particular version of dpdk , pls try to run sriov app with that to observe behaviour.
--> No iommu typically need kernel support as well. just check on that
--> there are other flags as well like "iommu" in amd and then virtual machine need some config as well for iommu support(just to test)

[diwakargoel]

Hi @amarsri28 , thanks for looking into this again.

With your pointers I switched to using the Vagrantfile from here: https://github.com/NetSys/bess/wiki/Build-and-Install-BESS which pulls and builds it's own version of dpdk.

This Vagrantfile sets up 2 virtio interfaces which I then bound to vfio-pci as shown below:

Before:

root@bessvm:/home/vagrant# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
<none>

Network devices using kernel driver
===================================
0000:00:03.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused= *Active*
0000:00:08.0 'Virtio network device 1000' if=eth1 drv=virtio-pci unused= 
0000:00:09.0 'Virtio network device 1000' if=eth2 drv=virtio-pci unused= 

Other Network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other Crypto devices
====================
<none>

Eventdev devices using DPDK-compatible driver
=============================================
<none>

Eventdev devices using kernel driver
====================================
<none>

Other Eventdev devices
======================
<none>

Mempool devices using DPDK-compatible driver
============================================
<none>

Mempool devices using kernel driver
===================================
<none>

Other Mempool devices
=====================
<none>
root@bessvm:/home/vagrant# 

Binding:

root@bessvm:/home/vagrant# modprobe vfio enable_unsafe_noiommu_mode=1
root@bessvm:/home/vagrant# modprobe vfio-pci
root@bessvm:/home/vagrant# echo 1 > /sys/module/vfio/parameters/enable_unsafe_noiommu_mode
root@bessvm:/home/vagrant# dpdk-devbind.py --bind=vfio-pci 00:08.0
root@bessvm:/home/vagrant# dpdk-devbind.py --bind=vfio-pci 00:09.0

After:

root@bessvm:/home/vagrant# dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:00:08.0 'Virtio network device 1000' drv=vfio-pci unused=
0000:00:09.0 'Virtio network device 1000' drv=vfio-pci unused=

Network devices using kernel driver
===================================
0000:00:03.0 'Virtio network device 1000' if=eth0 drv=virtio-pci unused=vfio-pci *Active*

Other Network devices
=====================
<none>

Crypto devices using DPDK-compatible driver
===========================================
<none>

Crypto devices using kernel driver
==================================
<none>

Other Crypto devices
====================
<none>

Eventdev devices using DPDK-compatible driver
=============================================
<none>

Eventdev devices using kernel driver
====================================
<none>

Other Eventdev devices
======================
<none>

Mempool devices using DPDK-compatible driver
============================================
<none>

Mempool devices using kernel driver
===================================
<none>

Other Mempool devices
=====================
<none>

But on running ./scripts/docker_setup.sh still get the same output:

+ docker exec bess ./bessctl run up4
E0711 23:40:16.708421776      14 fork_posix.cc:76]           Other threads are currently calling into gRPC, skipping fork() handlers
E0711 23:40:16.710408346      14 fork_posix.cc:76]           Other threads are currently calling into gRPC, skipping fork() handlers
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
To remove this DeprecationWarning exception, start IPDB(deprecation_warning=False, ...)
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
To remove this DeprecationWarning exception, start IPDB(deprecation_warning=False, ...)
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
To remove this DeprecationWarning exception, start IPDB(deprecation_warning=False, ...)
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
To remove this DeprecationWarning exception, start IPDB(deprecation_warning=False, ...)
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
To remove this DeprecationWarning exception, start IPDB(deprecation_warning=False, ...)
Deprecation warning https://docs.pyroute2.org/ipdb_toc.html
To remove this DeprecationWarning exception, start IPDB(deprecation_warning=False, ...)
*** Error: Unhandled exception in the configuration script (most recent call last)
  File "/opt/bess/bessctl/conf/up4.bess", line 136, in <module>
    p.init_port(idx, parser.mode)
  File "/opt/bess/bessctl/conf/ports.py", line 181, in init_port
    raise Exception(
Exception: Registered port for ens803f2 not detected!
  Command failed: run up4
max_ip_defrag_flows value not set. Not installing IP4Defrag module.
ip_frag_with_eth_mtu value not set. Not installing IP4Frag module.
Can't parse unix socket paths for notify! Setting it to default values (/tmp/notifycp)
Can't parse unix socket paths for end marker! Setting it to default values (/tmp/pfcpport)
Setting up port ens803f2 on worker ids [0]

Should I be using a different driver instead of vfio-pci to bind these virtio interfaces?

Would appreciate if you could take another look? Or instructions similar to https://github.com/NetSys/bess/wiki/Build-and-Install-BESS#start-up-bess-and-run-a-sample-configuration could also be provided for this project.

Thanks!

[amarsri28]

i understand you are running in vm over amd machine. i didnt tried with this config, few pointers will help.

1. use dpdk-20.11.3 to run sample dpdk app where you bind with noiommu in sriov and see if it works. on your setup. (i think hello-world does'nt do that ).

[github-actions]

This issue has been stale for 30 days and will be closed in 5 days. Comment to keep it open.