Comments (8)
krsna1729
commented on May 19, 2024
Vanilla BESS does not need DeInit. I ran up4 without this patch in a loop for 100 times without any crash or allocation failure.
Also the DeInit implemented in Exactmatch of your patch works fine. The issue is only with DeInit logic of patched wildcard
from upf.
amarsri28
commented on May 19, 2024
- Vanilla BESS also do allocations on heap, which doesnt seems getting free at reset , so its seems to be a memory leak, but of low amplitude(so may not cause issue if ran multiple times). In our case its visible because entire hash table doesn't get freed.
- I Used same nomenclature in wildcard & Exact Match for 'De-init'(code shared on local branch) but Wildcard crash and for now Exact match doesn't, We need to analyze the root cause in Wildcard first and also see if our way of 'Deinit()' in Exact Match need some improvement further. Code need more cleanup.
from upf.
krsna1729
commented on May 19, 2024
@amarsri28 this reminds me open an issue to implement size of the hash table configured at init instead of 1M entry hardcode currently in your patch.
from upf.
krsna1729
commented on May 19, 2024
sh: 1: insmod: not found
ESC[0;33mW0803 17:31:20.980811 1 vport.cc:330] Cannot load kernel module /bin/kmod/bess.ko
ESC[mI0803 17:31:20.980841 1 bessctl.cc:1931] Server listening on 0.0.0.0:10514
I0803 17:31:20.980880016 1 server_builder.cc:247] Synchronous server. Num CQs: 1, Min pollers: 1, Max Pollers: 1, CQ timeout (msec): 1000
# ./docker_setup.sh
I0803 17:31:29.652437 12 bessctl.cc:487] *** All workers have been paused ***
I0803 17:31:30.008656 37 worker.cc:319] Worker 0(0x7f8137ffd700) is running on core 12 (socket 0)
I0803 17:31:30.012262 12 packet_pool.cc:70] PacketPool2 requests for 65535 packets
I0803 17:31:30.030340 12 packet_pool.cc:157] PacketPool2 has been created with 65535 packets
I0803 17:31:30.047395 27 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:31:30.066031 39 packet_pool.cc:70] PacketPool3 requests for 65535 packets
I0803 17:31:30.084606 39 packet_pool.cc:157] PacketPool3 has been created with 65535 packets
I0803 17:31:30.100592 27 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:31:30.159476 43 bessctl.cc:691] Checking scheduling constraints
ESC[0;33mW0803 17:31:30.159970 39 metadata.cc:77] Metadata attr timestamp/8 of module access_measure has no upstream module that sets the value!
ESC[mESC[0;33mW0803 17:31:30.159986 39 metadata.cc:77] Metadata attr timestamp/8 of module core_measure has no upstream module that sets the value!
ESC[mI0803 17:31:30.160085 39 bessctl.cc:516] *** Resuming ***
# 1 # docker exec -it bess ./bessctl run up4
I0803 17:32:22.919605 39 bessctl.cc:487] *** All workers have been paused ***
I0803 17:32:22.919838 43 bessctl.cc:457] *** ResetAll requested ***
I0803 17:32:23.016655 43 bessctl.cc:1186] *** All modules have been destroyed ***
I0803 17:32:23.016901 43 bessctl.cc:977] *** All ports have been destroyed ***
I0803 17:32:23.016988 43 bessctl.cc:537] *** All workers have been destroyed ***
I0803 17:32:23.017004 37 worker.cc:325] Worker 0(0x7f8137ffd700) is quitting... (core 12, socket 0)
I0803 17:32:23.352613 73 worker.cc:319] Worker 0(0x7f8137ffd700) is running on core 12 (socket 0)
I0803 17:32:23.355507 39 packet_pool.cc:70] PacketPool4 requests for 65535 packets
I0803 17:32:23.373627 39 packet_pool.cc:157] PacketPool4 has been created with 65535 packets
I0803 17:32:23.390179 74 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:23.409282 75 packet_pool.cc:70] PacketPool5 requests for 65535 packets
I0803 17:32:23.427696 75 packet_pool.cc:157] PacketPool5 has been created with 65535 packets
I0803 17:32:23.443737 43 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:23.527647 79 bessctl.cc:691] Checking scheduling constraints
ESC[0;33mW0803 17:32:23.528151 75 metadata.cc:77] Metadata attr timestamp/8 of module access_measure has no upstream module that sets the value!
ESC[mESC[0;33mW0803 17:32:23.528168 75 metadata.cc:77] Metadata attr timestamp/8 of module core_measure has no upstream module that sets the value!
ESC[mI0803 17:32:23.528280 75 bessctl.cc:516] *** Resuming ***
# 2 # docker exec -it bess ./bessctl run up4
I0803 17:32:36.294975 75 bessctl.cc:487] *** All workers have been paused ***
I0803 17:32:36.295223 79 bessctl.cc:457] *** ResetAll requested ***
I0803 17:32:36.387238 79 bessctl.cc:1186] *** All modules have been destroyed ***
I0803 17:32:36.387444 79 bessctl.cc:977] *** All ports have been destroyed ***
I0803 17:32:36.387531 79 bessctl.cc:537] *** All workers have been destroyed ***
I0803 17:32:36.387549 73 worker.cc:325] Worker 0(0x7f8137ffd700) is quitting... (core 12, socket 0)
I0803 17:32:36.720604 104 worker.cc:319] Worker 0(0x7f8137ffd700) is running on core 12 (socket 0)
I0803 17:32:36.723242 75 packet_pool.cc:70] PacketPool6 requests for 65535 packets
I0803 17:32:36.741298 75 packet_pool.cc:157] PacketPool6 has been created with 65535 packets
I0803 17:32:36.755770 79 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:36.769212 106 packet_pool.cc:70] PacketPool7 requests for 65535 packets
I0803 17:32:36.786192 106 packet_pool.cc:157] PacketPool7 has been created with 65535 packets
I0803 17:32:36.798449 75 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:36.856086 75 bessctl.cc:691] Checking scheduling constraints
ESC[0;33mW0803 17:32:36.856565 110 metadata.cc:77] Metadata attr timestamp/8 of module access_measure has no upstream module that sets the value!
ESC[mESC[0;33mW0803 17:32:36.856582 110 metadata.cc:77] Metadata attr timestamp/8 of module core_measure has no upstream module that sets the value!
ESC[mI0803 17:32:36.856685 110 bessctl.cc:516] *** Resuming ***
# 3 # docker exec -it bess ./bessctl run up4
I0803 17:32:37.614322 75 bessctl.cc:487] *** All workers have been paused ***
I0803 17:32:37.614543 110 bessctl.cc:457] *** ResetAll requested ***
I0803 17:32:37.702212 110 bessctl.cc:1186] *** All modules have been destroyed ***
I0803 17:32:37.702373 110 bessctl.cc:977] *** All ports have been destroyed ***
I0803 17:32:37.702441 110 bessctl.cc:537] *** All workers have been destroyed ***
I0803 17:32:37.702459 104 worker.cc:325] Worker 0(0x7f8137ffd700) is quitting... (core 12, socket 0)
I0803 17:32:38.040553 138 worker.cc:319] Worker 0(0x7f8137ffd700) is running on core 12 (socket 0)
I0803 17:32:38.042585 75 packet_pool.cc:70] PacketPool8 requests for 65535 packets
I0803 17:32:38.059653 75 packet_pool.cc:157] PacketPool8 has been created with 65535 packets
I0803 17:32:38.068555 75 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:38.081611 139 packet_pool.cc:70] PacketPool9 requests for 65535 packets
I0803 17:32:38.098451 139 packet_pool.cc:157] PacketPool9 has been created with 65535 packets
I0803 17:32:38.107899 110 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:38.165206 143 bessctl.cc:691] Checking scheduling constraints
ESC[0;33mW0803 17:32:38.165705 110 metadata.cc:77] Metadata attr timestamp/8 of module access_measure has no upstream module that sets the value!
ESC[mESC[0;33mW0803 17:32:38.165719 110 metadata.cc:77] Metadata attr timestamp/8 of module core_measure has no upstream module that sets the value!
ESC[mI0803 17:32:38.165830 110 bessctl.cc:516] *** Resuming ***
# 4 # docker exec -it bess ./bessctl run up4
I0803 17:32:38.892449 110 bessctl.cc:487] *** All workers have been paused ***
I0803 17:32:38.892699 143 bessctl.cc:457] *** ResetAll requested ***
I0803 17:32:38.979990 143 bessctl.cc:1186] *** All modules have been destroyed ***
I0803 17:32:38.980185 143 bessctl.cc:977] *** All ports have been destroyed ***
I0803 17:32:38.980265 143 bessctl.cc:537] *** All workers have been destroyed ***
I0803 17:32:38.980274 138 worker.cc:325] Worker 0(0x7f8137ffd700) is quitting... (core 12, socket 0)
I0803 17:32:39.316596 168 worker.cc:319] Worker 0(0x7f8137ffd700) is running on core 12 (socket 0)
I0803 17:32:39.318600 110 packet_pool.cc:70] PacketPool10 requests for 65535 packets
I0803 17:32:39.335528 110 packet_pool.cc:157] PacketPool10 has been created with 65535 packets
I0803 17:32:39.344996 143 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:39.357714 170 packet_pool.cc:70] PacketPool11 requests for 65535 packets
I0803 17:32:39.374604 170 packet_pool.cc:157] PacketPool11 has been created with 65535 packets
I0803 17:32:39.381403 110 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:39.438671 174 bessctl.cc:691] Checking scheduling constraints
ESC[0;33mW0803 17:32:39.439173 110 metadata.cc:77] Metadata attr timestamp/8 of module access_measure has no upstream module that sets the value!
ESC[mESC[0;33mW0803 17:32:39.439188 110 metadata.cc:77] Metadata attr timestamp/8 of module core_measure has no upstream module that sets the value!
ESC[mI0803 17:32:39.439293 110 bessctl.cc:516] *** Resuming ***
# 5 # docker exec -it bess ./bessctl run up4
I0803 17:32:40.188390 110 bessctl.cc:487] *** All workers have been paused ***
I0803 17:32:40.188624 174 bessctl.cc:457] *** ResetAll requested ***
I0803 17:32:40.275872 174 bessctl.cc:1186] *** All modules have been destroyed ***
I0803 17:32:40.276037 174 bessctl.cc:977] *** All ports have been destroyed ***
I0803 17:32:40.276116 174 bessctl.cc:537] *** All workers have been destroyed ***
I0803 17:32:40.276137 168 worker.cc:325] Worker 0(0x7f8137ffd700) is quitting... (core 12, socket 0)
I0803 17:32:40.612597 202 worker.cc:319] Worker 0(0x7f8137ffd700) is running on core 12 (socket 0)
I0803 17:32:40.614571 110 packet_pool.cc:70] PacketPool12 requests for 65535 packets
I0803 17:32:40.631520 110 packet_pool.cc:157] PacketPool12 has been created with 65535 packets
I0803 17:32:40.638339 174 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:40.651021 203 packet_pool.cc:70] PacketPool13 requests for 65535 packets
I0803 17:32:40.667881 203 packet_pool.cc:157] PacketPool13 has been created with 65535 packets
I0803 17:32:40.679661 174 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:40.738204 207 bessctl.cc:691] Checking scheduling constraints
ESC[0;33mW0803 17:32:40.738693 174 metadata.cc:77] Metadata attr timestamp/8 of module access_measure has no upstream module that sets the value!
ESC[mESC[0;33mW0803 17:32:40.738711 174 metadata.cc:77] Metadata attr timestamp/8 of module core_measure has no upstream module that sets the value!
ESC[mI0803 17:32:40.738822 174 bessctl.cc:516] *** Resuming ***
# 6 # docker exec -it bess ./bessctl run up4
I0803 17:32:41.488739 174 bessctl.cc:487] *** All workers have been paused ***
I0803 17:32:41.488981 207 bessctl.cc:457] *** ResetAll requested ***
I0803 17:32:41.577760 207 bessctl.cc:1186] *** All modules have been destroyed ***
I0803 17:32:41.577958 207 bessctl.cc:977] *** All ports have been destroyed ***
I0803 17:32:41.578055 207 bessctl.cc:537] *** All workers have been destroyed ***
I0803 17:32:41.578064 202 worker.cc:325] Worker 0(0x7f8137ffd700) is quitting... (core 12, socket 0)
I0803 17:32:41.916596 235 worker.cc:319] Worker 0(0x7f8137ffd700) is running on core 12 (socket 0)
I0803 17:32:41.918604 174 packet_pool.cc:70] PacketPool14 requests for 65535 packets
I0803 17:32:41.935632 174 packet_pool.cc:157] PacketPool14 has been created with 65535 packets
I0803 17:32:41.942441 207 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:41.955417 236 packet_pool.cc:70] PacketPool15 requests for 65535 packets
I0803 17:32:41.972183 236 packet_pool.cc:157] PacketPool15 has been created with 65535 packets
I0803 17:32:41.978857 207 dpdk.cc:72] USER1: rte_ip_frag_table_create: allocated of 6291584 bytes at socket -1
I0803 17:32:41.999366 240 dpdk.cc:72] HASH: memory allocation failed <----------------
I0803 17:32:42.039973 207 bessctl.cc:691] Checking scheduling constraints
# docker exec bess-pfcpiface pfcpiface -config /conf/upf.json -simulate create
I0803 17:35:04.343660 3108 dpdk.cc:72] HASH: memory allocation failed
I0803 17:35:04.344419 3108 dpdk.cc:72] HASH: memory allocation failed
from upf.
krsna1729
commented on May 19, 2024
./docker_setup.sh
i=0; while [ $i -ne 100 ]; do i=$(($i+1)); echo "$i"; docker exec -it bess ./bessctl run up4; done
from upf.
krsna1729
commented on May 19, 2024
void WildcardMatch::DeInit() {
for (auto &tuple : tuples_) {
if (!tuple.ht)
continue;
tuple.ht->DeInit();
tuple.ht = NULL;
}
}@amarsri28 the above DeInit with NULL check works
from upf.
amarsri28
commented on May 19, 2024
Sai Crashes with above changes in Exact Match : logs
I0809 11:34:19.375598 10 bessctl.cc:537] *** All workers have been destroyed ***
I0809 11:34:19.375619 34 worker.cc:325] Worker 0(0x7f0f8b7fc700) is quitting... (core 12, socket 0)
F0809 11:34:19.420536 10 debug.cc:405] A critical error has occured. Aborting...
Signal: 11 (Segmentation fault), si_code: 1 (SEGV_MAPERR: address not mapped to object)
pid: 1, tid: 10, address: 0, IP: 0x5640055e44e8
Backtrace (recent calls first) ---
(0): bessd(_ZN10ExactMatch6DeInitEv+0x8) [0x5640055e44e8]
(1): bessd(_ZN6Module7DestroyEv+0x2a1) [0x564005415f31]
(2): bessd(_ZN11ModuleGraph12CreateModuleERK13ModuleBuilderRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN6google8protobuf3AnyEPN4bess2pb5ErrorE+0x33b) [0x56400543c79b]
(3): bessd(_ZN15BESSControlImpl12CreateModuleEPN4grpc13ServerContextEPKN4bess2pb19CreateModuleRequestEPNS4_20CreateModuleResponseE+0x297) [0x56400544b327]
(4): bessd(ZNSt17_Function_handlerIFN4grpc6StatusEPN4bess2pb11BESSControl7ServiceEPNS0_13ServerContextEPKNS3_19CreateModuleRequestEPNS3_20CreateModuleResponseEESt7_Mem_fnIMS5_FS1_S8_SB_SD_EEE9_M_invokeERKSt9_Any_dataOS6_OS8_OSB_OSD+0x44) [0x564005534014]
(5): bessd(_ZN4grpc16RpcMethodHandlerIN4bess2pb11BESSControl7ServiceENS2_19CreateModuleRequestENS2_20CreateModuleResponseEE10RunHandlerERKNS_13MethodHandler16HandlerParameterE+0x167) [0x56400557a0f7]
(6): bessd(_ZN4grpc6Server24SyncRequestThreadManager6DoWorkEPvb+0x22a) [0x56400605ec9a]
(7): bessd(_ZN4grpc13ThreadManager12MainWorkLoopEv+0xfa) [0x56400606100a]
(8): bessd(_ZN4grpc13ThreadManager12WorkerThread3RunEv+0xb) [0x56400606116b]
(9): bessd(+0x1459ebe) [0x5640061a4ebe]
(10): /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa2) [0x7f0f9284dfa2]
start_thread at ??:?
(11): /lib/x86_64-linux-gnu/libc.so.6(clone+0x3e) [0x7f0f925dc4ce]
clone at ??:?
from upf.
github-actions
commented on May 19, 2024
This issue has been stale for 30 days and will be closed in 5 days. Comment to keep it open.
from upf.
Related Issues (20)
- Support Largest burst sizes in pipeline HOT 2
- E2E integration test for UP4 Error HOT 2
- TCP Out-Of-Order / Dup ACK While streaming from Local nginx App HOT 1
- upf-0 stuck in PodInitializing state indefinitely HOT 1
- Work on multiple lcores/cores
- Run bess in dpdk mode using Mellanox VFs HOT 3
- protobuf python package and protoc version incompatibility when building docker image HOT 2
- Unable to run `./bessctl run up4` HOT 7
- PFCP messages bad UDP checksum HOT 2
- issue in Upf compilation HOT 2
- Patch not able to Merge due to failing background check HOT 13
- UPF performance issue for TCP traffic HOT 22
- code compilation issue -sim mode HOT 7
- Temporary issue deploying the UPF in standalone mode due to CNDP integration HOT 2
- Error deleting PDRs when using PFCP in simulation mode (-simulate delete) HOT 13
- Add documentation about configuring and deploying the UPF in af_packet and af_xdp modes HOT 15
- IPFrag module initialization fails when hugepage is not enabled HOT 6
- UPF dpdk-mode with virtio-net don't process downlink traffic HOT 11
- QUESTION: Is there a release schedule? HOT 1
- Possible need to revert PR #698 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from upf.