Inconsistent xApp while testing uplink traffic with multiple UEs

[mas-oliveira]

Inconsistent xApp while testing uplink traffic with multiple UEs

[yagoda]

Hi,

This error is coming from FlexRic so we cant say for sure what could be the issue as this is not our software.
Perhaps if you share the e2.pcap this could shed some light?

Regards,
Justin

[mas-oliveira]

Hi,

This error is coming from FlexRic so we cant say for sure what could be the issue as this is not our software. Perhaps if you share the e2.pcap this could shed some light?

Regards, Justin

Hello Justin,
e2ap.zip
These are the packets that I was able to capture during the execution with the filter sctp.port == 36421

[mas-oliveira]

Sorry, I have accidentally closed the issue.

[pgawlowicz]

Hi @mas-oliveira,
could you check the CPU load during the emulation? The E2 Agent is executed in a low-priority thread, therefore if the gnb has a lot to do, the E2 agent will struggle. This issue should be only present with zmq based setup. To reduce the CPU load you can increase the slow-down factor.

[pgawlowicz]

Also, please note that in order to generate uplink traffic you should rather use the following commands

sudo ip netns exec ue1 iperf -c 10.45.1.1 -i 1 -t 9000 -u -b 2M
sudo ip netns exec ue2 iperf -c 10.45.1.1 -i 1 -t 9000 -u -b 2M
sudo ip netns exec ue3 iperf -c 10.45.1.1 -i 1 -t 9000 -u -b 2M

All UEs will send UDP traffic to the core network IP. Note that when running in UDP mode, iperf client does not require server to be running. In the case of iperf3 server has to always run.

[pgawlowicz]

hi @mas-oliveira,
seems there was a bug in reporting of some metric. Could you try the following patch
e2sm_kpm.patch.txt

[mas-oliveira]

Hi @mas-oliveira, could you check the CPU load during the emulation? The E2 Agent is executed in a low-priority thread, therefore if the gnb has a lot to do, the E2 agent will struggle. This issue should be only present with zmq based setup. To reduce the CPU load you can increase the slow-down factor.

Thanks. I set the value up to 14 and it was possible to connect the 3 UEs, the problem is related to the CPU Load

[mas-oliveira]

Also, please note that in order to generate uplink traffic you should rather use the following commands

sudo ip netns exec ue1 iperf -c 10.45.1.1 -i 1 -t 9000 -u -b 2M
sudo ip netns exec ue2 iperf -c 10.45.1.1 -i 1 -t 9000 -u -b 2M
sudo ip netns exec ue3 iperf -c 10.45.1.1 -i 1 -t 9000 -u -b 2M

All UEs will send UDP traffic to the core network IP. Note that when running in UDP mode, iperf client does not require server to be running. In the case of iperf3 server has to always run.

Yes thanks, I had noticed that I had the inverse ways, I prefer to use the iPerf3 because it is possible to get more info about the connection.

[pgawlowicz]

Hi @mas-oliveira, did you try the provided patch? does it solve you issue?

[mas-oliveira]

Hi @mas-oliveira, did you try the provided patch? does it solve you issue?

Hello @pgawlowicz, sorry but I can't apply the patch:
marco@marco-NUC7i7BNH:~/srsRAN_Project$ git apply e2sm_kpm.patch
error: patch failed: lib/e2/e2sm/e2sm_kpm/e2sm_kpm_du_meas_provider_impl.cpp:544
error: lib/e2/e2sm/e2sm_kpm/e2sm_kpm_du_meas_provider_impl.cpp: patch does not apply

I'll send my current version of e2sm_kpm_du_meas_provider_impl.cpp
e2sm_kpm_du_meas_provider_impl.zip

[pgawlowicz]

could you try to apply the patch manually? Those are only two small fixes (each touching one line) in two files

[mas-oliveira]

could you try to apply the patch manually? Those are only two small fixes (each touching one line) in two files

Yes sure, I applied on scheduler_metric_handler.cpp but on e2sm_kpm_du_meas_provider_impl.cpp I wasn't able to find for (unsigned idx = 0; ue_aggr_rlc_metrics.size(); idx++) {
Am I working with the wrong version?

[pgawlowicz]

ok, seems you are using an older version, but this one does not contain the bug the patch is fixing. So it should be good. Now, you should be able to request RRU.Prb* and DRB.PacketSuccessRateUlgNBUu metrics. Could you confirm?

[mas-oliveira]

ok, seems you are using an older version, but this one does not contain the bug the patch is fixing. So it should be good. Now, you should be able to request RRU.Prb* and DRB.PacketSuccessRateUlgNBUu metrics. Could you confirm?

Hello again Piotr,

First of all thanks again for all of the support.

I tried to run with the following metrics:
{ name = "DRB.UEThpDl" },
{ name = "DRB.UEThpUl" },
{ name = "DRB.RlcPacketDropRateDl" },
{ name = "DRB.RlcSduTransmittedVolumeDL" },
{ name = "DRB.RlcSduTransmittedVolumeUL" },
{ name = "DRB.PacketSuccessRateUlgNBUu" },
{ name = "RRU.PrbAvailDl" },
{ name = "RRU.PrbAvailUl" }
And everything worked properly. But when I try to add the { name = "RRU.PrbTotUl" } the xApp just stay stucked at some point.

Generated of req_id = 1
E42_RIC_SUBSCRIPTION_REQUEST 31
adding event fd = 6 ev-> 6
[xApp]: RIC SUBSCRIPTION REQUEST sent
[xApp]: SUBSCRIPTION RESPONSE received
Pending event size before remove = 1
[xApp]: Successfully SUBSCRIBED to ran function = 2

I get this message but then I don't receive on my xApp any KPM message. This behavior is verified when I add the request to get the total number of PRB from any direction, uplink (RRU.PrbTotUl or RRU.PrbTotDl)

[pgawlowicz]

could you try to use the newest srsgnb version and apply the patch there?

[mas-oliveira]

could you try to use the newest srsgnb version and apply the patch there?

The behavior stills the same with the newer version and the patch already applied.

When I try to subscribe to the total amount of PRBs:

Near RT RIC logs:

[iApp]: E42 SETUP-REQUEST received
[iApp]: E42 SETUP-RESPONSE sent
[iApp]: SUBSCRIPTION-REQUEST xapp_ric_id->ric_id.ran_func_id 2
[E2AP] SUBSCRIPTION REQUEST generated
[NEAR-RIC]: nb_id 411 port = 25031

xApp Logs:

Generated of req_id = 1
E42_RIC_SUBSCRIPTION_REQUEST 31
adding event fd = 6 ev-> 6
[xApp]: RIC SUBSCRIPTION REQUEST sent
[xApp]: SUBSCRIPTION RESPONSE received
Pending event size before remove = 1
[xApp]: Successfully SUBSCRIBED to ran function = 2

And the xApp it gets everything stuck after that.
Without the total PRBs and only with the available metrics the xApp still running properly.

[pgawlowicz]

Hi @mas-oliveira, I have just tried with the newest gnb version (commit: 2f90c8b60e9396a7aed59645c98dbcbccda2bf7c), I have applied the patch:

diff --git a/lib/e2/e2sm/e2sm_kpm/e2sm_kpm_du_meas_provider_impl.cpp b/lib/e2/e2sm/e2sm_kpm/e2sm_kpm_du_meas_provider_impl.cpp
index 2874b0cb0..3b81d5e3c 100644
--- a/lib/e2/e2sm/e2sm_kpm/e2sm_kpm_du_meas_provider_impl.cpp
+++ b/lib/e2/e2sm/e2sm_kpm/e2sm_kpm_du_meas_provider_impl.cpp
@@ -533,7 +533,7 @@ bool e2sm_kpm_du_meas_provider_impl::get_drb_ul_success_rate(
     float              success_rate    = 0;
     uint32_t           total_lost_pdus = 0;
     uint32_t           total_pdus      = 0;
-    for (unsigned idx = 0; ue_aggr_rlc_metrics.size(); idx++) {
+    for (unsigned idx = 0; idx != ue_aggr_rlc_metrics.size(); idx++) {
       rlc_metrics& rlc_metric = ue_aggr_rlc_metrics[idx];
       total_lost_pdus += rlc_metric.rx.num_lost_pdus;
       total_pdus += rlc_metric.rx.num_pdus;
diff --git a/lib/scheduler/logging/scheduler_metric_handler.cpp b/lib/scheduler/logging/scheduler_metric_handler.cpp
index 400f385d3..8e419c2c7 100644
--- a/lib/scheduler/logging/scheduler_metric_handler.cpp
+++ b/lib/scheduler/logging/scheduler_metric_handler.cpp
@@ -272,6 +272,7 @@ scheduler_metrics_handler::ue_metric_context::compute_report(std::chrono::millis
   ret.dl_mcs        = sch_mcs_index{mcs};
   mcs               = data.nof_puschs > 0 ? std::roundf(static_cast<float>(data.ul_mcs) / data.nof_puschs) : 0;
   ret.ul_mcs        = sch_mcs_index{mcs};
+  ret.nof_prbs      = nof_prbs;
   ret.dl_prbs_used  = data.nof_dl_cws > 0 ? static_cast<double>(data.dl_prbs_used / data.nof_dl_cws) : 0;
   ret.ul_prbs_used  = data.nof_puschs > 0 ? static_cast<double>(data.ul_prbs_used / data.nof_puschs) : 0;
   ret.dl_brate_kbps = static_cast<double>(data.sum_dl_tb_bytes * 8U) / metric_report_period.count();

and both RRU.PrbTotUl and RRU.PrbTotDl work correctly.

[mas-oliveira]

Hello @pgawlowicz,

Sorry, it was a problem of my xApp, I'm now running with the default one and it's everything ok.
Thanks for the help!