[BUG] ping failed and rtps test failed.

[ZhenshengLee]

[BUG] ping failed and rtps test failed.

[jayvanmali]

Hi @akampmann

Great Library! Extremely cool to see the RTPS-DDS running on a Micro-controller

I have been trying to get the example project working (the new branch called /feature/track_latest) and i have had some issues trying to get the example working. I have used the branch as is and set my Unix machine Networking Information to:

Address: 192.168.1.104
Network: 255.255.0.0
Gateway: 192.168.1.104
DNS: 192.168.1.104

I have also been using the latest version of FastDDS

I have connected the STM32F7 directly to my computer and noticed that it was hanging when it was trying to match the publisher and subscriber.

I did some debugging and found that it was failing to join IGMP multicast group (The printf verbose debugging is extremely useful). Inside the STM32F7 in the stm32f7xx_hal_eth.c file i had to change

macinit.PromiscuousMode = ETH_PROMISCUOUS_MODE_DISABLE;
macinit.MulticastFramesFilter = ETH_MULTICASTFRAMESFILTER_PERFECT;

to
macinit.PromiscuousMode = ETH_PROMISCUOUS_MODE_ENABLE;
macinit.MulticastFramesFilter = ETH_MULTICASTFRAMESFILTER_NONE;

After changing this i was able to successfully join the IGMP group however i now get the error "ParticipantProxyData deserializtaion failed" when the handleSPDPPackage function is called.

I haven't found the cause of the "ParticipantProxyData deserializtaion failed" but am continuing to debug. If you have any comments on potential causes or tests to help narrow the problem down it would be great to hear them!

Cheers

Jay

[akampmann]

I will have a look and come back to you. I haven't been running embeddedRTPS on the STM32 code for a while. I've been mainly developing for the Xilinx UltraScale+ Cortex R5.

[akampmann]

@ZhenshengLee I've created a new branch that tracks the latest embeddedRTPS, CubeIDE, fw v1.16.2, FastDDS v2.3.2 (probably also works with the latest version):

https://github.com/embedded-software-laboratory/embeddedRTPS-STM32/tree/feature/track_latest

STM32 is now able to talk again to the Linux side, could you give it a try on your side?

I seem to have sporadic issues with pinging the STM, but that occurs regardless of including embeddedRTPS, please let me know if you observe the same behavior.

[ZhenshengLee]

STM32 is now able to talk again to the Linux side, could you give it a try on your side?

@akampmann Thanks for your quick reply!

I am on my vacation now, I'll give it a test in 10 days.

[jayvanmali]

Hi @akampmann, Thanks for your prompt reply.

Im currently away at the moment, but will test it with the FastDDS v2.3.2 version when i'm back (with in a day or so).

In regards to the MAC configuration changes, Once i get the example working, I will change it back to the same settings are provided in the example and see if it works. This should help understand the MAC configuration change.

Cheers

Jay

[akampmann]

@jayvanmali

this has most likely to do with the recent changes introduced that avoids adding remote locators that are not on the same subnet as embeddedRTPS. Could you please try to change this line to:

https://github.com/embedded-software-laboratory/embeddedRTPS/blob/68915c8ccc744db171a202182dce2ef55b7728ca/src/discovery/ParticipantProxyData.cpp#L187

to

return true;

The function returns false to indicate that the locator is not added, but that return is incorrectly causes the deserialization to stop. It worked on my side because since the first remote locator transmitted by remote participants was on the local network....

Do you mind confirming that this fixes the issue on your side?

I will then create a fix on the master branch shortly.

[akampmann]

@jayvanmali

I've updated https://github.com/embedded-software-laboratory/embeddedRTPS-STM32/tree/feature/track_latest to incorporate the fix. It works for me on the Cortex R5 now. Would you mind giving it a try on your side?

[jayvanmali]

Hi @akampmann

Apologies for the delayed response. I have been away for the past week and will back end of this week, Ill confirm the branch then and let you know my findings.

Cheers

[akampmann]

Hi,

do I understand correctly that you are not able to ping the STM?

If you are unable to ping the STM then something with your network or lwip config on the STM side is wrong. Can run your project without starting embeddedRTPS (comment out

embeddedRTPS-STM32/stm32/Src/main.cpp

Line 535 in f21d3fa

startRTPStest();

) and see if you are able to ping the STM?

[ZhenshengLee]

@akampmann thanks for your quick reply!

do I understand correctly that you are not able to ping the STM?

Yes, ping 192.168.1.103(stm32) outputs unreachable

If you are unable to ping the STM then something with your network or lwip config on the STM side is wrong. Can run your project without starting embeddedRTPS

Yes, I can ping the stm32 without starting embeddedRTPS, is there something wrong with the prio setting of threads? I've print the task info above.

This behavior happens also in microROS-ertps demo in https://github.com/micro-ROS/micro_ros_stm32cubemx_utils
this issue is related micro-ROS/micro_ros_stm32cubemx_utils#46

As for this repo, I've changed some code, but I believe the changing is about

• static ip resetting
• cubemx upgrade
• print2serial and gettaskinfo function.

You can check with this ZhenshengLee@3b983a3

Thanks.

[akampmann]

@jayvanmali thank you for trying out the new branch and providing feedback. I'm curious why the changes to the MAC configuration were not necessary on my side.

Regarding the deserialization failure: Could you please try FastDDS v2.3.2? That is the exact version that I tested on my side. There are probably changes to the SPDP messages in newer FastDDS version which have not been incorporated in embeddedRTPS yet. Such changes were also necessary in the past. I will try to find time to look into this. Meanwhile, could you try 2.3.2 on your side and come back to me?

[jayvanmali]

Hi @akampmann

So i have installed FastDDS v2.3.4 (I couldn't find the files for v2.3.2) and i still get the error:

[SPDP] ParticipantProxyData deserializtaion failed

Just to be safe, i cloned the files from GitHub (And swapped to the /feature/track_latest branch) and did the following (Determined by trial and error as well as the github read-me):

1. Setup my Unix IP to 192.168.1.104.

2. Deleted the G_Test folder (It was causing compilation issues).

3. Changed the name if the file config_stm to config and updated the IP address to 192.168.1.103 (The STM32 IP set by LwIP).

4. Added the IGMP flag to netif->flags in ethernetif.

5. Changed SPDP_WRITER_PRIO, THREAD_POOL_WRITER_PRIO, THREAD_POOL_READER_PRIO to be 24.

6. Updated Drivers/STM32F7xx_HAL_Driver/Src/stm32f7xx_eth.c to change:

   macinit.PromiscuousMode = ETH_PROMISCUOUS_MODE_DISABLE;
macinit.MulticastFramesFilter = ETH_MULTICASTFRAMESFILTER_PERFECT;

   to

   macinit.PromiscuousMode = ETH_PROMISCUOUS_MODE_ENABLE;
macinit.MulticastFramesFilter = ETH_MULTICASTFRAMESFILTER_NONE;

Here is a copy of the log from terminal: debuglog.txt

If you have any comments on potential causes or tests to help narrow the problem down it would be great to hear them!

I'm still making my way through debugging and will let you know when i manage to get it working!

Cheers

Jay

[jayvanmali]

Hi @akampmann

Great! ill give it a test a bit later this afternoon, Thanks for looking into it so quickly!

[ZhenshengLee]

@ZhenshengLee I've created a new branch that tracks the latest embeddedRTPS, CubeIDE, fw v1.16.2, FastDDS v2.3.2 (probably also works with the latest version):

https://github.com/embedded-software-laboratory/embeddedRTPS-STM32/tree/feature/track_latest

STM32 is now able to talk again to the Linux side, could you give it a try on your side?

@akampmann

The code has been altered to https://github.com/ZhenshengLee/embeddedRTPS-STM32/tree/outdoor/feature/track_latest/ to get a better debug experience.

• Tested with fastdds2.3.4, everything is OK
• linux side receiving well, sending well
• ping stm well
• stm side, doesn't konw if it works well

issue

the serial output behavior(the callback of rtpstest thread) is abnormal

expected

received data from Linux is output per second

actual

a lot of received data from Linux flushes out for once

screenshot

see the timestamps from the ternimal

screenshot

comment

is the serial or the callback work abnoraml?

Great! ill give it a test a bit later this afternoon, Thanks for looking into it so quickly!

EDIT: the test happens in 2022-0509-08:41 of your timezone, after your branch being updated for @jayvanmali 's issue

zs@zs-3630:~$ TZ='Europe/Berlin' date
2022年 05月 09日 星期一 08:41:15 CEST

[ZhenshengLee]

@ZhenshengLee I've created a new branch that tracks the latest embeddedRTPS, CubeIDE, fw v1.16.2, FastDDS v2.3.2 (probably also works with the latest version):

https://github.com/embedded-software-laboratory/embeddedRTPS-STM32/tree/feature/track_latest

STM32 is now able to talk again to the Linux side, could you give it a try on your side?

@akampmann

issue

oh, hello_world coredump after around 10 minutes

screenshot

screenshot

[ZhenshengLee]

I seem to have sporadic issues with pinging the STM, but that occurs regardless of including embeddedRTPS, please let me know if you observe the same behavior.

@akampmann

after a 2-hour test of ping, I think there's no issue of the ping function of the stm

here is my board and toolchain info:

• nucleo stm32f767zi
• stm32ide 1.9.0
• gcc: 10.3-2021.10 (builtin of the ide 1.9.0)
• stm32cubemx 6.2
• stm32cubeFW_F7 v1.16.2

screenshot

screenshot

[ZhenshengLee]

@akampmann

I think your new branch https://github.com/embedded-software-laboratory/embeddedRTPS-STM32/tree/feature/track_latest of work can be merged as the basic function(ping and rtpstest) has been recovered.

So, after the feature branch being merged into master

• this issue be closed
• two new issues be created to track the received data from Linux issue and "hello_world coredump" issue I mentioned above
• @jayvanmali would you agree that the ParticipantProxyData deserializtaion failed issue being tracked standalone?

[akampmann]

issue

the serial output behavior(the callback of rtpstest thread) is abnormal

expected

received data from Linux is output per second

actual

a lot of received data from Linux flushes out for once

screenshot

see the timestamps from the ternimal
screenshot

comment

is the serial or the callback work abnoraml?

@ZhenshengLee Can you add a newline character at the end of the output? Maybe the output buffer is only cleared and transmitted once full or after occurrence of a new line:

printf("Received data from Linux\n");

---

@jayvanmali would you agree that the ParticipantProxyData deserializtaion failed issue being tracked standalone?

@ZhenshengLee @jayvanmali This should be fixed on the latest master.

---

oh, hello_world coredump after around 10 minutes

@ZhenshengLee Could you please run with gdb and provide the output of backtrace once it runs into the segmentation fault?

[jayvanmali]

Hi @akampmann @ZhenshengLee

I have tried the new commit and the error for ParticipantProxyData deserializtaion failed has disappeared, however i still cant seem to get an output on either side. Wireshark shows that both the Unix machine and the STM32 are sending RTPS packets to 239.255.0.1 so i am not completely sure why i am not getting an output. The debug log however seems to freeze just after the ./hello_world linux application is started so i suspect it may be something to do with the RTOS threads.

Ive attached the RTPS debug log: RTPS_DEBUG.txt

Im also curious to know why i have to adjust the stm32f7xx_eth.c and the ethernetif file to successfully join the IGMP group. @ZhenshengLee did you have this issue when running this example?

[akampmann]

@jayvanmali Are you running this by any chance in a virtual machine? Could you provide a wireshark recording of the entire interaction?

[ZhenshengLee]

Im also curious to know why i have to adjust the stm32f7xx_eth.c and the ethernetif file to successfully join the IGMP group. @ZhenshengLee did you have this issue when running this example?

Nope, during my test, I did not change anything in eth setting of the stm32.

[jayvanmali]

Hi @akampmann

I'm running this on a Linux laptop with a HP USB-C to RJ45 Adaptor as the laptop does not have a Ethernet Port.

I have attached the RTPS Log from the STM32F7 and the Wireshark Recording:

RTPS Log: STM32_Log.txt
Wireshark Log: Wireshark_Recording.zip

After it appears to freeze, the pinging does not work anymore, I am doing some debugging to workout where its locking up but i cant immediately see anything failing.

Be happy to try any suggestions you recommend.

Cheers

Jay

[jayvanmali]

Hi @ZhenshengLee

That is interesting, Are you able to confirm if your stm32f7xx_eth has promiscuous mode enabled?

macinit.PromiscuousMode = ETH_PROMISCUOUS_MODE_ENABLE;
macinit.MulticastFramesFilter = ETH_MULTICASTFRAMESFILTER_NONE;

Can you also check your ethernetif and see if you have the IGMP flag set: NETIF_FLAG_IGMP

if i don't enable these i cant seem to join and IGMP group.

Much Appreciated

Jay

[akampmann]

Hi @akampmann

I'm running this on a Linux laptop with a HP USB-C to RJ45 Adaptor as the laptop does not have a Ethernet Port.

I have attached the RTPS Log from the STM32F7 and the Wireshark Recording:

RTPS Log: STM32_Log.txt Wireshark Log: Wireshark_Recording.zip

After it appears to freeze, the pinging does not work anymore, I am doing some debugging to workout where its locking up but i cant immediately see anything failing.

Be happy to try any suggestions you recommend.

Cheers

Jay

@jayvanmali

Something is off with the IP address configuration. The subnet assumed by embeddedRTPS is wrong (it sends messages to 192.168.42.44 (thus it assumes that this locator is on its own subnet)), but IP packets are actually sent from 192.168.1.103.

Can you confirm that the IP addresses here:

embeddedRTPS-STM32/stm32/LWIP/App/lwip.c

Line 69 in 6f9e84a

IP_ADDRESS[0] = 192;

And here:

embeddedRTPS-STM32/stm32/Core/Inc/rtps/config.h

Line 39 in 6f9e84a

const std::array<uint8_t, 4> IP_ADDRESS = {

match?

Currently configuring the IP address is a bit messy, as it has to be changed in two places. That is mainly because there does not seem to be a consistent way of configuring the IP address of the lwIP interface on different platforms (its different on the Cortex R5). I will have to add an assert at some point to warn of this misconfiguration during embeddedRTPS init.

[takasehideki]

This is just an FYI. I've confirmed feature/track_latest branch could work well in my environment as is (just changing the IP address). I have the utmost respect for @akampmann 's work!!

[akampmann]

@takasehideki thank you so much!

@jayvanmali did you have a chance to look at the IP config?

[akampmann]

Hi @jayvanmali did you have a chance to try the fix?

[jayvanmali]

Hi @akampmann, apologies for the delay, An Illness left me out of action for a while. I'll test it within the next day and report back, I'll attach the trace log and Wireshark log as well

Cheers

Jay

[jayvanmali]

Hi @akampmann

I've started trying the fixed, but have had network issues on the F7 which i have narrowed down to hardware issues, I'll continue testing the fix when i get access to a second F7 (within a day or so).

Cheers

[ZhenshengLee]

@jayvanmali

That is interesting, Are you able to confirm if your stm32f7xx_eth has promiscuous mode enabled?

macinit.PromiscuousMode = ETH_PROMISCUOUS_MODE_ENABLE; macinit.MulticastFramesFilter = ETH_MULTICASTFRAMESFILTER_NONE;

yes, you can check in https://github.com/ZhenshengLee/ros2_mcu/blob/0cdedf3f9f07599bb58fbd42e61ac5204fdfddd0/libs/custom_stm32f767zi_ertps/Drivers/STM32F7xx_HAL_Driver/Src/stm32f7xx_hal_eth.c#L1877-L1878

Can you also check your ethernetif and see if you have the IGMP flag set: NETIF_FLAG_IGMP

yes, you can check in https://github.com/ZhenshengLee/ros2_mcu/blob/0cdedf3f9f07599bb58fbd42e61ac5204fdfddd0/libs/custom_stm32f767zi_ertps/LWIP/Target/ethernetif.c#L241

[akampmann]

Hi @akampmann

I've started trying the fixed, but have had network issues on the F7 which i have narrowed down to hardware issues, I'll continue testing the fix when i get access to a second F7 (within a day or so).

Cheers

hey @jayvanmali, did you have a chance to try with a new F7?

[jayvanmali]

hey @akampmann, Apologies for the radio silence, I have been away. Sadly due to supply issues, i have not been able to get an F7 but i do have a H723. I'm going to retry and get this working on the H7.

I'm fully back on deck so i have more time and ill keep you updated on the progress, ill also use the newest version of embeddedRTPS. I have done a quick integration of it but it seems to fail joining the IGMP group so i need to check the MAC filtering.

Cheers for everything so far!

[jayvanmali]

Hey @akampmann

I have managed to test the new code on a STM32H723, Sorry for the late response. I managed to fix the IP Config

I still seem to have a problem with the matching of the Publishers and Subscribers. I have attached the Wireshark Log and the Embedded RTPS DDS Log.

If you have some thoughts, Happy to try any solution.

Cheers

Jay

RTPS_Wireshark.zip

RTPS_LOG.txt

[akampmann]

@jayvanmali

Hi,

the problem is simple, your interface IP address does not match the embeddedRTPS config. You are sending data from 192.168.1.47, but the discovery advertises 192.168.1.103 as the IP address of your STM.

You need to adapt this parameter to 192.168.1.47

embeddedRTPS-STM32/stm32/Core/Inc/rtps/config.h

Line 39 in b9a8bed

const std::array<uint8_t, 4> IP_ADDRESS = {

Unfortunately, you are not the first one to encounter this issue. I will try to include a warning or assert in the embeddedRTPS code to catch this problem.

[jayvanmali]

Hi @akampmann

Good catch! I fixed that change, but i still seem to have the STM32 code remain waiting for the Publisher and Subscribers to match.

Any thoughts on what might be causing that?

Cheers

Jay

[akampmann]

Can you provide a Wireshark recording with the adapted config?

Are you able to ping the STM?

[jayvanmali]

Hey @akampmann

I can successfully ping the STM32H7 on 192.168.1.47.

Here are the new New Logs

MCU.txt
Wrieshark Log.zip

Here is the RTPS config file:
config.zip

What factors would stop the publishers and subscribers from matching?

Cheers

Jay

[akampmann]

Hi @jayvanmali

your machine does not respond to the STM32.

Here are some thoughts:

• Are you using the FastDDS commit incorporate as a submodule in this repo?
• Could you provide a trace from the beginning of the communication?
• Also, do you have a router in between, that might block multicast? Could you try hooking up the STM directly with your machine?

Best regards
Alex

[jayvanmali]

Hey @akampmann

I think i have manged to get something working, It appears to be a issue with the version of FastDDS used in my ROS distribution (Locked to 2.6) and a second issue with the Firewall on my laptop

On a downgrade to 2.3 and just using FastDDS directly, it appears to be working.

Do you know what changes are potentially required to enable it working with FastDDS 2.6? Happy to look into it if you have any comments.

Best Regards

Jaynesh

[akampmann]

hey @jayvanmali

in the past, the main issues were increased discovery message sizes, due to additional fields. I will try to replicate this problem.

Have you tried turning verbosity on?

Best regards
Alex

[jayvanmali]

Hey @akampmann

I've turned on the debugging and am slowly going through the different versions of FastDDS and seeing when and if it fails and finding out why it does at those points, I will keep you updated with what i find.

One thing i have noticed is that when i try to add another participant in the same domain, i seem to keep getting a hard-fault even though my MAX_NUM_PARTICIPANTS in the config file is 2. I have made sure my FreeRTOS threads have a stack size of 5000. After stepping though the debugger it seems to fail at the "addBuiltInEndpoints". Do you have any idea of why this could be the case? or could i be missing a config setting to have multiple publishers and subscribers?

Cheers for all the help

Best Regards

Jay

[akampmann]

Hey @jayvanmali

I've finalized my thesis and now had some time to look into this.

I have tested embeddedRTPS both with FastDDS v2.6.0 and v2.8.0 - It worked with both versions on my side without requiring modifications to embeddedRTPS

Just to be clear: The issue on your side occurs in conjunction with ROS on top of embeddedRTPS, right? Can you confirm that FastDDS 2.6.0 and embeddedRTPS (without ROS2) works on your side? From my experience of using embeddedRTPS in a different context - these kind of issues have almost always been related to insufficient stack sizes of the reader workers. Depending on what you are doing in the context of the callback...

Can you try to use the following code snippet to print the stack consumption? Make sure to run it periodically or at least close to where you see your application crashing.

void printStackUsage() {
  UBaseType_t uxArraySize = uxTaskGetNumberOfTasks();
  unsigned long ulTotalRunTime, ulStatsAsPercentage;

  static TaskStatus_t* pxTaskStatusArray = static_cast<TaskStatus_t*>(
      pvPortMalloc(uxArraySize * sizeof(TaskStatus_t)));

  if (pxTaskStatusArray != NULL) {
    uxArraySize =
        uxTaskGetSystemState(pxTaskStatusArray, uxArraySize, &ulTotalRunTime);

    for (int x = 0; x < uxArraySize; x++) {
      printf("%s %u \n", pxTaskStatusArray[x].pcTaskName,
             static_cast<int>(
                 uxTaskGetStackHighWaterMark(pxTaskStatusArray[x].xHandle)));
    }
  }
}

Most likely you will have to increase (at least) the following stack sizes

const int THREAD_POOL_WRITER_STACKSIZE; 
const int THREAD_POOL_READER_STACKSIZE;

[akampmann]

@ZhenshengLee the core dump is also fixed on master, I was not using FastDDS correctly.