Total crash after remaining in the settings/radio menu

[rgr101]

Total crash after remaining in the settings/radio menu

[rgr101]

Just installed it -
After I had disabled the router's 2.4G channel autochange some weeks ago I didn't encounter any more outages.
And together with your latest optimizations I'm very optimistic that this issue will be finally resolved 😇.

(Anyway - since I'm running several other ESPs, I'll keep in mind that routers' channel autochanges in a 'wifi mesh' environment is nothing an ESP really seems to like...)

[rstrouse]

Interesting. Do you think you managed to get it to remain with a half-open socket? In this condition only one side of the bi-directional socket remains open. Do you remember whether the interface appeared to be still connected or was there an error on the screen?

[rgr101]

Hm - I'm afraid this is beyond my knowledge of networks and sockets...
I'm not sure anymore, but I think I left the WebUI open on the iPhone and came back to it after some time (probably at least half an hour, more likely) and all I saw was the faded, still open application and the never ending spinning wheel. But as said, I can't be 100% sure. After that, I tried several things; reloading the URL in Safari, WiFi reset/reconnect of the iPhone, pinging the ESP, accessing it via HA, ... - The only thing that finally helped was a hard reset of the ESP.

[rstrouse]

Thanks that looks like the iphone disconnected the socket but ESPSomfy RTS still thought it was alive. It should have dropped the iphone connection but it looks like it never received a disconnect notice. It should have checked that it still existed every 20 seconds. I will have to do some research to see why it is not receiving that.

[rgr101]

Hi again. I'm totally frustrated:

The breakdown occured again and meanwhile it repeats more or less daily!

So I suspect that it cannot be due to the earlier suspicion of the sockets still being "half-open"; I am currently using the application exclusively via the native HA API.
I almost think it could be due to the wifi connection.
I have a wifi mesh with automatic channel switching (AVM Fritz! Box router), roaming is enabled in ESP.

I have checked the event logs of the Fritz! Boxes and have not detected any channel changes. But roaming switches could still have taken place; unfortunately I cannot trace them in the Mesh logs...

Unfortunately, I can't yet find out whether the whole ESP has crashed, or is it "just" a Wifi reconnect fail?

=> This leads me back to the topic of whether the Serial Monitor outputs could be redirected to a network monitor as an alternative? (Of course, this wouldn't help 100% either, especially if you want to track the network connection; but at least you could see up to the point of termination). Or is there another logging option that I don't know about yet?

I really don't know what else the problem could be. Maybe I need to check the (already optimized) power supply again?

[rgr101]

Or, maybe it's a problem with the Lolin/Wemos platform? Can you suggest me another, "rock solid" board?

[rstrouse]

I am investigating some potential issues with the ArduinoJson module and the socket serialization. Using some clues from your previous posts I have set up a device to see if I can get it to crash. Interestingly I managed to get it to crash once and this crash looked like the json serialization chews up a lot of memory so in v2.4.2 I have removed the json serialization from the equation.

My generic ESP32 devkit board for my production has been rock solid. It rebooted 3 weeks ago when I installed v2.4.1 on it.
However, I just built this https://github.com/rstrouse/ESPSomfy-RTS/wiki/Mini-ESP32s3-Device-Build and it will become my new production unit.

[rgr101]

thank you so much for your efforts - ah, you mean that it might not be due to an wifi issue at all, but a memory overflow - or possibly even a memory leak within the JSON serialization library? - could be the reason.
Perhaps in my case, the situation is aggravated by the fact that I make 2x 20 individual shade movements via HA every day and the ESP also has to process several remote control receipts, which then sporadically lead to a memory overflow...
(I know you recommend grouping the roller shutters to reduce the commands, but I haven't got around to it yet, not least because the grouping options are also a bit more complicated for me due to several exceptions...) ?!?

[rstrouse]

That is correct. Install the v2.4.2 pre-release and give it a shot.

EDIT: It is not a leak in the Json module which makes it insidious it simply uses a large chunk of memory that is needed by the socket processor to send the state data.

[rgr101]

Installed. Still works after 2 days... keep you posted

Ah I see. So, are there ESPs with more memory than the initial WROOM? Does the PSRAM of some variants help?

[rstrouse]

Unfortunately, PSRAM is one of those things that you must specifically compile the code for. Since I still need to support the 4mb boards that is my target.

My only option is to continue to reduce the code size and point in time memory usage. However, the S3 chips compile to a smaller code base and leave some breathing room.

[rgr101]

S3: sounds promising 🤞

[rstrouse]

Yeah and the one I am using for testing (soon to be switched over to my production) is so tiny. Did you see my S3 build?

[rgr101]

yes - amazing!

[rgr101]

Unfortunately, this morning, it died again!

But I have made some interesting observations this time:

• The HA automation obviously still managed to to raise ONE roller shutter before it got out. I had noticed this several times before, it's a different one each time (the sequence triggered by HA seems to be relatively random?!?) But I don't think that's the key point, because:

• while the ESP was unresponsive, I had manually raised two shutters with the remote control. However, the ESP must still have been "alive" and received and evaluated the radio signal from the remote control, because after the reboot the HA status of both roller shutters was "up".

This leads to the conclusion that it was not the ESP itself having crashed, but simply that the Wifi connection was lost and could not be re-established.

I actually observe a similar effect sporadically with another self-made ESP8266 LED Driver.
I suspect the automatic Wifi channel change of the router. However, I don't want to switch this off, as it does have a purpose - I have already checked some web sources, there does seem to be a problem in the Wifi library of the 8266, which should be fixed in the ESP32, but nobody seems to know exactly.
I still need to check the router's event log to see when the last channel change took place...

[rstrouse]

I believe you are correct. The ESP32 did not crash but the wifi connection was toast. I too have suspected the channel change being an issue and with 2.4.2 there is code that checks periodically to see if there is a stronger channel to be had and switch to the said channel. However, if the driver is crashing then it probably cannot scan for a better channel anyway.

Please do check to see if a channel changed in the router event log as ESP32 does not have a dual WiFi radio like many devices do. It must disconnect before connecting to another channel.

[rstrouse]

Please install the changes I assed to the v2.4.2 pre-release. I have found a couple of places to make some improvements.

[rgr101]

I already had installed v2.4.2 pre-release...

I believe you are correct...

The event log shows some channel changes, but somehow they are not related to the time of the outage - but I can't reliably track this because I don't know exactly when the outage occurred.
I'm not a radio or wifi expert, and I don't understand the mechanism in detail... as far as I can tell, the AP initiates the channel change when it detects that there is a better/less busy channel.
My environment consists of 2x AVM Fritz!Box as AP and 1x Fritz! Repeater, which form a so called "Mesh Wifi". According to AVM this is IEEE 802.11k and v (but not r). All APs/repeaters run with the same SSID for 2.4 and 5 GHz and determine their active channels themselves. So they can run on the same or different channels at the same time, and they can also change channels independently.
If I understand this correctly, the channel switching function of the ESP is equivalent to searching for an alternative AP with better signal strength, possibly on a different channel -> i.e. roaming initiated by the ESP?
Anyway - could it be that something is counterproductive here? The AP actively logs off the clients just before a channel change (this can be seen in the event log), i.e. at that moment the ESP loses its connection and tries to establish a new one. At the same time, the AP performs its channel change and is ready to accept new connections. However, it could be that the ESP has found another AP in the "mesh" and is connecting to it even though it is de facto weaker, but the stronger one that was busy changing channels was not available. However, this does not explain why the ESP then loses the connection completely or is unable to re-establish it...?
In an ideal world, you would trust that all devices comply with the standards and you wouldn't have to worry about such low-level problems... But as far as I can tell, ESP doesn't support 802.11k and v (yet)...
A possible workaround could be to disable the automatic channel switching of the APs for 2.4GHz and not to optimize the channel assignment (Automatic channel switching can remain active for 5GHz because ESP doesn't care...)
I'll do this now and see if the system becomes more stable overall, or if it runs into new problems due to channel congestion...

[rstrouse]

Actually, the pre-releases are updated without a change to the version number. Update it again.

The ESP32 wifi chip does not support the elegant network assisted roaming that you find on 11v, 11k and 11r. It does support some aspects of 11r on the recent v5.1 release. However, this has not bubbled into the Arduino core yet. Even then it is simply a notification that the channel is changing. The ESP32 will still need to be the initiator in the end.

Since there is only one radio on an ESP32 it cannot channel scan without disconnect. Dual radio designs keep an ear open for other signals and are constantly searching the real estate listings for a better place to live. Honestly, I think the assertion that ESP32 goes seeking for a new home and never really finds it is not likely.

When the AP logs off the ESP32 it does not immediately attempt to reconnect to the old BSID. That would be counterproductive. It must perform a network scan for the SSID network. If there is more than one BSID (AP) on the identified network these are sorted in signal strength order where the best RSSI is at the top of the list and it picks that one.

[rgr101]

OK, updated to the latest 2.4.2 (after some hurdles, see backup file format issue) and disabled the 2.4G channel auto switching. Let's wait and see...

Honestly, I'd rather switch to a wired ESP (as all of my chained 4 unmanaged budget LAN switches just run 365x24 trouble-free...), but I read in another issue that even these boards aren't always 100% rock solid?

[rstrouse]

Hate to have you do this again but please update to the official v2.4.2 release. There have been some updates that improve the long term stability of the firmware.

[rgr101]

No problem - I‘ll do so asap.
I would do it anyway, as now it is an official- and no longer pre-release 😎

[rgr101]

update done, went well

[rstrouse]

v2.4.3 has been released and I believe we have solved this issue. I am going to close this for now.

[rgr101]

👍🙏🙋‍♂️