Sporadic segfaults in a python module when using ApiClient to query a Tapo P110

[gsaviane]

Sporadic segfaults in a python module when using ApiClient to query a Tapo P110

[mihai-dinculescu]

Thank you for raising the issue. Debugging this is going to be fun :)
Are you able to isolate the Python code that's causing the issue and share it?

[gsaviane]

Thank you for raising the issue. Debugging this is going to be fun :) Are you able to isolate the Python code that's causing the issue and share it?

Unfortunately stderr is only receiving this from the dying process

Thread 0x0000007fa31164c0 (most recent call first):
(no Python frame)
Fatal Python error: Segmentation fault

The code causing the segfault is attached here [removed]

[gsaviane]

OK, I might have found the cause. This little piece of python code is executed as a telegraf input plugin to collect data from the Tapos and forward it to an MQTT topic. The plugin was set to run every 5 secs with a 4 sec timeout if it does not complete in that time window. Normally the plugin takes less than 1 sec to complete, but in some occasions (network lags, device not ready) it goes past the timeout, and Telegraf preempts it with a SIGTERM. Just by increasing the timeout it's not happening again, and the problem is reproducible. If you have a Tapo P1XX device, just execute it as a normal python program and send it a SIGTERM before it exits (probably you would need a sleep()). It may lack some graceful disposal of the threads created in by tokio upon a SIGTERM

[mihai-dinculescu]

Thank you for the update.

I was not able to replicate the issue without TaskGroups. Did you?
Have you tried handling the SIGTERM on the Python side and cancelling the TaskGroups?

PS: I removed the zip file you've attached because it looked like it contained your Tapo password. You might want to change it.

[gsaviane]

Good catch! What an airhead I am, I changed it right away. Tomorrow I will try what you suggested Thanks! Il sab 15 giu 2024, 21:43 Mihai Dinculescu ***@***.***> ha scritto:

…

Thank you for the update. I was not able to replicate the issue without TaskGroups. Did you? Have you tried handling the SIGTERM on the Python side and cancelling the TaskGroups? PS: I removed the zip file you've attached because it looked like it contained your Tapo password. You might want to change it. — Reply to this email directly, view it on GitHub <#228 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC5KCCWC6PRMGXVXGCKCZ63ZHSKMLAVCNFSM6AAAAABJHGMLTKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNZQGU4TKMRVGU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[gsaviane]

Hi, I tried with a SIGTERM handler to cancel the tasks but it fails in the same way. However, now I can get logged more details about the error

Thread 0x0000007f9df514c0 (most recent call first):

Signal 15 received. Cancelling tasks ...
Error unhandled errors in a TaskGroup (1 sub-exception)
Traceback (most recent call last):
File "/usr/local/bin/tapo-telegraf-multi-async.py", line 149, in
asyncio.run(main())
File "/usr/lib/python3.11/asyncio/runners.py", line 190, in run
return runner.run(main)
^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/asyncio/base_events.py", line 653, in run_until_complete
return future.result()
^^^^^^^^^^^^^^^
File "/usr/local/bin/tapo-telegraf-multi-async.py", line 143, in main
if task.result() == "":
^^^^^^^^^^^^^
File "/usr/local/bin/tapo-telegraf-multi-async.py", line 109, in gen_influx_str_from_ip
p110 = await init_p110(ip)
^^^^^^^^^^^^^^^^^^^
File "/usr/local/bin/tapo-telegraf-multi-async.py", line 73, in init_p110
p110 = await client.p110(ip)
^^^^^^^^^^^^^^^^^^^^^
Exception: Tapo(Unknown(1003))
Signal 15 received. Cancelling tasks ...
thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.20.3/src/gil.rs:199:21:
assertion left != right failed: The Python interpreter is not initialized and the auto-initialize feature is not enabled.

Consider calling pyo3::prepare_freethreaded_python() before attempting to use Python APIs.
left: 0
right: 0
stack backtrace:
0: rust_begin_unwind
at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/std/src/panicking.rs:645:5
1: core::panicking::panic_fmt
at ./rustc/9b00956e56009bab2aa15d7bff10916599e3d6d6/library/core/src/panicking.rs:72:14
2: core::panicking::assert_failed_inner
3: core::panicking::assert_failed
4: parking_lot::once::Once::call_once_force::{{closure}}
5: parking_lot::once::Once::call_once_slow
6: pyo3::gil::GILGuard::acquire
7: <pyo3_asyncio::tokio::TokioRuntime as pyo3_asyncio::generic::Runtime>::spawn::{{closure}}
8: tokio::runtime::task::core::Core<T,S>::poll
9: tokio::runtime::task::harness::Harness<T,S>::poll
10: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
11: tokio::runtime::scheduler::multi_thread::worker::Context::run
12: tokio::runtime::context::set_scheduler
13: tokio::runtime::context::runtime::enter_runtime
14: tokio::runtime::scheduler::multi_thread::worker::run
15: <tokio::runtime::blocking::task::BlockingTask as core::future::future::Future>::poll
16: tokio::runtime::task::core::Core<T,S>::poll
Signal 15 received. Cancelling tasks ...
Signal 15 received. Cancelling tasks ...
Fatal Python error: Segmentation fault

[mihai-dinculescu]

Are you able to try out the suggested solution to use the pyo3 auto-initiatize feature?