Is it possible that callbacks can run even after resetCallbacks() has been called?

[themightyoarfish]

Is it possible that callbacks can run even after resetCallbacks() has been called?

[themightyoarfish]

Thank you for this explanation, i think in my case a lot of the problems where caused by my callbacks getting deleted before the peerconnection was destroyed. My callbacks were members of the object holding the connection, and the destruction order lead to them being deleted before the connection, which would try to invoke them while closing.

[paullouisageneau]

I am debugging a weird shutdown problem in which PeerConnections are outfitted with callbacks that will cause deletion of self from the server which bookkeeps them. So for instance, when the track closes, this should trigger an erase() call on the collection holding the connection.

I have mutexes where appropriate, but address and thread sanitizers on my program give me grief. Before destroying peer connection and track, i call resetCallbacks() on both, and only then the connection gets destroyed.

You should not use resetCallbacks() in this scenario as it will cause deadlocks if multiple threads call it on the same object at the same time: since the method must wait for all other callbacks to return before proceeding, if two threads call resetCallbacks() at the same time they will end up waiting for each other. You simply need to call close(), which will take care of race conditions and reset callbacks for you. After close() has returned, no callbacks will be called (callbacks can still be called during the execution of close()).

Now often I get this problem that (seemingly) the server object has already run its destructor and gets deleted, but then some callback of the track in my connection tries to access it.

As this is a relatively complex program, I cannot easily reproduce it, but I would like to know how the event processing system works. Is it possible that some events are queued, then the callbacks get reset, and the object destroyed, but still the events propagate through the system and the previously non-null callbacks get invoked? It looks like tasks are enqueued for the thread pool, which means that they may be pending while I reset the callbacks in another thread. Is there a way to deal with this? Is calling rtc::Cleanup().wait() the intended remedy?

It's not possible that previous callbacks are called after a change: the operation is synchronized and waits for any other thread running the callback to finish first, so once the method has returned the previous callback function can't be called anymore. This is also the case when setting null callbacks, once resetCallbacks() has returned, even if tasks are enqueued they won't do anything as callbacks are null.

rtc::Cleanup() is only to ensure everything is cleaned up at the end of your program. Waiting on the returned promise for an undefined time can be dangerous as all rtc objects must be destroyed before it can return, so a leak will make it hang.

Probably an unrelated issue is that I experience infrequent shutdown deadlocks, where my connection destructor hangs in resetCallbacks()

What could cause this, since nothing should compete for the mutex of the just default-constructed function object from nullptr?

This probably means that it has to wait to reset a callback because the callback is being executed by another thread blocked in user code. It could be the issue mentioned earlier with resetting callbacks from two callbacks at the same time, or it could a deadlock with a user mutex in another callback.