Adding and Removing Tracks Without Creating New Connections

[Abeja27]

Adding and Removing Tracks Without Creating New Connections

[paullouisageneau]

After some usage, I close the tracks and call setLocalDescription(). However, a new SDP is not sent to the browser. I understand that this may be the expected behavior, but is there a way to force sending a new SDP?

Tracks closing is not signaled as there is nothing new to negotiate media-wise. Closing a track will just mark it inactive next time something new is negotiated. Of course you can still add your own business logic to your signaling protocol to inform the other side about a change.

I create a new audio and video track and call setLocalDescription(). A new SDP is sent to the browser, but it doesn't contain all the necessary data to create the new tracks. It seems to notify that the tracks were closed but doesn't include the new ones.

I suspect the issue may be due to reusing the trackids for the newly created tracks. Is it necessary for them to be different? I've tried using different trackIds, which results in successful creation, but the problem is that the old tracks are never removed from the SDP. As a result, the SDP becomes longer with each new track creation and old track closure. Is there a way to remove the previous tracks from the SDP?

Tracks are identified by their mid, so when you create a track with the same mid, it does not create a new track in the SDP, it updates the existing one. The offer sent to the browser contains updated media information. However, expect bugs with this as support is shaky.

There is currently no way to automatically remove tracks from SDP. Browsers have the same issue as media entries are not supposed to be removed from SDP.

I'm confident that I'm doing something wrong, but I can't figure out what. If you need more specific details, please let me know, and I'll provide them. Thanks in advance.

I think you are not using tracks as they are designed to be used here. Media signaling is not made for sending an end-of-file signal on tracks. The idea of tracks is to abstract sharing different kind of media during a session, for instance screen sharing alongside of videoconferencing, but you are not supposed to create tracks indefinitely. This can however be done with Data Channels, for instance to transfer an unlimited number of files during the session.

Note that it is fine to stop sending in a track for a while, then resume sending.

[Abeja27]

Big thanks for the quick response- your input has been a huge help. I've been testing, but unfortunately, I haven't been able to find a 100% functional solution yet.

I think you are not using tracks as they are designed to be used here.

My project is a streaming application similar to Twitch, where the server side sends audio and video to multiple clients. The server (streamer) can start and stop the stream at any time, with a different number of video and audio tracks each time, each in various qualities. Since there can be many clients, I didn't want to create a new connection from scratch. Also, since I don't know how many tracks there will be, I cannot pre-create them. Is this use of tracks correct?

Note that it is fine to stop sending in a track for a while, then resume sending.

Currently, I use this concept to switch between different qualities. All tracks are pre-created, and only the ones that correspond to the selected quality are used.

Now that you have more context - I have been experimenting with the two possible options that come to mind, and in both cases, I am encountering issues:

1. Not reusing the tracks and assuming that the SDP gets longer each time. This alternative generally works, but sometimes (I haven't been able to determine the cause), when sending the new SDP to the browser, it starts sending ICE candidates again. When this happens, libdatachannel throws an error when calling addRemoteCandidate() with the message: "Got a remote candidate without ICE transport." Do you know what could be causing this error?

2. Attempting to reuse the tracks by creating them with the same MID as their already closed counterpart.
   In this case, I haven't been able to make it work. The behavior varies depending on the browser. Chromium browsers manage to create new tracks, but the RTCRtpTransceiver remains "stopped". In Firefox, no new tracks are created directly. I am attaching the two SDPs I send: the first when the tracks are created and the second, when the tracks are updated. I think that the second SDP lacks information about the updated tracks but that's as far as I can figure out. What are your thoughts about this?

Thank you once again for your help.

SDP1.txt
SDP2.txt

[paullouisageneau]

My project is a streaming application similar to Twitch, where the server side sends audio and video to multiple clients. The server (streamer) can start and stop the stream at any time, with a different number of video and audio tracks each time, each in various qualities. Since there can be many clients, I didn't want to create a new connection from scratch. Also, since I don't know how many tracks there will be, I cannot pre-create them. Is this use of tracks correct?

For different qualities of the same content, you should not use different tracks. This is called simulcast, and is typically achieved with multiple restriction identifiers (rid) inside a single track. The only case where you must negotiate another track is when a new stream appears, like a screen share in addition to the camera.

Pausing a stream should not be signaled by removing and re-adding the track later with libdatachannel, you should just stop sending. What you could do is signal the direction is inactive. The library doesn't have a mechanism to do it automatically, you have to the direction manually to inactive and re-add the same track to trigger renegotiation.

Currently, I use this concept to switch between different qualities. All tracks are pre-created, and only the ones that correspond to the selected quality are used.

The concept is good, but the dedicated mechanism is simulcast with different rid in a single track.

Not reusing the tracks and assuming that the SDP gets longer each time. This alternative generally works, but sometimes (I haven't been able to determine the cause), when sending the new SDP to the browser, it starts sending ICE candidates again. When this happens, libdatachannel throws an error when calling addRemoteCandidate() with the message: "Got a remote candidate without ICE transport." Do you know what could be causing this error?

I think the only case where you can get such an error is a race condition on a new peer connection where you set a remote offer, then set a remote candidate, then set the local description. You should set the local description immediately after setting the remote one, then set remote candidates.

Attempting to reuse the tracks by creating them with the same MID as their already closed counterpart.
In this case, I haven't been able to make it work. The behavior varies depending on the browser. Chromium browsers manage to create new tracks, but the RTCRtpTransceiver remains "stopped". In Firefox, no new tracks are created directly. I am attaching the two SDPs I send: the first when the tracks are created and the second, when the tracks are updated. I think that the second SDP lacks information about the updated tracks but that's as far as I can figure out. What are your thoughts about this?

Sorry, I wasn't clear, I meant that if you add a track with the same mid, it's basically the same track. You should not close the tracks on libdatachannel side in that case: libdatachannel tracks are actually transceivers, so it will mark them as removed in the SDP, the browser will stop the transceiver on its side too, and you are not supposed to re-use stopped transceivers.

[Abeja27]

Thank you for the help; everything is much clearer now. The only problem I have left to solve is the reconnection error ("Got a remote candidate without ICE transport"). Although, in the future, I will change the behavior to reuse the same tracks without closing them. For now, I would like to make it work by closing the tracks and creating new ones each time the livestream restarts.

I think the only case where you can get such an error is a race condition on a new peer connection where you set a remote offer, then set a remote candidate, then set the local description. You should set the local description immediately after setting the remote one, then set remote candidates.

I have been reviewing the code, and there is no race condition. But the flow I follow is different from what you suggest:

[libdatachannel server]

1. I close the tracks that are being used.
2. I create new tracks, and these are opened. (Does it make sense for them to be opened before negotiation?)
3. I call setLocalDescription()

[browser]

4. I call setRemoteDescription()
5. I call setLocalDescription()

[libdatachannel server]

6. I call setRemoteDescription()
7. I call addRemoteCandidate(), causing the error.

I have been reviewing the logs, and the problem is between steps 3 and 4. Right after calling setRemoteDescription() in the browser, the connection is completely closed, preventing the iceTransport from being created and leading to the error I described. Certainly, I shouldn't call addRemoteCandidate after the connection has been closed, but why does it get close in the first place? Here I attach the logs where you can see how it disconnects completely.

2024-03-02 16:41:32.266 DEBUG [17744] [rtc::impl::DtlsTransport::doRecv@975] TLS connection cleanly closed
2024-03-02 16:41:32.266 INFO  [17744] [rtc::impl::DtlsTransport::doRecv@992] DTLS closed
2024-03-02 16:41:32.267 INFO  [17744] [rtc::impl::PeerConnection::changeState@1179] Changed state to disconnected
2024-03-02 16:41:32.267 INFO  [17744] [rtc::impl::SctpTransport::incoming@458] SCTP disconnected
2024-03-02 16:41:32.267 INFO  [17744] [rtc::impl::PeerConnection::changeIceState@1197] Changed ICE state to closed
2024-03-02 16:41:32.267 INFO  [17744] [rtc::impl::PeerConnection::changeState@1179] Changed state to closed
2024-03-02 16:41:32.267 DEBUG [17744] [rtc::impl::SctpTransport::~SctpTransport@316] Destroying SCTP transport
2024-03-02 16:41:32.268 DEBUG [17744] [rtc::impl::DtlsTransport::stop@860] Stopping DTLS transport
2024-03-02 16:41:32.268 DEBUG [17744] [rtc::impl::DtlsTransport::stop@860] Stopping DTLS transport
2024-03-02 16:41:32.268 DEBUG [17744] [rtc::impl::DtlsTransport::stop@860] Stopping DTLS transport
2024-03-02 16:41:32.268 DEBUG [17744] [rtc::impl::DtlsTransport::~DtlsTransport@831] Destroying DTLS transport
2024-03-02 16:41:32.268 DEBUG [17744] [rtc::impl::IceTransport::~IceTransport@160] Destroying ICE transport

It seems like the browser is ending the connection; however, on the browser side, the connection does not end but rather starts looking for iceCandidates.

• The signalingState changes to have-remote-offer.
• The iceConnectionState changes to checking.
• The iceGatheringState changes to gathering.

Is it possible that libdatachannel does not support re-exchanging ice candidates, and a new connection has to be created every time this happens?

[paullouisageneau]

Is it possible that libdatachannel does not support re-exchanging ice candidates, and a new connection has to be created every time this happens?

libdatachannel does not support ICE or DTLS restart (the corresponding issue is #545). It appears here the browser closes the DTLS connection, so libdatachannel closes everything as a result. This should not happen when renegotiating new tracks, maybe all existing tracks have been removed so it decides to reset everything (possibly changing DTLS roles).

[Abeja27]

Okey, thank you for the support!