Providing multiple RTSP streams under single output from RtspServer

[juliusfriedman]

Providing multiple RTSP streams under single output from RtspServer

[juliusfriedman]

I can confirm our recent changes in RtspClient / RtpClient result in a StackOverflow exception which is exceedingly hard to track down without a stack trace. (It seems like Rtp over Udp is timing out and causing it somehow combined with MonitorProtocol)

I will continue to look into this when I have time but it probable will not be until the weekend.

Please let me know if you find it before me.

P.s. Tcp still works well (the problem only happens when using Udp)

[juliusfriedman]

I pushed up some code which swtiches between the 2 streams, you can setup the stream by pressing B after the server is running.

It seems to work okay for me, I need to look into why your cameras have so many artifacts in the stream, it seems like a bug on those cameras firmware or sometime of bandwidth exhaustion scenario...

I will let you know what I find ASAP.

[juliusfriedman]

It seems something broke in the RtspClient, its missing the Content of the Describe request sometimes...

If you add this logic (hack) it seems to work..

Describe:
                    response = SendRtspMessage(describe, out error, true, true, m_MaximumTransactionAttempts) ?? m_LastTransmitted;

                    if (!response.ContainsHeader(RtspHeaders.ContentLength)) goto Describe;
                    

It's too late for me to check into this further but at least you know works over TCP.

Please try and review your changes in the RtspClient around SendDescribe to see why the RtspClient is failing.

I believe it might have something to do with the modified return but I cannot spot it easily.

Talk to you soon!

[juliusfriedman]

It's just a bit more complex than it needs to be in order to work.

If you already have your output stream then its just a matter of calling packetize from the appropriate source..

You don't need the OutOfBandData and I personally would work with the RtspStream from the RtspServer rather than at the RtpClient level as it will decrease the complexity greatly.

All you need is an output stream:

Media.Rtsp.Server.MediaTypes.RtpAudioSink outputStream = new Rtsp.Server.MediaTypes.RtpAudioSink("Output", null, 0, 1, 8000);

Then from there you will attach to the output stream from the desired input stream like so:

someStream.RtpClient.RtpPacketReceieved += (s, p, tc) => outputStream.RtpClient.OnRtpPacketReceieved(p, tc);

This will cause the packets to begin to flow from someStream to outputStream.

Then upon a timer elapsing or otherwise you can easily stitch together different streams to create a tour provided your coded is the same for each stream (and the settings for the codec are relatively similar)

Remember to disconnect the events from the someStream when you are done with that particular instance so this way both streams are not flowing to the output at the same time.

If you need more help please let me know explicitly what.

Thank you again for your other contributions.

[juliusfriedman]

I made a small push which now includes your streams, I will get to the composite stuff tonight or this weekend and if not sometimes next week.

[juliusfriedman]

I pushed up some changes, it seems your camera has a lot of extra data on the TCP side, If you run the RtspClient test and attach to the Interleave events with the I key you will see there is always extra data.

I have verified this with Wireshark and although I am not sure where this is coming from yet or how VLC seems to deal with it so gracefully (without any debug messages in the log etc) it's something to be aware of for sure...

I will continue to look into this when I have time.

[juliusfriedman]

I made a few hacks to the code to get your cameras to work under udp mode and switch gracefully.

It was a combination of a bug (using the Connection: close header) and handling when the 2nd describe would not work.

Vlc doens't seems to have that issue and I will continue to look into this to make it better where possible.

[juliusfriedman]

Found that you can disable the keep alive requests with your cameras and it seems to reduce artifacts in interleaved mode especially considering all the malformed packets.

DisableKeepAliveRequests = true.

I will continue to look into this to see if there is anything else I am missing as well as investigate why VLC is providing so much better quality.

FWIW, When using 0 as the bufferSize I get much less artifacts than I do even with a larger buffer (65540 -> 10*1024)

[juliusfriedman]

I made some log statements only appear in debug and I still get a lot of artifacts from your stream, I am pretty sure its not performance related (or correctness related) as Wireshark reveals similar errors however both popular unmanaged libraries seems to be able to play from the stream without any artifacts...

This leaves a bug in VLC/Live555 / ffmpeg of which I have not had the time to review their latest code in detail but its possible especially for Live555 to have some type of protocol bugs because of the way it implements RTSP... (http://www.live555.com/liveMedia/doxygen/html/RTSPClient_8hh_source.html)

I also tried ffplay just to be sure and it seems that the streams play without artifacts using it so it just makes me very curious...

ffplay.exe -i "rtsp://8.15.251.101:1935/rtplive/R2_059" -rtsp_transport tcp -loglevel debug

You can see that there are some long pauses but there aren't really any major artifacts like there are with the server, its a result maybe of how they are holding onto packets for a longer time and re-ordering them before sending to the decoder BUT that shouldn't matter when going through the RtspServer or using the RtspClient.

I will continue to look into this and let you know what I find but in the meantime you might have better quality pulling down the stream with ffmpeg / ffplay and making something like a FFMpegSource which can run from the command line and output over rtsp locally so it can be consumed from the server again with less artifacts.

Thanks for your continued contributions and testing resources!

[juliusfriedman]

I pushed up a quick few other fixes from testing, it seems that the camera sometimes would send a binary packet which was getting my logic tripped up, the packet was never parsed but it wasted some cpu...

I adjusted for this logic (short packets) by ensuring the frame header was completely read before returning any frameLength, I also changed a few checks to look for frameChannel.HasValue

It didn't happen a lot but when it did it seems to cause some of the pauses and artifacts.

I am going to sleep now, will check back again in the morning.

[dimhotepus]

Thank you!

I'll definitely check you suggestions in a few days!

[juliusfriedman]

So I start to see some artifacts on ffplay as well... they eventually resolve but they are there especially more times than others.

I will continue to look into what can be done to reduce the artifacts, it seems your cameras have a VBR and are sensitive to sequence number and timestamp changes as a result.

When I play your streams individually through the server all the artifacts eventually clear up (even with VLC). and the Framerate is good.

It's when we starting to use the Composite stream I notice most of the artifacts and lack of framerate and TBH its somewhat expected because of the way we are mixing streams from multiple different sources into a single stream and hoping the decoder does the right thing.

Its more likely to work with less artifacts if all cameras have the same settings and (VBR) doesn't seems to lend itself to the best I Frame interval for doing this type of switching.

I will continue to look into this but I will be away the next few days, let me know what you find as well!

[juliusfriedman]

I found that if we lower the clock rate slightly and force a timestamp jump right at the point of switching it reduces the artifacts greatly and VLC plays the stream much better. I pushed up those changes for you to take a look at. FF Play seems to be about the same but I will continue to test and look into this.

Overall since the stream is stable minus these few artifacts which resolve themselves I think we can consider this issue closed after you verify the recent changes with the smaller ClockRate and TimeStamp changes right at the point of switching.

Let me know what you think!

[juliusfriedman]

You want to do something like a camera tour where you tour between each stream?

It should be very easy to achieve provided the underlying streams are using the same codec, you can essentially just create a RtspStream and a timer and switch which underlying packets are going to the output RtspStream based on your desired logic, e.g. a counter to select from the proper underlying streams packets.

The logic would essentially on elapse of the timer attach to the events from the desired stream and enqueue them into the output stream for playback.

Give me a try and show me what you have tried and I can better assist you from there!

The code would be something like this:

someStream.RtpClient.RtpPacketReceieved += (s, p, tc) => outputStream.RtpClient.OnRtpPacketReceieved(p, tc);

Where someStream is the underlying stream you want to temporarily tour to and outputStream is the stream you have setup for playback.

Take a look @ https://github.com/juliusfriedman/net7mma_core/blob/master/UnitTests/Program.cs#L2125
(TestServer) for inspiration!

[dimhotepus]

Well, may be it is dumb question.

I've tried to use TestServer method as a source.

int serverPort = 8000 + Media.Rtsp.RtspMessage.ReliableTransportDefaultPort;
System.Net.IPAddress serverIp =
    Media.Common.Extensions.Socket.SocketExtensions.GetFirstUnicastIPAddress(
        System.Net.Sockets.AddressFamily.InterNetwork);

Console.WriteLine("Server Starting on: {0}:{1}", serverIp, serverPort);

using (Media.Rtsp.RtspServer server = new Media.Rtsp.RtspServer(serverIp, serverPort)
{
    Logger = new Media.Rtsp.Server.Loggers.RtspServerConsoleLogger(),
    ClientSessionLogger = new Media.Rtsp.Server.Loggers.RtspServerConsoleLogger()
})
{
    server.TryAddMedia(
        new Media.Rtsp.Server.MediaTypes.RtspSource(
            "R2_059",
            "rtsp://8.15.251.101:1935/rtplive/R2_059"
        )
    );

    server.Start();

    // Wait for the server to start.
    while (!server.IsRunning) System.Threading.Thread.Sleep(0);

    Console.WriteLine("Listening on: " + server.LocalEndPoint);

    while (true)
    {
        ConsoleKeyInfo keyInfo = Console.ReadKey(true);
        if (keyInfo.Key == ConsoleKey.Q) break;
    }

    Console.WriteLine("Server Streamed : " + server.TotalStreamedBytes);
    Console.WriteLine("Rtsp Sent : " + server.TotalRtspBytesSent);
    Console.WriteLine("Rtsp Recieved : " + server.TotalRtspBytesRecieved);
    Console.WriteLine("Stopping Server");

    server.Stop();

    Console.WriteLine("Server Stopped");
}

I can play original stream in VLC, but stream from server doesn't work in VLC.

When I try to connect to rtsp://192.168.0.85:8554/live/R2_059 via VLC I see this:

Received Invalid Message: * RTSP/0.0

Full log:

Server Starting on: 192.168.0.85:8554
Server Started @ 10/25/2023 3:08:31 AM
Listening on: 192.168.0.85:8554
Starting Stream: R2_059 Id=d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2
RestartFaultedStreams
DisconnectAndRemoveInactiveSessions
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve: LengthInWordsMinusOne overflowed. Cannot store a number lower than 2 or higher than 0 in a 65535 structure.
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve - Begin
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve: LengthInWordsMinusOne overflowed. Cannot store a number lower than 2 or higher than 0 in a 65535 structure.
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve - Begin
RestartFaultedStreams
DisconnectAndRemoveInactiveSessions
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve: LengthInWordsMinusOne overflowed. Cannot store a number lower than 2 or higher than 0 in a 65535 structure.
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve - Begin
Adding Client: eb6753ba-bb47-4671-9ff0-6d7db93646e7
Accepted Client: eb6753ba-bb47-4671-9ff0-6d7db93646e7 @ 192.168.0.85:56194
10/25/2023 3:09:13 AM Added =True
Request=> OPTIONS rtsp://192.168.0.85:8554/live/R2_059 RTSP/1.0
CSeq: 2
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)

 Session=> eb6753ba-bb47-4671-9ff0-6d7db93646e7

RTSP/1.0 200
CSeq: 2
Public: OPTIONS, DESCRIBE, SETUP, PLAY, PAUSE, TEARDOWN, GET_PARAMETER
Allow: ANNOUNCE, RECORD, SET_PARAMETER
Cache-Control: no-cache
Supported: play.basic,con.persistent,setup.playing
Server: ASTI Media Server RTSP\1.0


Request=> DESCRIBE rtsp://192.168.0.85:8554/live/R2_059 RTSP/1.0
CSeq: 3
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Accept: application/sdp

 Session=> eb6753ba-bb47-4671-9ff0-6d7db93646e7

RTSP/1.0 200
CSeq: 3
Content-Type: application/sdp
Cache-Control: no-cache
Content-Base: rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/
Content-Length: 448
Content-Encoding: utf-8
Server: ASTI Media Server RTSP\1.0

v=0
o=ASTI-Media-Server 16781262518891554167 -1665481554806099096 IN IP4 192.168.0.85
s=ASTI-Streaming-Session R2_059
c=IN IP4 192.168.0.85
a=sdplang:en
a=range:npt=now-
a=control:*
t=0 0
m=video 0 RTP/AVP 97
a=rtpmap:97 H264/90000
a=fmtp:97 packetization-mode=1;profile-level-id=42C01E;sprop-parameter-sets=Z0LAHtoFB+wEQAAAAwBAAAAHo8WLqA==,aM48gA==
a=cliprect:0,0,240,320
a=framesize:97 320-240
a=framerate:15.0
a=control:trackID=1

Request=> SETUP rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/trackID=1 RTSP/1.0
CSeq: 4
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Transport: RTP/AVP;unicast;client_port=55634-55635

 Session=> eb6753ba-bb47-4671-9ff0-6d7db93646e7

RTSP/1.0 200
CSeq: 4
Transport: RTP/AVP;source=192.168.0.85;unicast;client_port=55634-55635;server_port=30000-30001;ssrc=1358227F
Server: ASTI Media Server RTSP\1.0
Session: 1634402757;timeout=30


Request=> PLAY rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/ RTSP/1.0
CSeq: 5
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: 1634402757
Range: npt=0.000-

 Session=> eb6753ba-bb47-4671-9ff0-6d7db93646e7

Media.Rtp.RtpClient-11202aa1-0e86-4303-9a04-47a977c10704@SendRecieve - Begin
Media.Rtp.RtpClient-11202aa1-0e86-4303-9a04-47a977c10704@ParseAndCompleteData - Remaining= 4
RTSP/1.0 200
Session: 1634402757;timeout=30
CSeq: 5
Range: npt=now-
RTP-Info: url=rtsp://192.168.0.85/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/video;ssrc=1358227F
Server: ASTI Media Server RTSP\1.0


Session:eb6753ba-bb47-4671-9ff0-6d7db93646e7 Attempting to complete previous mesage with buffer of 4 bytes.
Session:eb6753ba-bb47-4671-9ff0-6d7db93646e7 used 4 of buffer bytes
Request=> PLAY rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/ RTSP/1.0
CSeq: 5
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: 1634402757
Range: npt=0.000-

 Session=> eb6753ba-bb47-4671-9ff0-6d7db93646e7

RTSP/1.0 200
Session: 1634402757;timeout=30
CSeq: 5
Range: npt=now-
RTP-Info: url=rtsp://192.168.0.85/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/video;ssrc=1358227F
Server: ASTI Media Server RTSP\1.0


Received Invalid Message: * RTSP/0.0


For Session:eb6753ba-bb47-4671-9ff0-6d7db93646e7
Media.Rtp.RtpClient-11202aa1-0e86-4303-9a04-47a977c10704@ParseAndCompleteData - Remaining= 4
Session:eb6753ba-bb47-4671-9ff0-6d7db93646e7 Attempting to complete previous mesage with buffer of 4 bytes.
Session:eb6753ba-bb47-4671-9ff0-6d7db93646e7 used 4 of buffer bytes
Request=> PLAY rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/ RTSP/1.0
CSeq: 5
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: 1634402757
Range: npt=0.000-

 Session=> eb6753ba-bb47-4671-9ff0-6d7db93646e7

RTSP/1.0 200
Session: 1634402757;timeout=30
CSeq: 5
Range: npt=now-
RTP-Info: url=rtsp://192.168.0.85/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/video;ssrc=1358227F
Server: ASTI Media Server RTSP\1.0


Received Invalid Message: * RTSP/0.0


For Session:eb6753ba-bb47-4671-9ff0-6d7db93646e7
Media.Rtp.RtpClient-11202aa1-0e86-4303-9a04-47a977c10704@SendRecieve RtpSocket - SocketError = TimedOut lastOperation = 10/25/2023 3:09:13 AM taken = 00:00:00.0618179
RestartFaultedStreams
DisconnectAndRemoveInactiveSessions
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve: LengthInWordsMinusOne overflowed. Cannot store a number lower than 2 or higher than 0 in a 65535 structure.
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve - Begin
Media.Rtp.RtpClient-11202aa1-0e86-4303-9a04-47a977c10704@SendRecieve: LengthInWordsMinusOne overflowed. Cannot store a number lower than 2 or higher than 0 in a 65535 structure.
Media.Rtp.RtpClient-11202aa1-0e86-4303-9a04-47a977c10704@SendRecieve - Begin
Request=> TEARDOWN rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/ RTSP/1.0
CSeq: 6
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: 1634402757

 Session=> eb6753ba-bb47-4671-9ff0-6d7db93646e7

RTSP/1.0 200
Session: 1634402757;timeout=30
CSeq: 6
Server: ASTI Media Server RTSP\1.0


Session Inactive - 1634402757
Session Context Removed - 324543103@1634402757
Session Deactivated - 1634402757
Media.Rtp.RtpClient-11202aa1-0e86-4303-9a04-47a977c10704@SendRecieve - Exit
Adding Client: ce6e7802-b59d-4f0f-a506-668ea743d9ef
Request=> OPTIONS rtsp://192.168.0.85:8554/live/R2_059 RTSP/1.0
CSeq: 2
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)

 Session=> ce6e7802-b59d-4f0f-a506-668ea743d9ef

Accepted Client: ce6e7802-b59d-4f0f-a506-668ea743d9ef @ 192.168.0.85:56196
10/25/2023 3:09:24 AM Added =True
RTSP/1.0 200
CSeq: 2
Public: OPTIONS, DESCRIBE, SETUP, PLAY, PAUSE, TEARDOWN, GET_PARAMETER
Allow: ANNOUNCE, RECORD, SET_PARAMETER
Cache-Control: no-cache
Supported: play.basic,con.persistent,setup.playing
Server: ASTI Media Server RTSP\1.0


Request=> DESCRIBE rtsp://192.168.0.85:8554/live/R2_059 RTSP/1.0
CSeq: 3
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Accept: application/sdp

 Session=> ce6e7802-b59d-4f0f-a506-668ea743d9ef

RTSP/1.0 200
CSeq: 3
Content-Type: application/sdp
Cache-Control: no-cache
Content-Base: rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/
Content-Length: 448
Content-Encoding: utf-8
Server: ASTI Media Server RTSP\1.0

v=0
o=ASTI-Media-Server 16781262565443746052 -1665481508265769481 IN IP4 192.168.0.85
s=ASTI-Streaming-Session R2_059
c=IN IP4 192.168.0.85
a=sdplang:en
a=range:npt=now-
a=control:*
t=0 0
m=video 0 RTP/AVP 97
a=rtpmap:97 H264/90000
a=fmtp:97 packetization-mode=1;profile-level-id=42C01E;sprop-parameter-sets=Z0LAHtoFB+wEQAAAAwBAAAAHo8WLqA==,aM48gA==
a=cliprect:0,0,240,320
a=framesize:97 320-240
a=framerate:15.0
a=control:trackID=1

Request=> SETUP rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/trackID=1 RTSP/1.0
CSeq: 4
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Transport: RTP/AVP/TCP;unicast;interleaved=0-1

 Session=> ce6e7802-b59d-4f0f-a506-668ea743d9ef

RTSP/1.0 200
CSeq: 4
Transport: RTP/AVP/TCP;source=192.168.0.85;interleaved=0-1;ssrc=1358227F
Server: ASTI Media Server RTSP\1.0
Session: 151472816;timeout=30


Request=> PLAY rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/ RTSP/1.0
CSeq: 5
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: 151472816
Range: npt=0.000-

 Session=> ce6e7802-b59d-4f0f-a506-668ea743d9ef

Media.Rtp.RtpClient-042e3c94-3f8e-452a-9ab3-2e5f363fcf5a@SendRecieve - Begin
RTSP/1.0 200
Session: 151472816;timeout=30
CSeq: 5
Range: npt=now-
RTP-Info: url=rtsp://192.168.0.85/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/video;ssrc=1358227F
Server: ASTI Media Server RTSP\1.0


Media.Rtp.RtpClient-042e3c94-3f8e-452a-9ab3-2e5f363fcf5a@SendRecieve RtpSocket - SocketError = TimedOut lastOperation = 10/25/2023 3:09:24 AM taken = 00:00:00.0000014
042e3c94-3f8e-452a-9ab3-2e5f363fcf5aProcessFrameData - ParseAndHandleData
Media.Rtp.RtpClient-042e3c94-3f8e-452a-9ab3-2e5f363fcf5a@ProcessFrameData - raisedEvent for frameLength: 4 remainingInBuffer=0
Session:ce6e7802-b59d-4f0f-a506-668ea743d9ef Attempting to complete previous mesage with buffer of 4 bytes.
Session:ce6e7802-b59d-4f0f-a506-668ea743d9ef used 4 of buffer bytes
Request=> PLAY rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/ RTSP/1.0
CSeq: 5
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: 151472816
Range: npt=0.000-

 Session=> ce6e7802-b59d-4f0f-a506-668ea743d9ef

RTSP/1.0 200
Session: 151472816;timeout=30
CSeq: 5
Range: npt=now-
RTP-Info: url=rtsp://192.168.0.85/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/video;ssrc=1358227F
Server: ASTI Media Server RTSP\1.0


Received Invalid Message: * RTSP/0.0


For Session:ce6e7802-b59d-4f0f-a506-668ea743d9ef
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve: LengthInWordsMinusOne overflowed. Cannot store a number lower than 2 or higher than 0 in a 65535 structure.
Media.Rtp.RtpClient-8b095ebc-88d1-4d25-82cb-703ba2e41b02@SendRecieve - Begin
RestartFaultedStreams
DisconnectAndRemoveInactiveSessions
042e3c94-3f8e-452a-9ab3-2e5f363fcf5aProcessFrameData - ParseAndHandleData
Media.Rtp.RtpClient-042e3c94-3f8e-452a-9ab3-2e5f363fcf5a@ProcessFrameData - raisedEvent for frameLength: 4 remainingInBuffer=0
Session:ce6e7802-b59d-4f0f-a506-668ea743d9ef Attempting to complete previous mesage with buffer of 4 bytes.
Session:ce6e7802-b59d-4f0f-a506-668ea743d9ef used 4 of buffer bytes
Request=> PLAY rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/ RTSP/1.0
CSeq: 5
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: 151472816
Range: npt=0.000-

 Session=> ce6e7802-b59d-4f0f-a506-668ea743d9ef

RTSP/1.0 200
Session: 151472816;timeout=30
CSeq: 5
Range: npt=now-
RTP-Info: url=rtsp://192.168.0.85/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/video;ssrc=1358227F
Server: ASTI Media Server RTSP\1.0


Received Invalid Message: * RTSP/0.0


For Session:ce6e7802-b59d-4f0f-a506-668ea743d9ef
Media.Rtp.RtpClient-042e3c94-3f8e-452a-9ab3-2e5f363fcf5a@SendRecieve: LengthInWordsMinusOne overflowed. Cannot store a number lower than 2 or higher than 0 in a 65535 structure.
Media.Rtp.RtpClient-042e3c94-3f8e-452a-9ab3-2e5f363fcf5a@SendRecieve - Begin
Media.Rtp.RtpClient-042e3c94-3f8e-452a-9ab3-2e5f363fcf5a@ProcessFrameData - raisedEvent for frameLength: 183 remainingInBuffer=20
Request=> TEARDOWN rtsp://192.168.0.85:8554/live/d71eb5b9-09a9-48b8-b6a5-c2eda11e91f2/ RTSP/1.0
CSeq: 6
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: 151472816

 Session=> ce6e7802-b59d-4f0f-a506-668ea743d9ef

042e3c94-3f8e-452a-9ab3-2e5f363fcf5aProcessFrameData - ParseAndHandleData
RTSP/1.0 200
Session: 151472816;timeout=30
CSeq: 6
Server: ASTI Media Server RTSP\1.0

[dimhotepus]

#24 doen't help, lines LengthInWordsMinusOne overflowed. Cannot store a number lower than 2 or higher than 0 in a 65535 structure. are gone, but log the same and no video playing in VLC.

Tried to rollback to commit without all my PVS-Studio commits (8ad1f3f), but problem still remains.

[juliusfriedman]

You only changed values for MP4 Boxes / Atoms, I think the values are actually in BigEndian, I have to double check the spec...

I am pretty sure there is something else wrong I will check the tests asap and let you know

[juliusfriedman]

I checked your code, everything seems okay but I am able to confirm over UDP there is either something wrong with your server or the parsing logic with the SDP, please do a wireshark capture of the RtspClientTest on your stream and submit it here so I can take a look when I have time.

I verified over TCP your stream works correctly with the RtspClient as well as the server through VLC.

Please take a look at why the issue occurs over UDP and let me know if you need my help, especially if you dont find anything on your side which causes the UDP session to fail.

[juliusfriedman]

BTW, something is clearly happening with your server and UDP connections, it supplies and SDP the first time but then apparently stalls out and never finished the SDP for the subsequent describes...

In RtspClient @ StartPlaying you can also do something like this to verify it...

if (!string.IsNullOrEmpty(response.Body))
                                {
                                    //Try to create a session description even if there was no contentType so long as one was not specified against sdp.
                                    m_SessionDescription = new Sdp.SessionDescription(response.Body);
                                }
                                

[dimhotepus]

You want to do something like a camera tour where you tour between each stream?

It should be very easy to achieve provided the underlying streams are using the same codec, you can essentially just create a RtspStream and a timer and switch which underlying packets are going to the output RtspStream based on your desired logic, e.g. a counter to select from the proper underlying streams packets.

The logic would essentially on elapse of the timer attach to the events from the desired stream and enqueue them into the output stream for playback.

Give me a try and show me what you have tried and I can better assist you from there!

The code would be something like this:

someStream.RtpClient.RtpPacketReceieved += (s, p, tc) => outputStream.RtpClient.OnRtpPacketReceieved(p, tc);

Where someStream is the underlying stream you want to temporarily tour to and outputStream is the stream you have setup for playback.

Take a look @ https://github.com/juliusfriedman/net7mma_core/blob/master/UnitTests/Program.cs#L2125 (TestServer) for inspiration!

@juliusfriedman Could you please help me with this one more time?

I'm doing forwarding from input to output streams like this:

int serverPort = 8000 + Media.Rtsp.RtspMessage.ReliableTransportDefaultPort;
System.Net.IPAddress serverIp =
    Media.Common.Extensions.Socket.SocketExtensions.GetFirstUnicastIPAddress(
        System.Net.Sockets.AddressFamily.InterNetwork);

Console.WriteLine("Server Starting on: {0}:{1}", serverIp, serverPort);

using (Media.Rtsp.RtspServer server = new Media.Rtsp.RtspServer(serverIp, serverPort)
{
    Logger = new Media.Rtsp.Server.Loggers.RtspServerConsoleLogger(),
    ClientSessionLogger = new Media.Rtsp.Server.Loggers.RtspServerConsoleLogger()
})
{
    server.Start();

    while (!server.IsRunning) System.Threading.Thread.Sleep(0);

    var input = new Media.Rtsp.Server.MediaTypes.RtspSource(
        "R2_059",
        "rtsp://8.15.251.101:1935/rtplive/R2_059",
        Rtsp.RtspClient.ClientProtocolType.Tcp
    );

    // rtsp://8.15.251.101:1935/rtplive/R2_059 is H264 - MPEG-4 AVC (part 10) (h264) Video resolution: 320x240 Frame rate: 15
    Media.Rtsp.Server.MediaTypes.RFC6184Media output = new Rtsp.Server.MediaTypes.RFC6184Media(320, 240, "OUT");

    input.Start();
    input.RtpClient.OutOfBandData += (s, d, o, l) =>
    {
        output.RtpClient.OnOutOfBandData(d, o, l);
    };

    input.RtpClient.RtpFrameChanged += (s, p, tc, f) =>
    {
        output.RtpClient.OnRtpFrameChanged(p, tc, f);
    };
    //inputStream1.RtpClient.RtpPacketSent += (s, p, tc) =>
    //{
    //    outputStream.RtpClient.OnRtpPacketSent(p, tc);
    //};
    input.RtpClient.RtpPacketReceieved += (s, p, tc) =>
    {
        output.RtpClient.OnRtpPacketReceieved(p, tc);
    };

    input.RtpClient.RtcpPacketReceieved += (s, p, tc) =>
    {
        output.RtpClient.OnRtcpPacketReceieved(p, tc);
    };
    //inputStream1.RtpClient.RtcpPacketSent += (s, p, tc) =>
    //{
    //    outputStream.RtpClient.OnRtcpPacketSent(p, tc);
    //};

    server.TryAddMedia(output);

    Console.WriteLine("Listening on: " + server.LocalEndPoint);

    while (true)
    {
        ConsoleKeyInfo keyInfo = Console.ReadKey(true);
        if (keyInfo.Key == ConsoleKey.Q) break;
    }

    Console.WriteLine("Server Streamed : " + server.TotalStreamedBytes);
    Console.WriteLine("Rtsp Sent : " + server.TotalRtspBytesSent);
    Console.WriteLine("Rtsp Recieved : " + server.TotalRtspBytesRecieved);
    Console.WriteLine("Stopping Server");

    server.Stop();

    Console.WriteLine("Server Stopped");
}

But looks like I do not understand your idea correctly and unable to play rtsp://192.168.0.77:8554/live/OUT in VLC.
What I'm doing wrong here?

[dimhotepus]

It's just a bit more complex than it needs to be in order to work.

If you already have your output stream then its just a matter of calling packetize from the appropriate source..

You don't need the OutOfBandData and I personally would work with the RtspStream from the RtspServer rather than at the RtpClient level as it will decrease the complexity greatly.

All you need is an output stream:

Media.Rtsp.Server.MediaTypes.RtpAudioSink outputStream = new Rtsp.Server.MediaTypes.RtpAudioSink("Output", null, 0, 1, 8000);

Then from there you will attach to the output stream from the desired input stream like so:

someStream.RtpClient.RtpPacketReceieved += (s, p, tc) => outputStream.RtpClient.OnRtpPacketReceieved(p, tc);

This will cause the packets to begin to flow from someStream to outputStream.

Then upon a timer elapsing or otherwise you can easily stitch together different streams to create a tour provided your coded is the same for each stream (and the settings for the codec are relatively similar)

Remember to disconnect the events from the someStream when you are done with that particular instance so this way both streams are not flowing to the output at the same time.

If you need more help please let me know explicitly what.

Thank you again for your other contributions.

Ok, I've done exactly as needed:

int serverPort = 8000 + Media.Rtsp.RtspMessage.ReliableTransportDefaultPort;
System.Net.IPAddress serverIp =
    Media.Common.Extensions.Socket.SocketExtensions.GetFirstUnicastIPAddress(
        System.Net.Sockets.AddressFamily.InterNetwork);

Console.WriteLine("Server Starting on: {0}:{1}", serverIp, serverPort);

using (Media.Rtsp.RtspServer server = new Media.Rtsp.RtspServer(serverIp, serverPort)
{
    Logger = new Media.Rtsp.Server.Loggers.RtspServerConsoleLogger(),
    ClientSessionLogger = new Media.Rtsp.Server.Loggers.RtspServerConsoleLogger()
})
{
    server.Start();

    while (!server.IsRunning) System.Threading.Thread.Sleep(0);

    var output = new Rtsp.Server.MediaTypes.RtpAudioSink("Output", null, 0, 1, 8000);

    // rtsp://8.15.251.101:1935/rtplive/R2_059 is H264 - MPEG-4 AVC (part 10) (h264) Video resolution: 320x240 Frame rate: 15    
    var input = new Media.Rtsp.Server.MediaTypes.RtspSource(
        "R2_059",
        "rtsp://8.15.251.101:1935/rtplive/R2_059",
        Rtsp.RtspClient.ClientProtocolType.Tcp
    );
    input.Start();
    input.RtpClient.RtpPacketReceieved += (s, p, tc) =>
    {
        output.RtpClient.OnRtpPacketReceieved(p, tc);
    };

    server.TryAddMedia(output);

    Console.WriteLine("Listening on: " + server.LocalEndPoint);

    while (true)
    {
        ConsoleKeyInfo keyInfo = Console.ReadKey(true);
        if (keyInfo.Key == ConsoleKey.Q) break;
    }

    Console.WriteLine("Server Streamed : " + server.TotalStreamedBytes);
    Console.WriteLine("Rtsp Sent : " + server.TotalRtspBytesSent);
    Console.WriteLine("Rtsp Recieved : " + server.TotalRtspBytesRecieved);
    Console.WriteLine("Stopping Server");

    server.Stop();

    Console.WriteLine("Server Stopped");
}

Query rtsp://192.168.0.77:8554/live/Output via VLC but still nothing.

Request=> PLAY rtsp://192.168.0.77:8554/live/59613cf9-49d5-4d0e-b2f9-51ee19e545ea/ RTSP/1.0
CSeq: 5
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: -1021074595
Range: npt=0.000-

 Session=> 6a00caca-58d7-42f4-8d43-ad03d5670948

Media.Rtp.RtpClient-311663aa-2cc0-48b9-8347-a7776065eb3b@SendRecieve - Begin
RTSP/1.0 200
Session: -1021074595;timeout=30
CSeq: 5
Range: npt=now-
RTP-Info: url=rtsp://192.168.0.77/live/59613cf9-49d5-4d0e-b2f9-51ee19e545ea/audio;ssrc=4601F9A6
Server: ASTI Media Server RTSP\1.0


Media.Rtp.RtpClient-311663aa-2cc0-48b9-8347-a7776065eb3b@SendRecieve RtpSocket - SocketError = TimedOut lastOperation = 10/25/2023 9:02:27 PM taken = 00:00:00.0000006
311663aa-2cc0-48b9-8347-a7776065eb3bProcessFrameData - ParseAndHandleData
Media.Rtp.RtpClient-311663aa-2cc0-48b9-8347-a7776065eb3b@ProcessFrameData - raisedEvent for frameLength: 4 remainingInBuffer=0
Session:6a00caca-58d7-42f4-8d43-ad03d5670948 Attempting to complete previous mesage with buffer of 4 bytes.
Session:6a00caca-58d7-42f4-8d43-ad03d5670948 used 4 of buffer bytes
Request=> PLAY rtsp://192.168.0.77:8554/live/59613cf9-49d5-4d0e-b2f9-51ee19e545ea/ RTSP/1.0
CSeq: 5
User-Agent: LibVLC/3.0.19 (LIVE555 Streaming Media v2016.11.28)
Session: -1021074595
Range: npt=0.000-

 Session=> 6a00caca-58d7-42f4-8d43-ad03d5670948

RTSP/1.0 200
Session: -1021074595;timeout=30
CSeq: 5
Range: npt=now-
RTP-Info: url=rtsp://192.168.0.77/live/59613cf9-49d5-4d0e-b2f9-51ee19e545ea/audio;ssrc=4601F9A6
Server: ASTI Media Server RTSP\1.0


Received Invalid Message: * RTSP/0.0


For Session:6a00caca-58d7-42f4-8d43-ad03d5670948

[juliusfriedman]

Did you verify if you can play your source stream from the server by itself? If so please check out this code which should work with only minor changes:

//Make a new session description
                                Media.Sdp.SessionDescription compositeDescription = new Sdp.SessionDescription(0, "Bandit", "Composite");

                                //Add some required lines.
                                compositeDescription.Add(new Sdp.Lines.SessionConnectionLine()
                                {
                                    ConnectionNetworkType = Sdp.Lines.SessionConnectionLine.InConnectionToken,
                                    ConnectionAddressType = Sdp.SessionDescription.WildcardString,
                                    ConnectionAddress = System.Net.IPAddress.Any.ToString()
                                });

                                //you need to access the session description from the media you wish to composite.

                                //Add the first
                                compositeDescription.Add(tcpStream.SessionDescription.MediaDescriptions.FirstOrDefault(), false);

                                //Add the second
                                compositeDescription.Add(sampleStream.SessionDescription.MediaDescriptions.FirstOrDefault(), false);

                                Media.Rtsp.Server.MediaTypes.RtpSource compositeSource = new Rtsp.Server.MediaTypes.RtpSource("Composite", compositeDescription);

                                var compositeContext = compositeSource.RtpClient.GetTransportContexts().First();

                                //Todo, Context.Synchronize or Context.Clone..
                                //Make the context a copy of the of the context to which it represents.

                                compositeContext.MinimumSequentialValidRtpPackets = 0;

                                compositeContext.AllowOutOfOrderPackets = true;

                                compositeContext.SynchronizationSourceIdentifier = tcpStream.RtpClient.GetTransportContexts().First().SynchronizationSourceIdentifier;

                                compositeContext.RemoteSynchronizationSourceIdentifier = tcpStream.RtpClient.GetTransportContexts().First().RemoteSynchronizationSourceIdentifier;

                                compositeContext.IsRtcpEnabled = false;

                                compositeContext.SendInterval = compositeContext.ReceiveInterval = Media.Common.Extensions.TimeSpan.TimeSpanExtensions.InfiniteTimeSpan;

                                compositeContext = compositeSource.RtpClient.GetTransportContexts().Last();

                                //

                                compositeContext.MinimumSequentialValidRtpPackets = 0;

                                compositeContext.AllowOutOfOrderPackets = true;

                                compositeContext.SynchronizationSourceIdentifier = sampleStream.RtpClient.GetTransportContexts().First().SynchronizationSourceIdentifier;

                                compositeContext.RemoteSynchronizationSourceIdentifier = sampleStream.RtpClient.GetTransportContexts().First().RemoteSynchronizationSourceIdentifier;

                                compositeContext.IsRtcpEnabled = false;

                                compositeContext.SendInterval = compositeContext.ReceiveInterval = Media.Common.Extensions.TimeSpan.TimeSpanExtensions.InfiniteTimeSpan;

                                int sharedClock = -1; ;

                                //Determine if packets or events will be handled
                                if (tcpStream.RtpClient.FrameChangedEventsEnabled)
                                {
                                    compositeSource.RtpClient.FrameChangedEventsEnabled = true;

                                    //Todo, Delcare a function so what comes in can be buffered if required.
                                    tcpStream.RtpClient.RtpFrameChanged += (sender, frame, transportContext, final) =>
                                    {
                                        if (final)
                                        {

                                            if (sharedClock.Equals(-1))
                                            {
                                                sharedClock = frame.Timestamp;
                                            }
                                            else
                                            {
                                                sharedClock += frame.Timestamp - sharedClock;
                                            }

                                            compositeSource.RtpClient.OnRtpFrameChanged(frame, compositeSource.RtpClient.GetContextBySourceId(frame.SynchronizationSourceIdentifier), final);
                                        }
                                    };
                                }
                                else
                                {
                                    compositeSource.RtpClient.FrameChangedEventsEnabled = false;

                                    //Todo, Delcare a function so what comes in can be buffered if required.
                                    tcpStream.RtpClient.RtpPacketReceieved += (sender, packet, transportContext) =>
                                    {

                                        if (packet.Marker)
                                        {
                                            if (sharedClock.Equals(-1))
                                            {
                                                sharedClock = packet.Timestamp;
                                            }
                                            else
                                            {
                                                sharedClock += packet.Timestamp - sharedClock;
                                            }
                                        }
                                        else if (sharedClock.Equals(-1))
                                        {
                                            sharedClock = packet.Timestamp;
                                        }

                                        compositeSource.RtpClient.HandleIncomingRtpPacket(sender, packet, null);
                                    };
                                }

                                //The same for the other stream
                                if (sampleStream.RtpClient.FrameChangedEventsEnabled)
                                {
                                    //Todo, Delcare a function so what comes in can be buffered if required.
                                    sampleStream.RtpClient.RtpFrameChanged += (sender, frame, transportContext, final) =>
                                    {
                                        if (final) compositeSource.RtpClient.OnRtpFrameChanged(frame, compositeSource.RtpClient.GetContextBySourceId(frame.SynchronizationSourceIdentifier), final);
                                    };
                                }
                                else
                                {
                                    //Todo, Delcare a function so what comes in can be buffered if required.
                                    sampleStream.RtpClient.RtpPacketReceieved += (sender, packet, transportContext) =>
                                    {
                                        compositeSource.RtpClient.HandleIncomingRtpPacket(sender, packet, null);
                                    };
                                }

                                //switch sources after x frames in events, retimestamp data or otherwise and send out.

                                //Finish the SDP

                                compositeDescription.Add(new Sdp.Lines.SessionConnectionLine()
                                {
                                    ConnectionNetworkType = Sdp.Lines.SessionConnectionLine.InConnectionToken,
                                    ConnectionAddressType = Sdp.SessionDescription.WildcardString,
                                    ConnectionAddress = System.Net.IPAddress.Any.ToString()
                                });

                                //Indicate control to each media description contained can be attained from the main uri
                                //e.g. you can pause both streams at once.
                                compositeDescription.Add(new Sdp.SessionDescriptionLine("a=control:*"));

                                //Ensure the session members know they SHOULD only receive
                                compositeDescription.Add(new Sdp.SessionDescriptionLine("a=sendonly")); // this directive is for their `tools`

                                //that this a broadcast....
                                compositeDescription.Add(new Sdp.SessionDescriptionLine("a=type:broadcast"));

                                var md = compositeDescription.MediaDescriptions.First();

                                if (false.Equals(object.ReferenceEquals(md.ControlLine, null)))
                                {
                                    md.Remove(md.ControlLine);
                                }

                                //How to control only this track, any valid grammar value should work but some libraries KISS/SUCK so...
                                md.Add(new Sdp.SessionDescriptionLine("a=control:trackID=1"));

                                //May already have a name line...
                                //md.Add(new Media.Sdp.Lines.SessionNameLine("Pics Stream"));

                                md = compositeDescription.MediaDescriptions.Last();

                                if (false.Equals(object.ReferenceEquals(md.ControlLine, null)))
                                {
                                    md.Remove(md.ControlLine);
                                }

                                //How to control only that track
                                md.Add(new Sdp.SessionDescriptionLine("a=control:trackID=2"));

                                //May already have a name line.
                                //md.Add(new Media.Sdp.Lines.SessionNameLine("Bandit Stream"));

                                //Start the source now...
                                server.TryAddMedia(compositeSource);

                                System.Console.WriteLine("Started Composite");

                                //Could add third stream here...

                                break;

Please keep in mind in this example both streams flow to output at the same time. I didn't add logic here to stop the flow from either stream, just wanted to show how to create the composite stream.

P.s. This was already asked for and achieved before:

https://github.com/juliusfriedman/net7mma/blob/master/UnitTests/Program.cs#L2498

[juliusfriedman]

P.s. The above code will allow your users to select a stream through VLC or whatever player they are using.

If you don't need the stream to be selectable then you can skip having multiple tracks in the SDP and you can just use the code with a timer to switch out which source is attached to the compositeStream.

Let me know if you have any further questions!

[dimhotepus]

ThreadExtensions.MinimumStackSize = 1 causes StackOverflow for me. Solved by setting one to 64 * 1024

[dimhotepus]

My last attempt below. But it still causes artifacts in result video stream, hunging streaming, etc.
Result url is rtsp://<local ip address>:8554/live/carousel.

class TimedRingBuffer<T>
{
    private readonly T[] buffer;
    private readonly TimeSpan spanBetweenBufferIndexChange;
    private readonly int timesToUseNoIndex;

    private int currentIndex;
    private DateTime? currentIndexStartDateTime;

    public TimedRingBuffer(T[] buffer, TimeSpan spanBetweenBufferIndexChange)
    {
        if (buffer is null)
        {
            throw new ArgumentNullException(nameof(buffer));
        }

        if (buffer.Length == 0)
        {
            throw new ArgumentException("Sources should not be empty.", nameof(buffer));
        }

        if (timesToUseNoIndex < 0)
        {
            throw new ArgumentException($"Old index use time {timesToUseNoIndex} can't be < 0", nameof(timesToUseNoIndex));
        }

        this.buffer = buffer;
        this.spanBetweenBufferIndexChange = spanBetweenBufferIndexChange;

        this.currentIndex = 0;
    }

    public bool IsActiveNow(T item)
    {
        if (!currentIndexStartDateTime.HasValue)
        {
            this.currentIndexStartDateTime = DateTime.UtcNow;
        }

        if (currentIndexStartDateTime + spanBetweenBufferIndexChange <= DateTime.UtcNow)
        {
            this.currentIndex = (this.currentIndex + 1) % this.buffer.Length;
            this.currentIndexStartDateTime = DateTime.UtcNow;
        }

        return Equals(item, buffer[currentIndex]);
    }
}

static RtpSource CreateCompositeSource(RtspSource[] sources, string originator, string name, TimeSpan switchSourceAfterTimeSpan)
{
    // Make a new session description
    Sdp.SessionDescription compositeSession = new(0, originatorAndSession: originator, sessionName: name)
    {
        // Add some required lines.
        new Sdp.Lines.SessionConnectionLine
        {
            ConnectionNetworkType = Sdp.Lines.SessionConnectionLine.InConnectionToken,
            ConnectionAddressType = Sdp.SessionDescription.WildcardString,
            ConnectionAddress = System.Net.IPAddress.Any.ToString()
        }
    };

    foreach (var source in sources)
    {
        var mediaDescription =
            source.SessionDescription.MediaDescriptions.FirstOrDefault()
            ?? throw new Exception($"Source '{source.Source}' missed media description.");

        compositeSession.Add(mediaDescription, false);
    }

    var compositeSource = new RtpSource(name, compositeSession);
    var compositeSourceTransportContexts =
        compositeSource.RtpClient.GetTransportContexts().ToArray();

    if (sources.Length != compositeSourceTransportContexts.Length)
    {
        throw new Exception(
            $"Sources length ({sources.Length}) is not same as composite source contexts one ({compositeSourceTransportContexts.Length}).");
    }

    var timedRingBuffer = new TimedRingBuffer<RtspSource>(sources, switchSourceAfterTimeSpan);

    for (int i = 0; i < sources.Length; i++)
    {
        var source = sources[i];
        var compositeContext = compositeSourceTransportContexts[i];

        // Make the context a copy of the of the context to which it represents.
        SetupCompositeContextFromSource(source, compositeContext);

        // Determine if packets or events will be handled.
        ForwardCompositeEventsFromSource(source, compositeSource, timedRingBuffer.IsActiveNow);
    }

    // Switch sources after x frames in events, retimestamp data or otherwise and send out.

    // Finish the SDP
    compositeSession.Add(new Sdp.Lines.SessionConnectionLine
    {
        ConnectionNetworkType = Sdp.Lines.SessionConnectionLine.InConnectionToken,
        ConnectionAddressType = Sdp.SessionDescription.WildcardString,
        ConnectionAddress = System.Net.IPAddress.Any.ToString()
    });

    // Indicate control to each media description contained can be attained from the main uri
    // e.g. you can pause both streams at once.
    compositeSession.Add(new Sdp.SessionDescriptionLine("a=control:*"));
    // Ensure the session members know they SHOULD only receive. This directive is for their `tools`
    compositeSession.Add(new Sdp.SessionDescriptionLine("a=sendonly"));
    // That this a broadcast....
    compositeSession.Add(new Sdp.SessionDescriptionLine("a=type:broadcast"));

    var compositeMediaDescriptions = compositeSession.MediaDescriptions.ToArray();
    if (sources.Length != compositeMediaDescriptions.Length)
    {
        throw new Exception(
            $"Sources length ({sources.Length}) is not same as composite media descriptions one ({compositeMediaDescriptions.Length}).");
    }

    for (int i = 0; i < compositeMediaDescriptions.Length; i++)
    {
        var compositeDescription = compositeMediaDescriptions[i];
        var controlLine = compositeDescription.ControlLine;
        if (controlLine is not null)
        {
            compositeDescription.Remove(controlLine);
        }

        // How to control only this track, any valid grammar value should work but some libraries KISS/SUCK so...
        // compositeDescription.Add(new Sdp.SessionDescriptionLine($"a=control:trackID={i + 1}"));

        // May already have a name line...
        // compositeDescription.Add(new Sdp.Lines.SessionNameLine(sources[i].Name));
    }

    return compositeSource;
}

private static void ForwardCompositeEventsFromSource(RtspSource inputSource, RtpSource compositeSource, Func<RtspSource, bool> shouldForwardSource)
{
    RtpClient sourceClient = inputSource.RtpClient, compositeClient = compositeSource.RtpClient;

    bool isFrameChangedEventsEnabled = sourceClient.FrameChangedEventsEnabled;
    compositeClient.FrameChangedEventsEnabled = isFrameChangedEventsEnabled;

    if (isFrameChangedEventsEnabled)
    {
        sourceClient.RtpFrameChanged += (sender, frame, transportContext, final) =>
        {
            if (final && shouldForwardSource(inputSource))
            {
                var compositeTransportContext =
                    compositeClient.GetContextBySourceId(frame.SynchronizationSourceIdentifier);
                compositeClient.OnRtpFrameChanged
                (
                    frame,
                    compositeTransportContext,
                    final
                );
            }
        };
    }
    else
    {
        sourceClient.RtpPacketReceieved += (sender, packet, transportContext) =>
        {
            if (shouldForwardSource(inputSource))
            {
                compositeClient.HandleIncomingRtpPacket(sender, packet, null);
            }
        };
    }

    sourceClient.RtcpPacketReceieved += (sender, packet, transportContext) =>
    {
        if (shouldForwardSource(inputSource))
        {
            compositeClient.HandleIncomingRtcpPacket(sender, packet, null);
        }
    };
}

private static void SetupCompositeContextFromSource(RtspSource inputSource, RtpClient.TransportContext compositeContext)
{
    var transportContext = inputSource.RtpClient.GetTransportContexts().FirstOrDefault()
        ?? throw new Exception($"Source '{inputSource.Source}' has missed transport context.");

    compositeContext.MinimumSequentialValidRtpPackets = 0;
    compositeContext.AllowOutOfOrderPackets = true;
    compositeContext.SynchronizationSourceIdentifier =
        transportContext.SynchronizationSourceIdentifier;
    compositeContext.RemoteSynchronizationSourceIdentifier =
        transportContext.RemoteSynchronizationSourceIdentifier;
    compositeContext.IsRtcpEnabled = false;
    compositeContext.SendInterval =
        compositeContext.ReceiveInterval =
            Media.Common.Extensions.TimeSpan.TimeSpanExtensions.InfiniteTimeSpan;
}

static void TestServer()
{
    System.Runtime.GCLatencyMode oldMode = System.Runtime.GCSettings.LatencyMode;

    try
    {
        System.Runtime.GCSettings.LatencyMode = System.Runtime.GCLatencyMode.LowLatency;

        int serverPort = 8000 + Rtsp.RtspMessage.ReliableTransportDefaultPort;
        System.Net.IPAddress serverIp =
            Common.Extensions.Socket.SocketExtensions.GetFirstUnicastIPAddress(
                System.Net.Sockets.AddressFamily.InterNetwork);

        Console.WriteLine("Server Starting on: {0}:{1}", serverIp, serverPort);

        using (Rtsp.RtspServer server = new Rtsp.RtspServer(serverIp, serverPort)
        {
            Logger = new Rtsp.Server.Loggers.RtspServerConsoleLogger(),
            ClientSessionLogger = new Rtsp.Server.Loggers.RtspServerConsoleLogger()
        })
        {
            server.Start();

            // Wait for the server to start.
            while (!server.IsRunning) System.Threading.Thread.Sleep(0);

            var inputSource1 = new RtspSource(
                "R2_059",
                "rtsp://8.15.251.101:1935/rtplive/R2_059",
                Rtsp.RtspClient.ClientProtocolType.Tcp
            );
            inputSource1.Start();

            var inputSource2 = new RtspSource(
                "R2_052",
                "rtsp://8.15.251.101:1935/rtplive/R2_052",
                Rtsp.RtspClient.ClientProtocolType.Tcp
            );
            inputSource2.Start();

            var compositeSource = CreateCompositeSource
            (
                new[] { inputSource1, inputSource2 },
                originator: "nuSIM",
                name: "carousel",
                switchSourceAfterTimeSpan: TimeSpan.FromSeconds(5)
            );

            server.TryAddMedia(compositeSource);

            inputSource1.RtpClient.ThreadEvents = false;

            Console.WriteLine("Listening on: " + server.LocalEndPoint);

            while (true)
            {
                ConsoleKeyInfo keyInfo = Console.ReadKey(true);
                if (keyInfo.Key == ConsoleKey.Q) break;
            }

            Console.WriteLine("Server Streamed : " + server.TotalStreamedBytes);
            Console.WriteLine("Rtsp Sent : " + server.TotalRtspBytesSent);
            Console.WriteLine("Rtsp Recieved : " + server.TotalRtspBytesRecieved);
            Console.WriteLine("Stopping Server");

            server.Stop();

            Console.WriteLine("Server Stopped");
        }
    }
    finally
    {
        System.Runtime.GCSettings.LatencyMode = oldMode;
    }
}

[dimhotepus]

[juliusfriedman]

It seems like you had data flowing from both streams to the output source at the same time...

This is what caused the artifacts I assume, if you just leave one stream flowing through the compositeStream then you should not see any artifacts?

I think all that your missing is the removal of the events from the stream you don't want to display, It seems like why there are artifacts, because your sending data from both streams at the same time.

I dont' see how the TimedRingBuffer is useful yet, I don't think its needed because the Queue on each stream acts like a RingBuffer but I see your using it to switch the index of the stream so I will look into that as I don't have a lot of time right now, perhaps over the weekend.

I will check out your code but if you could provide me the link to 2 working streams it would be helpful for me to setup an example for you when I get time.

[dimhotepus]

rtsp://8.15.251.101:1935/rtplive/R2_051
rtsp://8.15.251.101:1935/rtplive/R2_059

both are working.

[dimhotepus]

Even if i do not do composite streaming at all, just

server.Start();

// Wait for the server to start.
while (!server.IsRunning) System.Threading.Thread.Sleep(0);

var inputSource1 = new RtspSource(
    "R2_059",
    "rtsp://8.15.251.101:1935/rtplive/R2_059",
    Rtsp.RtspClient.ClientProtocolType.Tcp
);
inputSource1.Start();
server.TryAddMedia(inputSource1);

Wait a bit and you see artifacts in result stream. May be it is related to TCP proto, but VLC plays original stream without artifacts.

[dimhotepus]

[juliusfriedman]

I doo see the artifacts, they seem to be reduced with larger buffer size...

server.TryAddMedia(new Media.Rtsp.Server.MediaTypes.RtspSource("R2_051", "rtsp://8.15.251.101:1935/rtplive/R2_051", Rtsp.RtspClient.ClientProtocolType.Tcp, 65540));
                    server.TryAddMedia(new Media.Rtsp.Server.MediaTypes.RtspSource("R2_059", "rtsp://8.15.251.101:1935/rtplive/R2_059", Rtsp.RtspClient.ClientProtocolType.Tcp, 65540));

I am seeing a lot of strange messages about buffer size exceeded, I will have to look into this to see exactly what is causing it.

[juliusfriedman]

I pushed up another small commit, it seems ThreadEvents can give your stream a little extra processing power and the artifacts seem to go away much quicker (but are still present during switching).

I will continue to look into this and let you know what I can find.

[juliusfriedman]

I found a small bug where packets were being missed when the buffer was empty, its a small change but seems to remove a lot of artifacts.

I will continue to look into why the remaining artifacts are present. (BTW if you use a really large buffer 10*1024) then it seems the artifacts clear up eventually.

[juliusfriedman]

I pushed up another small change but overall I think we will have better luck with the RtpVideoSink I will see if I can get an example working tomorrow, with that change we just have to add the frame and the Timestamp and Sequence numbers will be automatically calculated correctly without hacks...

[juliusfriedman]

I pushed up that change, it runs in VLC for hours at a time without any issues from what I can see, the artifacts come and go but I am sure we will track that down with time.

It should be trivial for your you to add streams to this example.

Please let me know if I can be of any other help!

[juliusfriedman]

BTW, also to reduce the artifacts when switching you can create a keyframe with sps and pps and send it right when the events are attached, you should get the sps and pps from the mediaDescription of the source stream, here is the example...

 var frame = new RFC6184Media.RFC6184Frame(97);

                                        var mediaDescription = currentStream.SessionDescription.MediaDescriptions.FirstOrDefault();

                                        var spsPPs = mediaDescription.FmtpLine.ToString().Split(';').Last().Replace("sprop-parameter-sets=", string.Empty).Split(",");

                                        frame.Packetize(Convert.FromBase64String(spsPPs[0]));
                                        frame.Packetize(Convert.FromBase64String(spsPPs[1]));
                                        frame.SynchronizationSourceIdentifier = compositeContext.SynchronizationSourceIdentifier;

                                        compositeSource.Frames.Enqueue(frame);

If you do this then it helps to reduce the artifacts a bit right at the time of switching.

[juliusfriedman]

Still looking into this, it seems your camera is sending some other type of data (maybe sps and pps) which should be included in teh data stream.

I am not sure how live 555/vlc parses it so elegantly especially with all the strange malformed packets but its something I will continue to look into and see where I can get.

I will keep you updated.

(BTW Thanks for your continued work on the code)

[juliusfriedman]

Made another few small improvements to the sending of data so its not as much of a hack. Looking at the Wire capture there is definitely bandwidth exhaustion on the camera, possibly hardware failure. (I get a lot of malformed packets)

Why VLC doesn't seem to have any issues or display any log errors / debug warnings is something I am still curious about and will continue to look into.

[juliusfriedman]

In the latest code (Release) I can run your streams for over an hour and the artifacts are very minimal, please take a look and let me know what you think.

[juliusfriedman]

A quick update before I head off for the night, I updated the source code to show you that PerPacket is working a lot better with ffplay, I see VLC also is improved but still has artifacts.

It may save you having to create a FFMPEG Source after all...

Please test and let me know how you find it!

[juliusfriedman]

Things looked good from over here, I was able to run your streams for hours without any issues.

I find VLC has more artifacts than ffplay but the artifacts come and go with time on VLC, when using FFplay it was much harder to notice any artifacts at all even during switching of streams.

I think PerPacket is the way to go for your cameras, see how I setup the Traffic streams in the example server.

I am going to close this issue if I don't hear back from you with confirmation, Thank you again for your contributions and testing resources!

[juliusfriedman]

Just pushed up a few more fixes, please let me know what you think asap!

(Funny note, your cameras reply to a DESCRIBE * but then stop responding in the stream directly after that) You can press "X" when running the RtspClient test to see it) it will drop all RTP traffic for some reason right after it gets a DESCRIBE *....

Let me know if you notice anything else when testing!

[juliusfriedman]

All PR's are merged and I did some basic testing, everything looks good to me especially with PerPacket turned on the for streams in question.

Thanks again for your contributions and testing resources!

[juliusfriedman]

I pushed up a few fixes in anticipation of RTSP 2 support on these cameras... unfortunately they don't seem to support anything but 1.0.

Still looking into why exactly the stream artifacts are so heavy when going through the rtsp server (especially on the composite stream)

It would be ideal to get this to work without changing the cameras GOP size and I Frame interval.

I will let you know what I can find if anything.