RTMP2WebRTC: Stuttering when using flutter WebRTC

[winlinvip]

RTMP2WebRTC: Stuttering when using flutter WebRTC

[suzp1984]

Steps to reproduce

1. download this video sample recorded from the problem drone. https://drive.google.com/file/d/1UEMoMxv76D9W5pzOE9RQuBQ1bT4qIp06/view?usp=sharing
2. start srs: ./objs/srs -c conf/rtmp2rtc.conf, don't forget to config env $CANDIDATE.
3. rtmp publish step 1 video: ffmpeg -re -stream_loop -1 -i drone_1.flv -vcodec copy -f flv rtmp://localhost/live/livestream
4. check webrtc play in ios flutter_live app. if the main branch of flutter_live can't compile well with your flutter toolchain, use this branch https://github.com/suzp1984/flutter_live/tree/debug/4052-webrtc
5. compare the video with the chrome pc version. the chrome pc webrtc play well, the ios webrtc play stuttering.
6. check the safari mac: the webrtc play also has same problem, the http-flv can't play only show one picture with error indicator.

What I found have three problems: ios flutter_live webrtc play, mac safari webrtc play and mac safari http-flv & http-ts play.

Note: use export SRS_VHOST_HTTP_REMUX_HAS_AUDIO=off; ./objs/srs -c conf/http.ts.live.conf to reproduce http-ts play problem.

Cause

the ios & mac hardware decoder (Video Toolbox) dislike to feed it so many SEI NALU between NonIDR and IDR NALU packets.
The general NALU type sequence of a Normal rtmp video stream is like this:
SEI -> IDR -> SEI -> IDR -> ... NonIDR -> IDR -> ... NonIDR
But the problem video stream is like this:
SEI -> IDR -> SEI -> IDR -> SEI -> NonIDR -> (repeat SEI, NonIDR | IDR).

Solutions

add filter to delete the SEI NALU for rtc, which is works well.
for the http-flv on safari mac, need to do similar thing to filter the flv video tags with SEI NALU.(not yet verified).

[zeldan]

I believe this problem is related to #4050 (see my comment #4050 (comment)), where I am experiencing the same issue (but in the browser, not in the application).

[dancixx]

Same issue for me too. I get the same codec information in all cases which is h264 but on iOS in some cases, I have a huge fps drop. I thought there was some experimental codec issue related to VP9/AV1, but the default SRS settings for transcoding is the copy engine, so actually there is no transcoding process.

[DanielZelei]

@winlinvip

In the previously closed issue, you mentioned that the profile should be set to baseline.

I am unable to make any changes to the drone (there are no available options).

Where should I change this setting? Could you please point out where and what I should change in the SRS configuration?

I also tried transcoding with ultrafast and baseline settings, but it was slow (around 2-3 seconds), and I need the streaming to be under 0.5 seconds.

[suzp1984]

Let's reproduce this issue first on iOS device.

My Env

MacOS 14.4.1 M2 chip
iphone 11 + ios 17.4.1
My MacOS's ip address: 192.168.12.100

Here is my steps to do it.

1. prepare a video source to testing

I used this video source https://www.youtube.com/watch?v=-B2jRTTmYwk, restore it to mp4 file by youtube-dl.
(I like this song, you're welcome).
store the video source as test.mp4.

ffprobe test.mp4 and ffprobe -show_frames | grep -i 'pict_type' to check the mp4 file video codec profile and I/P/B frames.

Above video is h.264 high profile without b-frame.

2. start srs

export CANDIDATE=192.168.12.100
./objs/srs -c conf/rtmp2rtc.conf

replace ip address to yours.

3. rtmp publish to srs

1. publish to srs with b-frame

ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec h264 -profile:v high -acodec copy -f flv rtmp://localhost/live/livestream

2. publish to srs without b-frame

ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec h264 -profile:v high -bf 0 -acodec copy -f flv rtmp://localhost/live/livestream
or
ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec copy -acodec copy -f flv rtmp://localhost/live/livestream
or 
ffmpeg -re -stream_loop -1 -i test.mp4 -vcodec h264 -profile:v baseline -acodec copy -f flv rtmp://localhost/live/livestream

4. build and install flutter_live

use the branch in https://github.com/ossrs/flutter_live/pull/20.
In https://github.com/ossrs/flutter_live/pull/20/files#diff-69396fa518b81aa9d2099ead13c33e18bc8e570098408f2f50ee114adf6898d1, I upload my DEVELOPMENT_TEAM = 4X75RZ9C85; , that's a mistake, if you knows a tiny of xcode, you know how to replace this identify to your own one.

What I did in this draft is to upgrade flutter and flutter_webrtc and remove the camera_with_rtmp dependency.

5. play webrtc videos in ios

click webrtc option.
input: webrtc://192.168.12.100/live/livestream. (replace the ip address to yours).
click play.
compare the video quality.

My conclusion

I think the video stream without b-frame seems not that bad, just the same result to play webrtc in web browser (both safari and chrome in MacOS, check my comments in #4050)