Video frames lost at end of Grabbing and Recording session

[bytedeco]

Video frames lost at end of Grabbing and Recording session

[techstroke]

Well, strange enough but changing the Video codec to AV_CODEC_ID_MPEG4 from AV_CODEC_ID_H264 , fixed the problem for now , I'll still keep this issue open for you to look into ! & Thanks a lot for this beautiful library , makes it easier to work with ffmpeg and opencv on android 👍

[saudet]

Most (all?) codecs are going to fail when trying to setTimestamp() something that is <= getTimestamp(), so doing something like this should fix the error:

if (recorder.getTimestamp() < frameGrabber.getTimestamp()) {
    recorder.setTimestamp(frameGrabber.getTimestamp());
}

Let me know if that doesn't do the trick though, thanks!

[techstroke]

Thanks for the code !, Well this doesn't work even after replacing the setTimestamp() code with yours . So, I assume this is a bug in H264 codec implementation , as I am in hurry I'll rather change my codec to MPEG4 for now , till this thing gets fixed !

[saudet]

Well anyway, it's related to the call to setTimestamp(). If you figure out that this is indeed a problem with FFmpegFrameRecorder and not something wrong with your code or your data, please let me know so I can fix it, thanks!

[shuky19]

Hi,

I am having the same problem with my code, I am recording a movie from the native camera's API on an Android devices, later on I am using JavaCV to upload it to an RTMP server.

I have looked at @saudet comment, and implemented the same, but still the last few seconds in my videos are being trimed.

Here is my code (I have closed issue #47 since after further investigation I understood it is due to this exact problem):

    FFmpegFrameRecorder mFrameRecorder = new FFmpegFrameRecorder(urlPath, videoProfile.width, videoProfile.height, 1);
    mFrameRecorder.setFormat("flv");
    mFrameRecorder.setSampleRate(videoProfile.audioSampleRate);
    mFrameRecorder.setFrameRate(videoProfile.frameRate);
    mFrameRecorder.setVideoBitrate(videoProfile.bitRate);
    mFrameRecorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
    mFrameRecorder.setVideoOption("preset", "ultrafast");
    mFrameRecorder.start();

    FFmpegFrameGrabber mGrabber = new FFmpegFrameGrabber(mFilePath);
    mGrabber.setSampleRate(videoProfile.audioSampleRate);
    mGrabber.setFrameRate(videoProfile.frameRate);
    mGrabber.setVideoBitrate(videoProfile.bitRate);
    mGrabber.setVideoCodec(avcodec.AV_CODEC_ID_H264);
    mGrabber.start();


    Frame frame;
    while ((frame = getFrame()) != null) {
          if (frame != null) {
                // If frame contains image
                if (frame.image != null) {
                    // If need to rotate
                    if (mRotation != 0) {
                        frame.image = rotateImage(frame.image);
                    }
                }

                if (mGrabber.getTimestamp() > mFrameRecorder.getTimestamp()) {
                    mFrameRecorder.setTimestamp(mGrabber.getTimestamp());
                } else {
                    Log.w(TAG, "Incorrect timestamp: " + mGrabber.getTimestamp() + " when Recorder on: " + mFrameRecorder.getTimestamp());
                }

                try {
                    mFrameRecorder.record(frame);
                } catch (FrameRecorder.Exception e) {
                    Log.e(TAG, e.getMessage());
                    e.printStackTrace();
                }
            }
    }

Can you take a look? also can you reopen this issue?

thanks,
Shuky

[saudet]

@shuky19 Does the same thing happen with Java SE? Or does this happen only with Android?

[shuky19]

I tried it only on android...

[saudet]

So, could you please try it on Java SE as well?

[saudet]

Does this still happen with the latest version of JavaCV? If so, let's reopen this as a bug. Thanks!

[saudet]

I think this commit should fix this: fb2206d
Let me know! Thanks

[liangkuai]

Hi, guys
When I upload file to RTMP server, I get a fault all the time.

[libx264 @ 00000000266ccc00] non-strictly-monotonic PTS

And when I open rtmp stream in VLC, the video play fast abnormally.
I need some help...

[kmandayam]

Can anyone confirm if this if this works? I am using 1.3.1 and I am still facing this issue

[saudet]

@kmandayam Please provide a code snippet to reproduce this issue, thank you!

[kmandayam]

I'm trying to merge audio while video recording. I'm doing it onPreviewFrame:

    @Override
    public void onPreviewFrame(byte[] data, Camera camera)
    {
        try
        {
            if (songPath != null)
            {
                audioFrame = audioGrabber.grabFrame(true, false, false, false);
            }

        }
        catch (FrameGrabber.Exception e)
        {
            e.printStackTrace();
        }
        /* get video data */
        if (yuvIplImage != null && isRecording)
        {
            try
            {
                ((ByteBuffer) yuvIplImage.image[0].position(0)).put(data);
                timestamp = 1000 * (System.currentTimeMillis() - startTime);
                if (songPath == null)
                {
                    if (timestamp > recorder.getTimestamp())
                    {
                        recorder.setTimestamp(timestamp);
                    }
                }
                else
                {
                    // Updating timestamp if audio is chosen from the library
                    if (recorder.getTimestamp() <= audioGrabber.getTimestamp() )
                    {
                        recorder.setTimestamp(audioGrabber.getTimestamp());
                    }
                }

                if (useFilter)
                {
                    Frame frame2;
                    if (useNexusFilter())
                    {
                        nexusFilter.push(yuvIplImage);
                        while ((frame2 = nexusFilter.pull()) != null)
                        {

                            if (audioFrame != null)
                            {
                                recorder.record(audioFrame);
                            }
                            recorder.record(frame2);
                        }
                    }
                    else
                    {
                        filter.push(yuvIplImage);
                        while ((frame2 = filter.pull()) != null)
                        {

                            if (audioFrame != null)
                            {
                                recorder.record(audioFrame);
                            }
                            recorder.record(frame2);
                        }
                    }
                }
                else
                {
                    if (audioFrame != null)
                    {
                        recorder.record(audioFrame);
                    }
                    recorder.record(yuvIplImage);
                }
            }
            catch (NullPointerException e)
            {
                e.printStackTrace();
            }
            catch (FFmpegFrameRecorder.Exception | FrameFilter.Exception | BufferOverflowException e)
            {
                e.printStackTrace();
            }
        }
    }

I'm initializing and updating the FrameGrabber and recorder in surfaceCreated:

@OverRide
public void surfaceCreated(SurfaceHolder holder)
{
try
{
stopPreview();
camera.setPreviewDisplay(holder);
this.camera.setPreviewCallback(CameraView.this);
if (songPath != null)
{
// Initialize audioGrabber
audioGrabber = new FFmpegFrameGrabber(songPath);
audioGrabber.start();
audioGrabber.setFormat("mp3");
audioGrabber.setFrameRate(FRAME_RATE);
audioGrabber.setSampleRate(AUDIO_RATE);

                // Update recorder settings
                recorder.setSampleRate(audioGrabber.getSampleRate());
                recorder.setFrameRate(FRAME_RATE);
                recorder.setAudioChannels(audioGrabber.getAudioChannels());
                recorder.setSampleRate(AUDIO_RATE);
            }
        }
        catch (FrameGrabber.Exception e)
        {
            e.printStackTrace();
        }
        catch (IOException exception)
        {
            camera.release();
            camera = null;
        }
    }

I'm also losing few seconds of audio before the video ends. Please take a look and let me know if I am going wrong somewhere

[saudet]

Does this happen only on Android or can you reproduce that with Java SE as well?

[kmandayam]

@saudet I've only tried it on android. Not sure how to reproduce it with Java SE

[saudet]

@kmandayam Could you elaborate on why you're not sure how to reproduce this issue with Java SE?

[kmandayam]

@saudet Sorry for the late reply. I don't have a lot of experience working with javacv, so I don't know how to replicate this problem.

[kmandayam]

It works fine when I don't try to merge a separate audio file with the video (ie using the mic). All the frames are retained.

[saudet]

I see, audio and video need to have the same length, or the shortest might get cut, yes.

[kmandayam]

Well, since I am trying to merge audio while I record video, even if I stop recording mid way through the audio file, it still gets cropped. Could it have to do anything with the sample rate or frame rate?

[saudet]

It's possible that more frames would be needed because of incorrect rates, yes.