Attachment shows up as FFV1 track

[mediaarea]

Attachment shows up as FFV1 track

[kieranjol]

Also the rawcooked command originally would have been with 18.10
rawcooked input --check full -o ffv1_path

when i rechecked with git-master of rawcooked, i just did rawcooked path/to/dpx

and now i'm trying with git master:
rawcooked --all path/to/dpx

[kieranjol]

I got the same ffmpeg error with the file created via rawcooked git-master. The final check still seems to be running though.

[kieranjol]

The final post transcode check just said that everything was fine btw.

[JeromeMartinez]

Looks like a FFmpeg has some difficulties to decode the file created by itself. I heard that there is some heuristical checks for detecting corrupted streams, maybe the issue is there. RAWcooked has its own Matroska parser, so if reversibility is fine with RAWcooked I guess that this is on FFmpeg side which can not demux the file.

I tested your FFmpeg command line having errors with their git-master, no issue, maybe it would be fine with this version of FFmpeg?

[kieranjol]

Which version of ffmpeg do you mean,current git-master? I compiled ffmpeg after that fix was out in.
I've only ever seen this happen with one single MKV file created by rawcooked. Should I go to ffmpeg for further help,or is it worth trying to figure out what it is about this one rawcooked job that is causing issues?

[JeromeMartinez]

Currently RAWcooked has no encoder & muxer, it uses FFmpeg, you can see that by running rawcooked -d YourDpxDirectoryName, then launching the FFmpeg command displayed. If you try to dig with FFmpeg people, it is better to show them the FFmpeg command line (+maybe provide the attachment file).

Anyway, the main issue for the moment is to be able to reproduce the issue with a small input so I or Ffmpeg people can reproduce the issue. I didn't download the files (I was too late), do you mean by "as well as the source sequence" that the package you provided we permitting me to reproduce the error? If not, is it possible to have the 1st MB (should be enough for getting the 1st FFmpeg error) ?

[kieranjol]

Ahhh,use dd or something to just slice of the first megabyte of the MKV file? I can do that!

…

On Thu, 15 Aug 2019, 17:35 Jérôme Martinez, ***@***.***> wrote: Currently RAWcooked has no encoder & muxer, it uses FFmpeg, you can see that by running rawcooked -d YourDpxDirectoryName, then launching the FFmpeg command displayed. If you try to dig with FFmpeg people, it is better to show them the FFmpeg command line (+maybe provide the attachment file). Anyway, the main issue for the moment is to be able to reproduce the issue with a small input so I or Ffmpeg people can reproduce the issue. I didn't download the files (I was too late), do you mean by "as well as the source sequence" that the package you provided we permitting me to reproduce the error? If not, is it possible to have the 1st MB (should be enough for getting the 1st FFmpeg error) ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#244?email_source=notifications&email_token=AAITFPWAPX63WYQIBDHVBWDQEWAWLA5CNFSM4IHZYGF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4MKHAI#issuecomment-521708417>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAITFPV34HZR3OA7DMJDNF3QEWAWLANCNFSM4IHZYGFQ> .

[JeromeMartinez]

Ahhh,use dd or something to just slice of the first megabyte of the MKV file?

Right, just need that. I am not a Linux/Mac user, so not sure, but looks like that head head -c 1M input.file > output.file could also do the job.

[kieranjol]

Hi Jerome, i emailed the first meg - hope it helps with the test..

[JeromeMartinez]

IIUC the issue, the attachment is huge (1 GB) and FFmpeg does not appreciate that (looks like it considers the stream as buggy in the case the attachment size is > 256 MB, I understand a bit that as a 1 GB attachment is not classic).

I'll try to fix FFmpeg for that.

Note that the attachment is big because I need to store 8 KB of "random" data for each DPX frame, for reversibility, looks like the DPX files have 8 KB of extra data after the real content, I'll also check that this is coherent.

[JeromeMartinez]

Additional remark: IMO this MKV file is conform to spec, I don't remind that there is a limitation for theFileData Matroska element. @robUx4 would you mind to confirm that there is no limitation in element size even for attachments, or am I wrong there?

[JeromeMartinez]

@kieranjol beside the issue, I see that your DPX has a weird size.
File size is 0xC04000 (and DPX dedicated field says same)
Header size is 0x2000
Content size is 0xC0000 (0x800 x 0x600 x 4 bytes per pixel)
So remaining 0x2000 bytes at the end are present, and they look like random (so can not be compressed, so a big RAWcooked side car data file!). Is it wanted?

[JeromeMartinez]

The issue can be reproduced without RAWcooked, just by using FFmpeg, so it is not a RAWcooked issue (except blaming it for creating a big attachment), I have open a dedicated ticket https://trac.ffmpeg.org/ticket/8083 and I'll try to provide a patch to FFmpeg.

[kieranjol]

@kieranjol beside the issue, I see that your DPX has a weird size.
File size is 0xC04000 (and DPX dedicated field says same)
Header size is 0x2000
Content size is 0xC0000 (0x800 x 0x600 x 4 bytes per pixel)
So remaining 0x2000 bytes at the end are present, and they look like random (so can not be compressed, so a big RAWcooked side car data file!). Is it wanted?

Woah - it's not wanted, this sequence was delivered by a vendor so I don't have much further info.

The issue can be reproduced without RAWcooked, just by using FFmpeg, so it is not a RAWcooked issue (except blaming it for creating a big attachment), I have open a dedicated ticket https://trac.ffmpeg.org/ticket/8083 and I'll try to provide a patch to FFmpeg.

Thanks for opening the ticket - the conversation is very interesting. I concur that it's not a RAWcooked issue. I think from my side, I'll either ZIP this sequence and bypass rawcooked, or else I'll hold off for the ffmpeg issue to be fixed an then process it with rawcooked.

Thanks so much for the detailed analysis.

[JeromeMartinez]

I'll either ZIP this sequence and bypass rawcooked,

IMO RAWcooked is still interesting here for other advantages (VLC playback, compression performance), just that you can not handle the file with FFmpeg, consider the resulting file as a compressed file better than a ZIP.

In other words, in both cases (RAWcooked and ZIP), you can not use the file directly with current FFmpeg, but with RAWcooked you keep the other advantages as well as expecting to have FFmpeg support in a near future.

[robUx4]

Additional remark: IMO this MKV file is conform to spec, I don't remind that there is a limitation for theFileData Matroska element. @robUx4 would you mind to confirm that there is no limitation in element size even for attachments, or am I wrong there?

No limitation on the size apart from the fact it needs to fit in 56 bits unsigned. 1GB is fine.