Encodes RAW audio-visual data into the Matroska container (MKV), using the video codec FFV1 for the image and audio codec FLAC for the sound.
Home Page: https://mediaarea.net/RAWcooked
License: BSD 2-Clause "Simplified" License
RAWcooked encodes RAW audio-visual data into the Matroska container (MKV), using the video codec FFV1 for the image and audio codec FLAC for the sound.
RAWcooked is ready for production. As with any other tool, keep doing basic checks e.g. testing reversibility with your own files.
Information on this project and builds for different platforms can be found at https://mediaarea.net/RAWcooked.
BSD 2-Clause License.
Some alternate open source licenses are also available.
When unpacking the rawcooked Matroska file, I get "600" set as default permissions for the created files.
Is that on purpose?
For example:
Input is:
-rwxrwxrwx 1 pb pb 17080320 Dec 6 13:17 pb_test1/reel_1/0092274.dpx
Output is:
-r-------- 1 pb pb 17080320 Mar 27 19:01 pb_test1.mkv.RAWcooked/pb_test1/reel_1/0092274.dpx
This applies to all files created by RAWcooked.
Setup:
From: #154 (comment)
Would be better to have a single video stream for that kind of directory.
Being careful about time stamping when muxing.
Why is context=0 chosen at https://github.com/MediaArea/RAWcooked/blob/master/Source/CLI/Main.cpp#L384? I would have set context=1, but I may be wrong.
-g 1?-level 3?I personally don’t see and therefore I suggest to delete the possibility to change those values.
When running RAWcooked decoding twice for the same MKV file, it just overwrites the already existing target DPXs.
Would it be possible to give a classic "already exists, overwrite Y/N?" option?
What are the milestones for moving out of alpha, and what conditions are to be met for this tool being 'production ready'?
On our end, I think I'd be satisfied with having a workflow/script in place that performs the normalisation and then performs a losslessness check. I'm curious to know what others think.
We have tested several sequences with RAWcooked. It seems that the MKV file contains all the frames and seems to be lossless, but when running RAWcooked on the MKV file, only a portion of the files are regenerated. I looked at the hears of some of the source TIFFs around the point where the regeneration fails, and I can't see any change in the properties of the file or in the number run of the sequence.
This has happened with 16-bit TIFF and 16-bit DPX.
I'm not sure if i will be able to provide you with the sequence, but I could enquire?
rawcooked is built from source:
$ rawcooked --version
RAWcooked 18.10
License detected but decoding not supported in this version.
Using default license.
Consider to support RAWcooked: https://MediaArea.net/RAWcooked
command line output:
$ rawcooked 419e0c5f-cc55-4863-a948-ee43e522d96b.mkv
License detected but decoding not supported in this version.
Using default license.
Consider to support RAWcooked: https://MediaArea.net/RAWcooked
58 MiB/s, 0.07x realtime
Files are in 419e0c5f-cc55-4863-a948-ee43e522d96b.mkv.RAWcooked/.
telecine:2_mkv administrator$
Here's a few medainfos:
rawcooked MKV
$ mediainfo TEST_CENTRE/TESTING_FOR_ACCESSIONING/RAWCOOKED/TESTS/test_2/2_mkv/419e0c5f-cc55-4863-a948-ee43e522d96b.mkv
General
Unique ID : 88811273795883654014970224587938745385 (0x42D06DDD727DB23E0101BD343BA84C29)
Complete name : TEST_CENTRE/TESTING_FOR_ACCESSIONING/RAWCOOKED/TESTS/test_2/2_mkv/419e0c5f-cc55-4863-a948-ee43e522d96b.mkv
Format : Matroska
Format version : Version 4 / Version 2
File size : 5.66 GiB
Duration : 22 s 667 ms
Overall bit rate : 2 145 Mb/s
Writing application : Lavf58.12.100
Writing library : Lavf58.12.100
ErrorDetectionType : Per level 1
Attachments : RAWcooked reversibility data
Video
ID : 1
Format : FFV1
Format version : Version 3.4
Format settings, GOP : N=1
Codec ID : V_MS/VFW/FOURCC / FFV1
Duration : 22 s 667 ms
Bit rate : 2 103 Mb/s
Width : 2 048 pixels
Height : 1 536 pixels
Display aspect ratio : 4:3
Frame rate mode : Constant
Frame rate : 24.000 FPS
Color space : RGB
Bit depth : 16 bits
Scan type : Progressive
Compression mode : Lossless
Bits/(Pixel*Frame) : 27.851
Stream size : 5.55 GiB (98%)
Writing library : Lavc58.18.100 ffv1
Default : Yes
Forced : No
coder_type : Range Coder
MaxSlicesCount : 64
ErrorDetectionType : Per slice
The original DPX sequence
$ mediainfo TEST_CENTRE/TESTING_FOR_ACCESSIONING/RAWCOOKED/TESTS/test_2/1_seq/test_4_dpx_2/af9628_ifard2017036.19_spike_ivan_the_terrible.0000000.dpx
General
Complete name : TEST_CENTRE/TESTING_FOR_ACCESSIONING/RAWCOOKED/TESTS/test_2/1_seq/test_4_dpx_2/af9628_ifard2017036.19_spike_ivan_the_terrible.0000000.dpx
CompleteName_Last : TEST_CENTRE/TESTING_FOR_ACCESSIONING/RAWCOOKED/TESTS/test_2/1_seq/test_4_dpx_2/af9628_ifard2017036.19_spike_ivan_the_terrible.0000543.dpx
Format : DPX
Format version : Version 2.0
File size : 9.56 GiB
Duration : 22 s 667 ms
Overall bit rate : 3 624 Mb/s
Encoded date : 2017-09-22T13:56:51
Writing library : AvidImageSequencer AMA Plug-in
Video
Format : DPX
Format version : Version 2.0
Format_Compression : Raw
Format settings : Big / Filled A
Duration : 22 s 667 ms
Bit rate : 3 624 Mb/s
Width : 2 048 pixels
Height : 1 536 pixels
Display aspect ratio : 4:3
Frame rate : 24.000 FPS
Color space : RGB
Bit depth : 16 bits
Scan type : Progressive
Compression mode : Lossless
Bits/(Pixel*Frame) : 48.005
Stream size : 9.56 GiB (100%)
Writing library : AvidImageSequencer AMA Plug-in
Encoded date : 2017-09-22T13:56:51
This is the source mediatrace for the last file that is correctly restored by rawcooked
$ mediainfo --File_TestContinuousFileNames=0 --Details=1
TEST_CENTRE/TESTING_FOR_ACCESSIONING/RAWCOOKED/TESTS/test_2/1_seq/test_4_dpx_2/af9628_ifard2017036.19_spike_ivan_the_terrible.0000165.dpx
0000000 -------------------------
0000000 --- DPX, accepted ---
0000000 -------------------------
0000000 Generic section header (1664 bytes)
0000000 File information (768 bytes)
0000000 Magic number: SDPX
0000004 Offset to image data: 2048 (0x00000800)
0000008 Version number of header format: V2.0
0000010 Total image file size: 18876416 (0x01200800)
0000014 Ditto Key: 1 (0x00000001)
0000018 Generic section header length: 1664 (0x00000680)
000001C Industry specific header length: 384 (0x00000180)
0000020 User-defined header length: 0 (0x00000000)
0000024 FileName:
0000088 Creation Date: 2017:09:22:13:57:34Z
00000A0 Creator: AvidImageSequencer AMA Plug-in
0000294 Encryption key: 4294967295 (0xFFFFFFFF)
0000298 Reserved for future use: (104 bytes)
0000300 Image information (640 bytes)
0000300 Image orientation: 0 (0x0000) - Left to right, Top to bottom
0000302 Number of image elements: 1 (0x0001)
0000304 Pixels per line: 2048 (0x00000800)
0000308 Lines per image element: 1536 (0x00000600)
000030C image element (72 bytes)
000030C Data sign: 0 (0x00000000) - unsigned
0000310 Reference low data code value: 0 (0x00000000)
0000314 Reference low quantity represented: 0.000
0000318 Reference high data code value: 65535 (0x0000FFFF)
000031C Reference high quantity represented: 0.000
0000320 Descriptor: 50 (0x32) - R,G,B
0000321 Transfer characteristic: 0 (0x00) -
0000322 Colorimetric specification: 0 (0x00) -
0000323 Bit depth: 16 (0x10) - integer
0000324 Packing: 1 (0x0001) - Filled A
0000326 Encoding: 0 (0x0000) - Raw
0000328 Offset to data: 2048 (0x00000800)
000032C End-of-line padding: 0 (0x00000000)
0000330 End-of-image padding: 0 (0x00000000)
0000334 Description of image element:
0000354 Padding: (504 bytes)
000054C Reserved for future use: (52 bytes)
0000580 Image source information (256 bytes)
0000580 X Offset: 4294967295 (0xFFFFFFFF)
0000584 Y Offset: 4294967295 (0xFFFFFFFF)
0000588 X center: 0.000
000058C Y center: 0.000
0000590 X original size: 4294967295 (0xFFFFFFFF)
0000594 Y original size: 4294967295 (0xFFFFFFFF)
0000598 Source image filename:
00005FC Source image date/time:
0000614 Input device name:
0000634 Input device serial number:
0000654 Border validity (8 bytes)
0000654 XL border: 65535 (0xFFFF)
0000656 XR border: 65535 (0xFFFF)
0000658 YT border: 65535 (0xFFFF)
000065A YB border: 65535 (0xFFFF)
000065C Pixel ratio : horizontal: 4294967295 (0xFFFFFFFF)
0000660 Pixel ratio : vertical: 4294967295 (0xFFFFFFFF)
0000664 Additional source image information (28 bytes)
0000664 X scanned size: 0.000
0000668 Y scanned size: 0.000
000066C Reserved for future use: (20 bytes)
0000680 Industry specific header (384 bytes)
0000680 Motion-picture film information (256 bytes)
0000680 Film mfg. ID code:
0000682 Film type:
0000684 Offset in perfs:
0000686 Prefix:
000068C Count:
0000690 Format - e.g. Academy:
00006B0 Frame position in sequence: 4294967295 (0xFFFFFFFF)
00006B4 Sequence length (frames): 4294967295 (0xFFFFFFFF)
00006B8 Held count (1 = default): 4294967295 (0xFFFFFFFF)
00006BC Frame rate of original (frames/s): 24.000
00006C0 Shutter angle of camera in degrees: 0.000
00006C4 Frame identification - e.g. keyframe:
00006E4 Slate information:
0000748 Reserved for future use: (56 bytes)
0000780 Television information (128 bytes)
0000780 SMPTE time code: 5850112 (0x00594400)
0000784 SMPTE user bits: 4294967295 (0xFFFFFFFF)
0000788 Interlace: 255 (0xFF) - 2:1 interlace
0000789 Field number: 255 (0xFF)
000078A Video signal standard: 255 (0xFF) - Reserved for future high-definition progressive
000078B Zero: 255 (0xFF)
000078C Horizontal sampling rate (Hz): 0.000
0000790 Vertical sampling rate (Hz): 0.000
0000794 Temporal sampling rate or frame rate (Hz): 24.000
0000798 Time offset from sync to first pixel (ms): 0.000
000079C Gamma: 0.000
00007A0 Black level code value: 0.000
00007A4 Black gain: 0.000
00007A8 Breakpoint: 0.000
00007AC Reference white level code value: 65535.000
00007B0 Integration time (s): 0.000
00007B4 Reserved for future use: (76 bytes)
0000800 Image Data (18874368 bytes)
0000800 Data: (18874368 bytes)
1200800 ------------------------
1200800 --- DPX, filling ---
1200800 ------------------------
1200800 -------------------------
1200800 --- DPX, finished ---
1200800 -------------------------
and this is the next file, which along with all subsequence frames, is not restored by rawcooked
$ mediainfo --File_TestContinuousFileNames=0 --Details=1 TEST_CENTRE/TESTING_FOR_ACCESSIONING/RAWCOOKED/TESTS/test_2/1_seq/test_4_dpx_2/af9628_ifard2017036.19_spike_ivan_the_terrible.0000166.dpx
0000000 -------------------------
0000000 --- DPX, accepted ---
0000000 -------------------------
0000000 Generic section header (1664 bytes)
0000000 File information (768 bytes)
0000000 Magic number: SDPX
0000004 Offset to image data: 2048 (0x00000800)
0000008 Version number of header format: V2.0
0000010 Total image file size: 18876416 (0x01200800)
0000014 Ditto Key: 1 (0x00000001)
0000018 Generic section header length: 1664 (0x00000680)
000001C Industry specific header length: 384 (0x00000180)
0000020 User-defined header length: 0 (0x00000000)
0000024 FileName:
0000088 Creation Date: 2017:09:22:13:57:35Z
00000A0 Creator: AvidImageSequencer AMA Plug-in
0000294 Encryption key: 4294967295 (0xFFFFFFFF)
0000298 Reserved for future use: (104 bytes)
0000300 Image information (640 bytes)
0000300 Image orientation: 0 (0x0000) - Left to right, Top to bottom
0000302 Number of image elements: 1 (0x0001)
0000304 Pixels per line: 2048 (0x00000800)
0000308 Lines per image element: 1536 (0x00000600)
000030C image element (72 bytes)
000030C Data sign: 0 (0x00000000) - unsigned
0000310 Reference low data code value: 0 (0x00000000)
0000314 Reference low quantity represented: 0.000
0000318 Reference high data code value: 65535 (0x0000FFFF)
000031C Reference high quantity represented: 0.000
0000320 Descriptor: 50 (0x32) - R,G,B
0000321 Transfer characteristic: 0 (0x00) -
0000322 Colorimetric specification: 0 (0x00) -
0000323 Bit depth: 16 (0x10) - integer
0000324 Packing: 1 (0x0001) - Filled A
0000326 Encoding: 0 (0x0000) - Raw
0000328 Offset to data: 2048 (0x00000800)
000032C End-of-line padding: 0 (0x00000000)
0000330 End-of-image padding: 0 (0x00000000)
0000334 Description of image element:
0000354 Padding: (504 bytes)
000054C Reserved for future use: (52 bytes)
0000580 Image source information (256 bytes)
0000580 X Offset: 4294967295 (0xFFFFFFFF)
0000584 Y Offset: 4294967295 (0xFFFFFFFF)
0000588 X center: 0.000
000058C Y center: 0.000
0000590 X original size: 4294967295 (0xFFFFFFFF)
0000594 Y original size: 4294967295 (0xFFFFFFFF)
0000598 Source image filename:
00005FC Source image date/time:
0000614 Input device name:
0000634 Input device serial number:
0000654 Border validity (8 bytes)
0000654 XL border: 65535 (0xFFFF)
0000656 XR border: 65535 (0xFFFF)
0000658 YT border: 65535 (0xFFFF)
000065A YB border: 65535 (0xFFFF)
000065C Pixel ratio : horizontal: 4294967295 (0xFFFFFFFF)
0000660 Pixel ratio : vertical: 4294967295 (0xFFFFFFFF)
0000664 Additional source image information (28 bytes)
0000664 X scanned size: 0.000
0000668 Y scanned size: 0.000
000066C Reserved for future use: (20 bytes)
0000680 Industry specific header (384 bytes)
0000680 Motion-picture film information (256 bytes)
0000680 Film mfg. ID code:
0000682 Film type:
0000684 Offset in perfs:
0000686 Prefix:
000068C Count:
0000690 Format - e.g. Academy:
00006B0 Frame position in sequence: 4294967295 (0xFFFFFFFF)
00006B4 Sequence length (frames): 4294967295 (0xFFFFFFFF)
00006B8 Held count (1 = default): 4294967295 (0xFFFFFFFF)
00006BC Frame rate of original (frames/s): 24.000
00006C0 Shutter angle of camera in degrees: 0.000
00006C4 Frame identification - e.g. keyframe:
00006E4 Slate information:
0000748 Reserved for future use: (56 bytes)
0000780 Television information (128 bytes)
0000780 SMPTE time code: 5850113 (0x00594401)
0000784 SMPTE user bits: 4294967295 (0xFFFFFFFF)
0000788 Interlace: 255 (0xFF) - 2:1 interlace
0000789 Field number: 255 (0xFF)
000078A Video signal standard: 255 (0xFF) - Reserved for future high-definition progressive
000078B Zero: 255 (0xFF)
000078C Horizontal sampling rate (Hz): 0.000
0000790 Vertical sampling rate (Hz): 0.000
0000794 Temporal sampling rate or frame rate (Hz): 24.000
0000798 Time offset from sync to first pixel (ms): 0.000
000079C Gamma: 0.000
00007A0 Black level code value: 0.000
00007A4 Black gain: 0.000
00007A8 Breakpoint: 0.000
00007AC Reference white level code value: 65535.000
00007B0 Integration time (s): 0.000
00007B4 Reserved for future use: (76 bytes)
0000800 Image Data (18874368 bytes)
0000800 Data: (18874368 bytes)
1200800 ------------------------
1200800 --- DPX, filling ---
1200800 ------------------------
1200800 -------------------------
1200800 --- DPX, finished ---
1200800 -------------------------
Hi - I am having an issue with one DPX sequence only. In all my testing, rawcooked makes a fully reversible sequence - it just appears that there are issues with the Matroska file itself when read in ffmpeg.
There appears to be two issues:
I tested this with Rawcooked 18.10 on Windows 7 and OSX, and I just finished testing on OSX El Capitan with rawcooked git master. The same issue occurs. I tried to replicate the issue with just 25000 or 50000 files and the issue did not appear. I was hoping to figure out at what point the issue occurs.
This is what appears with 4.1.1 which was the version of ffmpeg that rawcooked used
$ ffmpeg -i /Volumes/kieran_tests/oe13172/f69fb94e-412b-45d8-bb21-1988399e2ccc/objects/f69fb94e-412b-45d8-bb21-1988399e2ccc.mkv.RAWcooked/24e6bb3c-fd29-47bc-bf3e-b4d64ed0c23f.mkv
ffmpeg version 4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags=-I/System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers/ --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr
libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
libpostproc 55. 3.100 / 55. 3.100
[matroska,webm @ 0x7f9e5c000000] Invalid length 0x36b16ede > 0x10000000 for syntax element 5
[matroska,webm @ 0x7f9e5c000000] Duplicate element
[matroska,webm @ 0x7f9e5c000000] Read error at pos. 187 (0xbb)
[matroska,webm @ 0x7f9e5c000000] Duplicate element
Last message repeated 4 times
[matroska,webm @ 0x7f9e5c000000] Invalid length 0x36b16ede > 0x10000000 for syntax element 5
[matroska,webm @ 0x7f9e5c000000] Duplicate element
[matroska,webm @ 0x7f9e5c000000] incomplete attachment
Last message repeated 1 times
[matroska,webm @ 0x7f9e5c000000] Could not find codec parameters for stream 1 (Video: ffv1 (FFV1 / 0x31564646), none, 2048x1536): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, matroska,webm, from '/Volumes/kieran_tests/oe13172/f69fb94e-412b-45d8-bb21-1988399e2ccc/objects/f69fb94e-412b-45d8-bb21-1988399e2ccc.mkv.RAWcooked/24e6bb3c-fd29-47bc-bf3e-b4d64ed0c23f.mkv':
Metadata:
ENCODER : Lavf58.20.100
Duration: 01:28:52.71, start: 0.000000, bitrate: 350886 kb/s
Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gbrp10le, 2048x1536, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
Metadata:
ENCODER : Lavc58.35.100 ffv1
DURATION : 01:28:52.709000000
Stream #0:1: Video: ffv1 (FFV1 / 0x31564646), none, 2048x1536, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
Metadata:
ENCODER : Lavc58.35.100 ffv1
DURATION : 01:28:52.709000000
At least one output file must be specified
and here is what appears with ffmpeg git master - it seems like the same errors combined with the general Matroska errors that I reported here: http://ffmpeg.org/pipermail/ffmpeg-user/2019-July/044725.html
$ ./ffmpeg -i /Volumes/kieran_tests/oe13172/f69fb94e-412b-45d8-bb21-1988399e2ccc/objects/f69fb94e-412b-45d8-bb21-1988399e2ccc.mkv.RAWcooked/24e6bb3c-fd29-47bc-bf3e-b4d64ed0c23f.mkv
ffmpeg version N-94421-gb3b7523feb Copyright (c) 2000-2019 the FFmpeg developers
built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
configuration:
libavutil 56. 32.100 / 56. 32.100
libavcodec 58. 55.100 / 58. 55.100
libavformat 58. 30.100 / 58. 30.100
libavdevice 58. 9.100 / 58. 9.100
libavfilter 7. 58.100 / 7. 58.100
libswscale 5. 6.100 / 5. 6.100
libswresample 3. 6.100 / 3. 6.100
[matroska,webm @ 0x7ffd13808600] Invalid length 0x36b16ede > 0x10000000 for element with ID 0x465C at 0x31c
[matroska,webm @ 0x7ffd13808600] Duplicate element
[matroska,webm @ 0x7ffd13808600] 0x00 at pos 186 (0xba) invalid as first byte of an EBML number
[matroska,webm @ 0x7ffd13808600] Duplicate element
[matroska,webm @ 0x7ffd13808600] Element at 0x61 ending at 0x3c4dbb8c53abee exceeds containing master element ending at 0x140d
[matroska,webm @ 0x7ffd13808600] Duplicate element
[matroska,webm @ 0x7ffd13808600] Element at 0x70 ending at 0x8b4dbb8e5423 exceeds containing master element ending at 0x141c
[matroska,webm @ 0x7ffd13808600] Element at 0x7f ending at 0x16b1f4 exceeds containing master element ending at 0x142b
[matroska,webm @ 0x7ffd13808600] Element at 0x90 ending at 0x1675c6 exceeds containing master element ending at 0x143c
[matroska,webm @ 0x7ffd13808600] Invalid length 0x36b16ede > 0x10000000 for element with ID 0x465C at 0x31c
[matroska,webm @ 0x7ffd13808600] Duplicate element
[matroska,webm @ 0x7ffd13808600] incomplete attachment
Last message repeated 1 times
[matroska,webm @ 0x7ffd13808600] Could not find codec parameters for stream 1 (Video: ffv1 (FFV1 / 0x31564646), none, 2048x1536): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, matroska,webm, from '/Volumes/kieran_tests/oe13172/f69fb94e-412b-45d8-bb21-1988399e2ccc/objects/f69fb94e-412b-45d8-bb21-1988399e2ccc.mkv.RAWcooked/24e6bb3c-fd29-47bc-bf3e-b4d64ed0c23f.mkv':
Metadata:
ENCODER : Lavf58.20.100
Duration: 01:28:52.71, start: 0.000000, bitrate: 350886 kb/s
Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gbrp10le, 2048x1536, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
Metadata:
ENCODER : Lavc58.35.100 ffv1
DURATION : 01:28:52.709000000
Stream #0:1: Video: ffv1 (FFV1 / 0x31564646), none, 2048x1536, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
Metadata:
ENCODER : Lavc58.35.100 ffv1
DURATION : 01:28:52.709000000
At least one output file must be specified
I am not sure if these are relevant, but I'll attach the ffmpeg log of an earlier run, as well as the rawcooked_reversibility_data which I copy/pasted during the rawcooking. I do not know how to extract the attachment in the MKV file seeing as ffmpeg sees it as a video track.
Logs: https://we.tl/t-MfiegwDn4r
metadata - mediainfo/mediatrace of MKV file as well as the source sequence (in supplemental) https://we.tl/t-rNwSDKx9EP
In line https://github.com/MediaArea/RAWcooked/blob/master/Source/CLI/Main.cpp#L278 the full path to rawcooked.1 should be given, in order to work by drag & dropping the file into the Terminal as well.
From #154 (comment)
I would indeed suggest to test the continuity of the numbers. An additional flag could be set to either give an error message (default) or make different streams, when there are missing or «missing» frames. Would this be difficult to implement?
When encoding DPX (gbrp10le), I get warnings when encoding/decoding with rawcooked.
Encoding:
[matroska @ 0x1391ee0] Codec for stream 1 does not use global headers but container format requires global headers
[matroska @ 0x1391ee0] Codec for stream 2 does not use global headers but container format requires global headers
Decoding/Playback:
The tags at index 2 refer to a non-existent track 2.= 0B f=0/0
The tags at index 3 refer to a non-existent track 3.
The tags at index 5 refer to a non-existent track 2.
The tags at index 6 refer to a non-existent track 3.
[matroska,webm @ 0x7f702c000940] Could not find codec parameters for stream 1 (Attachment: none): unknown codec
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[matroska,webm @ 0x7f702c000940] Could not find codec parameters for stream 2 (Attachment: none): unknown codec
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Stream layout:
Input #0, matroska,webm, from 'test2.mkv':
Metadata:
ENCODER : Lavf56.40.101
Duration: 00:00:02.08, start: 0.000000, bitrate: 13027 kb/s
Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gbrp10le, 1920x1080, SAR 1:1 DAR 16:9, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
Metadata:
ENCODER : Lavc56.60.100 ffv1
DURATION : 00:00:02.084000000
Stream #0:1: Attachment: none
Metadata:
filename : test2/md5sums.md5
mimetype : application/octet-stream
Stream #0:2: Attachment: none
Metadata:
filename : RAWcooked reversibility data
mimetype : application/octet-stream
1.12 M-V: 0.003 fd= 2 aq= 0KB vq= 863KB sq= 0B f=0/0
Hi,
what is the status of audio encoding, did anybody encode anything larger than the testing files?
Because I found out that RAWcooked cannot encode anything larger than 6704894 bytes, regardless of type ('0x1' or '0xfffe') or bit depth (16 or 24), sampling (44100 or 48000), ChannelMask (mono or 5.1) etc..
It always ends up on this boundary.
One example:
$ ffmpeg -f lavfi -i sine -c:a pcm_s16le -t 76.018333 bla.wav
..
$ ls -l bla.wav
-rw-rw-r--. 1 user user 6704894 Aug 3 19:02 bla.wav
$ rawcooked bla.wav
..
(works)
$ ls -l bla.wav.mkv
-rw-rw-r--. 1 user user 918246 Aug 3 19:03 bla.wav.mkv
$ ffmpeg -f lavfi -i sine -c:a pcm_s16le -t 76.018334 bla.wav
..
$ ls -l bla.wav
-rw-rw-r--. 1 user user 6704896 Aug 3 19:04 bla.wav
$ rawcooked bla.wav
bla.wav is not small, expected to be an attachment? Please contact [email protected] if you want support of such file.
I was wondering if it would be possible to add an option to RAWCooked that would invoke the -report option in FFmpeg? This would allow a logfile to be created for the transcode. This would be extremely useful for an archive to store in an archival package as evidence of the normalisation process.
When encoding (ffmpeg-generated) rgb48le DPX files, the resulting FFV1/MKV file is stored as "yuv444p16le". Also tested with rgb48be file (Source: DaVinci/@retokromer).
Is this behavior intentionally?
Additionally, rawcooked outputs the following message during encoding:
Untested FFV1-SLICE-SliceContent:1, please contact [email protected] if you want support of such file
ffprobe outputs (sorry, not uncut 😉):
Stream #0:0: Video: dpx, rgb48le, 2048x1536 [SAR 1:1 DAR 4:3], 25 tbr, 25 tbn, 25 tbc
Input #0, matroska,webm, from 'tib-A_01_16bpc.mkv':
Metadata:
ENCODER : Lavf56.40.101
Duration: 00:00:05.21, start: 0.000000, bitrate: 1128931 kb/s
Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv444p16le, 2048x1536, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
Metadata:
ENCODER : Lavc56.60.100 ffv1
DURATION : 00:00:05.209000000
Stream #0:1: Attachment: none
Metadata:
filename : RAWcooked reversibility data
mimetype : application/octet-stream
Version: RAWcooked 18.06.20180630
I was wondering what format the actual reversibility attachment is? I extracted it with mkvextract and looked at it in a hex editor, but aside from some RAWCooked version info at the start, I can't figure it out. I assumed that it was perhaps a compress (with what?) header of the first DPX file?
My colleague @raecasey reported an error with ffmpeg/rawcooked regarding some weird frame sizes.
We have a sequence that is 3512x2640 (you can replicate with ffmpeg -f lavfi -i testsrc=size=3512x2640 -pix_fmt rgb48be -vframes 1 weird.dpx ) and the rawcooked job fails due to an ffmpeg error regarding slices. If I manually specify a slice count other than 128 then all is well. I tried 256 and 16 I think. So perhaps rawcooked needs to choose a different slice count for some rasters? :
We are using an older version of rawcooked in production but I compiled from git-master on OSX El Capitan and saw the same issue.
$ rawcooked --file weird.dpx
License detected but decoding not supported in this version.
Using default license.
Consider to support RAWcooked: https://MediaArea.net/RAWcooked
Track 1:
weird.dpx
DPX/Raw/RGB/16bit/BE
*** This input format flavor is not supported by the current license key. ***
One or more requested features are not supported with the current license key.
Please contact [email protected] for a quote or a temporary key.
Ignoring the license for the moment.
ffmpeg version 4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags=-I/System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers/ --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr
libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
libpostproc 55. 3.100 / 55. 3.100
[dpx_pipe @ 0x7fd20080c800] Stream #0: not enough frames to estimate rate; consider increasing probesize
Input #0, dpx_pipe, from 'weird.dpx':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: dpx, rgb48be, 3512x2640 [SAR 1:1 DAR 439:330], 24 tbr, 24 tbn, 24 tbc
File 'weird.dpx.mkv' already exists. Overwrite ? [y/N] y
Stream mapping:
Stream #0:0 -> #0:0 (dpx (native) -> ffv1 (native))
File weird.dpx.rawcooked_reversibility_data -> Stream #0:1
Press [q] to stop, [?] for help
[ffv1 @ 0x7fd203001e00] Unsupported number 128 of slices requested, please specify a supported number with -slices (ex:4,6,9,12,16, ...)
Error initializing output stream 0:0 -- Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Conversion failed!
I suggest to add a flag to the CLI rawcooked, allowing to choose a specific encoder or decoder to use:
I'm running through some --check / --hash tests and everything is running perfectly except for the check -o command which is throwing up this error:
C:\RAWcooked Beta>rawcooked --check "T:\COMPLETED\Scan01.mkv" -o "T:\COMPLETED\Scan01"
Using default license.
Consider to support RAWcooked: https://MediaArea.net/RAWcooked
114 MiB/s, 0.94x realtime
Reversability was checked, issues detected, see below.
Error: can not open file for reading.
Scan01/24198_01_000000.dpx
Scan01/24198_01_000001.dpx
Scan01/24198_01_000002.dpx
Scan01/24198_01_000003.dpx
Scan01/24198_01_000004.dpx
Scan01/24198_01_000005.dpx
Scan01/24198_01_000006.dpx
Scan01/24198_01_000007.dpx
Scan01/24198_01_000008.dpx
Scan01/24198_01_000009.dpx
...
Error: file names listed in source but not present.
Scan01/24198_01_000000.dpx
Scan01/24198_01_000001.dpx
Scan01/24198_01_000002.dpx
Scan01/24198_01_000003.dpx
Scan01/24198_01_000004.dpx
Scan01/24198_01_000005.dpx
Scan01/24198_01_000006.dpx
Scan01/24198_01_000007.dpx
Scan01/24198_01_000008.dpx
Scan01/24198_01_000009.dpx
...
Is the value 0x21 correct at line 29
RAWcooked/Source/Lib/RAWcooked/RAWcooked.cpp
Lines 28 to 29 in fb26f2c
0x22?I've been making initial tests of RawCooked and had a test scan directory containing DPX files and a single WAV. Using an older version of RAWCooked I attempted a MKV conversion on the /Scan01/ folder and received the error:
"Untested WAV FormatTag ChannelMask, please contact [email protected]."
I managed to convert the directory by redirecting the rawcooked command to the first .dpx file in the /Scan01/ directory. This MKV would not play in my Mac VLC player version 2.0.9, which I understand is too old a version now so I'll upgrade later today and test again. Upon re-expansion of the MKV I found that the WAV file was missing, and the resulting directory .01 GB smaller in size. The MKV was significantly smaller file size than the Scan 01 folder though, and a few random MD5 checksum tests of DPX files from original directory and MKV expanded directory were perfect matches.
When starting rawcooked with a "normal" FFV1/MKV file (=not produced by rawcooked), rawcooked processes the whole file (CPU at 100%) - but only at the very end says:
Untested More video frames in source content than saved frame headers in reversibility data, please contact [email protected] if you want support of such file
Would it be possible to tell the user earlier that nothing will be output?
Or even more daring:
Would it be possible to actually output a generated DPX sequence for such files?
(But that's wishlist of course 😉 )
Hi Everyone,
We've been testing out new RAWcooked conformance checking features, and we seem to be getting a whole lotta "non-conforming ditto key" messages. This is kinda how Genevieve and feel right about now:
Perhaps this is less a RAWcooked issue, and more a question to the larger cooking community, but does anyone out there have any opinions—should we be overly concerned about this? SMPTE spec and FADGI recommendations are pretty opaque when it comes to ditto key, and "Inconsistent. Sometimes used correctly, sometimes blank" feels like license to not care too much. Thoughts?
Ben
Users and potential users e.g. @retokromer @kieranjol @stephenmcconnachie, do you expect empty directory names to be kept i.e. empty directories are created during reverse?
Make a new alpha release 18.07 and update the Homebrew/Linuxbrew formula.
In the main GitHub page add
instead of Not yet ready for production use, for tests only. Sorry, I cannot open myself a PR for this.
This has low-to-no priority at all, but might be interesting for some:
I just tested a codec roundtrip:
Decompression back to original DPX files required to convert uncompressed back to FFV1.
Would it be possible to decode non-FFV1 directly, too (e.g. whatever ffmpeg supports)?
This would allow future codec format migrations, while keeping the DPX-reversibility 😎 😄
(Just curious, if you have time and interest to answer this 😉)
I presume that the description on
and come from a copy-and-paste.Spawned from issue #118 😄
Would it be possible to actually output a generated DPX sequence for regular FFV1/MKV files that were not created with rawcooked?
Basically the equivalent of:
ffmpeg -i INPUT.mkv %06d.dpxThis would reduce possible parametrization mistakes compared to ffmpeg (happens to everyone of use every now and then) - and might be easier/more welcoming for some users 😁
Priority: Frozen until sponsorship-request 😉
Hi,
I'm only starting to do some testing, so far, RAWCooked is very very slick :)
I was wondering if there is any kind of losslessness check involved in the DPX -> FFV1 process?
it does not appear to be the case. I think that some sort of PASS/FAIL test could increase trust in RAWCooked.
I am most likely going to write a python script that will use RAWCooked to encode, then it will perform a validation in the form of reconstructing the DPX sequence and checking if the whole file MD5s are the same.
I might also look into framemd5 validation, but it might not be needed if the goal of RAWCOoked is to have whole-file losslessness.
For the moment, I rely on the idea that the input DPX file is conform to specification, including annexes.
For DPX 10-bit Filled A, the pixel format should be, per 32-bit:
Up to now it was the case for all the files I tested and all the files tested by users, but one of the users found 1 DPX file (in a complete sequence!) with bits 1 & 0 are not zeroed! Obviously a corrupted file but it exists.
I could test and store the padding bits, but it would be double the read on the disk:
I am reluctant on that, as it is an extra read (so costly in time) for only buggy files but without that we could not guaranty the same hash for the file.
Another way is to force the verification of the reversibility after compression, so we are more sure it is reversible but similar issue about speed (double read of the input) so wondering if it should be activated by default.
Is something worth it to be changed here?
Thoughts from users?
I'm getting an error, "Error: unsupported DPX Offset to image data in bytes." when I attempt to process some DPX files.
See below. Please let me know if you need any additional info or samples.
rawcooked --version
RAWcooked 18.10.1.20190608
$ rawcooked --check-padding dpx_folder
Using default license.
Error: unsupported DPX Offset to image data in bytes.
mediainfo --Details=1 dpx_00000000.dpx
0000000000 -------------------------
0000000000 --- DPX, accepted ---
0000000000 -------------------------
0000000000 Generic section header (1664 bytes)
0000000000 File information (768 bytes)
0000000000 Magic number: SDPX
0000000004 Offset to image data: 2480 (0x000009B0)
0000000008 Version number of header format: V1.0
0000000010 Total image file size: 16378880 (0x00F9EC00)
0000000014 Ditto Key: 1 (0x00000001)
0000000018 Generic section header length: 1664 (0x00000680)
000000001C Industry specific header length: 384 (0x00000180)
0000000020 User-defined header length: 432 (0x000001B0)
0000000024 FileName: dpx_00000000.dpx
0000000088 Creation Date: 2019-06-06T13:44:27Z
00000000A0 Creator: US, Some/Info
0000000104 Project: Item ID, local, 1234; Collection ID, local, 5678
0000000294 Encryption key: 4294967295 (0xFFFFFFFF)
0000000298 Reserved for future use: (104 bytes)
0000000300 Image information (640 bytes)
0000000300 Image orientation: 0 (0x0000) - Left to right, Top to bottom
0000000302 Number of image elements: 1 (0x0001)
0000000304 Pixels per line: 2336 (0x00000920)
0000000308 Lines per image element: 1752 (0x000006D8)
000000030C image element (72 bytes)
000000030C Data sign: 0 (0x00000000) - unsigned
0000000310 Reference low data code value: 0 (0x00000000)
0000000314 Reference low quantity represented: 0.000
0000000318 Reference high data code value: 1023 (0x000003FF)
000000031C Reference high quantity represented: 2.047
0000000320 Descriptor: 50 (0x32) - R,G,B
0000000321 Transfer characteristic: 1 (0x01) - Printing density
0000000322 Colorimetric specification: 2 (0x02) -
0000000323 Bit depth: 10 (0x0A) - integer
0000000324 Packing: 1 (0x0001) - Filled A
0000000326 Encoding: 0 (0x0000) - Raw
0000000328 Offset to data: 8192 (0x00002000)
000000032C End-of-line padding: 0 (0x00000000)
0000000330 End-of-image padding: 0 (0x00000000)
0000000334 Description of image element: IMAGE DESCRIPTION DATA
0000000354 Padding: (504 bytes)
000000054C Reserved for future use: (52 bytes)
0000000580 Image source information (256 bytes)
0000000580 X Offset: 0 (0x00000000)
0000000584 Y Offset: 0 (0x00000000)
0000000588 X center: 0.000
000000058C Y center: 0.000
0000000590 X original size: 2336 (0x00000920)
0000000594 Y original size: 1752 (0x000006D8)
0000000598 Source image filename:
00000005FC Source image date/time:
0000000614 Input device name: Some Device Info
0000000634 Input device serial number: SNSA-1234
0000000654 Border validity (8 bytes)
0000000654 XL border: 65535 (0xFFFF)
0000000656 XR border: 65535 (0xFFFF)
0000000658 YT border: 65535 (0xFFFF)
000000065A YB border: 65535 (0xFFFF)
000000065C Pixel ratio : horizontal: 16777216 (0x01000000)
0000000660 Pixel ratio : vertical: 16777216 (0x01000000)
0000000664 Additional source image information (28 bytes)
0000000664 X scanned size: 0.000
0000000668 Y scanned size: 0.000
000000066C Reserved for future use: (20 bytes)
0000000680 Industry specific header (384 bytes)
0000000680 Motion-picture film information (256 bytes)
0000000680 Film mfg. ID code:
0000000682 Film type:
0000000684 Offset in perfs:
0000000686 Prefix:
000000068C Count:
0000000690 Format - e.g. Academy:
00000006B0 Frame position in sequence: 0 (0x00000000)
00000006B4 Sequence length (frames): 16777216 (0x01000000)
00000006B8 Held count (1 = default): 16777216 (0x01000000)
00000006BC Frame rate of original (frames/s): 18.000
00000006C0 Shutter angle of camera in degrees: 0.000
00000006C4 Frame identification - e.g. keyframe:
00000006E4 Slate information: SLATE INFO
0000000748 Reserved for future use: (56 bytes)
0000000780 Television information (128 bytes)
0000000780 SMPTE time code: 0 (0x00000000)
0000000784 SMPTE user bits: 0 (0x00000000)
0000000788 Interlace: 255 (0xFF) - 2:1 interlace
0000000789 Field number: 255 (0xFF)
000000078A Video signal standard: 255 (0xFF) - Reserved for future high-definition progressive
000000078B Zero: 255 (0xFF)
000000078C Horizontal sampling rate (Hz): 0.000
0000000790 Vertical sampling rate (Hz): 0.000
0000000794 Temporal sampling rate or frame rate (Hz): 18.000
0000000798 Time offset from sync to first pixel (ms): 0.000
000000079C Gamma: 0.000
00000007A0 Black level code value: 0.000
00000007A4 Black gain: 0.000
00000007A8 Breakpoint: 0.000
00000007AC Reference white level code value: 0.000
00000007B0 Integration time (s): 0.000
00000007B4 Reserved for future use: (76 bytes)
0000000800 User defined header (432 bytes)
0000000800 User identification: FADGI process history
0000000820 User defined: (400 bytes)
00000009B0 Image Data (16370688 bytes)
00000009B0 Data: (16370688 bytes)
0000F9D5B0 ------------------------
0000F9D5B0 --- DPX, filling ---
0000F9D5B0 ------------------------
0000F9D5B0 -------------------------
0000F9D5B0 --- DPX, finished ---
0000F9D5B0 -------------------------
Would be nice if the original timestamps of the source files could be preserved and restored when unpacking.
🍰
When calling rawcooked with an argument that does not yet exist, the following message appears:
Option -xxx not yet tested, please contact [email protected] if you want support of such option
I personally found this a bit confusing, since the same message appeared, regardless if my parameter made any sense whatsoever.
Would it be possible to show this message only for args that might really be a valid, but not-yet-tested option? 😄
And for everything else, you may output something like "Invalid option" - so noone's going to hunt ghosts? 😉
build: RAWcooked 18.06.20180616
I am using Windows 7 and rawcooked 18.10.1 . I think I've never tried RAWcooked in Win7 before, only Win10, Ubuntu and OSX.
When I just type rawcooked, all seems well, but when I use rawcooked -hor supply a file as input, the program crashes and here is the error report that Windows 7 provides:
C:\Users\hi>rawcooked
Usage: "rawcooked [options] DirectoryName [options]"
"rawcooked --help" for displaying more information
or "man rawcooked" for displaying the man page
C:\Users\hi>rawcooked -h
Problem signature:
Problem Event Name: APPCRASH
Application Name: RAWcooked.exe
Application Version: 0.0.0.0
Application Timestamp: 5beb1aa0
Fault Module Name: RAWcooked.exe
Fault Module Version: 0.0.0.0
Fault Module Timestamp: 5beb1aa0
Exception Code: c000001d
Exception Offset: 000000000001262e
OS Version: 6.1.7601.2.1.0.256.1
Locale ID: 6153
Additional Information 1: 8833
Additional Information 2: 8833468c23e2f968ed35f4b3a75fc117
Additional Information 3: 346d
Additional Information 4: 346d185dc088fbc65b53fa650d291f84
Read our privacy statement online:
http://go.microsoft.com/fwlink/?linkid=104288&clcid=0x0409
If the online privacy statement is not available, please read our privacy statement offline:
C:\Windows\system32\en-US\erofflps.txt
Apologies in advance for such a terrible, un-reproducible issue - I mostly just would like some guidance on how best to figure out what's happening with this issue, is there some way to get some verbose info on what might be happening under the hood?
I'm un-rawcooked a small 4K sequence, which is 98 gigs. When the un-rawcooking got to 71.46% it stalled, and I noticed that all of our memory was being used by Rawcooked - 48 gigs. This is on Win 7 Pro on a HP Z800 with dual Xeon Cpus.
My colleague Gavin Martin was noticing similar issues on OSX when unrawcooking large sequences (sorry, we've no significant further info on this, but we'll document things better next time it happens) - all memory would be used but at least the rawcooked process would continue, unlike on Windows when it stalls. My hope is that the current Windows process will continue eventually, but I was wondering how best to give you more info.
I will try to reproduce with lavfi input as that could be the best way to go.
The windows version is RAWcooked 18.10.1.20190112
Format : DPX
Format version : Version 1.0
File size : 98.4 GiB
Duration : 1 min 17 s
Overall bit rate : 10.9 Gb/s
Encoded date : 2019-01-30T13:21:08GMT
Writing library : daVinci
Video
Format : DPX
Format version : Version 1.0
Format_Compression : Raw
Format settings : Big / Packed
Duration : 1 min 17 s
Bit rate : 10.9 Gb/s
Width : 4 096 pixels
Height : 3 072 pixels
Display aspect ratio : 4:3
Frame rate : 24.000 FPS
Color space : RGB
Bit depth : 12 bits
Scan type : Progressive
Compression mode : Lossless
Bits/(Pixel*Frame) : 36.005
Stream size : 98.4 GiB (100%)
Writing library : daVinci
Encoded date : 2019-01-30T13:21:08GMT
Color primaries : Printing density
General
Unique ID : 2766380268760127251511
73 (0xD01E8BC4CCA9238E98A88BCC677471E1)
Complete name : F:\test_seq\21da0554-1
62515a2c2ed.mkv
Format : Matroska
Format version : Version 4
File size : 59.8 GiB
Duration : 1 min 17 s
Overall bit rate mode : Variable
Overall bit rate : 6 609 Mb/s
Writing application : Lavf58.25.100
Writing library : Lavf58.25.100
ErrorDetectionType : Per level 1
Attachments : RAWcooked reversibilit
Video
ID : 1
Format : FFV1
Format version : Version 3.4
Format settings, GOP : N=1
Codec ID : V_MS/VFW/FOURCC / FFV1
Duration : 1 min 17 s
Bit rate mode : Variable
Bit rate : 6 478 Mb/s
Width : 4 096 pixels
Height : 3 072 pixels
Display aspect ratio : 4:3
Frame rate mode : Constant
Frame rate : 24.000 FPS
Color space : RGB
Bit depth : 12 bits
Scan type : Progressive
Compression mode : Lossless
Bits/(Pixel*Frame) : 21.450
Stream size : 58.6 GiB (98%)
Writing library : Lavc58.43.101 ffv1
Default : Yes
Forced : No
coder_type : Range Coder
MaxSlicesCount : 256
ErrorDetectionType : Per slice
When using the option -level 1 on regular RGB48 content, I receive an error:
Verbose (-v like FFmpeg) option should display more details about the work done:
I'm getting an error, "Error: undecodable DPX Expected data size is bigger than real file size." when I attempt to process some DPX files.
See below. Please let me know if you need any additional info or samples.
rawcooked --version
RAWcooked 18.10.1.20190608
$ rawcooked --check-padding dpx_folder
[...]
Error: undecodable DPX Expected data size is bigger than real file size.
Error: unsupported DPX Offset to image data in bytes.
Info from terminal on Mac:
ls -al dpx_00000000.dpx
-rwxrwxrwx@ 1 some one 16373168 Jun 11 17:45 dpx_00000000.dpx
Windows properties:
Windows "Size" 16373168
Windows "Size on disk: 16373760
mediainfo --Details=1 dpx_00000000.dpx
0000000000 -------------------------
0000000000 --- DPX, accepted ---
0000000000 -------------------------
0000000000 Generic section header (1664 bytes)
0000000000 File information (768 bytes)
0000000000 Magic number: SDPX
0000000004 Offset to image data: 2480 (0x000009B0)
0000000008 Version number of header format: V1.0
0000000010 Total image file size: 16378880 (0x00F9EC00)
0000000014 Ditto Key: 1 (0x00000001)
0000000018 Generic section header length: 1664 (0x00000680)
000000001C Industry specific header length: 384 (0x00000180)
0000000020 User-defined header length: 432 (0x000001B0)
0000000024 FileName: dpx_00000000.dpx
0000000088 Creation Date: 2019-06-06T13:44:27Z
00000000A0 Creator: US, Some/Info
0000000104 Project: Item ID, local, 1234; Collection ID, local, 5678
0000000294 Encryption key: 4294967295 (0xFFFFFFFF)
0000000298 Reserved for future use: (104 bytes)
0000000300 Image information (640 bytes)
0000000300 Image orientation: 0 (0x0000) - Left to right, Top to bottom
0000000302 Number of image elements: 1 (0x0001)
0000000304 Pixels per line: 2336 (0x00000920)
0000000308 Lines per image element: 1752 (0x000006D8)
000000030C image element (72 bytes)
000000030C Data sign: 0 (0x00000000) - unsigned
0000000310 Reference low data code value: 0 (0x00000000)
0000000314 Reference low quantity represented: 0.000
0000000318 Reference high data code value: 1023 (0x000003FF)
000000031C Reference high quantity represented: 2.047
0000000320 Descriptor: 50 (0x32) - R,G,B
0000000321 Transfer characteristic: 1 (0x01) - Printing density
0000000322 Colorimetric specification: 2 (0x02) -
0000000323 Bit depth: 10 (0x0A) - integer
0000000324 Packing: 1 (0x0001) - Filled A
0000000326 Encoding: 0 (0x0000) - Raw
0000000328 Offset to data: 8192 (0x00002000)
000000032C End-of-line padding: 0 (0x00000000)
0000000330 End-of-image padding: 0 (0x00000000)
0000000334 Description of image element: IMAGE DESCRIPTION DATA
0000000354 Padding: (504 bytes)
000000054C Reserved for future use: (52 bytes)
0000000580 Image source information (256 bytes)
0000000580 X Offset: 0 (0x00000000)
0000000584 Y Offset: 0 (0x00000000)
0000000588 X center: 0.000
000000058C Y center: 0.000
0000000590 X original size: 2336 (0x00000920)
0000000594 Y original size: 1752 (0x000006D8)
0000000598 Source image filename:
00000005FC Source image date/time:
0000000614 Input device name: Some Input Device Info
0000000634 Input device serial number: SNSA-123
0000000654 Border validity (8 bytes)
0000000654 XL border: 65535 (0xFFFF)
0000000656 XR border: 65535 (0xFFFF)
0000000658 YT border: 65535 (0xFFFF)
000000065A YB border: 65535 (0xFFFF)
000000065C Pixel ratio : horizontal: 16777216 (0x01000000)
0000000660 Pixel ratio : vertical: 16777216 (0x01000000)
0000000664 Additional source image information (28 bytes)
0000000664 X scanned size: 0.000
0000000668 Y scanned size: 0.000
000000066C Reserved for future use: (20 bytes)
0000000680 Industry specific header (384 bytes)
0000000680 Motion-picture film information (256 bytes)
0000000680 Film mfg. ID code:
0000000682 Film type:
0000000684 Offset in perfs:
0000000686 Prefix:
000000068C Count:
0000000690 Format - e.g. Academy:
00000006B0 Frame position in sequence: 0 (0x00000000)
00000006B4 Sequence length (frames): 16777216 (0x01000000)
00000006B8 Held count (1 = default): 16777216 (0x01000000)
00000006BC Frame rate of original (frames/s): 18.000
00000006C0 Shutter angle of camera in degrees: 0.000
00000006C4 Frame identification - e.g. keyframe:
00000006E4 Slate information: SLATE INFO
0000000748 Reserved for future use: (56 bytes)
0000000780 Television information (128 bytes)
0000000780 SMPTE time code: 0 (0x00000000)
0000000784 SMPTE user bits: 0 (0x00000000)
0000000788 Interlace: 255 (0xFF) - 2:1 interlace
0000000789 Field number: 255 (0xFF)
000000078A Video signal standard: 255 (0xFF) - Reserved for future high-definition progressive
000000078B Zero: 255 (0xFF)
000000078C Horizontal sampling rate (Hz): 0.000
0000000790 Vertical sampling rate (Hz): 0.000
0000000794 Temporal sampling rate or frame rate (Hz): 18.000
0000000798 Time offset from sync to first pixel (ms): 0.000
000000079C Gamma: 0.000
00000007A0 Black level code value: 0.000
00000007A4 Black gain: 0.000
00000007A8 Breakpoint: 0.000
00000007AC Reference white level code value: 0.000
00000007B0 Integration time (s): 0.000
00000007B4 Reserved for future use: (76 bytes)
0000000800 User defined header (432 bytes)
0000000800 User identification: FADGI process history
0000000820 User defined: (400 bytes)
00000009B0 Image Data (16370688 bytes)
00000009B0 Data: (16370688 bytes)
0000F9D5B0 ------------------------
0000F9D5B0 --- DPX, filling ---
0000F9D5B0 ------------------------
0000F9D5B0 -------------------------
0000F9D5B0 --- DPX, finished ---
0000F9D5B0 -------------------------
The goal of RAWCooked is bit-by-bit reversible compression, and file system metadata is not stored in the bits. I am wondering if it would be possible to restore the file system dates when decoding from FFV1 back to DPX? This isn't a huge deal really, and I think on our end, we could accomplish this with a script if needed.
In order to avoid issues with older ffmpeg-libraries on certain Linux distros, I'd like to compile RAWcooked as static build against locally compiled ffmpeg libs.
I'd be very grateful for hints on how to properly do that.
🍰 🍒
(Managed to compile RAWcooked, but not sure how to tell it where to take its ffmpeg libs from)
This kind of directory structure is expected to be handled:
├── 99075c82-2770-4020-a289-d191dab0b852_1
│ ├── 99075c82-2770-4020-a289-d191dab0b852_1_00086400.dpx
│ ├── (...)
├── 99075c82-2770-4020-a289-d191dab0b852_2
│ ├── 99075c82-2770-4020-a289-d191dab0b852_2_00086400.dpx
│ ├── (...)
├── 99075c82-2770-4020-a289-d191dab0b852_3
│ ├── 99075c82-2770-4020-a289-d191dab0b852_300086400.dpx
│ ├── (...)
├── (...)
├── 99075c82-2770-4020-a289-d191dab0b852_9
│ ├── 99075c82-2770-4020-a289-d191dab0b852_900086400.dpx
│ ├── (...)
├── 99075c82-2770-4020-a289-d191dab0b852_10
│ ├── 99075c82-2770-4020-a289-d191dab0b852_1000086400.dpx
│ ├── (...)
├── (...)
FFmpeg does not appreciate to put 0 byte files in MKV attachments ("Could not open attachment file") and rejects the command.
Quick "fix" for this issue is to ignore such useless file, but we would not be "fully reversible" (name of the empty file not kept), so maybe we should store the name in RAWcooked file (similar to empty directory case).
Reversing the MKV back to DPX sequences works properly, but there is no progress indicator whatsoever.
It would be nice to have some means of showing any progress (or to see that rawcooked is still alive).
Just an idea 😃
I can use -o for stipulating the ffv1 filename for the DPX-> FFV1 process, but is there a way to set the output directory for the FFV1 -> DPX process?
$ rawcooked '/tmp/04ac946a-5e37-4ef8-bdca-24474bc02965/df909af4-57b1-4234-b211-fd901e662d11.mkv' -o '/home/kieranjol/Downloads'
Unsupported output file extension. Please contact [email protected] if you want support of such output format.
Hi All,
For some reason, I've been experiencing some sort of indeterminate RAWcooked "could not open file/could not open attachment" issue; here's the error messaging that I've been receiving:
MY-LPAMI-056430:~ benjaminturkus$ rawcooked --all /Volumes/NYPF257402/Film/data/PreservationMasters/10-bit_DPX/myd_380086_v01_pm_DPX
Using default license.
Consider to support RAWcooked: https://MediaArea.net/RAWcooked
Warning: non-conforming DPX Ditto key (x251).
At least 1 file is not conform to specifications.
Do you want to continue despite warnings ? [y/N] y
Track 1:
myd_380086_v01_pm.wav
WAV/PCM/96kHz/24bit/1ch/LE
*** This input format flavor is not supported by the current license key. ***
Track 2:
myd_380086_v01_pm_%07d.dpx
(0000000 --> 0064056)
DPX/Raw/RGB/10bit/FilledA/BE
One or more requested features are not supported with the current license key.
Please contact [email protected] for a quote or a temporary key.
Ignoring the license for the moment.
ffmpeg version 4.1.3 Copyright (c) 2000-2019 the FFmpeg developers
built with Apple LLVM version 10.0.1 (clang-1001.0.46.4)
configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.3_1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags='-I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include -I/Library/Java/JavaVirtualMachines/adoptopenjdk-11.0.2.jdk/Contents/Home/include/darwin' --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr
libavutil 56. 22.100 / 56. 22.100
libavcodec 58. 35.100 / 58. 35.100
libavformat 58. 20.100 / 58. 20.100
libavdevice 58. 5.100 / 58. 5.100
libavfilter 7. 40.101 / 7. 40.101
libavresample 4. 0. 0 / 4. 0. 0
libswscale 5. 3.100 / 5. 3.100
libswresample 3. 3.100 / 3. 3.100
libpostproc 55. 3.100 / 55. 3.100
Guessed Channel Layout for Input Stream #0.0 : mono
Input #0, wav, from '/Volumes/NYPF257402/Film/data/PreservationMasters/10-bit_DPX/myd_380086_v01_pm_DPX/myd_380086_v01_pm.wav':
Metadata:
title : myd_380086_v01_pm_Audio 96.0KHz 24bit.wav
copyright : Lasergraphics Inc
encoded_by : Lasergraphics
date : 2018-05-23
creation_time : 17:20:37
time_reference : 0
coding_history : A=PCM,F=96000,W=24,M=mono,T=libsndfile-1.0.20
:
Duration: 00:44:29.04, bitrate: 2304 kb/s
Stream #0:0: Audio: pcm_s24le ([1][0][0][0] / 0x0001), 96000 Hz, mono, s32 (24 bit), 2304 kb/s
[image2 @ 0x7fc4d1052400] Could not open file : /Volumes/NYPF257402/Film/data/PreservationMasters/10-bit_DPX/myd_380086_v01_pm_DPX/myd_380086_v01_pm_0000000.dpx
[image2 @ 0x7fc4d1052400] Could not find codec parameters for stream 0 (Video: dpx, none): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #1, image2, from '/Volumes/NYPF257402/Film/data/PreservationMasters/10-bit_DPX/myd_380086_v01_pm_DPX/myd_380086_v01_pm_%07d.dpx':
Duration: 00:44:29.04, start: 0.000000, bitrate: N/A
Stream #1:0: Video: dpx, none, 24 tbr, 24 tbn, 24 tbc
Could not open attachment file /Volumes/NYPF257402/Film/data/PreservationMasters/10-bit_DPX/myd_380086_v01_pm_DPX.rawcooked_reversibility_data.
In terms of troubleshooting, I've tried cooking a small subset of the DPX files without problem, and I've rolled back to an older version of RAWcooked (RAWcooked_CLI_18.10.1.20190112_Mac.dmg) and again, I haven't experienced the same issue. I'm currently on RAWcooked 18.10.1, downloaded direct from Github.
Thanks for any help!
Ben
When using the option -level 0 on regular RGB48 content, I receive an error:
Hi, I installed the latest 16.04 Ubuntu build.
I made a dpx sequence with
ffmpeg -f lavfi -i testsrc=size=2048x1556 -pix_fmt rgb48le -t 1 /home/kieran110s/dpxo/out_%06d.dpx
I get 'Untested DPX file size' when I try to encode with -e
$ rawcooked -e out_000001.dpx
Untested DPX file size, please contact [email protected] if you want support of such file
$ rawcooked /home/kieran110s/dpxo
Segmentation fault (core dumped)
What I'm starting to think is that currently, the app should be used without -e, and instead the ffmpeg command should be copy pasted into a terminal?
The prompt says that there is a rawcooked --help, but is gives an error and not the existing help:
I made a PR for the man page.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.