implementation of LUTs

[ietf-wg-cellar]

implementation of LUTs

[robUx4]

I am implementing into a fork what my company

@retokromer did you do it ? If so would you like to share your solution and possibly make a pull request to add what you created to Matroska ?

[dericed]

I think LUT storage was discussed on CELLAR but AFAIK no Elements are yet proposed to contain them.

[retokromer]

As said there and at the Berlin Workshop, we use currently an image that we store like a «record cover», but I’m not happy with this solution.

I’m not the only one who tried that route, see e.g. this post by @ubitux in FFmpeg-devel: http://ffmpeg.org/pipermail/ffmpeg-devel/2013-May/144031.html

And, for a little background, in my opinion, the link in that post is useful as well: http://www.quelsolaar.com/technology/clut.html

I would like (and I’m working on) something more generic, which has similarities with Jim Lindner’s Filmic project. I’m currently doing my testings in NUT, but would like to switch to Matroska.

[retokromer]

To my knowledge, the most advanced solution has been presented by the ACES people: https://j.mp/S-2014-006

And a sample implementations is available here on GitHub: https://github.com/ampas/CLF

[ubitux]

No idea if that helps but haldclut and common lut3d (text-based) are implemented in the filters of these names in FFmpeg. And haldclut can indeed now handle a CLUT video stream instead of a single image, which is pretty useful.

[retokromer]

Thank you, @ubitux! My goal is precisely to generalise this very mechanism to the different CLUTs we use in the film digitising and restoration workflow. And I would like to store the whole «thing» in one single container.

[ubitux]

Well, any video container with multiple video streams support + lossless should do the trick? like, mkv+ffv1.

edit: oups, just realized I'm commenting on Matroska-Org, I guess you agreed on the container already...

[retokromer]

Indeed, Matroska/FFV1 is the dreamed solution. For the moment, we don’t have a standardised way to achieve this, and the purpose is to define one.

[retokromer]

We tested hyperspectral scanning with 8 and 16 LEDs (and processing the signals in Gray instead of classic binary code), which has some similarity with Jim Lindner’s Filmic project, if I understood carefully.

[robUx4]

I didn't know about this. This is definitely something that should be supported, especially for the FFV1/Matroska combination. But it should be supported in a proper way and preferrably existing using existing standards.

IMO this is part of how the codec encodes the colours so that they are displayed correctly in the end. It may be possible to reuse the same CLUT format between various codecs though. It's not clear to me if it is in addition or replaces the MatrixCoefficients we already have. It seems they both transform the colour (or say how the values should be interpreted). So maybe we should have one value that says the Matrix is taken from a CLUT field found elsewhere.

Since it is essential for playback it should not go in tags. And interleaving the image doesn't seem to make sense (unless it changes over time). I'd rather store it in the track itself as (similar to CodecPrivate) and likely with a picture codec ID (PNG, raw, etc).

[retokromer]

We try to follow what proposed by ACES, and I really suggest to do so.

[retokromer]

The current LUT are lists of values.

However, we already use and are intensively experimenting with modified LUTs, which are vectors or matrices of functions. We do this for film restoration purposes, e.g. recovering colour decay. An implementation allowing future development in this direction, would be our dreamed solution.

[dericed]

IIUC Aces is an xml representation of a lookup table. Are any custom elements needed? Could it be stored in an Attachment with an AttachmentLink for it. I'm thinking this may work similarly to how a subtitle track may depend on a font file in an Attachment for the correct presentation (though that scenario needs a better definition as well).

[retokromer]

The current LUTs are:

• different kind of text files (including XML)
• TIFF images
• other not-easy-to-read data formats

This can indeed be stored as an attachment! It’s what we actually do in other containers. In Martoska we currently store a classic text LUT as a TIFF image by declaring it as the cover of the album.

In my opinion, an elegant solution should allow future readers to process them directly when playing back the stream. If more than one LUT is stored, then the player should allow the user the choice between the possibilities (e.g. «Estamancolor» look versus «Technicolor» look of a movie).

[robUx4]

It's not exactly like a font as the font must be referenced in the subtitle codec. Codecs will not reference the LUT in some random format. There must be a way to link a track to a LUT. I suppose it's the same for the whole duration of the Segment ? If so an entry(s) in Track is probably the best candidate. And it's not codec private or block additional because it's not part of the codec. It should be something aside. And, like I said, it seems to be a specific case of MatrixCoefficients for colour mapping.

[retokromer]

Closing this one, as open since six months.

[dericed]

Although open since six months I don't think the need for this is gone.

[retokromer]

I am afraid, I cannot wait any longer, therefore I am implementing into a fork what my company is needing for its own work.

[retokromer]

Seems not to be of interest. I give up.

[JeromeMartinez]

Seems not to be of interest.

It is not.

[retokromer]

Then open another issue.

[JeromeMartinez]

There is some history (comments...), it is a bit a shame to see it closed and being forced to open another issue for linking to comments are in this ticket as a punishment for not being quick enough.

[retokromer]

OK, I’ll reopen and unwatch matroska-specification.

[JeromeMartinez]

unwatch matroska-specification

Different punishment :(.

[retokromer]

WTF: Why do I not have the right to stop loosing my time?

[dericed]

I tried to see if the OP of an issue could be changed, but I don't think it's possible. I agree with @JeromeMartinez that this topic is interesting and should persist.

[retokromer]

@robUx4 I did indeed implement «something» in Matroska, but then I switched «definitively» to NUT – simply because on my end it was quicker to do so. I will check what exactly was «something», but I cannot do this in the next days, probably only in February.

[retokromer]

@robUx4 I’ll try to prepare a PR the weekend after next (no guarantee). Here are two examples of logarithmic 3D LUTs, used by the industry, for the Kodak 2383 and Fuji 3513 stocks. To my taste, they are too hard for grading, but that’s just my personal taste.

[JeromeMartinez]

@retokromer would it be possible to have a sample file (.nut if this is the files you have) with real world content, both what you store and what you have after transformation to more "classic" format e.g. DPX 16-bit (or better, the FFmpeg command)? It would permit people not specialized in LUTs to well understand the purpose.

[retokromer]

@JeromeMartinez As presented in Vienna, today we mainly use OpenEXR or an expanded OpenEXR in an expanded NUT. There are no standard FFmpeg commands for… personal formats! We often use a DPX flavours, mainly 16-bit or 10-bit, as delivery format to the customers, at the end of our workflow. (We finished such a job this week, therefore I used DPX to stress-test RAWcooked in a real world situation and to earn some statistics about the encoding time and the writing time on LTO. I guess, you are referring to this.) Of course, I can upload such an expanded NUT, but you will not have a software to parse or play it. I will look for a short content that can be shared without legal issues, as we are not an archive and don’t have collections. (BTW: I am desperately seeking for some content on lenticular film to share on GitHub as test examples, if you should have any.)

[retokromer]

@JeromeMartinez This is the general DI Workflow at Haghefilm Digitaal, as presented by Gerard den Haan at The Reel Thing in Amsterdam last year. He discussed in-depth the example of the restoration of stencil coloured films. An extremely interesting talk, in my opinion.

[retokromer]

FYI: Tests with the new ProRes RAW were successful. However, this covers just a little part of my wish about LUTs.

[retokromer]

FYI: ProRes RAW in a film scanning context does offer a way to implement the illuminant/white point parameters. This is useful especially when working on severely faded prints or masked elements such as colour negatives or intermediates. In fact, those metadata allow to import nicely «even» files in the restoration suite. We’ll try to organise a report from the field on the late Summer, on the basis of a negative which has been cut using shots on different film stocks.

I’m now hesitating if this would best fit in the container (Matroska) or in the video codec (FFV1), hmm…

[kieranjol]

Just FYI, I'm getting quite interested in possibly using LUTs in this kind of way as well, mostly based on some work that @wendriftwood is doing with us at the moment. Looks like it could be a nice solution for film archives.