kfjahnke/envutil

utility to convert between lat/lon and cubemap environment maps, and to extract single images and image series in several projections and arbitrary orientation

envutil's Issues

integrate more of libffmpeg

ffmpeg is extremely capable when it comes to image processing - as it was pointed out by @chaosjug on hugin-ptx (see https://groups.google.com/g/hugin-ptx/c/C58NlKPYHbM/m/w02W0QD6BAAJ) the entire process of extracting a set of frames from an environment image and producing an output video from the result can be done using the ffmpeg v360 filter. Looking at this filter (see https://ffmpeg.org/ffmpeg-filters.html#v360), it offers a fair amount of extra options which go beyond what envutil currently provides, e.g. using double fisheyes as a substrate.

The v360 filter can also be used on single images (take care to work the data as TIFF to avoid compression artifacts), and I played with it a bit, and ascertained that the conversions compare well with what envutil produces, as long as the parameterization is compatible.

Can envutil be made to pass it's pixel pipeline through an ffmpeg filter? If the libffmpeg code can be leveraged to parse a filter specification (ffmpeg obviously can, the CL tool does just that) - it might be a matter of just passing through the textual spec and writing an interface which routes the data through the filter chain.

accept multiple mounted images, stitch

This would be a path towards stitching: several source images could be mounted with additional orientation information (currently, the mount is conceptually head-on, and any reorientation is via the virtual camera's y/p/r orientation). The next step would be to do the 'voronoi thing' (like animated sequences of PTOs in lux), which would already be 'good enough' to apply patches. Eventually, even an implementation of the Burt&Adelson image splining algorithm might be slotted in for 'proper' stitching - or some other method to blend the images seamlessly.

envutil's video output is not generally viable

So far, generating video output from a 'seq file' doesn't produce generally viable output. Here on my debian system, the 'videos' app displays the video, but VLC does not. On other systems, VLC also doesn't seem to play the video. Is this a VLC problem (missing/wrong codec?) or is there something wrong with the video?

I have 'brutalized' an example file from the ffmpeg project to produce the output, and my handling of ffmpeg may not be correct. I suspect that the problem is my end, because emitting single frames and generating the video from that with CL ffmpeg produces a generally viable output.

process all data in linear RGB internally, add display transform

Currently, envutil will process data as delivered by OIIO, without conversion to linear RGB if that is the native data format in the image file. This is not mathematically correct unless the data is in linear RGB already, and to be completely correct, the data should be scene-linear, and display-linear data would need to be converted back to scene-linear to do the math correctly.

The conversion of display-linear data or data in an output color space (like sRGB) may not be technically possible (if a 3D LUT was used, the conversion may not be unambiguously reversible, filters may be baked into the data) - but for the sake of processing, approaching scene-linear via a simple tone curve for all channels should produce better results, even if the correct reconstruction of the scene-linear data is not possible.

I think that the color space conversions could be done with OIIO or OCIO, but so far I don't have any code in place. Keep possibility of using EMoR in mind.

I need:

• inspection of the input file: what is it's color space
• if it's in an output color space, convert to display-linear
• invert the display transform
• on output, if the format requires, convert accordingly

If a 3D LUT was used, inverting it may be possible under constraints, investigate.

mounted images wit 360 degree hfov don't behave quite as 'proper' environment images do

This is especially noticeable when OIIO's lookup code is used (the mounted image doesn't always wrap around correctly, the visibility of the flaw depending on the interpolator) - with twining it's masked by the reflect boundary condition, so there is indeed a discontinuity but it's usually too slight to be visible. With OIIO lookup, the problem can be mitigated by passing appropriate boundary treatment options, but this is not automatic.

envutil code would need special treatment of wrap-around for mounted images with 360 degrees hfov (and proper handling of zenith and nadir for full sphericals) - probably it would be best to 'elevate' such images to 'environment image status' and use that code path, rather than the mounted-image path. Cyclindrical images with 360 degree hfov would need special treatment, since the environment code path currently only accepts full sphericals and cubemaps ('fully covered' input in lux speak).

extend and factor out twining code

Having worked with the twining 'meta filter', I get the impression that the method is very powerful and versatile. It may be a good idea to decouple twining from the remainder of the code and make it into a stand-alone library. It would need only zimt as a dependency. It may also be a good idea to move the twining code to the zimt library, which as already accreting a fair amount of collateral code. The base idea is similar to what I proposed as 'meta filter' in vspline (see https://bitbucket.org/kfj/vspline/src/master/example/metafilter3.cc), but the twining code is more efficient, because it does not require rotation operations on the coordinates.

The twining code generates the basis vectors for the sub-pick-ups from two additional target coordinates calculated from two 'canonical' neighbours, optionally projected to the tangent plane for mathematical correctness (which is usually overkill, so it's optional, see --twine_precise). Is there maybe a better way to obtain the derivative? I also think that maybe reducing the width of the 'canonical' step to, say, 1%, may approach the derivative better, making the projection to the tangent plane even less necessary.

Additionally, providing a 2D version of the twining code would be nice. For both the 3D and the 2D case, an alternative route of parameterization might directly provide the basis vectors. This may be even more versatile and allow application code to come up with it's own ways of generating the basis vectors, rather than relying on the 'canonical neighbours'.