mafiosnik777/enhancr

Video Frame Interpolation & Super Resolution using NVIDIA's TensorRT & Tencent's NCNN inference, beautifully crafted and packaged into a single app

enhancr's Issues

AV1 encoder quits halfway without reason with a completed message

Hi Team,
First of all, I am very impressed by enhance. It's really good. However, I am facing problems with its reliability. AV1 encodes more than 70% of the times do not finish. The frames encoded will encode till just over half the frames then it will end saying the encoding is complete. It is very frustrating since the CPU based AV1 encoding takes almost 90 mins to 120 mins and suddenly after an hour it will just finish abruptly. I do not face this issue if I use the AV1 NVENC hardware encoders. This is not a cache problem too since I always maintained 100 GB of free space.

This is the settings what I use:
-c:v libsvtav1 -qp 50 -preset 7 -svtav1-params tune=0:enable-tf=0:enable-overlays=1:enable-qm=1 -pix_fmt yuv420p10le

Note: qp is 50 since its Anime and I don't see much quality degradation.

Regards,
Neo

Selecting an external model crashes the encoder

I'm trying to use external models to upscale my video, specifically 2x VHS upscale and denoise Film and 4x UniScale Balanced.

Settings: x265, "Upscaling - RealESRGAN (DirectML)", 2x

Whether I use pth or onnx files, the result is always:

[enhancr] Preparing media for upscaling process..
[enhancr] Using custom model: C:\Users\AMD\AppData\Roaming\.enhancr\models\RealESRGAN\4x-UniScale-Balanced [72000g].onnx
[enhancr] Encoding parameters: -c:v hevc_nvenc -preset p6 -b_adapt 1 -rc-lookahead 30 -qp 23 -qp_cb_offset -2 -qp_cr_offset -2 -pix_fmt nv12
[enhancr] Scale: 2x
[enhancr] Mode: Upscaling - RealESRGAN (DirectML)
[enhancr] Starting upscaling process..

[Pipe] Starting video output | Threads: 16 | Streams: 2
[Pipe] Frame: 1/43230
[Pipe] Error: Failed to retrieve frame 13 with error: operator (): 'ortapi->Run( resource.session, nullptr, &resource.input_name, &resource.input_tensor, 1, &resource.output_name, 1, &resource.output_tensor )' failed: D:\a\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\ExecutionProvider.cpp(898)\onnxruntime.dll!00007FF9E82E45B4: (caller: 00007FF9E82E4C0F) Exception(2) tid(13a0) 887A0006 The GPU will not respond to more commands, most likely because of an invalid command passed by the calling application.

Any suggestions?

missing colorspace parameter

the input video is mp4 hvc1

`[enhancr] Preparing media for restoration process..
[enhancr] Encoding parameters: -c:v h264_nvenc -preset p6 -b_adapt 1 -rc-lookahead 30 -qp 18 -qp_cb_offset -2 -qp_cr_offset -2 -pix_fmt nv12
[enhancr] Mode: Restoration - DPIR (DirectML)
[enhancr] Starting restoration process..

[Pipe] Starting video output | Threads: 8 | Streams: 2
[Pipe] Script evaluation done in 1.62 seconds
[Pipe] Error: Failed to retrieve frame 0 with error: Resize error: Resize error 3074: unrecognized transfer characteristics (0/0/0 => 0/0/0). May need to specify additional colorspace parameters.
Output 8 frames in 0.02 seconds (343.94 fps)
[Pipe]
[enhancr] Finishing up restoration..
[enhancr] Muxing in streams..
[enhancr] Muxing in streams..
[enhancr] Completed restoring
[enhancr] Completed queue with 0 error(s).`

Hardware accelerated decoding

Was wondering if the option to pass --hwaccel to ffmpeg could be added. For some of the models I tend to run into a CPU bottleneck before CUDA.

Upscaled video appears glitched when video width or height is not a multiple of 4 pixels

When upscaling a video with a width or height that is a multiple of 4 pixels (ex. 360), the video appears fine.
If the width or height is not a multiple of 4 pixels (ex. 362), the video appears glitched, with the frames being two-tone colors, or something like that.

Version used is 0.9.8 (Free).
Engine used is ShuffleCUGAN (NCNN).

Add Support for Other Hardware Encoders and Formats.

Due to the massive speedups from Tensor RT it is possible for both CAIN and RIFE to both be bottle necked by transcoding speed instead of the interpolation itself. To Avoid this there is already support for NVENC. However enhancr also supports AMD gpus and soon apple silicon which both don't have NVENC. It would be nice for people on AMD and apple silicon to have hardware encode support to avoid transcoding and cpu bottlenecks.

Currently enhancr only supports HEVC with NVENC as well. It would be nice to support more formats like the older h.264 for compatibility reasons as well as newer formats like AV1 who are file size concerned.

Restore doesn't work

-Load video
-Add to que
-Click enhance
The cue is immediately completed, nothing is processed and the interface is now unusable and stuck in an infinite loading glitch.

All settings are stock standard.
Project source and destination in Documents folder.
Latest installer on github from releases.
Test with multiple input videos and formats.
Windows 11, everything up to date.

Video gets flipped 90°

I am trying to upscale a vertical video (taken with a smartphone camera; 848x480p). But after the upscale (used base settings, realesrgan with 4x scale) the video is flipped by 90° and the quality is absolutly horrible.

"nvenc No capable devices found" with software encoding @2x720p RealESRGAN (NCNN)

I can run neither upscaling nor restoration with RealESRGAN (NCNN) on anything 720p and above.
Hardware encoding is turned off in settings. 480*360 worked fine with 3x, 720p and above fails with 2x.
ShuffleCUGAN (NCNN) also works fine.

`•─────────•°• enhancr •°•─────────•
OS: Microsoft Windows 10 Pro x64
CPU: AMD Ryzen 7 5800X3D 8-Core Processor | [8C/16T]
RAM: 32694 MB
GPU: NVIDIA GeForce RTX 3080 | 10240 MB
Status: Ready
•°•────────────────────────────────•°•
[enhancr] 'vid_720p.mp4': Successfully added to Queue
[enhancr] Preparing media for upscaling process..
[enhancr] Using custom model: C:\Users\xxxxxx\AppData\Roaming.enhancr\models\RealESRGAN
[enhancr] Encoding parameters: -c:v libx264 -crf 16 -preset slow -x264-params direct=spatial:me=umh -pix_fmt yuv420p
[enhancr] Scale: 2x
[enhancr] Mode: Upscaling - RealESRGAN (NCNN)
[enhancr] Starting upscaling process..

[Pipe] [0 NVIDIA GeForce RTX 3080] queueC=2[8] queueG=0[16] queueT=1[2]
[0 NVIDIA GeForce RTX 3080] bugsbn1=0 bugbilz=0 bugcopc=0 bugihfa=0
[0 NVIDIA GeForce RTX 3080] fp16-p/s/a=1/1/1 int8-p/s/a=1/1/1
[0 NVIDIA GeForce RTX 3080] subgroup=32 basic=1 vote=1 ballot=1 shuffle=1
[Pipe] Starting video output | Threads: 8 | Streams: 2
[Pipe] Script evaluation done in 1.22 seconds
[Pipe] [h264_nvenc @ 00000250f3d8cb00] No capable devices found
[Pipe] Error: fwrite() call failed when writing frame: 3, plane: 0, errno: 32
Output 12 frames in 0.95 seconds (12.68 fps)
[Pipe] [enhancr] Finishing up upscaling..
[enhancr] Muxing in streams..
[Pipe] [matroska,webm @ 00000205eda8f900] EBML header parsing failed
[Pipe] D:\temp\enhancr\1684753613579.mkv: Invalid data found when processing input
[enhancr] Completed upscaling
[enhancr] Completed processing queue successfully.`

`[enhancr] 'vid_720p.mp4': Successfully added to Queue
[enhancr] Preparing media for upscaling process..
[enhancr] Using custom model: C:\Users\xxxxxx\AppData\Roaming.enhancr\models\RealESRGAN
[enhancr] Encoding parameters: -c:v ffv1 -coder 2 -context 1 -level 3 -slices 12 -g 1
[enhancr] Scale: 2x
[enhancr] Mode: Upscaling - RealESRGAN (NCNN)
[enhancr] Starting upscaling process..

[Pipe] [0 NVIDIA GeForce RTX 3080] queueC=2[8] queueG=0[16] queueT=1[2]
[0 NVIDIA GeForce RTX 3080] bugsbn1=0 bugbilz=0 bugcopc=0 bugihfa=0
[0 NVIDIA GeForce RTX 3080] fp16-p/s/a=1/1/1 int8-p/s/a=1/1/1
[0 NVIDIA GeForce RTX 3080] subgroup=32 basic=1 vote=1 ballot=1 shuffle=1
[Pipe] Starting video output | Threads: 8 | Streams: 2
[Pipe] Frame: 1/85050
[Pipe] [h264_nvenc @ 0000019c1dad8c80] No capable devices found
[Pipe] Error: fwrite() call failed when writing frame: 3, plane: 0, errno: 32
Output 13 frames in 1.02 seconds (12.79 fps)
[Pipe] [enhancr] Finishing up upscaling..
[enhancr] Muxing in streams..
[Pipe] [matroska,webm @ 0000027f0569f900] EBML header parsing failed
D:\temp\enhancr\1684753681678.mkv: Invalid data found when processing input
[enhancr] Completed upscaling
[enhancr] Completed processing queue successfully.
`

Need internet connection to use app?

If I join the patreon to use the app, does it need to be connected to the internet every time I use it? If I cancel the patreon subscription can I still use the last version of the app that I have installed? Or am I paying $120 a year to use the app?

Unable to select preferred GPU

On Asus ZenBook 14x with dual Intel Iris Xe integrated graphics and Nvidia RTX 3050 dedicated GPU, enhancr does not detect the RTX 3050 and shows the Intel Iris Xe graphics as the selected GPU in the settings information panel as follows:

•─────────•°• enhancr •°•─────────•
OS: Microsoft Windows 11 Home x64
CPU: Intel Gen Intel® Core™ i9-13900H | [14C/20T]
RAM: 32372 MB
GPU: Intel(R) Iris(R) Xe Graphics | 1024 MB
Status: Ready
•°•────────────────────────────────•°•

I have also confirmed that there is no activity on the RTX 3050 GPU when processing videos in enhancr. All hardware decoding/encoding is limited to the Intel Iris Xe integrated GPU, which is obviously very slow when compared to the RTX 3050.

There doesn't appear to be any way to change the GPU selection in enhancer settings, so it would be very helpful if an option could be added to select the preferred GPU.

can't get normal esrgan to work

got this error:
clip = core.trt.Model(clip, engine_path=engine, num_streams=streams) File "src\cython\vapoursynth.pyx", line 2565, in vapoursynth.Function.__call__vapoursynth.Error: operator (): no valid optimization profile found

i used a model called 2X_DigitalFilmV5_Lite.pth
btw the realesrgan models worked just fine just the esrgan one that i had this problem with
here is the log
log.txt

Request for inclusion of basicVSR++ model

Hi,

I hope this issue finds you well. I am a user of this repository and I have noticed that the basicVSR++ model is currently not included.

I would like to request that the basicVSR++ model be added to the repository. It has excellent video super-resolution performance.

Thank you for your time and consideration.

Best regards

Offline Installer

Due to the unstable nature of my internet connection, I am unable to complete the installation of the application through the provided installer, as it does not offer a retry or resume functionality similar to that of a web browser or download manager. Consequently, I am seeking an offline installer. Additionally, if there are direct links to the primary files, I would be able download them via IDM or web browser.
Thank you in advance

Error with Restore

I installed the software and I get this error

[Pipe] [NULL @ 00000226c616dc00] Unable to find a suitable output format for '='
=: Invalid argument
[Pipe] Error: fwrite() call failed when writing frame: 1, plane: 0, errno: 32
Output 13 frames in 1.54 seconds (8.46 fps)
[Pipe] [enhancr] Finishing up restoration..
[enhancr] Muxing in streams..
[enhancr] Muxing in streams..
[Pipe] [matroska,webm @ 000002e39978ecc0] EBML header parsing failed
C:\Users\i5\AppData\Local\Temp\enhancr\1685654715695.mkv: Invalid data found when processing input
[enhancr] Completed restoring
[enhancr] Completed processing queue successfully.

Support Real-ESRGAN x2 Plus

I've run a bunch of tests on upscaling SD movies (VHS and DVD) and found that using that using Real-ESRGAN x2 Plus (RealESRGAN_x2plus.pth) is offering much better quality than the included RealESRGAN encoder, which tends to smooth out details so people look like they're made of plastic.

Unfortunately, using x2 Plus as custom model has a severe impact on the transcoding speed (going from approx 60-110fps for RealESRGAN TensorRT 2x to around 13-24fps for the custom model converted to TensorRT engine).

Is there any way to speed this up, or is there any way you could implement a fast x2 (or x4) upscaler that is much better for realistic movies (as opposed to anime content)?

No Uninstaller:)

My gpu isnt support and its not uninstalling;(

AV1 in enhancr does not support 120fps videos on non-interpolation modes, slows it down instead.

When upscaling/restoring 120fps videos and setting the codec to av1, the output video is slowed down, my guess 2x to 60fps. Why? Av1 does support HFR videos, but why not in enhancr?

enhancr does not like certain MKV files

I tried using a 480p anime (Dragon Ball Z). And I believe that since it has chapters, Enhancr does not like it. please fix

Error with video

Im getting the following error show below when trying to upscale a video:

[enhancr] Preparing media for upscaling process..

[enhancr] Encoding parameters: -c:v libx264 -crf 16 -preset slow -x264-params direct=spatial:me=umh -pix_fmt yuv420p

[enhancr] Scale: 2x

[enhancr] Mode: Upscaling - RealESRGAN (DirectML)

[enhancr] Starting upscaling process..


[Pipe] Starting video output | Threads: 8 | Streams: 2

[Pipe] Script evaluation done in 2.02 seconds
[Pipe] 2023-06-13 18:49:43.3519796 [E:onnxruntime:, sequential_executor.cc:514 onnxruntime::ExecuteKernel] Non-zero status code returned

[Pipe] while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: D:\a\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\FusedGraphKernel.cpp(365)\onnxruntime.dll!00007FFA8BA1E00A: (caller: 00007FFA8BA1C7FC) Exception(2) tid(5780) 8007000E Not enough memory resources are available to complete this operation.
[Pipe] Error: Failed to retrieve frame 1 with error: operator (): 'ortapi->RunWithBinding(resource.session, nullptr, resource.binding)' failed: Non-zero status code returned while running DmlFusedNode_0_0 node. Name:'DmlFusedNode_0_0' Status Message: D:\a\onnxruntime\onnxruntime\onnxruntime\core\providers\dml\DmlExecutionProvider\src\FusedGraphKernel.cpp(365)\onnxruntime.dll!00007FFA8BA1E00A: (caller: 00007FFA8BA1C7FC) Exception(2) tid(5780) 8007000E Not enough memory resources are available to complete this operation.

[Pipe] Output 8 frames in 16.06 seconds (0.50 fps)
[Pipe] 

[enhancr] Finishing up upscaling..

[enhancr] Muxing in streams..

[Pipe] [matroska,webm @ 00000264e9d5af40] Duplicate element

[matroska,webm @ 00000264e9d5af40] 0x00 at pos 110 (0x6e) invalid as first byte of an EBML number

[matroska,webm @ 00000264e9d5af40] Duplicate element

[matroska,webm @ 00000264e9d5af40] 0x00 at pos 202 (0xca) invalid as first byte of an EBML number

[matroska,webm @ 00000264e9d5af40] Element at 0x67 ending at 0x7cec0100000070 exceeds containing master element ending at 0x1413

C:\Users\KKart\AppData\Local\Temp\enhancr\1686707366850.mkv: End of file

[enhancr] Completed upscaling

[enhancr] Completed queue with 0 error(s). // Not true

This happens to a specific video and I've tried reencoding it still doesn't work.

Which engine gives the best quality? [QUESTION]

Which engine gives the best quality? NCNN, DirectML, or TensorRT on Windows? I would also like to know which one is the fastest. Would be nice to have this information as a hover tooltip.

"free" and "pro" versions

What the difference between "free" and "pro" versions?
How to obtain "pro" versions?

vapoursynth.Error: RIFE: gpu_thread must be between 1 and 1 (inclusive)

I cannot run any interpolation, upscale, or restore models. Logs below. I have an integrated GPU and a dedicated GPU. I used this installer https://github.com/mafiosnik777/enhancr/releases/download/0.9.9/enhancr-setup-free-0.9.9.exe

•─────────•°• enhancr •°•─────────•
OS: Microsoft Windows 11 Home x64
CPU: Intel Core™ i5-10500H | [6C/12T]
RAM: 16173 MB
GPU: NVIDIA GeForce RTX 3060 Laptop GPU | 6144 MB
Status: Ready
•°•────────────────────────────────•°•
[enhancr] 'XMassTestClip.mp4': Successfully added to Queue
[enhancr] Encoding parameters: -c:v h264_nvenc -preset p6 -b_adapt 1 -rc-lookahead 30 -qp 18 -qp_cb_offset -2 -qp_cr_offset -2 -pix_fmt nv12
[enhancr] Model: RIFE - v4.6
[enhancr] Starting interpolation process..

[Pipe] [0 Intel(R) UHD Graphics] queueC=0[1] queueG=0[1] queueT=0[1]
[0 Intel(R) UHD Graphics] bugsbn1=0 bugbilz=0 bugcopc=0 bugihfa=0
[0 Intel(R) UHD Graphics] fp16-p/s/a=1/1/1 int8-p/s/a=1/1/1
[0 Intel(R) UHD Graphics] subgroup=32 basic=1 vote=1 ballot=1 shuffle=1
[Pipe] [1 NVIDIA GeForce RTX 3060 Laptop GPU] queueC=2[8] queueG=0[16] queueT=1[2]
[1 NVIDIA GeForce RTX 3060 Laptop GPU] bugsbn1=0 bugbilz=0 bugcopc=0 bugihfa=0
[1 NVIDIA GeForce RTX 3060 Laptop GPU] fp16-p/s/a=1/1/1 int8-p/s/a=1/1/1
[1 NVIDIA GeForce RTX 3060 Laptop GPU] subgroup=32 basic=1 vote=1 ballot=1 shuffle=1
[Pipe] Script evaluation failed:
Python exception: RIFE: gpu_thread must be between 1 and 1 (inclusive)

Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 3115, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 3116, in vapoursynth._vpy_evaluate
File "D:\Videos\enhancr\resources\inference\rife.py", line 67, in
clip = core.rife.RIFE(clip, model=model, factor_num=2, gpu_id=0, gpu_thread=threading(), tta=TTA, uhd=UHD, skip=frameskip, sc=sceneDetection)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "src\cython\vapoursynth.pyx", line 2847, in vapoursynth.Function.call
vapoursynth.Error: RIFE: gpu_thread must be between 1 and 1 (inclusive)
[Pipe]
[enhancr] An error has occured.
[enhancr] Completed queue with 1 error(s).

Integrated + External GPU not supported

I am trying to run this on a windows 11 laptop with an external nvidia gpu, though the tool keeps only recognizing the igpu of the laptops cpu, but not the external gpu. I have tried to force it to use the gpu, but it didnt work. Settings only show the igpu and upscaling something results in an error, that my gpu is unsupported (no capable devices found).
This seems to only be an issue with x265, x264 works on my igpu, though i would like to use x265 and my external gpu

Uninstaller doesn't uninstall, runs install again

Steps to reproduce:

1. Open Windows installed application list
2. Select Enhancr to uninstall and begin the uninstall process.

Tiny window- can't resize or increase font size

At 4K I can barely read the window text on a 42" monitor.
No maximize option?
No resize window option.
Windows 10

Implement NVIDIA OFA FRUC as an option for video frame interpolation

https://docs.nvidia.com/video-technologies/optical-flow-sdk/nvfruc-programming-guide/index.html

Example built win64 binary: https://github.com/brokeDude2901/NvOFFRUCSample

Pros:

• Low VRAM usage, only use Tensor Cores

Cons:

• Require at least a NVIDIA GPU Turing