holywu/vs-rife

RIFE function for VapourSynth

RIFE

Real-Time Intermediate Flow Estimation for Video Frame Interpolation, based on https://github.com/hzwer/Practical-RIFE.

Dependencies

• PyTorch 2.5.0.dev
• VapourSynth R66 or later
• vs-miscfilters-obsolete (only needed for scene change detection)

trt requires additional Python packages:

• TensorRT 10.1.0
• Torch-TensorRT 2.5.0.dev

To install or update PyTorch and Torch-TensorRT to the latest nightly build, run pip install --pre -U torch torchvision torch_tensorrt --index-url https://download.pytorch.org/whl/nightly/cu124 --extra-index-url https://pypi.nvidia.com

Installation

pip install -U vsrife
python -m vsrife

Usage

from vsrife import rife

ret = rife(clip)

See __init__.py for the description of the parameters.

vs-rife's Issues

TRT optimization pass will fail in UHD at high scales and Ensemble=True

As we know, RIFE doesn't like patterns.
I'm using high scales, even in UHD, to fight these distorted / disordered patterns that you would get in some interpolation frames.

But I can't get the trt optimization pass to run successfully with scale=4 and ensemble=True in UHD.

Will succeed:

RIFE(clip, model='4.4', scale=4, num_streams=1, sc=True, sc_threshold=0.12, ensemble=False, trt=True)

Will fail:

RIFE(clip, model='4.4', scale=4, num_streams=1, sc=True, sc_threshold=0.12, ensemble=True, trt=True)

I can only guess that the GPU memory is running out?
On my old Nvidia 1070 (8GB) trt optimization would already fail in UHD with scale=1.
Now with my Nvidia 4090 (24GB) trt optimization fails with scale=4 and ensemble=True. (for all RIFE models)

The script...

import os, sys
import vapoursynth as vs
core = vs.core

sys.path.append(r"P:\_Apps\StaxRip\StaxRip 2.29.0-x64 (VapourSynth)\Apps\Plugins\VS\Scripts")
core.std.LoadPlugin(r"P:\_Apps\StaxRip\StaxRip 2.29.0-x64 (VapourSynth)\Apps\Plugins\Dual\L-SMASH-Works\LSMASHSource.dll", altsearchpath=True)
clip = core.lsmas.LibavSMASHSource(r"T:\Test\tst.mp4")
from vsrife import RIFE
import torch
os.environ["CUDA_MODULE_LOADING"] = "LAZY"
clip = vs.core.resize.Bicubic(clip, format=vs.RGBH, matrix_in_s="709")

# RIFE calls with different workspace sizes that didn't help
#clip = RIFE(clip, model='4.4', scale=4, num_streams=1, sc=True, sc_threshold=0.12, ensemble=True, trt=True, trt_max_workspace_size=536870912)
#clip = RIFE(clip, model='4.4', scale=4, num_streams=1, sc=True, sc_threshold=0.12, ensemble=True, trt=True, trt_max_workspace_size=4294967296)

clip = RIFE(clip, model='4.4', scale=4, num_streams=1, sc=True, sc_threshold=0.12, ensemble=True, trt=True)
clip = vs.core.resize.Bicubic(clip, format=vs.YUV420P8, matrix_s="709")
clip.set_output()

...will produce lots of these warnings while processing...

...
[10/01/2023-11:53:31] [TRT] [E] 2: [virtualMemoryBuffer.cpp::nvinfer1::StdVirtualMemoryBufferImpl::resizePhysical::140] Error Code 2: OutOfMemory (no further information)
[10/01/2023-11:53:31] [TRT] [E] 2: [virtualMemoryBuffer.cpp::nvinfer1::StdVirtualMemoryBufferImpl::resizePhysical::140] Error Code 2: OutOfMemory (no further information)
[10/01/2023-11:53:31] [TRT] [W] Requested amount of GPU memory (8589934592 bytes) could not be allocated. There may not be enough free memory for allocation to succeed.
...

...and eventually end with the error output:

Python exception: 

Traceback (most recent call last):
  File "src\cython\vapoursynth.pyx", line 2866, in vapoursynth._vpy_evaluate
  File "src\cython\vapoursynth.pyx", line 2867, in vapoursynth._vpy_evaluate
  File "T:\Test\tst.vpy", line 12, in <module>
    clip = RIFE(clip, model='4.4', scale=4, num_streams=1, sc=True, sc_threshold=0.12, ensemble=True, trt=True)
  File "D:\Python\Python310\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "D:\Python\Python310\lib\site-packages\vsrife\__init__.py", line 219, in RIFE
    flownet = lowerer(
  File "D:\Python\Python310\lib\site-packages\torch_tensorrt\fx\lower.py", line 316, in __call__
    return do_lower(module, inputs)
  File "D:\Python\Python310\lib\site-packages\torch_tensorrt\fx\passes\pass_utils.py", line 118, in pass_with_validation
    processed_module = pass_(module, input, *args, **kwargs)
  File "D:\Python\Python310\lib\site-packages\torch_tensorrt\fx\lower.py", line 313, in do_lower
    lower_result = pm(module)
  File "D:\Python\Python310\lib\site-packages\torch\fx\passes\pass_manager.py", line 238, in __call__
    out = _pass(out)
  File "D:\Python\Python310\lib\site-packages\torch\fx\passes\pass_manager.py", line 238, in __call__
    out = _pass(out)
  File "D:\Python\Python310\lib\site-packages\torch_tensorrt\fx\passes\lower_pass_manager_builder.py", line 202, in lower_func
    lowered_module = self._lower_func(
  File "D:\Python\Python310\lib\site-packages\torch_tensorrt\fx\lower.py", line 178, in lower_pass
    interp_res: TRTInterpreterResult = interpreter(mod, input, module_name)
  File "D:\Python\Python310\lib\site-packages\torch_tensorrt\fx\lower.py", line 130, in __call__
    interp_result: TRTInterpreterResult = interpreter.run(
  File "D:\Python\Python310\lib\site-packages\torch_tensorrt\fx\fx2trt.py", line 252, in run
    assert engine
AssertionError

Tried with 0.25/0.5x/2x/4x the workspace size but didn't help.

Solution for 'There is no attribute or namespace named misc'

So if you get a There is no attribute or namespace named misc error with current VapourSynth versions,
grab the MiscFilter.dll from here and drop it in core\plugins of your VapourSynth installation.

The error would show up as soon as you enable sc and set the sc_theshold. (e.g. sc=True and sc_threshold=0.12 or something)

how to use this as a filter?

Hi, any way to use this as a filter - vapoursynth?

Script evaluation failed

[vapoursynth] clip = rife(clip,factor_num=2,factor_den=1,sc_threshold=0.1,trt=True)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
[vapoursynth] return func(*args, **kwargs)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\vsrife_init_.py", line 208, in rife
[vapoursynth] flownet = lowerer(
[vapoursynth] ^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch_tensorrt\fx\lower.py", line 316, in call
[vapoursynth] return do_lower(module, inputs)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch_tensorrt\fx\passes\pass_utils.py", line 118, in pass_with_validation
[vapoursynth] processed_module = pass_(module, input, *args, **kwargs)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch_tensorrt\fx\lower.py", line 313, in do_lower
[vapoursynth] lower_result = pm(module)
[vapoursynth] ^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch\fx\passes\pass_manager.py", line 238, in call
[vapoursynth] out = _pass(out)
[vapoursynth] ^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch\fx\passes\pass_manager.py", line 238, in call
[vapoursynth] out = _pass(out)
[vapoursynth] ^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch_tensorrt\fx\passes\lower_pass_manager_builder.py", line 202, in lower_func
[vapoursynth] lowered_module = self._lower_func(
[vapoursynth] ^^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch_tensorrt\fx\lower.py", line 178, in lower_pass
[vapoursynth] interp_res: TRTInterpreterResult = interpreter(mod, input, module_name)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch_tensorrt\fx\lower.py", line 130, in call
[vapoursynth] interp_result: TRTInterpreterResult = interpreter.run(
[vapoursynth] ^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch_tensorrt\fx\fx2trt.py", line 251, in run
[vapoursynth] engine = self.builder.build_engine(self.network, builder_config)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[vapoursynth] clip = rife(clip,factor_num=2,factor_den=1,sc_threshold=0.1,trt=True)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context
[vapoursynth] return func(*args, **kwargs)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\vsrife_init_.py", line 227, in rife
[vapoursynth] clip = sc_detect(clip, sc_threshold)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[vapoursynth] File "C:\Python311\Lib\site-packages\vsrife_init_.py", line 279, in sc_detect
[vapoursynth] sc_clip = clip.resize.Bicubic(format=vs.GRAY8, matrix_s="709").misc.SCDetect(threshold)
[vapoursynth] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[vapoursynth] File "src\cython\vapoursynth.pyx", line 2007, in vapoursynth.VideoNode.getattr
[vapoursynth] AttributeError: There is no attribute or namespace named misc
[vapoursynth]
[vapoursynth] could not init VS

vs-rife + latest vs-dpir don't work

When using just vs-rife:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'C:\Users\Selur\Desktop\VTS_01_1.VOB'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading C:\Users\Selur\Desktop\VTS_01_1.VOB using D2VSource
clip = core.d2v.Source(input="E:/Temp/vob_941fdaaeda22090766694391cc4281d5_853323747.d2v")
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip, mode=7, rate=10, dupThresh=0.04, vidThresh=3.50, sceneThresh=15.00)# new fps: 10
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
clip = core.misc.SCDetect(clip=clip,threshold=0.150)
from vsrife import RIFE
# adjusting color space from YUV420P8 to RGBS for VsTorchRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# adjusting frame count&rate with RIFE (torch)
clip = RIFE(clip, multi=3, device_type='cuda', device_index=0) # new fps: 20
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 30.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=30, fpsden=1)
# Output
clip.set_output()

everything works.
But when I add latest vs-dpir:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import os
import site
# Import libraries for onnxruntime
from ctypes import WinDLL
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
WinDLL(path+'cublas64_11.dll')
WinDLL(path+'cudart64_110.dll')
WinDLL(path+'cudnn64_8.dll')
WinDLL(path+'cudnn_cnn_infer64_8.dll')
WinDLL(path+'cudnn_ops_infer64_8.dll')
WinDLL(path+'cufft64_10.dll')
WinDLL(path+'cufftw64_10.dll')
WinDLL(path+'nvinfer.dll')
WinDLL(path+'nvinfer_plugin.dll')
WinDLL(path+'nvparsers.dll')
WinDLL(path+'nvonnxparser.dll')
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'C:\Users\Selur\Desktop\VTS_01_1.VOB'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading C:\Users\Selur\Desktop\VTS_01_1.VOB using D2VSource
clip = core.d2v.Source(input="E:/Temp/vob_941fdaaeda22090766694391cc4281d5_853323747.d2v")
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip, mode=7, rate=10, dupThresh=0.04, vidThresh=3.50, sceneThresh=15.00)# new fps: 10
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
from vsdpir import DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=15.000, task="denoise", provider=1, device_id=0)
clip = core.resize.Bicubic(clip=clip, format=vs.YUV444P16, matrix_s="470bg", range_s="limited")
clip = core.misc.SCDetect(clip=clip,threshold=0.150)
from vsrife import RIFE
# adjusting color space from YUV444P16 to RGBS for VsTorchRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# adjusting frame count&rate with RIFE (torch)
clip = RIFE(clip, multi=3, device_type='cuda', device_index=0) # new fps: 20
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 30.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=30, fpsden=1)
# Output
clip.set_output()

I get:

Python exception: [WinError 127] Die angegebene Prozedur wurde nicht gefunden. Error loading "I:\Hybrid\64bit\Vapoursynth\Lib/site-packages\torch\lib\cudnn_cnn_train64_8.dll" or one of its dependencies.

Using just vs-dpir:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
import os
import site
# Import libraries for onnxruntime
from ctypes import WinDLL
path = site.getsitepackages()[0]+'/onnxruntime_dlls/'
WinDLL(path+'cublas64_11.dll')
WinDLL(path+'cudart64_110.dll')
WinDLL(path+'cudnn64_8.dll')
WinDLL(path+'cudnn_cnn_infer64_8.dll')
WinDLL(path+'cudnn_ops_infer64_8.dll')
WinDLL(path+'cufft64_10.dll')
WinDLL(path+'cufftw64_10.dll')
WinDLL(path+'nvinfer.dll')
WinDLL(path+'nvinfer_plugin.dll')
WinDLL(path+'nvparsers.dll')
WinDLL(path+'nvonnxparser.dll')
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/DeinterlaceFilter/TIVTC/libtivtc.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/d2vSource/d2vsource.dll")
# source: 'C:\Users\Selur\Desktop\VTS_01_1.VOB'
# current color space: YUV420P8, bit depth: 8, resolution: 720x480, fps: 29.97, color matrix: 470bg, yuv luminance scale: limited, scanorder: telecine
# Loading C:\Users\Selur\Desktop\VTS_01_1.VOB using D2VSource
clip = core.d2v.Source(input="E:/Temp/vob_941fdaaeda22090766694391cc4281d5_853323747.d2v")
# Setting color matrix to 470bg.
clip = core.std.SetFrameProps(clip, _Matrix=5)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=5)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=5)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 29.970
clip = core.std.AssumeFPS(clip=clip, fpsnum=30000, fpsden=1001)
# Deinterlacing using TIVTC
clip = core.tivtc.TFM(clip=clip)
clip = core.tivtc.TDecimate(clip=clip, mode=7, rate=10, dupThresh=0.04, vidThresh=3.50, sceneThresh=15.00)# new fps: 10
# make sure content is preceived as frame based
clip = core.std.SetFieldBased(clip, 0)
from vsdpir import DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# denoising using DPIRDenoise
clip = DPIR(clip=clip, strength=15.000, task="denoise", provider=1, device_id=0)
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 10.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=10, fpsden=1)
# Output
clip.set_output()

works fine.

-> do you have an idea how I could fix this?

Wrong output framerate

That -

vs-rife/vsrife/__init__.py

Line 280 in 91e894f

format_clip = clip.std.BlankClip(

- doesn't work as expected because of two reasons:

1. clip.fps.numerator / denominator can be 0 / 1 (from the docs: "It is 0/1 when the clip has a variable framerate")
2. there's a frame duration attached to each frame, and it seems like FrameEval(frame_adjuster) return frames with the original durations, not the ones from format_clip

A quick fix that works:

    clip0 = vs.core.std.Interleave([clip] * factor_num)
    if factor_den>1:
        clip0 = clip0.std.SelectEvery(cycle=factor_den,offsets=0)
    clip1 = clip.std.DuplicateFrames(frames=clip.num_frames - 1).std.DeleteFrames(frames=0)
    clip1 = vs.core.std.Interleave([clip1] * factor_num)
    if factor_den>1:
        clip1 = clip1.std.SelectEvery(cycle=factor_den,offsets=0)

How to set clip.num_frames

How to set the frames numbers?I only found the "multi: int ="in "init.py".Can I set the whole number of the frames numbers?Like 60 fps?Thanks!

Use dynamic shapes for TRT backend?

Would it be possible to include a toggle to build a dynamic shapes engine instead of static as people like me who vary their input resolutions a lot will benefit much more from not having to build engine over the ~5% speed decrease in inference (referring to vs-mlrt wiki benchmarks)

Many thanks!

Could you?

please implement DDColor to vapoursynth?
the best image colorization I have ever seen.
https://github.com/piddnad/DDColor
thank you very much in advance!

How to get Torch 2.5.0.dev and TensorRT 2.5.0.ev

I'm trying to updated to the latest version using:

F:\Hybrid\64bit\Vapoursynth>python -m pip install --pre torch_tensorrt --index-url https://download.pytorch.org/whl/nightly/cu124
Looking in indexes: https://download.pytorch.org/whl/nightly/cu124
Requirement already satisfied: torch_tensorrt in f:\hybrid\64bit\vapoursynth\lib\site-packages (2.4.0.dev20240518+cu121)
Requirement already satisfied: torch<2.5.0,>=2.4.0.dev in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch_tensorrt) (2.4.0.dev20240523+cu121)
Requirement already satisfied: tensorrt<10.1.0,>=10.0.1 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch_tensorrt) (10.0.1)
Requirement already satisfied: packaging>=23 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch_tensorrt) (24.0)
Requirement already satisfied: numpy in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch_tensorrt) (1.26.4)
Requirement already satisfied: typing-extensions>=4.7.0 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch_tensorrt) (4.12.0)
Requirement already satisfied: tensorrt-cu12 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from tensorrt<10.1.0,>=10.0.1->torch_tensorrt) (10.0.1)
Requirement already satisfied: filelock in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (3.14.0)
Requirement already satisfied: sympy in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (1.12)
Requirement already satisfied: networkx in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (3.3)
Requirement already satisfied: jinja2 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (3.1.4)
Requirement already satisfied: fsspec in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (2024.5.0)
Requirement already satisfied: mkl<=2021.4.0,>=2021.1.1 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (2021.4.0)
Requirement already satisfied: intel-openmp==2021.* in f:\hybrid\64bit\vapoursynth\lib\site-packages (from mkl<=2021.4.0,>=2021.1.1->torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (2021.4.0)
Requirement already satisfied: tbb==2021.* in f:\hybrid\64bit\vapoursynth\lib\site-packages (from mkl<=2021.4.0,>=2021.1.1->torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (2021.12.0)
Requirement already satisfied: MarkupSafe>=2.0 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from jinja2->torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (2.1.5)
Requirement already satisfied: mpmath>=0.19 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from sympy->torch<2.5.0,>=2.4.0.dev->torch_tensorrt) (1.3.0)

F:\Hybrid\64bit\Vapoursynth>python -m pip install tensorrt==10.0.1 tensorrt-cu12_bindings==10.0.1 tensorrt-cu12_libs==10.0.1 --extra-index-url https://pypi.nvidia.com
Looking in indexes: https://pypi.org/simple, https://pypi.nvidia.com
Requirement already satisfied: tensorrt==10.0.1 in f:\hybrid\64bit\vapoursynth\lib\site-packages (10.0.1)
Requirement already satisfied: tensorrt-cu12_bindings==10.0.1 in f:\hybrid\64bit\vapoursynth\lib\site-packages (10.0.1)
Requirement already satisfied: tensorrt-cu12_libs==10.0.1 in f:\hybrid\64bit\vapoursynth\lib\site-packages (10.0.1)
Requirement already satisfied: tensorrt-cu12 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from tensorrt==10.0.1) (10.0.1)
Requirement already satisfied: nvidia-cuda-runtime-cu12 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from tensorrt-cu12_libs==10.0.1) (12.5.39)

F:\Hybrid\64bit\Vapoursynth>python -m pip install -U vsrife==5.2.0
Collecting vsrife==5.2.0
  Using cached vsrife-5.2.0-py3-none-any.whl.metadata (1.8 kB)
Requirement already satisfied: numpy in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsrife==5.2.0) (1.26.4)
Requirement already satisfied: requests in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsrife==5.2.0) (2.32.2)
INFO: pip is looking at multiple versions of vsrife to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement torch>=2.5.0.dev (from vsrife) (from versions: 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1)
ERROR: No matching distribution found for torch>=2.5.0.dev

using the calls from the README.md, but I only get Torch 2.4.0.dev and TensorRT 2.4.0.dev and thus can't install vsrife 5.2.0.
Any advice how to get this working?

Requesting integration with MPV

Can this be integrated into MPV? MPV allows usage of vaoursynth filters so is it possible to use this a vf filter?

Vapoursynth R58 and Python 3.10 compatibilty

trying to install vs-rife in Vapoursynth R58 I get:

I:\Hybrid\64bit\Vapoursynth>python -m pip install --upgrade vsrife
Collecting vsrife
  Using cached vsrife-2.0.0-py3-none-any.whl (32.5 MB)
Requirement already satisfied: torch>=1.9.0 in i:\hybrid\64bit\vapoursynth\lib\site-packages (from vsrife) (1.11.0+cu113)
Requirement already satisfied: numpy in i:\hybrid\64bit\vapoursynth\lib\site-packages (from vsrife) (1.22.3)
Collecting VapourSynth>=55
  Using cached VapourSynth-57.zip (567 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      Traceback (most recent call last):
        File "C:\Users\Selur\AppData\Local\Temp\pip-install-s7976394\vapoursynth_701a37362cd045f58da4818d07217c99\setup.py", line 64, in <module>
          dll_path = query(winreg.HKEY_LOCAL_MACHINE, REGISTRY_PATH, REGISTRY_KEY)
        File "C:\Users\Selur\AppData\Local\Temp\pip-install-s7976394\vapoursynth_701a37362cd045f58da4818d07217c99\setup.py", line 38, in query
          reg_key = winreg.OpenKey(hkey, path, 0, winreg.KEY_READ)
      FileNotFoundError: [WinError 2] Das System kann die angegebene Datei nicht finden

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\Selur\AppData\Local\Temp\pip-install-s7976394\vapoursynth_701a37362cd045f58da4818d07217c99\setup.py", line 67, in <module>
          raise OSError("Couldn't detect vapoursynth installation path")
      OSError: Couldn't detect vapoursynth installation path
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

any idea how to fix it?

Potential Speedup with number of threads

@HolyWu

I commented out:

#if num_streams > vs.core.num_threads:
#    raise vs.Error("rife: setting num_streams greater than `core.num_threads` is useless")

and then set the number of streams to 24. It gave me 125-135 Frames per second, instead of 115 Frames per second with 12 streams.
If I set it to 25/26 streams, it's getting very slow like 55fps, because of the new cuda fallback policy, which literally would end the program instead of working slowly.

My hardware:
I5-10500H Laptop with 6Cores / 12 threads.
RTX 3060 with 6144mb

The video I have tested, just the first minute:
720p anime video.

Suggestion:

I will try to look up with the help ai coding tools, if we can slowly increase the number of streams up until the point, where the vram is full with a 10% percent margin. So we might get the biggest boost. I do not think we should set a default num_stream number or at least set it to 1, because, the numofstreams value may change with the resoultion of the video.

A 4k video is 9 times bigger than a 720p video, which means, you cannot have "numofstreams=24", which would not fit into the memory. A dynamic approach to this would be better.

I will close this issue, since this is just a documentation for me and if you think, this is important for you.

RIFE 4.8 is available

Would be nice if you could update to support 4.8 :)

Solved: "No module named 'packaging'"

D:\dependencies>python -m vsrife
Traceback (most recent call last):
  File "<frozen runpy>", line 189, in _run_module_as_main
  File "<frozen runpy>", line 148, in _get_module_details
  File "<frozen runpy>", line 112, in _get_module_details
  File "D:\Python\Python311\Lib\site-packages\vsrife\__init__.py", line 12, in <module>
    from torch_tensorrt.fx import LowerSetting
  File "D:\Python\Python311\Lib\site-packages\torch_tensorrt\fx\__init__.py", line 1, in <module>
    from .converters import *  # noqa: F403 F401
    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python\Python311\Lib\site-packages\torch_tensorrt\fx\converters\__init__.py", line 5, in <module>
    from .adaptive_avgpool import *  # noqa: F401 F403
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "D:\Python\Python311\Lib\site-packages\torch_tensorrt\fx\converters\adaptive_avgpool.py", line 7, in <module>
    from .converter_utils import extend_mod_attr_to_tuple, mark_as_int8_layer
  File "D:\Python\Python311\Lib\site-packages\torch_tensorrt\fx\converters\converter_utils.py", line 23, in <module>
    from ..utils import torch_dtype_from_trt
  File "D:\Python\Python311\Lib\site-packages\torch_tensorrt\fx\utils.py", line 3, in <module>
    from packaging import version
ModuleNotFoundError: No module named 'packaging'

Error will go away after:

pip install packaging

New RIFE_m model

https://github.com/hzwer/arXiv2020-RIFE/blob/main/model/IFNet_m.py

https://drive.google.com/file/d/147XVsDXBfJPlyct2jfo9kpbL944mNeZr/view?usp=sharing

Cannot import name 'SourceIR' from 'torch_tensorrt.fx.converters.converter_utils'

• created 'Vapoursynth'-folder
• downloaded 'Windows embeddable package (64-bit)' from https://www.python.org/downloads/release/python-3123/
• extracted it into the 'Vapoursynth'-folder
• downloaded 'VapourSynth64-Portable-R68' from https://github.com/vapoursynth/vapoursynth/releases
• extracted it into the 'Vapoursynth'-folder
• finished Vapoursynth portable installation:
  • adjusted python312._pth by adding:

  Scripts
  Lib\site-packages
  

  • installed pip:
    • download get-pip.py from https://bootstrap.pypa.io/get-pip.py into the 'Vapoursynth'-folder
    • python get-pip.py
  • python -m pip install wheel\VapourSynth-68-cp312-cp312-win_amd64.whl
• installed torch dependencies:
  • python -m pip install wheel
    • install TensorRT FX
    • python -m pip install torch_tensorrt_fx_only
  • install Pytorch 2.4 dev (uninstalled torch 2.3.0 and installed Pytorch 2.4.0.dev20240519+cu121)
    • python -m pip install --pre torch torchvision --index-url https://download.pytorch.org/whl/nightly/cu121
  • install TensorRT
  • python -m pip install https://github.com/HolyWu/vs-dpir/releases/download/v4.1.0/torch_tensorrt-2.4.0.dev20240518+cu121-cp312-cp312-win_amd64.whl
  • python -m pip install tensorrt==10.0.1 tensorrt-cu12_bindings==10.0.1 tensorrt-cu12_libs==10.0.1 --extra-index-url https://pypi.nvidia.com
• installed 'vsrife':
  • python -m pip install -U vsrife
  • python -m vsrife
    Calling it with:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
import sys
import os
core = vs.core
# Import scripts folder
scriptPath = 'F:/Hybrid/64bit/vsscripts'
sys.path.insert(0, os.path.abspath(scriptPath))
# loading plugins
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="F:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/LSMASHSource.dll")
# Import scripts
import ChangeFPS
import validate
# Source: 'G:\TestClips&Co\files\test.avi'
# Current color space: YUV420P8, bit depth: 8, resolution: 640x352, frame rate: 25fps, scanorder: progressive, yuv luminance scale: limited, matrix: 470bg
# Loading G:\TestClips&Co\files\test.avi using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/test.avi", format="YUV420P8", stream_index=0, cache=0, fpsnum=25, prefer_hw=0)
frame = clip.get_frame(0)
# Setting detected color matrix (470bg).
clip = core.std.SetFrameProps(clip=clip, _Matrix=5)
# setting color transfer (170), if it is not set.
if validate.transferIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Transfer=6)
# setting color primaries info (to 470), if it is not set.
if validate.primariesIsInvalid(clip):
  clip = core.std.SetFrameProps(clip=clip, _Primaries=5)
# setting color range to TV (limited) range.
clip = core.std.SetFrameProps(clip=clip, _ColorRange=1)
# making sure frame rate is set to 25fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# making sure the detected scan type is set (detected: progressive)
clip = core.std.SetFrameProps(clip=clip, _FieldBased=0) # progressive
original = clip
from vsrife import rife
# adjusting color space from YUV420P8 to RGBS for vsTorchRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# adjusting frame count&rate with RIFE (torch)
clip = rife(clip, sc_threshold=0.150) # new fps: 50
# adjusting frame count by adding duplicate frames
original = ChangeFPS.ChangeFPS(clip=original,target_fps_num=50,target_fps_den=1)
# adjusting output color from: YUV420P8 to YUV420P10 for NVEncModel
original = core.resize.Bicubic(clip=original, format=vs.YUV420P10, range_s="limited")
# adjusting output color from: RGBS to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_s="limited", dither_type="error_diffusion")
original = core.text.Text(clip=original,text="Original",scale=1,alignment=7)
clip = core.text.Text(clip=clip,text="Filtered",scale=1,alignment=7)
stacked = core.std.StackHorizontal([original,clip])
# set output frame rate to 50fps (progressive)
stacked = core.std.AssumeFPS(clip=stacked, fpsnum=50, fpsden=1)
# output
stacked.set_output()

results in:

Python exception: cannot import name 'SourceIR' from 'torch_tensorrt.fx.converters.converter_utils' (F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\fx\converters\converter_utils.py)

Traceback (most recent call last):
File "src\\cython\\vapoursynth.pyx", line 3365, in vapoursynth._vpy_evaluate
File "src\\cython\\vapoursynth.pyx", line 3366, in vapoursynth._vpy_evaluate
File "J:\tmp\tempPreviewVapoursynthFile20_19_40_586.vpy", line 40, in 
clip = rife(clip, sc_threshold=0.150) # new fps: 50
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsrife\__init__.py", line 199, in rife
from .IFNet_HDv3_v4_15_lite import IFNet
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsrife\IFNet_HDv3_v4_15_lite.py", line 4, in 
from .interpolate import interpolate
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\vsrife\interpolate.py", line 11, in 
from torch_tensorrt.dynamo._SourceIR import SourceIR
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\__init__.py", line 133, in 
from torch_tensorrt.dynamo import backend # noqa: F401
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\dynamo\__init__.py", line 10, in 
from ._compiler import compile, convert_module_to_trt_engine
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\dynamo\_compiler.py", line 14, in 
from torch_tensorrt.dynamo import _defaults, partitioning
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\dynamo\partitioning\__init__.py", line 1, in 
from ._adjacency_partitioner import partition as fast_partition
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\dynamo\partitioning\_adjacency_partitioner.py", line 20, in 
from torch_tensorrt.dynamo.conversion._ConverterRegistry import (
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\dynamo\conversion\__init__.py", line 1, in 
from . import aten_ops_converters, ops_evaluators, prims_ops_converters
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\dynamo\conversion\aten_ops_converters.py", line 11, in 
from torch_tensorrt.dynamo.conversion import impl
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\dynamo\conversion\impl\__init__.py", line 1, in 
from torch_tensorrt.fx.converters.impl import convolution
File "F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\fx\converters\impl\convolution.py", line 9, in 
from torch_tensorrt.fx.converters.converter_utils import (
ImportError: cannot import name 'SourceIR' from 'torch_tensorrt.fx.converters.converter_utils' (F:\Hybrid\64bit\Vapoursynth\Lib\site-packages\torch_tensorrt\fx\converters\converter_utils.py)

calling python -m pip list
shows:

Package                  Version
------------------------ ------------------------
annotated-types          0.7.0
antlr4-python3-runtime   4.9.3
beautifulsoup4           4.12.3
blis                     0.7.11
Bottleneck               1.3.8
catalogue                2.0.10
certifi                  2024.2.2
charset-normalizer       3.3.2
click                    8.1.7
cloudpathlib             0.16.0
colorama                 0.4.6
confection               0.1.4
contourpy                1.2.1
cupy-cuda11x             13.1.0
cycler                   0.12.1
cymem                    2.0.8
einops                   0.8.0
fastai                   1.0.60
fastprogress             1.0.3
fastrlock                0.8.2
filelock                 3.14.0
fonttools                4.52.1
fsspec                   2024.5.0
huggingface-hub          0.23.1
idna                     3.7
imageio                  2.34.1
intel-openmp             2021.4.0
Jinja2                   3.1.4
kiwisolver               1.4.5
kornia                   0.7.2
kornia_rs                0.1.3
langcodes                3.4.0
language_data            1.2.0
lazy_loader              0.4
llvmlite                 0.42.0
marisa-trie              1.1.1
markdown-it-py           3.0.0
MarkupSafe               2.1.5
matplotlib               3.9.0
mdurl                    0.1.2
mkl                      2021.4.0
mpmath                   1.3.0
murmurhash               1.0.10
networkx                 3.3
numba                    0.59.1
numexpr                  2.10.0
numpy                    1.26.4
nvidia-cuda-runtime-cu12 12.5.39
nvidia-ml-py3            7.352.0
omegaconf                2.3.0
opencv-contrib-python    4.9.0.80
opencv-python            4.9.0.80
packaging                24.0
pandas                   2.2.2
Pillow                   10.1.0
pip                      24.0
preshed                  3.0.9
pydantic                 2.7.1
pydantic_core            2.18.2
Pygments                 2.18.0
pyparsing                3.1.2
python-dateutil          2.9.0.post0
pytz                     2024.1
PyYAML                   6.0.1
requests                 2.32.2
rich                     13.7.1
safetensors              0.4.3
scikit-image             0.23.2
scipy                    1.13.1
setuptools               70.0.0
six                      1.16.0
smart-open               6.4.0
soupsieve                2.5
spacy                    3.7.4
spacy-legacy             3.0.12
spacy-loggers            1.0.5
srsly                    2.4.8
stgpytools               1.0.5
sympy                    1.12
tbb                      2021.12.0
tensorrt                 10.0.1
tensorrt-cu12            10.0.1
tensorrt-cu12_bindings   10.0.1
tensorrt-cu12_libs       10.0.1
thinc                    8.2.3
tifffile                 2024.5.22
timm                     1.0.3
torch                    2.4.0.dev20240523+cu121
torch_tensorrt           2.4.0.dev20240518+cu121
torch-tensorrt-fx-only   1.4.0.dev0
torchvision              0.19.0.dev20240525+cu121
tqdm                     4.66.4
typer                    0.9.4
typing_extensions        4.12.0
tzdata                   2024.1
urllib3                  2.2.1
VapourSynth              68
vsanimesr                1.0.0
vscodeformer             1.0.1
vsddcolor                1.0.1
vsdeoldify               3.5.3
vsdpir                   4.1.0
vsgan                    1.6.4
vsgmfss_fortuna          1.0.0
vsgrlir                  1.1.1
vshinet                  1.0.0
vsrealesrgan             4.0.0
vsrife                   5.0.0
vsscunet                 1.0.0
vsswinir                 2.0.0
vstools                  3.1.0
vsutil                   0.8.0
wasabi                   1.1.2
weasel                   0.3.4
wheel                    0.43.0

Any idea what I'm missing / doing wrong?

VapourSynth-RIFE-ncnn-Vulkan (RIFE-r9-win64.7z)

Did you make the plugin "https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan" (RIFE-r9-win64.7z)?
and because I wanted to use rife-v4.8
but it gives the error "layer MemoryData not exists or registered"
I already tried using the version of "https://github.com/styler00dollar/VapourSynth-RIFE-ncnn-Vulkan"
but it doesn't work on my pc.
with RIFE.dll I can use even rife-v4.0
I would like to know if you still have the "Source code (zip)"
because "https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan" is no longer on github

Could you please port EMA-VFI?

Results are much better than RIFE.
Thank you very much in advance

[Feature request] Live feed

Thank you for great repo!
I'm interesting in way to use it with live feed, for ex. cv2 webcam.

[Q] 0bit models in the repo

Hi

i see in the model folders, have a files (models?) with 0bits, i presume when the plugin "learn", the models is filled with the data

this is correct?

then, in a system with install this plugin as system-wide, these models should be have a write permissions? (in case of linux)

greetings

half the image is broken when using 4k content

I get a broken output (see attachment), when using:

# Imports
import vapoursynth as vs
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/MiscFilter/MiscFilters/MiscFilters.dll")
core.std.LoadPlugin(path="i:/Hybrid/64bit/vsfilters/SourceFilter/LSmashSource/vslsmashsource.dll")
# source: 'G:\TestClips&Co\files\MPEG-4 H.264\4k\Back to the Future (1985) 4k 10bit - 0.10.35-0.11.35.mkv'
# current color space: YUV420P10, bit depth: 10, resolution: 3840x2076, fps: 23.976, color matrix: 2020ncl, yuv luminance scale: limited, scanorder: progressive
# Loading G:\TestClips&Co\files\MPEG-4 H.264\4k\Back to the Future (1985) 4k 10bit - 0.10.35-0.11.35.mkv using LWLibavSource
clip = core.lsmas.LWLibavSource(source="G:/TestClips&Co/files/MPEG-4 H.264/4k/Back to the Future (1985) 4k 10bit - 0.10.35-0.11.35.mkv", format="YUV420P10", cache=0, fpsnum=24000, fpsden=1001, prefer_hw=1)
# Setting color matrix to 2020ncl.
clip = core.std.SetFrameProps(clip, _Matrix=9)
clip = clip if not core.text.FrameProps(clip,'_Transfer') else core.std.SetFrameProps(clip, _Transfer=9)
clip = clip if not core.text.FrameProps(clip,'_Primaries') else core.std.SetFrameProps(clip, _Primaries=9)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
# making sure frame rate is set to 23.976
clip = core.std.AssumeFPS(clip=clip, fpsnum=24000, fpsden=1001)
clip = core.misc.SCDetect(clip=clip,threshold=0.150)
from vsrife import RIFE
# adjusting color space from YUV420P10 to RGBS for VsTorchRIFE
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="2020ncl", range_s="limited")
# adjusting frame count&rate with RIFE (torch)
clip = RIFE(clip, scale=0.5, multi=3, device_type='cuda', device_index=0, fp16=True) # new fps: 71.928
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="2020ncl", range_s="limited", dither_type="error_diffusion")
# set output frame rate to 71.928fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=8991, fpsden=125)
# Output
clip.set_output()

tried different scale values, fp16 disabled, without scene change detection and other values for mult, nothing helped.
https://github.com/HomeOfVapourSynthEvolution/VapourSynth-RIFE-ncnn-Vulkan works fine.
2k content also works fine.
I tried different source filters and different files.
Would be nice if this could be fixed.

attachment was too large: https://ibb.co/WGT9pvL

Model 4.9 is available

Could you add support for 4.9 ?

Please add new 4.7 to 4.12 models (now with ensemble support)

Hey @HolyWu.
hzwer was so kind to add ensemble support for 4.7+ models in https://github.com/hzwer/Practical-RIFE and he also added some new ones.

Could you please add them to vs-rife too?

Note:
As far as I understand it we wouldn't need the revision/sub-minor number for mode switches in vs-rife.
According to hzwer only ensemble support was added, so I guess it's just updated inference scripts.
But please, check it out for yourself and ultimately it's your decision of course!

Install problems - solved

Ran into 2 problems, installing the current vs-rife version (on Windows) today, that cost me some time to resolve and here are the solutions:

Problem 1:

Collecting tensorrt>=8.5.2.2 (from vsrife)
  Using cached tensorrt-8.6.1.post1.tar.gz (18 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [21 lines of output]
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\Samhayne\AppData\Local\Temp\pip-install-2qleovil\tensorrt_1b55b028ef7f49f3b31b96c06fee5dd8\setup.py", line 103, in <module>
          if disable_internal_pip or nvidia_pip_index_url in parent_command_line() or nvidia_pip_index_url in pip_config_list():
                                                             ^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\Samhayne\AppData\Local\Temp\pip-install-2qleovil\tensorrt_1b55b028ef7f49f3b31b96c06fee5dd8\setup.py", line 96, in parent_command_line
          return subprocess.check_output(["ps", "-p", str(pid), "-o", "command", "--no-headers"]).decode()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "D:\Python\Python311\Lib\subprocess.py", line 466, in check_output
          return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "D:\Python\Python311\Lib\subprocess.py", line 548, in run
          with Popen(*popenargs, **kwargs) as process:
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "D:\Python\Python311\Lib\subprocess.py", line 1026, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "D:\Python\Python311\Lib\subprocess.py", line 1538, in _execute_child
          hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      FileNotFoundError: [WinError 2] Das System kann die angegebene Datei nicht finden
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Fixed by installing psutil:

pip install psutil

Problem 2:

Looking in indexes: https://pypi.org/simple, https://pypi.nvidia.com
Collecting tensorrt-libs
  Using cached tensorrt-libs-8.6.1.tar.gz (6.8 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'D:\Python\Python311\python.exe' -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\Samhayne\\AppData\\Local\\Temp\\pip-install-tkxmv8xe\\tensorrt-libs_454e03b7bfb443408d87c2e2b512cda7\\setup.py'"'"'; __file__='"'"'C:\\Users\\Samhayne\\AppData\\Local\\Temp\\pip-install-tkxmv8xe\\tensorrt-libs_454e03b7bfb443408d87c2e2b512cda7\\setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\Samhayne\AppData\Local\Temp\pip-pip-egg-info-fuzvb0d_'
       cwd: C:\Users\Samhayne\AppData\Local\Temp\pip-install-tkxmv8xe\tensorrt-libs_454e03b7bfb443408d87c2e2b512cda7\
  Complete output (15 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "C:\Users\Samhayne\AppData\Local\Temp\pip-install-tkxmv8xe\tensorrt-libs_454e03b7bfb443408d87c2e2b512cda7\setup.py", line 137, in <module>
      raise RuntimeError(open("ERROR.txt", "r").read())
  RuntimeError:
  ###########################################################################################
  The package you are trying to install is only a placeholder project on PyPI.org repository.
  This package is hosted on NVIDIA Python Package Index.

  This package can be installed as:
  ```
  $ pip install --extra-index-url https://pypi.nvidia.com tensorrt-libs
  ```
  ###########################################################################################

  ----------------------------------------

Looks like at PyPI there's no Windows version of TensorRT available at the moment. (see: NVIDIA/TensorRT#2933 (comment))
(Couldn't get an older version either from there)

Solution: Install manually from zip as described here:
https://docs.nvidia.com/deeplearning/tensorrt/install-guide/index.html#installing-zip

Issue running

Hi HolyWu,

Thanks for your hard work, but after many hours troubleshooting I have no choice but to ask for help. I hope that's ok.

My script throws an error, and after troubleshooting, I narrowed it down to torch_tensorrt. This is what happens when I try to import torch_tensorrt in Python:

Python 3.12.0 (tags/v3.12.0:0fb18b0, Oct 2 2023, 13:03:39) [MSC v.1935 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

import torch_tensorrt
Traceback (most recent call last):
File "", line 1, in
File "C:\Python34\Lib\site-packages\torch_tensorrt_init_.py", line 127, in
from torch_tensorrt import ts
File "C:\Python34\Lib\site-packages\torch_tensorrt\ts_init_.py", line 1, in
from torch_tensorrt.ts._compile_spec import TensorRTCompileSpec # noqa: F401
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python34\Lib\site-packages\torch_tensorrt\ts_compile_spec.py", line 7, in
import torch_tensorrt._C.ts as _ts_C
ImportError: DLL load failed while importing _C: The specified procedure could not be found.
exit()

I know the folder says Python34, but I assure you I have Python 3.12 installed.

Here is my pip list:

tensorrt 10.0.1
tensorrt-cu12 10.2.0.post1
tensorrt-cu12_bindings 10.0.1
tensorrt-cu12_libs 10.0.1
TIVTC 2
torch 2.5.0.dev20240802+cu124
torch_tensorrt 2.5.0.dev20240722+cu124
torchaudio 2.4.0.dev20240802+cu124
torchvision 0.20.0.dev20240802+cu124
tqdm 4.66.1
TTempSmooth 4.1
typing_extensions 4.8.0
tzlocal 1.5.1
urllib3 2.2.2
VapourSynth 69
virtualenv 16.6.1
virtualenv-clone 0.5.3
vsbasicvsr 1.0.0
vsgan 1.2.1
vsrealesrgan 1.0.0
vsrife 5.2.0
Waifu2x_NCNN_Vulkan 5
Waifu2x_NCNN_Vulkan_models 0.1
whichcraft 0.6.0
yapf 0.31.0
ZNEDI3 2.1

Am I doing something wrong?

4K video stuttering with this as filter in MPV

I simply cannot get 4K video to work properly, no matter how many parameters I change. It works fine for the first second, then goes wildly down the drain. It's like if it was a 4fps video.

This is despite tweaking the different parameters, and trying the lowest values possible.
MPV also reports that the video fps is ~60, and my GPU isn't always maxed out, so the issue is probably not that it can't keep up?
Weirdly though, MPV reports the estimated screen refresh rate dips to around 13, which I have not have happen before.

Current VapourSynth script:

# Loosely based on https://raw.githubusercontent.com/hooke007/MPV_lazy/50ebf2c6570aa2db45bb158c3da2cbc8e3fb013e/portable_config/vs/rife_2x.vpy

from fractions import Fraction
import vapoursynth as vs
from vsrife import rife

output_format = vs.YUV444P10 # YUV444P10 (or YUV420P10) for full 10-bit color YUV420P8 for 8-bit color.
display_peak_brightness=800

# Disable RIFE when above threshold.
# Skip interpolation for >4K or >=60 Hz content due to performance
disable_fps_threshold = 59
disable_width_threshold = 3840
disable_height_threshold = 2160

# Switch to half scale when above threshold.
reduced_scale_width_threshold = disable_width_threshold * 0.85
reduced_scale_height_threshold = disable_height_threshold * 0.85
reduced_scale = 0.5

# Uses nvidia RT cores. Will require an RTX GPU (not tested on anything other than RTX 4080)
# It also takes a million years to build an RT engine for each resolution and config, but it is much faster than regular.
gpu_tensorrt=True
tensorrt_debug=True # Enable for TensorRT debug logging.

gpu_streams=4 # This setting very quickly explodes your VRAM usage.
gpu_format=vs.RGBH # RGBH is faster, RGBS is more accurate
ensemble=True # Produces better results but also more expensive.


core = vs.core
clip = video_in

def aprox(num, target, margin = 0.05):
    return target - margin <= num <= target + margin

if not (clip.width > disable_width_threshold or clip.height > disable_height_threshold or container_fps > disable_fps_threshold):
    sup  = core.mv.Super(clip, pel=1)
    vec = core.mv.Analyse(sup, blksize=8, isb=True)
    clip = core.mv.SCDetection(clip=clip, vectors=vec, thscd1=240, thscd2=130)

    target_fps = 60
    target_frac = Fraction(target_fps / container_fps).limit_denominator(100)

    print(f"Target frac={target_frac.numerator}/{target_frac.denominator}={target_frac.numerator/target_frac.denominator}")

    uhd = clip.width >= reduced_scale_width_threshold and clip.height >= reduced_scale_height_threshold

    if uhd:
        print("Clip is UHD. Adjusting settings...")

    scale = reduced_scale if uhd else 1
    ensemble = False if uhd else ensemble
    print(f"Expensive settings:\ngpu_streams={gpu_streams},scale={scale},ensemble={ensemble},gpu_tensorrt={gpu_tensorrt}")

    clip = core.resize.Lanczos(clip=clip, format=gpu_format, matrix_in_s='709')
    clip = rife(clip=clip, model="4.16.lite", factor_num=target_frac.numerator, factor_den=target_frac.denominator, device_index=0, num_streams=gpu_streams, scale=scale, ensemble=ensemble, sc=True, trt=gpu_tensorrt, trt_debug=tensorrt_debug)
    clip = core.resize.Lanczos(clip=clip, format=output_format, matrix_s="2020ncl", nominal_luminance=display_peak_brightness)

    new_fps = Fraction(container_fps * target_frac).limit_denominator(100)
    print(f"New fps_frac={new_fps.numerator}/{new_fps.denominator}={new_fps.numerator/new_fps.denominator}")
    clip = core.std.AssumeFPS(clip=clip, fpsnum=new_fps.numerator, fpsden=new_fps.denominator)

else:
    print("Video is too expensive to interpolate. Ignoring...")

clip.set_output()

RIFE model 4.15

Would be nice if 4.15 models could be supported.

Requesting example vapoursynth script

I tried to create a valid script for a while, but I can't make it run.

from vsrife import RIFE
import vapoursynth as vs
core = vs.core
core.std.LoadPlugin(path='/usr/lib/x86_64-linux-gnu/libffms2.so')
clip = core.ffms2.Source(source='test.webm')
print(clip) # YUV420P8
clip = vs.core.resize.Bicubic(clip, format=vs.RGBS)
print(clip) # RGBS
clip = RIFE(clip)
clip.set_output()

vspipe --y4m inference.py - | x264 - --demuxer y4m -o example.mkv

Error: Failed to retrieve frame 0 with error: Resize error: Resize error 3074: no path between colorspaces (2/2/2 => 0/2/2). May need to specify additional colorspace parameters.

Can I get an example that should actually work?

error when run cmd "python -m vsrife"

Got this error when run cmd "python -m vsrife"

C:\Windows\System32>python -m vsrife
Traceback (most recent call last):
File "", line 189, in run_module_as_main
File "", line 148, in get_module_details
File "", line 112, in get_module_details
File "C:\Program Files\Python311\Lib\site-packages\vsrife_init.py", line 13, in
from torch_tensorrt.fx import compile
File "C:\Program Files\Python311\Lib\site-packages\torch_tensorrt\fx_init.py", line 1, in
from .converters import * # noqa: F403 F401
^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Program Files\Python311\Lib\site-packages\torch_tensorrt\fx\converters_init.py", line 2, in
import tensorrt as trt
ModuleNotFoundError: No module named 'tensorrt'

C:\Windows\System32>

Also, where do I put the files i downloaded from after i uncompress them:

https://github.com/HolyWu/vs-rife/releases/download/v3.1.0/CUDA-11.7_cuDNN-8.6.0_TensorRT-8.5.2.2_win64.7z

Does it work on linux

Hello
Does it work on linux?

Small feature request for RIFEv4: target fps as alternative to multiplier

I would it be possible to allow setting a target fps instead of a multiplier when using RIFEv4?
When going from for example 23.976 (24000/1001) to 60 fps, having to use (60 * 1001 / 24000 =) 2,5025 is kind of annoying. ;)
I know could write a wrapper arount the rife.RIFE but I suspect depending on the resulting float it would be more accurate if this was done inside the filter.

performance issue?

I am unable to reach 1x speed when upscaling 24fps to 48fps, however my system is only running at 800/2000mhz, 8/24 vram, 10% cpu. increasing streams did not help. has anyone experience this or know how to improve utilized performance?

google colab request

I'd suggest making a colab notebook for this project as a demonstration of how to use this amazing repo, thanks so much!

RIFE 4.7 tuned for anime

https://github.com/hzwer/Practical-RIFE added RIFE 4.7 an updated vs-rife would be nice

How to set 'clip.num_frames

How to set the frames numbers?I only found the "multi: int ="in "init.py".Can I set the whole number of the frames numbers?Like 60 fps?Thanks!

' Add 4.12.lite model' pip release?

Seeing that 'pip install -U vsrife' returns v4.2.0 I was wondering if 4.12.lite model support is finished. :)

Getting Error when interpolating

    model.load_model(os.path.join(os.path.dirname(__file__), model_dir), -1)
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\site-packages\vsrife\RIFE_HDv2.py", line 164, in load_model
    convert(torch.load('{}/flownet.pkl'.format(path), map_location=self.torch_device)))
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\serialization.py", line 608, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
  File "C:\Users\\AppData\Local\Programs\Python\Python39\lib\site-packages\torch\serialization.py", line 777, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input  ```

Source file is a 720p 30fps mp4, loaded into VS through Lsmash source, set the format to RGBS. Nothing else
System specs are R7 3700x, 32GB of ram and a RTX 3060