holywu/vs-dpir

DPIR function for VapourSynth

DPIR

Plug-and-Play Image Restoration with Deep Denoiser Prior, based on https://github.com/cszn/DPIR.

Dependencies

• NumPy
• PyTorch >=2.0.1
• VapourSynth >=R60

trt requires additional runtime libraries:

• CUDA Toolkit 11.8
• cuDNN 8.9
• TensorRT >=8.6.1.6

For ease of installation on Windows, you can download the 7z file on Releases which contains required runtime libraries and Python wheel file. Either add the unzipped directory to your system PATH or copy the DLL files to a directory which is already in your system PATH. Finally pip install the Python wheel file.

Installation

pip install -U vsdpir
python -m vsdpir

Usage

from vsdpir import dpir

ret = dpir(clip)

See __init__.py for the description of the parameters.

vs-dpir's Issues

torch 2.0 compatibility?

Using: python -m pip install --upgrade vsdpir==3.0.1
I get:

Collecting vsdpir==3.0.1
  Downloading vsdpir-3.0.1-py3-none-any.whl (13 kB)
Requirement already satisfied: numpy in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.0.1) (1.24.3)
Requirement already satisfied: requests in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.0.1) (2.30.0)
Requirement already satisfied: tensorrt>=8.5.3.1 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.0.1) (8.6.1)
Collecting torch-tensorrt-fx-only>=1.3.0 (from vsdpir==3.0.1)
  Downloading torch_tensorrt_fx_only-1.3.0-py3-none-any.whl (128 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 128.4/128.4 kB 7.9 MB/s eta 0:00:00
Requirement already satisfied: torch>=1.13.0 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.0.1) (2.0.1+cu117)
Requirement already satisfied: tqdm in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.0.1) (4.65.0)
Requirement already satisfied: vapoursynth>=55 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.0.1) (62)
Requirement already satisfied: vsutil in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.0.1) (0.8.0)
Requirement already satisfied: filelock in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=1.13.0->vsdpir==3.0.1) (3.12.0)
Requirement already satisfied: typing-extensions in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=1.13.0->vsdpir==3.0.1) (4.5.0)
Requirement already satisfied: sympy in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=1.13.0->vsdpir==3.0.1) (1.12)
Requirement already satisfied: networkx in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=1.13.0->vsdpir==3.0.1) (3.1)
Requirement already satisfied: jinja2 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=1.13.0->vsdpir==3.0.1) (3.1.2)
INFO: pip is looking at multiple versions of torch-tensorrt-fx-only to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install vsdpir and vsdpir==3.0.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    vsdpir 3.0.1 depends on torch>=1.13.0
    torch-tensorrt-fx-only 1.3.0 depends on torch<1.14.0 and >=1.13.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

Could you update torch-tensorrt-fx-only and vs-dpir to work with torch 2.0.1?

Many errors

Getting massive amounts of script errors related to Pytorch 1.10, _c and nn

missing onnxruntime on setup.cfg(?)

Hi

i'm not python coder, but, should be added onnxruntime on the install_requires field into setup.cfg file?

greetings

Vapoursynth R58

Trying to install vs-dpir to latest portable Vapoursynth R58-RC1,
but I get:

I:\Hybrid\64bit\Vapoursynth>python -m pip install --upgrade vsdpir
Collecting vsdpir
  Downloading vsdpir-2.1.0-py3-none-any.whl (7.0 kB)
Collecting VapourSynth>=55
  Downloading VapourSynth-57.zip (567 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 567.1/567.1 KB 7.1 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [15 lines of output]
      Traceback (most recent call last):
        File "C:\Users\Selur\AppData\Local\Temp\pip-install-8umebnq_\vapoursynth_2f81ae9c21b34d768097c5a208b5989a\setup.py", line 64, in <module>
          dll_path = query(winreg.HKEY_LOCAL_MACHINE, REGISTRY_PATH, REGISTRY_KEY)
        File "C:\Users\Selur\AppData\Local\Temp\pip-install-8umebnq_\vapoursynth_2f81ae9c21b34d768097c5a208b5989a\setup.py", line 38, in query
          reg_key = winreg.OpenKey(hkey, path, 0, winreg.KEY_READ)
      FileNotFoundError: [WinError 2] Das System kann die angegebene Datei nicht finden

      During handling of the above exception, another exception occurred:

      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "C:\Users\Selur\AppData\Local\Temp\pip-install-8umebnq_\vapoursynth_2f81ae9c21b34d768097c5a208b5989a\setup.py", line 67, in <module>
          raise OSError("Couldn't detect vapoursynth installation path")
      OSError: Couldn't detect vapoursynth installation path
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

any idea how to get vs-dpir setup with Vapoursynth R58?

ONNX Runtime vs. provider selection?

Can you shed some light on what runtime (onnxruntime, onnxruntime-gpu and onnxruntime-directml) is need for which provider?

When I install 'onnxruntime-gpu' and use provider=1, I get:

2022-03-19 20:14:30.0977965 [E:onnxruntime:Default, provider_bridge_ort.cc:995 onnxruntime::ProviderLibrary::Get] LoadLibrary failed with error 126 "Das angegebene Modul wurde nicht gefunden." when trying to load "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll" 2022-03-19 20:14:30.0978594 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:535 onnxruntime::python::CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.
Vapoursynth preview error: 2022-03-19 20:14:30.3973685 [E:onnxruntime:Default, provider_bridge_ort.cc:995 onnxruntime::ProviderLibrary::Get] LoadLibrary failed with error 126 "Das angegebene Modul wurde nicht gefunden." when trying to load "I:\Hybrid\64bit\Vapoursynth\Lib\site-packages\onnxruntime\capi\onnxruntime_providers_cuda.dll" 2022-03-19 20:14:30.3974014 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:535 onnxruntime::python::CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/reference/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.

(dpir still seems to work)
When I uninstall 'onnxruntime-gpu', install 'onnxruntime' and use provider=1, no error appears.
When I uninstall 'onnxruntime', install 'onnxruntime-directml' and use provider=1, no error appears. When use provider=3, no error appears and the processing is a lot faster.

-> Is there a downside of using onnxruntime-directml ?

Also using provider=0 and provider=1 doesn't seem to make a difference. From the looks of it cpu&gpu usage both times is the same.
Would be nice if you could shed some light on what is used when. :)

Problem installing

Trying to get a Vapoursynth R66 setup running installing vs-dpir I run into:

python -m pip install -U vsdpir==3.1.1
Collecting vsdpir==3.1.1
  Using cached vsdpir-3.1.1-py3-none-any.whl.metadata (3.1 kB)
Requirement already satisfied: numpy>=1.24.3 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.1.1) (1.26.4)
Requirement already satisfied: requests>=2.30.0 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.1.1) (2.31.0)
Requirement already satisfied: tensorrt>=8.6.1 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.1.1) (10.0.0b6)
Collecting torch-tensorrt-fx-only>=1.5.0.dev0 (from vsdpir==3.1.1)
  Using cached torch_tensorrt_fx_only-1.5.0.dev0-py3-none-any.whl.metadata (13 kB)
Requirement already satisfied: torch>=2.0.1 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.1.1) (2.2.0+cu121)
Requirement already satisfied: tqdm>=4.65.0 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.1.1) (4.66.2)
Requirement already satisfied: vapoursynth>=60 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from vsdpir==3.1.1) (66)
Collecting vstools>=2.1.0 (from vsdpir==3.1.1)
  Using cached vstools-3.1.0-py3-none-any.whl.metadata (1.3 kB)
Requirement already satisfied: charset-normalizer<4,>=2 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from requests>=2.30.0->vsdpir==3.1.1) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from requests>=2.30.0->vsdpir==3.1.1) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from requests>=2.30.0->vsdpir==3.1.1) (2.2.1)
Requirement already satisfied: certifi>=2017.4.17 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from requests>=2.30.0->vsdpir==3.1.1) (2024.2.2)
Requirement already satisfied: filelock in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=2.0.1->vsdpir==3.1.1) (3.13.4)
Requirement already satisfied: typing-extensions>=4.8.0 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=2.0.1->vsdpir==3.1.1) (4.11.0)
Requirement already satisfied: sympy in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=2.0.1->vsdpir==3.1.1) (1.12)
Requirement already satisfied: networkx in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=2.0.1->vsdpir==3.1.1) (3.3)
Requirement already satisfied: jinja2 in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=2.0.1->vsdpir==3.1.1) (3.1.3)
Requirement already satisfied: fsspec in f:\hybrid\64bit\vapoursynth\lib\site-packages (from torch>=2.0.1->vsdpir==3.1.1) (2024.3.1)
INFO: pip is looking at multiple versions of torch-tensorrt-fx-only to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install vsdpir and vsdpir==3.1.1 because these package versions have conflicting dependencies.

The conflict is caused by:
    vsdpir 3.1.1 depends on torch>=2.0.1
    torch-tensorrt-fx-only 1.5.0.dev0 depends on torch<2.2 and >=2.0.1

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

any idea how to fix this?

Make TensorRT dependencies optional

vs-dpir/vsdpir/__init__.py

Line 9 in b517f75

import tensorrt

If you do not want to use TensorRT, and do not have TensorRT installed, then continuing is not possible.
The same goes for the torch_tensorrt imports.

Support for task = 'sr' (super resolution)

Hello, there is any plan to add support to the task "super resolution" ?

fp16 mode for vs-swinir?

Sorry to ask here, but would it be possible to add FP16 mode to https://github.com/HolyWu/vs-swinir ?

Thank you.

name 'math' is not defined

Using v1.4.0 and:

# Imports
import vapoursynth as vs
import math
# getting Vapoursynth core
core = vs.core
# Loading Plugins
core.std.LoadPlugin(path="I:/Hybrid/64bit/vsfilters/SourceFilter/FFMS2/ffms2.dll")
# source: 'G:\TestClips&Co\files\test.avi'
# current color space: YUV420P8, bit depth: 8, resolution: 640x352, fps: 25, color matrix: 470bg, yuv luminance scale: limited, scanorder: progressive
# Loading source using FFMS2
clip = core.ffms2.Source(source="G:/TestClips&Co/files/test.avi",cachefile="E:/Temp/avi_6c441f37d9750b62d59f16ecdbd59393_853323747.ffindex",format=vs.YUV420P8,alpha=False)
# making sure input color matrix is set as 470bg
clip = core.resize.Bicubic(clip, matrix_in_s="470bg",range_s="limited")
# making sure frame rate is set to 25
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Setting color range to TV (limited) range.
clip = core.std.SetFrameProp(clip=clip, prop="_ColorRange", intval=1)
from vsdpir import DPIR
# adjusting color space from YUV420P8 to RGBS for vsDPIRDeblock
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_s="limited")
# deblocking using DPIRDeblock
clip = DPIR(clip=clip, strength=50.000, device_index=0, tile_x=320, tile_y=176, tile_pad=10)
# adjusting output color from: RGBS to YUV420P8 for x264Model
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P8, matrix_s="470bg", range_s="limited")
# set output frame rate to 25.000fps
clip = core.std.AssumeFPS(clip=clip, fpsnum=25, fpsden=1)
# Output
clip.set_output()

I get: name 'math' is not defined.
I tried whether adding import math would help, it did not.

Is it a bug or am I missing something?

"Only RGBS supported"

I tried to input video and the script gave me an error saying "Only RGBS supported". What even is RGBS video?

I tested converting to RGB64, RGB32 and RGB24 using Virtualdub, but all of them resulted in the same "Only RGBS supported" message. What am I supposed to do?

[SUGGESTION] Awesome-Super-Resolution

Since seems that you're very active in "porting" GANs to VapourSynth, we strongly recommend you to check @ChaofWang collection:

https://github.com/ChaofWang/Awesome-Super-Resolution

Lazy loading is not enabled.

Getting this warning message before the YUV4MPEG2 header.
Causes unrecognized headers in ffmpeg and mpv, can't do anything with it it seems.
A while later, the vspipe starts outputting stuff.

[06/01/2023-15:40:27] [TRT] [W] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage and speed up TensorRT initialization. See "Lazy Loading" section of CUDA documentation https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#lazy-loading

Use gray models when passing a GRAY clip

I realised there's gray models included with the original DPIR. Is there a reason those can't be used when passing a GRAY clip converted to RGBS (or if it's possible, just pure GRAY support)?

add 'NTIRE 2021 Video Super-Resolution'

Would be nice if 'NTIRE 2021 Video Super-Resolution' (see: https://forum.doom9.org/showthread.php?p=1974190#post1974190) could be added.

Random black specks on output compared to original DPIR

I can't give a better explanation than the title, so the issue can be clearly seen in this comparison. The other image in the comparison is the standalone DPIR, and the issue only happens with task='deblock'. task='denoise' works fine.

Tensorrt is optional to use, but is still always required to be installed?

I can set trt=False, but I still can't use this plugin if tensorrt is not installed. Has this a reason, or was this just an oversight?

Installing tensorrt takes a lot of space and time and can be rather complicated, so one might want to avoid it, especially if the GPU doesn't support it anyway.

Update dpir inline with the newer setup like vs-rife?

Would it be possible to update the Tensorrt methods to use one similar that of rife
it is much easier to setup and would probably help compatability when using both plugins since the cuDNN libs can sometimes clash

TensorRT whl for Vapoursynth R62, Python 3.11 ?

I'm trying to get vs-dpir working with the new Vapoursynth R62 which does not support Python 3.10 anymore.
Can you build and share a new tensoRTxxx.whl file which works with Python 3.11?

Failed to evaluate the script

my script:

from vsdpir import dpir
core = vs.core
ret= core.lsmas.LWLibavSource(r'C:\Users\Administrator\Desktop\12.mp4')
ret = core.resize.Bicubic(ret, format=vs.RGBS, matrix_in_s="470bg", dither_type="error_diffusion")
ret = dpir(ret)
ret = core.resize.Bicubic(ret, format=vs.YUV420P16, matrix_s="709", range_s="full")
ret.set_output()>

The script ran incorrectly：

Failed to evaluate the script:
Python exception: cannot import name 'dpir' from 'vsdpir' (D:\video\VapourSynth64Portablevb\VapourSynth64\Lib\site-packages\vsdpir_init_.py)
Traceback (most recent call last):
File "src\cython\vapoursynth.pyx", line 2866, in vapoursynth._vpy_evaluate
File "src\cython\vapoursynth.pyx", line 2867, in vapoursynth.vpy_evaluate
File "D:/video/VapourSynth64Portablevb/dpir.vpy", line 29, in
from vsdpir import dpir
ImportError: cannot import name 'dpir' from 'vsdpir' (D:\video\VapourSynth64Portablevb\VapourSynth64\Lib\site-packages\vsdpir_init.py)>

vs-famser is ok.