Vapoursynth DPIR Denoise (mlrt)

From wiki.selur.net
Jump to navigation Jump to search

DPIRDenoise (mlrt) removes noise with DRUNet, the denoiser network of DPIR, run as an ONNX model through vs-mlrt. It works with NVIDIA, Intel and AMD hardware and on the CPU, but the network is large: a GPU backend is strongly recommended.

Filtering → Vapoursynth → DeNoise

DPIRDenoise (mlrt) controls in Hybrid
DPIRDenoise (mlrt) controls in Hybrid

What it does

Every frame is passed through DRUNet (the color model for RGB, the gray model for gray clips). Strength tells the network how much noise to expect; higher values smooth more and lose more detail.

Requirements

  • Hybrid's vs-mlrt add-on with the DPIR models and at least one backend.
  • GPU backends: TensorRT and ONNX (CUDA) for NVIDIA, OpenVINO (GPU/NPU) for Intel, NCNN for Vulkan GPUs, DML for DirectX 12 GPUs; on macOS ONNX (CoreML).

Settings

Control Parameter Range / values Hybrid default Meaning
Strength strength 0.001 – 1000 5 Noise level the network assumes, on a 0–255 scale.
Tiles tiles 1 – 64 1 Splits the frame to save memory; may cost some quality.
Tile pad overlap 0 – 64 16 Overlap of the tiles; only with more than one tile.
Backend backend OpenVINO (CPU/GPU/NPU), ONNX (CPU/CUDA), DML (GPU), TensorRT, TensorRT RTX, NCNN (macOS: ONNX CoreML/CPU) OpenVINO (CPU) Runtime that executes the network.
Device device_id 0 – 4 0 GPU to use; not for CPU and CoreML backends.
Streams num_streams 1 – 64 1 Parallel streams; TensorRT, TensorRT RTX, OpenVINO (GPU), NCNN and CoreML only.
FP16 / BF16 fp16, bf16 on/off off / off Half precision; BF16 only with TensorRT and OpenVINO (CPU). The two exclude each other.
CUDA Graph use_cuda_graph on/off off TensorRT, TensorRT RTX and ONNX (CUDA) only.
TRT backend optimization builder_optimization_level 0 – 5 3 TensorRT engine build effort.
Dynamic Shapes (min/opt/max) min_shapes, opt_shapes, max_shapes 0 – 8192 unset TensorRT engine sizes; with a minimum size the engine becomes dynamic.

Example

Default settings on a 10-bit YUV 4:2:0 source (640×356, padded to mod 8):

clip = core.std.AddBorders(clip=clip, left=0, right=0, top=2, bottom=2) # add borders to achieve mod 8 (vsDPIRmlrtDenoise) - 640x360
# changing range from limited to full range for vsDPIRmlrtDenoise
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGBS for vsDPIRmlrtDenoise
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="full", range_s="full")
# denoising using DPIR (mlrt)
from vsmlrt import Backend
clip = vsmlrt.DPIR(clip, strength=5.000, model=1, backend=Backend.OV_CPU(fp16=False,bf16=False))
clip = core.std.Crop(clip=clip, left=0, right=0, top=2, bottom=2) # removing added borders from mod requirement (vsDPIRmlrtDenoise) -  640x356

How Hybrid calls it

  • The clip is padded to mod 8, converted to 32-bit float RGB (gray clips to gray; 16-bit float with TensorRT and FP16), and cropped and converted back afterwards.
  • Strength, the model and the backend options are always written; tiles and tile pad only when they differ from the defaults.
  • TensorRT builds an engine on the first run for each size. TensorRT cannot be used in vs-mlrt filters and torch filters in the same script.
  • DPIRDenoise (mlrt) also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).
  • The PyTorch version is DPIR (torch); for compression artifacts see DPIRDeblock (mlrt).

Background

vs-mlrt

vs-mlrt runs ONNX models in VapourSynth with several backends. Its DPIR page describes the denoise models with a default strength of 5 (the deblocking models use 50), normalised by 1/255, and tile sizes in multiples of 8. Strength can also be a gray clip for a per-pixel strength. The model is very slow on a CPU (about 0.05 fps at 360p); TensorRT is the fastest backend, and more tiles help when GPU memory runs out.

DPIR

DPIR (Zhang et al., TPAMI 2021) uses the DRUNet denoiser as a plug-in prior for image restoration. DRUNet was trained on noise levels of about 0–50.

Sources