Vapoursynth DPIR Deblock (mlrt)

From wiki.selur.net
Jump to navigation Jump to search

DPIR Deblock (mlrt) removes compression artifacts with the DPIR deblocking networks, run as ONNX models through vs-mlrt. Unlike DPIR Deblock (torch) it needs no PyTorch and runs on many kinds of hardware: CPU, Intel, NVIDIA, any Vulkan GPU or DirectML.

Filtering → Vapoursynth → DeBlock

DPIR Deblock (mlrt) controls in Hybrid
DPIR Deblock (mlrt) controls in Hybrid

What it does

Every frame is restored with DPIR's deblocking model (color model for RGB, gray model for gray clips). Strength tells the network how strong the compression artifacts are; higher values remove more blocking but also more detail.

Requirements

  • Windows: Hybrid's vs-mlrt add-on with the core component, the DPIR models and at least one backend. macOS: ONNX Runtime (CoreML or CPU) from the bundle.

Settings

Control Parameter Range / values Hybrid default Meaning
Strength strength 0.001 – 1000 50 Deblocking strength; vs-mlrt gives the network strength/255 as noise level.
Tiles tiles 1 – 64 1 Splits each frame dimension into this many tiles to save memory.
Tile pad overlap 0 – 64 16 Overlap of the tiles; only with more than one tile.
Backend backend OpenVINO (CPU), OpenVINO (GPU), ONNX (CPU), ONNX (CUDA), DML (GPU), TensorRT, TensorRT RTX, NCNN; macOS: ONNX (CoreML), ONNX (CPU) OpenVINO (CPU); macOS ONNX (CoreML) Inference engine; only the backends whose add-on component and hardware are present are offered.
Device device_id 0 – 4 0 GPU to use; not for the CPU and CoreML backends.
Streams num_streams 1 – 64 1 Parallel streams (TensorRT, TensorRT RTX, OpenVINO GPU, NCNN, CoreML).
FP16 fp16 on/off off Half precision (faster, less memory).
BF16 bf16 on/off off bfloat16 precision; TensorRT and OpenVINO (CPU) only, excludes FP16.
CUDA Graph use_cuda_graph on/off off Reduces CPU overhead; TensorRT, TensorRT RTX and ONNX (CUDA).
TRT backend optimization builder_optimization_level 0 – 5 3 Effort TensorRT spends on building the engine; TensorRT and TensorRT RTX.
Dynamic Shapes (min/opt/max) min_shapes, opt_shapes, max_shapes 0 = unset – Lets one TensorRT engine serve several frame sizes; TensorRT and TensorRT RTX.

Example

Default settings (OpenVINO on the CPU) on a 640×356 10-bit YUV 4:2:0 source (padded to 640×360 for mod 8):

clip = core.std.AddBorders(clip=clip, left=0, right=0, top=2, bottom=2) # add borders to achieve mod 8 (vsDPIRmlrtDeblock) - 640x360
# adjusting color space from YUV420P10 to RGBS for vsDPIRmlrtDeblock
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# deblocking using DPIR (mlrt)
from vsmlrt import Backend
clip = vsmlrt.DPIR(clip, strength=50.000, model=3, backend=Backend.OV_CPU(fp16=False,bf16=False))
clip = core.std.Crop(clip=clip, left=0, right=0, top=2, bottom=2) # removing added borders from mod requirement (vsDPIRmlrtDeblock) -  640x356
# adjusting output color from RGBS to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_in_s="full", range_s="limited", dither_type="error_diffusion")

How Hybrid calls it

  • The clip is padded to a multiple of 8, converted to 32-bit float RGB (16-bit with TensorRT and FP16; gray clips to gray), restored, cropped back and converted back.
  • The model is chosen automatically: deblocking color model for RGB, deblocking gray model for gray clips.
  • TensorRT engines are stored in the Engine Path folder of Hybrid's path settings (the temp folder if it is empty); the first run with a new size builds the engine.
  • DPIR Deblock (mlrt) also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

vs-mlrt (DPIR)

vs-mlrt runs ONNX models through several backends: vsov (OpenVINO, x86 CPU and Intel GPU), vsort (ONNX Runtime: CPU, CUDA, DirectML, CoreML), vstrt and vstrt_rtx (TensorRT, NVIDIA) and vsncnn (any Vulkan GPU). Its DPIR function offers two denoising models (default strength 5) and two deblocking models (default strength 50); the strength is on a 1/255 scale and can also be a clip. Without tiles the frame size must be a multiple of 8. TensorRT with FP16 is by far the fastest; on the CPU DPIR is very slow.

Sources