Vapoursynth SwinIR (mlrt)

From wiki.selur.net
Jump to navigation Jump to search

SwinIR (mlrt) runs the color denoising and JPEG artifact removal models of SwinIR, a Swin Transformer restoration network, as ONNX models through vs-mlrt at the input size. It works on NVIDIA and Intel hardware and on the CPU, but the network is large and practical only on a GPU; the super-resolution models are in Vapoursynth SwinIR Resize (mlrt).

Filtering → Vapoursynth → Other

SwinIR (mlrt) controls in Hybrid
SwinIR (mlrt) controls in Hybrid

What it does

Every frame is restored on its own by the chosen model. The denoise models are trained for one noise level (15, 25 or 50), the JPEG models for one JPEG quality (10 to 40); pick the one closest to the source.

Requirements

  • Hybrid's vs-mlrt add-on with the SwinIR (mlrt) models and at least one backend.
  • GPU backends: TensorRT and ONNX (CUDA) for NVIDIA, TensorRT RTX for RTX cards, OpenVINO (GPU) for Intel; on macOS ONNX (CoreML).

Settings

Control Parameter Range / values Hybrid default Meaning
Model model denoise 15, denoise 25, denoise 50, compression artifact removal jpeg10, jpeg20, jpeg30, jpeg40 denoise 15 Network: color denoising for a noise level, or JPEG artifact removal for a quality.
Backend backend ONNX (CPU/CUDA), OpenVINO (CPU/GPU), TensorRT, TensorRT RTX (macOS: ONNX CoreML/CPU) ONNX (CPU) Runtime that executes the network; only installed backends are listed.
Tiles tiles 1 – 64 1 Number of tiles the frame is split into to save memory; 1 = whole frame.
Tile pad overlap 0 – 64 16 Overlap of the tiles; only with more than one tile.
FP16 fp16 on/off off Half precision; greyed out for the CPU backends, which run at full precision. The GPU backends, OpenVINO (GPU) included, allow it.
BF16 bf16 on/off off bfloat16; TensorRT and OpenVINO (CPU) only. Switches FP16 off.
Device device_id 0 – 4 0 GPU to use; not for CPU and CoreML backends.
Streams num_streams 1 – 64 1 Parallel streams; TensorRT, TensorRT RTX, OpenVINO (GPU) and CoreML only.
CUDA Graph use_cuda_graph on/off off TensorRT, TensorRT RTX and ONNX (CUDA) only.
TRT backend optimization builder_optimization_level 0 – 5 3 TensorRT engine build effort.
Dynamic Shapes (opt/min/max) opt_shapes, min_shapes, max_shapes 0 – 8192 unset TensorRT engine sizes; with a minimum size the engine becomes dynamic.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356):

clip = core.std.AddBorders(clip=clip, left=0, right=0, top=2, bottom=2) # add borders to achieve mod 8 (vsSwinIRmlrt) - 640x360
# adjusting color space from YUV420P10 to RGBS for vsSwinIRmlrt
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# denoising using SwinIR (mlrt)
from vsmlrt import Backend
clip = vsmlrt.SwinIR(clip, model=17, backend=Backend.ORT_CPU(fp16=False,output_format=0))
clip = core.std.Crop(clip=clip, left=0, right=0, top=2, bottom=2) # removing added borders from mod requirement (vsSwinIRmlrt) -  640x356

How Hybrid calls it

  • The clip is converted to 32-bit float RGB (16-bit float with TensorRT and FP16), full range, and padded to mod 8.
  • Model and backend options are always written; tiles and tile pad only when they differ from the defaults.
  • TensorRT builds an engine on the first run for each size and stores it in the engine path from Hybrid's path settings (or the temp folder). TensorRT cannot be used in vs-mlrt filters and torch filters in the same script.
  • SwinIR (mlrt) also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

SwinIR

SwinIR (Liang et al., ICCV workshops 2021) is an image restoration network built from Swin Transformer blocks. Besides super-resolution it has models for gray and color denoising and for gray and color JPEG artifact reduction; this filter uses the color ones.

vs-mlrt

vs-mlrt runs ONNX models in VapourSynth with several backends. Its SwinIR wrapper expects 16- or 32-bit float input and uses an overlap of 16 between tiles unless told otherwise.

Sources