Vapoursynth SwinIR Resize (mlrt)

From wiki.selur.net
Jump to navigation Jump to search

SwinIR (mlrt) upscales with the super-resolution models of SwinIR, a Swin Transformer restoration network, run as ONNX models through vs-mlrt. It is one of the resizers of Vapoursynth Resizer and works on NVIDIA and Intel hardware and on the CPU.

Filtering → Vapoursynth → Frame → Resize

SwinIR (mlrt) controls in Hybrid
SwinIR (mlrt) controls in Hybrid

What it does

Every frame is enlarged by the factor of the chosen model (2, 3, 4 or 8). Lightweight and classical models were trained on clean downscaled images; the realSR (BSRGAN) models on real-world degradations, where GAN models look sharper and PSNR models smoother. Hybrid resizes the result to the exact output size afterwards.

Requirements

  • Hybrid's vs-mlrt add-on with the SwinIR (mlrt) models and at least one backend.
  • GPU backends: TensorRT and ONNX (CUDA) for NVIDIA, TensorRT RTX for RTX cards, OpenVINO (GPU) for Intel; on macOS ONNX (CoreML).

Settings

Control Parameter Range / values Hybrid default Meaning
Model model lightweight SR x2/x3/x4, classical SR x2/x3/x4/x8, realSR BSRGAN L GAN x4, L PSNR x4, M GAN x2, M PSNR x2, M GAN x4, M PSNR x4 lightweight SR x2 Network; the "xN" in the name is the upscale factor.
Backend backend ONNX (CUDA), ONNX (CPU), OpenVINO (GPU/CPU), TensorRT, TensorRT RTX (macOS: ONNX CoreML/CPU) ONNX (CPU) (macOS: CoreML) Runtime that executes the network; only installed backends are listed.
32bit fp16 on/off off Full precision; off runs in half precision. The CPU backends always run in full precision (forced on and greyed out); TensorRT RTX starts with it on; OpenVINO (GPU) also allows half precision.
BF16 bf16 on/off off bfloat16; TensorRT and OpenVINO (CPU) only. Switches 32bit on.
Device device_id 0 – 4 0 GPU to use; not for CPU and CoreML backends.
Streams num_streams 1 – 16 1 Parallel streams; TensorRT, TensorRT RTX, OpenVINO (GPU) and CoreML only.
Cuda Graph use_cuda_graph on/off off TensorRT, TensorRT RTX and ONNX (CUDA) only.
TRT backend optimization builder_optimization_level 0 – 5 3 TensorRT engine build effort.
TRT max workspace workspace 0 – 1048576 MiB 0 TensorRT workspace limit; 0 = not set.
Tile (wxh) tilesize 1 – 8192 off, 256×256 Splits the frame to save memory; off = whole frame.
Overlap overlap 2 – 624 16 Overlap of the tiles; only with tiling.
Dynamic Shapes (opt/min/max) opt_shapes, min_shapes, max_shapes 0 – 8192 unset TensorRT engine sizes; with a minimum size the engine becomes dynamic.
Resizer – point, rect, linear, cubic, lanczos, blackman, blackmanminlobe, spline36, spline64, spline144, spline, gauss, sinc spline64 Kernel that brings the result to the exact output size.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356), resized to 1280x712:

# changing range from limited to full range for vsSwinIRmlrtResize
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGBS for vsSwinIRmlrtResize
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="full", range_s="full")
# resizing using SwinIR (mlrt)
from vsmlrt import Backend
clip = vsmlrt.SwinIR(clip=clip, model=0, backend=Backend.ORT_CPU(fp16=False,output_format=0))
# adjusting output color from RGBS to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_in_s="full", range_s="full", dither_type="error_diffusion")
# changing range from full to limited range for matching filter target luma scale
clip = core.resize.Bicubic(clip, format=vs.YUV420P10,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})

How Hybrid calls it

  • Selected with Resize enabled and Resizer set to SwinIR (mlrt) (see Vapoursynth Resizer).
  • The clip is converted to 32-bit float RGB, or 16-bit float RGB with TensorRT and 32bit off, and treated as full range; there is no size multiple.
  • The output is input × the model's factor. If that differs from the target, fmtconv resizes it with the Resizer kernel to the exact size (half-float clips are converted to 32-bit float first). With separately filtered fields the target height is halved.
  • Model and backend options are always written; tile size and overlap only when tiling is enabled.
  • TensorRT builds an engine on the first run for each size. TensorRT cannot be used in vs-mlrt filters and torch filters in the same script.
  • The PyTorch version is SwinIR (torch).

Background

SwinIR

SwinIR (Liang et al., ICCV workshops 2021) is an image restoration network built from Swin Transformer blocks. The classical models are the medium-size SwinIR-M trained on DF2K, the lightweight ones the small SwinIR-S trained on DIV2K; the realSR models use the BSRGAN degradation model. SwinIR also has denoise and JPEG models, which this resizer does not offer.

vs-mlrt

vs-mlrt runs ONNX models in VapourSynth with several backends. Its SwinIR wrapper expects 16- or 32-bit float RGB and uses the whole frame as one tile unless a tile size is given.

Sources