Vapoursynth SwinIR Resize

From wiki.selur.net
Jump to navigation Jump to search

SwinIR (torch) upscales with SwinIR, an image restoration network built from Swin Transformer blocks. It is one of the resizers of Vapoursynth Resizer and runs on an NVIDIA GPU through PyTorch.

Filtering → Vapoursynth → Frame → Resize

SwinIR controls in Hybrid
SwinIR controls in Hybrid

What it does

Every frame is enlarged by the factor of the chosen model (2×, 3× or 4×). The SwinIR-S models were trained on clean downscaled images, SwinIR-L on real-world degradations, AnimeScale on anime. Hybrid resizes the result to the exact output size afterwards.

Requirements

  • Windows, Hybrid's torch add-on and an NVIDIA GPU with CUDA.
  • SwinIR is only listed as a resizer when the package with its models is installed; the upscale factor always comes from the chosen model.

Settings

Control Parameter Range / values Hybrid default Meaning
Model model SwinIR-S x2, SwinIR-S x3, SwinIR-S x4, SwinIR-L x4, AnimeScale x2 SwinIR-S x2 Network; "xN" is the upscale factor.
Tile (x/y) tile_w, tile_h 0 – 8192 0 / 0 Splits the frame into tiles to save GPU memory; 0 = no tiling.
Tile pad tile_pad 0 – 64 16 Overlap of the tiles; only with tiling.
GPU ID device_index 0 – 4 0 CUDA device.
FP16 – on/off off Half precision (16-bit float clip); faster and needs less memory.
Streams num_streams 1 – 64 1 Number of CUDA streams.
NVFuser nvfuser on/off off Uses PyTorch's NVFuser.
Cuda graphs cuda_graphs on/off off Uses CUDA graphs.
Resizer – point, rect, linear, cubic, lanczos, blackman, blackmanminlobe, spline36, spline64, spline144, spline, gauss, sinc spline64 Kernel that brings the result to the exact output size.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356), resized to 1280x712:

from vsswinir import swinir as SwinIR
# adjusting color space from YUV420P10 to RGBS for vsSwinIR
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# resizing using SwinIR
clip = SwinIR(clip, device_index=0) # 1280x712

How Hybrid calls it

  • Selected with Resize enabled and Resizer set to SwinIR (see Vapoursynth Resizer).
  • The clip is converted to 32-bit float RGB (16-bit float with FP16) and padded to mod 4; the padding is cropped afterwards.
  • The output is input × the model's factor. If that differs from the target, fmtconv resizes it with the Resizer kernel to the exact size. With separately filtered fields the target height is halved.
  • GPU ID is always written, and so are the tiles and tile pad while tiling is on; model, streams, NVFuser and Cuda graphs only when they differ from the defaults.
  • The vs-mlrt version, which also runs without NVIDIA hardware, is SwinIR (mlrt).

Background

SwinIR

SwinIR (Liang et al., ICCV workshops 2021) is an image restoration network for super-resolution, denoising and JPEG artifact removal. SwinIR-S is the lightweight model trained on DIV2K; SwinIR-L is the large real-world model trained with BSRGAN degradations and a GAN loss.

vs-swinir

HolyWu's vs-swinir runs SwinIR in VapourSynth with PyTorch. It accepts 16- or 32-bit float RGB.

Sources