Vapoursynth FeMaSR

From wiki.selur.net
Jump to navigation Jump to search

FeMaSR is a real-world blind super-resolution network that matches the features of the low-resolution picture against a learned codebook of high-resolution features, which helps it restore realistic textures. It is one of the resizers of Vapoursynth Resizer and runs on an NVIDIA GPU through PyTorch.

Filtering → Vapoursynth → Frame → Resize

FeMaSR controls in Hybrid
FeMaSR controls in Hybrid

What it does

Every frame is enlarged 2× or 4×, depending on the model. Hybrid resizes the result to the exact output size afterwards.

Requirements

  • Windows, Hybrid's torch add-on and an NVIDIA GPU with CUDA.
  • Tensor RT additionally needs Torch-TensorRT from the add-on; it cannot be combined with TensorRT in vs-mlrt filters in the same script.

Settings

Control Parameter Range / values Hybrid default Meaning
Model model FeMaSR_SRX2, FeMaSR_SRX4 FeMaSR_SRX2 Network: 2× or 4×.
GPU ID device_index 0 – 4 0 CUDA device.
Streams num_streams 1 – 64 1 Number of CUDA streams.
FP16 – on/off off Half precision (16-bit float clip), with or without Tensor RT.
Tensor RT trt on/off off Uses TensorRT; the engine is built on the first run and cached.
max WSS trt_workspace_size 0 – 1073741824 0 TensorRT workspace size; 0 = not set.
Opt. Level trt_optimization_level 0 – 5 3 TensorRT engine build effort; only with Tensor RT.
Resizer – point, rect, linear, cubic, lanczos, blackman, blackmanminlobe, spline36, spline64, spline144, spline, gauss, sinc spline64 Kernel that brings the result to the exact output size.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356), resized to 1280x712:

# adjusting color space from YUV420P10 to RGBS for vsFeMaSR
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# resizing using FeMaSR
from vsfemasr import femasr as FeMaSR
clip = FeMaSR(clip, device_index=0) # 1280x712

How Hybrid calls it

  • Selected with Resize enabled and Resizer set to FeMaSR (see Vapoursynth Resizer).
  • The clip is converted to 32-bit float RGB (16-bit float with FP16) and padded to mod 4; the plugin pads further on its own (32 for 2×, 16 for 4×) and crops again.
  • The output is input × the model's factor. If that differs from the target, fmtconv resizes it with the Resizer kernel to the exact size. With separately filtered fields the target height is halved.
  • GPU ID is always written, and with Tensor RT also the optimization level; model, streams and max WSS only when they differ from the defaults.
  • With Tensor RT the engines are cached in the engine path from Hybrid's path settings (or the temp folder).

Background

FeMaSR

FeMaSR (Chen et al., ACM Multimedia 2022) uses a pretrained VQGAN codebook of high-resolution features as a prior: features of the degraded input are matched to the codebook, and the matched features are decoded into the high-resolution picture.

vs-femasr

HolyWu's vs-femasr runs FeMaSR in VapourSynth with PyTorch, optionally with TensorRT.

Sources