Vapoursynth AnimeSR

From wiki.selur.net
Jump to navigation Jump to search

AnimeSR (TencentARC) is a real-world super-resolution network for animation videos that always upscales by 4. It is one of the resizers of Vapoursynth Resizer and runs on an NVIDIA GPU through PyTorch.

Filtering → Vapoursynth → Frame → Resize

AnimeSR controls in Hybrid
AnimeSR controls in Hybrid

What it does

The network is recurrent: each output frame and an internal state are passed on to the next frame. It therefore works best when the clip is rendered from start to end; jumping around in a preview can give different results. Hybrid resizes the 4× result to the exact output size afterwards.

Requirements

  • Windows, Hybrid's torch add-on and an NVIDIA GPU with CUDA.
  • Tensor RT additionally needs Torch-TensorRT from the add-on; it cannot be combined with TensorRT in vs-mlrt filters in the same script.

Settings

Control Parameter Range / values Hybrid default Meaning
Model model AnimeSR_v1, AnimeSR_v2 AnimeSR_v1 v1 is the model of the paper; v2 looks more natural with fewer artifacts. Both 4×.
GPU ID device_index 0 – 4 0 CUDA device.
Tensor RT trt on/off off Uses TensorRT; the engine is built on the first run and cached.
FP16 – on/off off Half precision (16-bit float clip), with and without Tensor RT.
max WSS trt_workspace_size 0 – 1073741824 0 TensorRT workspace size; 0 = not set.
Resizer – point, rect, linear, cubic, lanczos, blackman, blackmanminlobe, spline36, spline64, spline144, spline, gauss, sinc spline64 Kernel that brings the 4× result to the exact output size.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356), resized to 1280x712:

from vsanimesr import animesr as AnimeSR
# adjusting color space from YUV420P10 to RGBS for vsAnimeSR
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# resizing using AnimeSR
clip = AnimeSR(clip, model=0, device_index=0) # 2560x1424
# resizing 2560x1424 to 1280x712
clip = core.fmtc.resample(clip, w=1280, h=712, kernel="spline64", interlaced=False, interlacedd=False)

How Hybrid calls it

  • Selected with Resize enabled and Resizer set to AnimeSR (see Vapoursynth Resizer).
  • The clip is converted to 32-bit float RGB (16-bit float with FP16) and padded to mod 4.
  • The output is always 4× the input. If that differs from the target, fmtconv resizes it with the Resizer kernel to the exact size. With separately filtered fields the target height is halved.
  • Model and GPU ID are always written (the plugin's own default model is v2); max WSS only when it is not 0.
  • With Tensor RT the engines are cached in the engine path from Hybrid's path settings (or the temp folder). For clips larger than 1280×720 Hybrid raises the engine's maximum size to the clip size.

Background

AnimeSR

AnimeSR (Wu et al., NeurIPS 2022) learns the degradations of real animation videos from a set of basic operators instead of only synthetic ones. With it the authors published the AVC dataset of animation clips.

vs-animesr

HolyWu's vs-animesr runs AnimeSR in VapourSynth with PyTorch, optionally with TensorRT. It accepts 16- or 32-bit float RGB; the precision follows the input format. Its TensorRT engines are built for input sizes from 128×128 up to 1280×720 by default.

Sources