Vapoursynth BasicVSR++

From wiki.selur.net
Jump to navigation Jump to search

BasicVSR++ is a recurrent video network that uses neighbor frames to restore each frame; this filter offers its models for compressed-video enhancement, deblurring and denoising at the input size. It runs on an NVIDIA GPU through PyTorch; the 4× super-resolution models are in Vapoursynth BasicVSR++ Resize.

Filtering → Vapoursynth → Other

BasicVSR++ controls in Hybrid
BasicVSR++ controls in Hybrid

What it does

Frames are processed in sequences of Length frames; information is carried forward and backward through each sequence and aligned with optical flow. The result of a frame therefore depends on its neighbors, not only on the frame itself. The output keeps the input size.

Requirements

  • Windows, Hybrid's torch add-on and an NVIDIA GPU with CUDA.

Settings

Control Parameter Range / values Hybrid default Meaning
Model model NTIRE 2021 (1), (2), (3), Video Deblurring (DVD), Video Deblurring (GoPro), Video Denoising NTIRE 2021 (1) Network; the NTIRE 2021 models are the three tracks of the challenge on quality enhancement of compressed video.
Length length 1 – 1000 15 Frames per processed sequence; more frames need more GPU memory.
Device device_index 0 – 4 0 CUDA device.
Tile (x/y) tile 0 – 8192 0 / 0 Splits the frame into tiles to save GPU memory; 0 = no tiling (both values must be set).
Tile pad tile_pad 0 – 64 16 Overlap of the tiles; only with tiling.
FP16 – on/off off Half precision (16-bit float clip); faster and needs less memory.
CPU Cache cpu_cache on/off off Keeps intermediate features in system RAM; saves GPU memory, slower.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356):

clip = core.std.AddBorders(clip=clip, left=0, right=0, top=2, bottom=2) # add borders to achieve mod 8 (vsBasicVSRPPFilter) - 640x360
# adjusting color space from YUV420P10 to RGBS for vsBasicVSRPPFilter
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# Quality enhancement using BasicVSR++
from vsbasicvsrpp import basicvsrpp as BasicVSRPP
clip = BasicVSRPP(clip, model=4)
clip = core.std.Crop(clip=clip, left=0, right=0, top=2, bottom=2) # removing added borders from mod requirement (vsBasicVSRPPFilter) -  640x356

How Hybrid calls it

  • The clip is converted to 32-bit float RGB (16-bit float with FP16) and padded to mod 8; the padding is cropped afterwards.
  • The model is always written; length, device, tiles and tile pad only when they differ from the defaults.
  • BasicVSR++ also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

BasicVSR++

BasicVSR++ (Chan et al., CVPR 2022) improves BasicVSR with second-order grid propagation and flow-guided deformable alignment. It won tracks of the NTIRE 2021 challenges on video super-resolution and on compressed video enhancement.

vs-basicvsrpp

HolyWu's vs-basicvsrpp runs BasicVSR++ in VapourSynth with PyTorch and accepts 16- or 32-bit float RGB. It processes Length frames per call and keeps them in a cache, so playing or encoding front to back is much faster than jumping around in the clip.

Sources