Vapoursynth RVRT

From wiki.selur.net
Jump to navigation Jump to search

RVRT (Recurrent Video Restoration Transformer) restores video from many neighbor frames; this filter offers its denoising and deblurring models at the input size. It runs on an NVIDIA GPU through PyTorch, is very slow and needs a lot of GPU memory; the 4× super-resolution models are in Vapoursynth RVRT Resize.

Filtering → Vapoursynth → Other

RVRT controls in Hybrid
RVRT controls in Hybrid

What it does

Frames are processed in groups; each group is aligned to its neighbors and refined recurrently, so the result of a frame depends on the frames around it. denoise removes noise of an assumed strength, deblur removes motion blur. Long clips are split into overlapping chunks.

Requirements

  • Windows, Hybrid's torch add-on (with its vsrvrt build) and an NVIDIA GPU with CUDA.
  • The RVRT models are licensed for non-commercial use only.
  • The filter is only shown when the RVRT resizer is available.

Settings

Control Parameter Range / values Hybrid default Meaning
Mode – denoise, deblur denoise Denoising or deblurring model.
Sigma sigma 0 – 50 12 denoise: assumed noise level.
Model model gopro, dvd gopro deblur: training set (GoPro or DVD).
FP16 use_fp16 on/off on Half precision: about twice as fast with half the memory.
Force Tile size – on/off off Enables the tile size; otherwise the plugin picks tiles from the free GPU memory.
Tile T / H / W tile_size T 0 – 1024, H/W 0 – 1024 in steps of 8 64 / 256 / 256 Tile size in frames, height and width; 0/0/0 = no tiling.
Force Tile Overlap size – on/off off With Force Tile size: enables the tile overlap.
Overlap T / H / W tile_overlap 0 – 512 2 / 20 / 20 Overlap of the tiles in frames, height and width.
Chunking use_chunking on/off on Processes the clip in chunks of frames; off processes the whole clip in one pass.
Chunk size / overlap chunk_size, chunk_overlap 0 – 127 / 0 – 63 off, 64 / 16 Forces the chunk size and overlap in frames; otherwise the plugin's values are used.
Lazy chunking preview_mode on/off off Processes chunks on demand and independently; for quick previews.
GPU ID device 0 – 8 0 CUDA device.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356):

clip = core.std.AddBorders(clip=clip, left=0, right=0, top=2, bottom=2) # add borders to achieve mod 8 (vsRVRTFilter) - 640x360
# adjusting color space from YUV420P10 to RGBS for vsRVRTFilter
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# Denoising using RVRT
import vsrvrt
clip = vsrvrt.Denoise(clip, device="cuda:0", use_chunking=True)
clip = core.std.Crop(clip=clip, left=0, right=0, top=2, bottom=2) # removing added borders from mod requirement (vsRVRTFilter) -  640x356

How Hybrid calls it

  • The clip is converted to RGB and brought to mod 8; the output keeps the input size.
  • The device (from GPU ID) and Chunking are always written; sigma, model and FP16 only when they differ from the defaults; tile size and overlap only when forced; chunk size and overlap only when forced with Chunking on.
  • RVRT also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

RVRT

RVRT (Liang et al., NeurIPS 2022) splits a video into short clips, processes the frames of a clip in parallel and passes features recurrently from clip to clip, aligned with guided deformable attention.

vs-rvrt

Lyra-Vhess's vs-rvrt offers RVRT denoising, deblurring and 4× super-resolution for VapourSynth. Its README suggests spatial tiles that are multiples of 8 (256 as a starting point) and chunks below 128 frames. Without an explicit setting it chunks on its own when a clip is long or would need too much memory; lazy chunking drops the recurrence across chunks.

Sources