Vapoursynth RIFE (mlrt)

From wiki.selur.net
Jump to navigation Jump to search

RIFE (mlrt) runs the RIFE frame interpolation network as an ONNX model through vs-mlrt. Depending on the backend it works on the CPU or on NVIDIA, Intel and AMD hardware.

Filtering → Vapoursynth → Frame → Interpolation

RIFE (mlrt) controls in Hybrid
RIFE (mlrt) controls in Hybrid

What it does

RIFE estimates the optical flow between two frames and synthesizes the frames in between. An integer multiplier interleaves new frames; a fractional multiplier places the new frames at the exact time points. Frames marked as the last frame before a cut are duplicated instead of interpolated.

Requirements

  • Hybrid's vs-mlrt add-on with the RIFE models; without them the method is not offered.
  • GPU backends: TensorRT and ONNX (CUDA) for NVIDIA, OpenVINO (GPU) for Intel, NCNN for Vulkan GPUs, DML for DirectX 12 GPUs; on macOS ONNX (CoreML) or CPU.

Settings

Tick Frame Interpolation and pick RIFEmlrt in the method list.

Control Parameter Range / values Hybrid default Meaning
Factor/FPS – on/off on On: multiple of the input rate; off: absolute target rate.
Factor (num/den) multi 1 – 120000 2/1 Multiple of the input rate.
FPS (num/den) multi 1 – 120000 2/1 Target frame rate; the default of 2 fps must be changed before this mode does anything.
Model model 4.0, 4.2 – 4.26_heavy (incl. _lite/_heavy) 4.4 RIFE model version.
Ensemble ensemble on/off off Ensemble variant of the model (slower). Not for 4.21–4.23 and 4.25 or newer.
Scale scale 0.25, 0.5, 1.0, 2.0, 4.0 1.0 Resolution of the flow estimation (0.5 suits 4K); only for models 4.0–4.6 without ensemble.
Backend backend OpenVINO (CPU/GPU), ONNX (CPU/CUDA), DML, TensorRT, TensorRT RTX, NCNN (macOS: ONNX CoreML/CPU) OpenVINO (CPU); macOS ONNX (CoreML) Runtime that executes the network.
FP16 / BF16 fp16, bf16 on/off off / off Half precision; the two exclude each other and are disabled for OpenVINO.
Device ID device_id 0 – 16 0 GPU to use; not for CPU and CoreML backends.
Streams num_streams 1 – 16 1 Parallel streams; enabled only for the backends that use it (TensorRT, TensorRT RTX, OpenVINO (GPU), NCNN and CoreML).
CUDA Graphs use_cuda_graph on/off off TensorRT, TensorRT RTX and ONNX (CUDA) only.
TRT backend optimization builder_optimization_level 0 – 5 3 TensorRT engine build effort.
Implementation _implementation 1, 2 1 vs-mlrt's alternative ONNX implementation 2 (experimental, TensorRT); used only when its model file exists and scale is 1.
Dynamic Shapes (min/opt/max) min_shapes, opt_shapes, max_shapes sizes unset TensorRT engine sizes; with a minimum size the engine becomes dynamic.
Scene detection / threshold – on/off, 0.001 – 1 on, 0.1 Detects cuts before RIFE so they are not interpolated.
Manual scene changes – on/off, frame list off List of the last frames before each cut; excludes the automatic detection.
Tools: SCD – scd, misc (Windows), std, mv – Scene change detection, shown with Scene detection on; mv uses its own motion thresholds.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356, 25 fps), interpolated to 50 fps:

clip = misc.SCDetect(clip=clip, threshold=0.100, tools={'scd': 'scd'})
clip = core.std.AddBorders(clip=clip, left=0, right=0, top=14, bottom=14) # add borders to achieve mod 32 (vsRIFEmlrt) - 640x384
# adjusting color space from YUV420P10 to RGBS for vsRIFEmlrt
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# adjusting frame count&rate with RIFE (mlrt)
from vsmlrt import Backend
clip = vsmlrt.RIFE(clip, model=44, multi=Fraction(2, 1), backend=Backend.OV_CPU(fp16=False,bf16=False))
clip = core.std.Crop(clip=clip, left=0, right=0, top=14, bottom=14) # removing added borders from mod requirement (vsRIFEmlrt) -  640x356

How Hybrid calls it

  • The clip is converted to 32-bit float RGB (16-bit with FP16) and padded to the size the model and scale require; the padding is removed afterwards.
  • Factor and FPS are both turned into one exact multiplier; in FPS mode it is the target rate divided by the frame rate Hybrid tracks for the clip, so FPS mode also works for variable frame rate sources. If the multiplier is not above 1, the filter is skipped with a note in the script and the frame rate stays.
  • With scene detection Hybrid runs the detection itself before RIFE (RGB clips are converted to 16-bit YUV 4:4:4 for it); manual scene changes mark the listed frames instead.
  • The model and the backend are always written; scale only when it is not 1.0, ensemble only when ticked.
  • TensorRT builds an engine on the first run for each size.

Background

vs-mlrt

vs-mlrt's RIFE needs 16- or 32-bit float RGB. Its documentation recommends running a scene detection before RIFE, since the result depends strongly on it, and suggests scale 0.5 for 4K material. The frame size must be a multiple of 32 divided by the scale (some newer models need larger multiples). TensorRT is clearly the fastest backend.

Other RIFE versions

RIFE runs on Vulkan GPUs through ncnn, TorchRIFE in PyTorch on NVIDIA GPUs.

Sources