Vapoursynth ProPainter

From wiki.selur.net
Jump to navigation Jump to search

ProPainter is a video inpainting network: masked areas such as logos, watermarks, objects or damage are filled with content propagated from other frames. In outpainting mode it extends the frame to a larger size and invents the new borders. It runs on an NVIDIA GPU through PyTorch and needs a lot of resources.

Filtering → Vapoursynth → Other

ProPainter controls in Hybrid
ProPainter controls in Hybrid

What it does

Optical flow between frames is estimated and completed inside the masked area, the known pixels are propagated along that flow, and a transformer fills what is still missing. Because content comes from other frames, a static logo over a moving background is usually replaced well; areas that are never visible are invented. Frames are processed in sequences of Length frames.

Requirements

  • Windows, Hybrid's torch add-on and an NVIDIA GPU with CUDA (the CPU can be selected but is very slow).
  • For inpainting a mask: an image of the clip's size at that point in the filter chain (white = area to fill), a mask video, or a mask generated by Hybrid.

Settings

Control Parameter Range / values Hybrid default Meaning
Mode – inpainting, outpainting inpainting Fill masked areas, or extend the frame.
Mask type img_mask_path, clip_mask image, video, generated image Source of the inpainting mask; without a mask the filter is skipped.
Length length 12 – 10000 100 Frames per processed sequence; lower values need less memory but are not faster.
Dilation mask_dilation 0 – 32 4 Grows the mask; raise it if the mask misses the edges of the logo.
Enhance mask_enhance on/off on Inpainting: improves the edges of the mask for better recognition.
Neighbor length neighbor_length 1 – 100 10 Local neighbor frames; lower needs less memory, higher can help fast motion.
Ref. Stride ref_stride 1 – 100 10 Stride of the global reference frames; higher is faster and needs less memory but can lower the quality.
Raft Iter. raft_iter 1 – 100 20 Iterations of the optical flow estimation; lower is faster but can lower the quality.
Mask Region (w, h, x, y) mask_region on/off; w/h 128 – 8192 in steps of 8, x/y 0 – 8176 off; 256, 256, 0, 0 Processes only this area; much faster on HD material.
Debug – on/off off Preview only: draws the mask region instead of filtering.
SCThreshold sc_threshold 0 – 1 0.1 Scene change threshold used to pick the reference frames; 0 = no scene detection.
SC freq sc_min_freq 0 – 1024 0 Inpainting: minimum number of frames between two accepted scene changes (at least 5 when above 0); 0 = no spacing.
FP16 enable_fp16 on/off on Half precision; faster and needs less memory.
Device device_index −1 – 16 0 CUDA device; −1 = CPU.
Inference inference_mode on/off off Enables torch inference mode.
Outpaint resolution outpaint_size 256 – 8192 256 × 256 Outpainting: new frame size; must not be smaller than the current size and must be larger in at least one direction, else the filter is skipped.

Example

Mask image J:/tmp/claude/wiki/assets/logo_mask.png (640x356, white box in the top right corner; no mask is set by default), otherwise default settings, on a 10-bit YUV 4:2:0 source (640x356):

# changing color matrix from '470bg' to '709' for vsProPainter
clip = core.resize.Bicubic(clip, matrix_in_s="470bg", matrix_s="709")
# changing range from limited to full range for vsProPainter
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# Inpainting using ProPainter
from vspropainter import propainter as ProPainter
clip = ProPainter(clip, img_mask_path="J:/tmp/claude/wiki/assets/logo_mask.png", length=100, sc_threshold=0.10)
# changing color matrix from '709' to '470bg' undoing color matrix change for vsProPainter
clip = core.resize.Bicubic(clip, matrix_in_s="709", matrix_s="470bg")
# changing range from full to limited range for vsProPainter
clip = core.resize.Bicubic(clip, format=vs.YUV420P10,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})

How Hybrid calls it

  • The clip is requested as full range with the Rec.709 matrix; the plugin works in 8-bit RGB internally.
  • A mask video is read with LWLibavSource as 8-bit RGB.
  • Inpainting: length and scene change threshold are always written; the other options only when they differ from the defaults.
  • Outpainting: length and dilation are always written; Hybrid continues with the new frame size.
  • ProPainter also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

ProPainter

ProPainter (Zhou et al., ICCV 2023) combines image- and feature-level flow propagation with a sparse transformer for video inpainting. It uses RAFT for optical flow and a recurrent network for flow completion.

vs-propainter

dan64's vs-propainter brings ProPainter to VapourSynth, with inpainting from a mask image, mask clip or region and an outpainting function. Its README lists ways to save memory and time: a smaller mask region, a shorter length (memory only), fewer neighbor frames, a larger reference stride, FP16, and turning scene detection off.

Sources