Vapoursynth FaceRestore

From wiki.selur.net
Jump to navigation Jump to search

FaceRestore is a temporal face restoration filter: faces are tracked over neighbor frames and restored with the DicFace, KEEP or PGTFormer model, which keeps them steadier than per-frame restoration. It runs on an NVIDIA GPU through PyTorch.

Filtering → Vapoursynth → Other

FaceRestore controls in Hybrid
FaceRestore controls in Hybrid

What it does

Faces are detected, tracked over Radius frames before and after, restored by the chosen model and pasted back; coarse structure and tones come from the source face. Faces that are too small, blurry, covered or turned away are left untouched, and restored faces fade in and out at the ends of their track. With Scale above 1 the whole frame is upscaled as well. For per-frame restoration see CodeFormer.

Requirements

  • Windows, Hybrid's torch add-on and an NVIDIA GPU with CUDA.
  • Only models whose weights are installed are listed.

Settings

Control Parameter Range / values Hybrid default Meaning
Model model DicFace, KEEP, PGTFormer DicFace Restoration model.
Scale upscale 1 – 16 1 Upscale factor of the output.
Radius radius 14 – 60 (KEEP: 24 – 60) 15 Frames before and after used for tracking and smoothing.
Tone tone 0 – 4 0.5 Structure coarser than this many source pixels is taken from the source face.
Weight weight 0 – 1 1 0 = codebook only, 1 = closest to the input; not for KEEP.
Window window centre, script centre DicFace only: frames n−2 … n+2 or n … n+4.
Sample sample mean, random mean DicFace only.
Batch batch 1 – 8 1 Faces per network call; not for KEEP.
Crop crop vfhq, ffhq vfhq PGTFormer only: face crop style.
FP16 half on/off on Half precision for detection and face parsing; the restoration model stays at full precision.
Sharpness sharpness 0 – 1 0.04 Faces blurrier than this are skipped.
Min. eye dist. min_eye_dist 0 – 50 5 Faces with a smaller eye distance (in pixels) are skipped.
Min. visibility min_visibility 0 – 1 0.45 Faces less visible than this are skipped.
Fade fade 0 – Radius − 7 8 Frames over which a face fades in and out at the ends of its track.
Device device_index 0 – 4 0 CUDA device.
Upscaler bg_clip as CodeFormer Bilinear Only with Scale above 1: how the background is upscaled.
Adjuster – Bilinear, Bicubic, Point, Lanczos, Spline16, Spline36, Spline64 Bicubic Only with Scale above 1: resizer used after the filter.
Undo upscale – on/off on On: the result is scaled back to the input size with the Adjuster. Off: the larger size is kept and only reduced to the resize resolution if it is larger.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356):

# changing range from limited to full range for vsFaceRestore
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGBS for vsFaceRestore
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="full", range_s="full")
# Temporal face restoration using FaceRestore (DicFace)
import vsfacerestore
clip = vsfacerestore.restore(clip, model="dicface", upscale=1) # 640x356
# adjusting output color from RGBS to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_in_s="full", range_s="full", dither_type="error_diffusion")
# changing range from full to limited range for matching filter target luma scale
clip = core.resize.Bicubic(clip, format=vs.YUV420P10,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})

How Hybrid calls it

  • The clip is converted to full-range RGB.
  • Model and scale are always written; the other options only when they differ from the defaults, which equal the package defaults.
  • For an Upscaler other than Bilinear, Hybrid builds a background clip at the final size; network upscalers run on the selected device.
  • FaceRestore also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Sources

  • Hybrid's tooltips