Vapoursynth CodeFormer

From wiki.selur.net
Revision as of 09:48, 16 November 2026 by Selur (talk | contribs) (New filter page (via create-page on MediaWiki MCP Server))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

CodeFormer is a blind face restoration network: it finds the faces in each frame, rebuilds them from a learned codebook and pastes them back, optionally while upscaling the whole frame. It runs on an NVIDIA GPU through PyTorch.

Filtering → Vapoursynth → Other

CodeFormer controls in Hybrid
CodeFormer controls in Hybrid

What it does

Faces are detected, cropped and aligned, replaced by a reconstruction and blended back into the frame. The network invents plausible detail rather than recovering the real one, so faces can look different from the person and can flicker from frame to frame. With Scale above 1 the rest of the frame is upscaled as well. For a temporally steadier result see FaceRestore.

Requirements

  • Windows, Hybrid's torch add-on and an NVIDIA GPU with CUDA.
  • The dlib detector is only listed when dlib is installed.

Settings

Control Parameter Range / values Hybrid default Meaning
Scale upscale 1 – 16 1 Upscale factor of the output.
Detector detector retinaface, dlib retinaface Face detector.
Weight weight 0 – 1 0.5 Low values give prettier faces from the codebook, high values stay closer to the input.
Only center only_center_face on/off off Restores only the face nearest the center.
Streams num_streams 1 – 16 1 Number of CUDA streams; not more than VapourSynth's thread count.
Device device_index 0 – 4 0 CUDA device.
Upscaler bg_clip Bilinear, Bicubic, Point, Lanczos, Spline16, Spline36, Spline64, NNEDI3, RealESRGAN (general), FeMaSR (2x), FeMaSR (4x) Bilinear Only with Scale above 1: how the background is upscaled. Bilinear lets the plugin do it; the network upscalers are listed only when installed.
Adjuster – Bilinear, Bicubic, Point, Lanczos, Spline16, Spline36, Spline64 Bicubic Only with Scale above 1: resizer used after the filter.
Undo upscale – on/off on On: the result is scaled back to the input size with the Adjuster. Off: the larger size is kept and only reduced to the resize resolution if it is larger.

Example

Default settings on a 10-bit YUV 4:2:0 source (640x356):

# changing range from limited to full range for vsCodeFormer
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGB24 for vsCodeFormer
clip = core.resize.Bicubic(clip=clip, format=vs.RGB24, matrix_in_s="470bg", range_in_s="full", dither_type="error_diffusion", range_s="full")
# Blind Face Restoration using CodeFormer
for folder in ("F:/Hybrid/64bit/Vapoursynth/Lib/site-packages/nvidia/cuda_runtime/bin", "F:/Hybrid/64bit/Vapoursynth/Lib/site-packages/nvidia/cublas/bin"):
  if os.path.isdir(folder): os.add_dll_directory(folder)
from vscodeformer import codeformer as CodeFormer
clip = CodeFormer(clip, upscale=1) # 640x356
# adjusting output color from RGB24 to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_in_s="full", range_s="full")
# changing range from full to limited range for matching filter target luma scale
clip = core.resize.Bicubic(clip, format=vs.YUV420P10,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})

How Hybrid calls it

  • The clip is converted to 8-bit RGB, full range.
  • Scale is always written; the other options only when they differ from the defaults.
  • For an Upscaler other than Bilinear, Hybrid builds a background clip at the final size and passes it to the filter; RealESRGAN and FeMaSR backgrounds run on the selected device.
  • CodeFormer also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

CodeFormer

CodeFormer (Zhou et al., NeurIPS 2022) treats face restoration as a lookup in a discrete codebook of high-quality face parts: a transformer predicts the codebook entries from the degraded face. A fidelity weight balances between the codebook result and the input.

vs-codeformer

HolyWu's vs-codeformer runs CodeFormer in VapourSynth with PyTorch. It accepts 8-bit RGB only; a background clip has to be 8-bit RGB at exactly Scale × the input size.

Sources