Vapoursynth HAVC: Difference between revisions

From wiki.selur.net
Jump to navigation Jump to search
New page: HAVC (via create-page on MediaWiki MCP Server)
 
Update to current Hybrid behaviour; code as <pre> (via update-page on MediaWiki MCP Server)
 
Line 87: Line 87:
== Example ==
== Example ==
Default settings (exemplar model CMNET2 with references from HAVC) on a black-and-white 10-bit YUV 4:2:0 source:
Default settings (exemplar model CMNET2 with references from HAVC) on a black-and-white 10-bit YUV 4:2:0 source:
<syntaxhighlight lang="python">
<pre>
# changing color matrix from '470bg' to '709' for vsHAVC
# changing color matrix from '470bg' to '709' for vsHAVC
clip = core.resize.Bicubic(clip, matrix_in_s="470bg", matrix_s="709")
clip = core.resize.Bicubic(clip, matrix_in_s="470bg", matrix_s="709")
Line 94: Line 94:
# changing color matrix from '709' to '470bg' undoing color matrix change for vsHAVC
# changing color matrix from '709' to '470bg' undoing color matrix change for vsHAVC
clip = core.resize.Bicubic(clip, matrix_in_s="709", matrix_s="470bg")
clip = core.resize.Bicubic(clip, matrix_in_s="709", matrix_s="470bg")
</syntaxhighlight>
</pre>


== How Hybrid calls it ==
== How Hybrid calls it ==

Latest revision as of 09:50, 16 November 2026

HAVC (Hybrid Automatic Video Colorizer) colorizes black-and-white video with deep learning. It combines still-image colorizers (DeOldify, DDColor and others) and can then spread the colors of reference frames over each scene with an exemplar video model, so the colors stay stable over time.

Filtering → Vapoursynth → Color → Coloring

HAVC controls in Hybrid
HAVC controls in Hybrid

What it does

  1. Picture based colorization: two still-image models (Coloring, e.g. DeOldify + DDColor) colorize the frames; their results are merged (Combine) and stabilized (Stabilize).
  2. Exemplar models (optional, on by default): at scene changes reference frames are taken – from HAVC itself, from a folder of (corrected) reference frames or from a reference clip – and a video model (CMNET2, Deep-Exemplar, Deep-Remaster or ColorMNet) propagates their colors over the following frames.
  3. Post-processing: color noise reduction in one hue range (Color tweaks, Denoise), remapping of one color range to another (Color map), contrast and brightness tuning (B&W tune) and optional frame interpolation of the colors.

Requirements

  • Windows with an NVIDIA graphics card (CUDA, upstream recommends at least an RTX 3060) and Hybrid's torch add-on (package vshavc). Without them the filter is not offered.

Settings

Colorization

Control Parameter Values Hybrid default Meaning
Speed Preset placebo … veryfast medium Render size and speed of the picture based colorization.
Coloring ColorModel 15 combinations, e.g. Video+ModelScope, Video+Artistic, DeOldify or DDColor models alone Video+ModelScope The two still-image models used.
Combine CombMethod Simple, Constrained-Chroma, Luma-Masked, Adaptive-Luma, Chroma-Retention, ChromaBound Adaptive Simple How the results of the two models are merged.
Stabilize (spat) VideoTune verystable … veryvivid veryvivid Trade-off between vivid and stable colors.
Stabilize (temp) ColorTemp none, very high … very low none Temporal color stabilization (DDColor models).
Color tweaks ColorFix none, retinex/red, magenta, magenta/violet, violet, violet/red, blue/magenta, yellow, yellow/orange, yellow/green retinex/red Hue range in which color noise is reduced; disabled when Denoise is none.
Denoise ColorTune none, light, medium, strong light Strength of the Color tweaks correction.
Color map ColorMap none, blue->brown, blue->red, blue->green, green->brown, green->red, green->blue, redrose->brown, redrose->blue, red->brown, red->blue, yellow->rose red->brown Changes one color range into another.
Interpolation FrameInterp 0 – 10 0 0 = off; otherwise colors are computed every n frames and interpolated (below 5 with Deep-Exemplar, from 5 on with ColorMNet). Upstream suggests 3.
B&W tune BlackWhiteTune none, light, medium, strong light Contrast and brightness improvement of the colored frames.
B&W mode BlackWhiteMode CLAHE (luma), Simple RGB, CLAHE RGB, CLAHE (luma) + Simple RGB, ScaleAbs-LUT, Multi-Scale Retinex (HAVC), Multi-Scale Retinex (B&W) CLAHE (luma) Method of the improvement; only with B&W tune.
B&W blend BlackWhiteBlend on/off on Blend the tuned frames with the untuned ones; only with B&W tune.
FP16 enable_fp16 on/off on Half precision for DDColor.
GPU ID – – – Disabled: HAVC always runs on the first CUDA device.

Exemplar Models (group, on by default)

Control Parameter Values Hybrid default Meaning
Method DeepExMethod HAVC, HAVC + RF same as video, HAVC + RF different from video, RF same as video, RF different from video, clip same as video, clip different from video HAVC Where the reference frames come from: HAVC itself, HAVC plus a folder of reference frames (RF), only a folder, or a reference clip. "same/different from video" tells whether the references show the same scenes.
Model DeepExModel CMNET2, Deep-Exemplar, Deep-Remaster, ColorMNet CMNET2 Video model that propagates the reference colors.
Backbone DeepExBackbone dinov2, dinov3 dinov3 Key encoder of CMNET2; only for CMNET2.
Frames DeepExMaxMemFrames 0 – 15000 20 Reference frames kept in memory. CMNET2/Deep-Remaster: suggested 4–50 (up to 100 for CMNET2), 0 = 20. ColorMNet remote: 50–5000, 0 = 5000 or the clip length; local: about 1–2 frames per 8 GB GPU memory.
Preset DeepExPreset auto, fast, medium, slow, slower auto Render size of the exemplar model.
Vivid DeepExVivid on/off off More vivid colors (CMNET2 about +15 % saturation, Deep-Exemplar +25 %, Deep-Remaster +20 % and hue +10; ColorMNet: memory reset at each reference frame).
Ref merge / Weight DeepExRefMerge no, verylow, low, med, high, veryhigh no Merges the reference frames into the propagated frames with weight 0.3–0.7 (shown in Weight); only with methods HAVC, HAVC + RF same as video and clip same as video.
Reference frames only DeepExOnlyRefFrames on/off off Writes the generated reference frames into the reference folder so they can be checked and corrected; only with method HAVC and a folder set.
Ref Dir/File ScFrameDir folder or file – Folder of reference frames (RF methods) or reference clip (clip methods).
SC thresh ScThreshold 0 – 1 0.10 Scene change threshold that creates reference frames; 0 = none.
SC offset ScThtOffset 1 – 25 1 Compares frame n with n − offset; not for the RF-only methods.
SC min int ScMinInt 1 – 25 1 Minimum distance between two scene changes; not for the RF-only methods.
SC min freq ScMinFreq 0 – 1500 15 Forces a reference frame at least every n frames.
SC SSIM thresh ScThtSSIM 0 – 1 0 Drops reference frames that are too similar to already chosen ones; suggested 0.35–0.65.
normalize ScNormalize on/off off Normalizes the frames before scene detection; makes soft cuts easier to detect.

Upstream's recommended high-quality setting (RTX 4070 or better): Speed slower, Color map red->brown, Color tweaks retinex/red, Denoise medium, Stabilize balanced, B&W tune light with CLAHE (luma), Interpolation 3, CMNET2 in remote mode with Frames 0, Preset auto, Vivid on, Ref merge high, SC thresh 0.10, SC min freq 0, normalize on.

Example

Default settings (exemplar model CMNET2 with references from HAVC) on a black-and-white 10-bit YUV 4:2:0 source:

# changing color matrix from '470bg' to '709' for vsHAVC
clip = core.resize.Bicubic(clip, matrix_in_s="470bg", matrix_s="709")
# adding colors using HAVC
clip = havc.HAVC_main(clip, DeepExMethod=0, DeepExRefMerge=0, ScFrameDir=None, DeepExModel=0, DeepExEncMode=0, DeepExBackbone="dinov3", DeepExMaxMemFrames=20)
# changing color matrix from '709' to '470bg' undoing color matrix change for vsHAVC
clip = core.resize.Bicubic(clip, matrix_in_s="709", matrix_s="470bg")

How Hybrid calls it

  • HAVC accepts any format; it works in 8-bit RGB internally and returns the input format. Hybrid converts the color matrix to BT.709 before the filter and back afterwards, and makes width and height even.
  • Most settings are only written when they differ from the defaults. With the exemplar group on, Method, Model and the reference folder (None without one) are always written, Backbone always for CMNET2. The exemplar model always runs in remote mode (a separate process without GPU memory limit, DeepExEncMode=0); there is no control for it.
  • With the RF-only and clip methods the picture based settings (Speed, Coloring, Combine, Stabilize, Color tweaks) are not used and are disabled.
  • Hybrid skips the filter with a script comment instead of producing an error when a method needs a reference folder or clip and none is set, or when SC thresh and SC min freq are both 0.
  • Some settings combinations force single-threaded processing (Reference frames only with an SSIM threshold and SC min freq above 1).
  • HAVC also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

HAVC (HAVC_main)

HAVC by Dan64 (formerly vs-deoldify) is a VapourSynth package that combines DeOldify, DDColor and Zhang's colorization models with the exemplar video models CMNET2, Deep-Exemplar, Deep-Remaster and ColorMNet. It requires PyTorch 2.1.1+, VapourSynth R62+ and an NVIDIA GPU; the parameters are documented in the function's docstrings and in a HAVC User Guide (PDF) in the project's documentation folder.

Sources

  • vs-havc README and its documentation folder (HAVC User Guide)
  • Hybrid's tooltips