Vapoursynth SCUNet (mlrt)

From wiki.selur.net
Revision as of 03:24, 15 November 2026 by Selur (talk | contribs) (New filter page (via create-page on MediaWiki MCP Server))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

SCUNet (mlrt) runs the SCUNet denoising network as an ONNX model through vs-mlrt. It works with NVIDIA and Intel hardware and on the CPU (slow), and can restore the original colors afterwards (ColorFix).

Filtering → Vapoursynth → DeNoise

SCUNet (mlrt) controls in Hybrid
SCUNet (mlrt) controls in Hybrid

What it does

Every frame is passed through SCUNet ("Swin-Conv-UNet"). The real models handle real-world noise of unknown strength; the color and gray models are trained for Gaussian noise of a fixed level. The gray models only filter the luma plane; the chroma planes are kept.

Requirements

  • Hybrid's vs-mlrt add-on with the SCUNet models and at least one backend.
  • GPU backends: TensorRT and ONNX (CUDA) for NVIDIA, OpenVINO (GPU) for Intel; on macOS ONNX (CoreML).
  • Fix Color needs the torch add-on.

Settings

Control Parameter Range / values Hybrid default Meaning
Model model color_15/25/50, real_psnr, real_gan, gray_15/25/50 real_psnr Network to use; real_psnr keeps the colors better than real_gan.
Tiles tiles 1 – 64 1 Splits the frame to save memory.
Tile pad overlap 0 – 64 64 Overlap of the tiles.
Backend backend ONNX (CUDA/CPU), OpenVINO (GPU/CPU), TensorRT, TensorRT RTX (macOS: ONNX CoreML/CPU) ONNX (CPU) Runtime that executes the network.
Device device_id 0 – 4 0 GPU to use; not for CPU and CoreML backends.
Streams num_streams 1 – 64 1 Parallel streams; TensorRT, TensorRT RTX, OpenVINO (GPU) and CoreML only.
FP16 / BF16 fp16, bf16 on/off off / off Half precision; BF16 only with TensorRT and OpenVINO (CPU). The two exclude each other.
CUDA Graph use_cuda_graph on/off off TensorRT, TensorRT RTX and ONNX (CUDA) only.
TRT backend optimization builder_optimization_level 0 – 5 3 TensorRT engine build effort.
Dynamic Shapes (min/opt/max) min_shapes, opt_shapes, max_shapes 0 – 8192 unset TensorRT engine sizes.

Fix Color: the same options as in SCUNet (torch) (average or wavelet mode, radius, fast, wavelets, plane).

Example

Default settings on a 10-bit YUV 4:2:0 source (640×356, padded to mod 8):

clip = core.std.AddBorders(clip=clip, left=0, right=0, top=2, bottom=2) # add borders to achieve mod 8 (vsSCUNetmlrt) - 640x360
# changing range from limited to full range for vsSCUNetmlrt
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGBS for vsSCUNetmlrt
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="full", range_s="full")
# denoising using SCUNet (mlrt)
from vsmlrt import Backend
clip = vsmlrt.SCUNet(clip, model=3, backend=Backend.ORT_CPU(fp16=False,output_format=0))
clip = core.std.Crop(clip=clip, left=0, right=0, top=2, bottom=2) # removing added borders from mod requirement (vsSCUNetmlrt) -  640x356

How Hybrid calls it

  • The clip is padded to mod 8 and converted to 32-bit float RGB (16-bit float with FP16); for the gray models a float YUV clip is used and only its luma plane is filtered.
  • The model and the backend options are always written; tiles and tile pad only when they differ from the defaults.
  • TensorRT builds an engine on the first run for each size and cannot be mixed with torch TensorRT filters in the same script.
  • SCUNet (mlrt) also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

SCUNet

SCUNet (Zhang et al., Machine Intelligence Research 2023) combines Swin Transformer and residual convolution branches in a UNet and was trained only on synthetic degradations. vs-mlrt chooses real_psnr as its default model because the GAN model can shift colors.

Sources