Vapoursynth SCUNet (mlrt)

From wiki.selur.net
Jump to navigation Jump to search

SCUNet (mlrt) runs the SCUNet denoising network as an ONNX model through vs-mlrt. It works with NVIDIA and Intel hardware and on the CPU (slow), and can restore the original colors afterwards (ColorFix).

Filtering → Vapoursynth → DeNoise

SCUNet (mlrt) controls in Hybrid
SCUNet (mlrt) controls in Hybrid

What it does

Every frame is passed through SCUNet ("Swin-Conv-UNet"). The real models handle real-world noise of unknown strength; the color and gray models are trained for Gaussian noise of a fixed level. The gray models only filter the luma plane; the chroma planes are kept.

Requirements

  • Hybrid's vs-mlrt add-on with the SCUNet models and at least one backend.
  • GPU backends: TensorRT and ONNX (CUDA) for NVIDIA, OpenVINO (GPU) for Intel; on macOS ONNX (CoreML).
  • Fix Color needs the torch add-on (Windows only).

Settings

Control Parameter Range / values Hybrid default Meaning
Model model color_15/25/50, real_psnr, real_gan, gray_15/25/50 real_psnr Network to use; real_psnr keeps the colors better than real_gan.
Tiles tiles 1 – 64 1 Splits the frame to save memory.
Tile pad overlap 0 – 64 64 Overlap of the tiles.
Backend backend ONNX (CUDA/CPU), OpenVINO (GPU/CPU), TensorRT, TensorRT RTX (macOS: ONNX CoreML/CPU) ONNX (CPU) Runtime that executes the network.
Device device_id 0 – 4 0 GPU to use; not for CPU and CoreML backends.
Streams num_streams 1 – 64 1 Parallel streams; TensorRT, TensorRT RTX, OpenVINO (GPU) and CoreML only.
FP16 / BF16 fp16, bf16 on/off off / off Half precision; FP16 is greyed out for the CPU backends (full precision only) and allowed on the GPU backends, OpenVINO (GPU) included. BF16 only with TensorRT and OpenVINO (CPU). The two exclude each other.
CUDA Graph use_cuda_graph on/off off TensorRT, TensorRT RTX and ONNX (CUDA) only.
TRT backend optimization builder_optimization_level 0 – 5 3 TensorRT engine build effort.
Dynamic Shapes (min/opt/max) min_shapes, opt_shapes, max_shapes 0 – 8192 unset TensorRT engine sizes.

Fix Color: the same options as in SCUNet (torch) (average or wavelet mode, radius, fast, wavelets, plane).

Example

Default settings on a 10-bit YUV 4:2:0 source (640×356, padded to mod 8):

clip = core.std.AddBorders(clip=clip, left=0, right=0, top=2, bottom=2) # add borders to achieve mod 8 (vsSCUNetmlrt) - 640x360
# changing range from limited to full range for vsSCUNetmlrt
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGBS for vsSCUNetmlrt
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="full", range_s="full")
# denoising using SCUNet (mlrt)
from vsmlrt import Backend
clip = vsmlrt.SCUNet(clip, model=3, backend=Backend.ORT_CPU(fp16=False,output_format=0))
clip = core.std.Crop(clip=clip, left=0, right=0, top=2, bottom=2) # removing added borders from mod requirement (vsSCUNetmlrt) -  640x356

How Hybrid calls it

  • The clip is padded to mod 8 and converted to 32-bit float RGB (16-bit float with FP16); for the gray models a float YUV clip is used and only its luma plane is filtered.
  • The model and the backend options are always written; tiles and tile pad only when they differ from the defaults.
  • TensorRT builds an engine on the first run for each size and cannot be mixed with torch TensorRT filters in the same script.
  • SCUNet (mlrt) also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

SCUNet

SCUNet (Zhang et al., Machine Intelligence Research 2023) combines Swin Transformer and residual convolution branches in a UNet and was trained only on synthetic degradations. vs-mlrt chooses real_psnr as its default model because the GAN model can shift colors.

Sources