Vapoursynth HINet

From wiki.selur.net
Jump to navigation Jump to search

HINet (torch) runs Megvii's Half Instance Normalization Network, a neural network for image restoration. Hybrid offers its four trained models: deblurring (GoPro or REDS), denoising (SIDD) and deraining (Rain13k). The derain model can also be used to remove film scratches.

Filtering → Vapoursynth → DeNoise

HINet controls in Hybrid
HINet controls in Hybrid

What it does

Every frame is passed through the network of the chosen mode; there is no temporal part. With Scratch (derain mode only), Hybrid prepares the frame so that vertical film scratches look like rain streaks, lets the derain model remove them, and undoes the preparation afterwards.

Requirements

  • Windows, Hybrid's torch add-on and an NVIDIA GPU with CUDA; the models are about 355 MB each.

Settings

Control Parameter Range / values Hybrid default Meaning
Mode mode deblur - GoPro, deblur - REDS, denoise, derain deblur - GoPro The model that is used.
FP16 fp16 on/off off Half precision; not available for the deblur modes.
CPU device_type on/off off Run on the CPU (very slow).
Device device_index 0 – 4 0 CUDA device.
Tile (x/y) tile_x, tile_y 0 – 8192 0 / 0 Tile size to save GPU memory; 0 = no tiling.
Tile pad tile_pad 0 – 64 16 Overlap of the tiles; only with tiling.
Count – 1 – 10 1 Applies HINet several times in a row.

Scratch (derain only)

Control Range Hybrid default Meaning
Resolution mode divider, manual divider How the frame is resized before the network.
Divider (width / height) 1 – 100 3.5 / 1.5 The frame size is divided by these factors.
Resolution (manual) 0 – 8192 0 / 0 Fixed size; 0 keeps the source width or height.
In (min / max) 0 – 1 0.11 / 0.82 Input range of the levels adjustment.
Out (min / max) 0 – 1 0 / 1 Output range of the levels adjustment.
Gamma 0 – 1 0.8 Gamma of the levels adjustment.

Example

Default settings on a 10-bit YUV 4:2:0 source (640×356, padded to mod 8):

clip = core.std.AddBorders(clip=clip, left=0, right=0, top=2, bottom=2) # add borders to achieve mod 8 (vsHINet) - 640x360
# adjusting color space from YUV420P10 to RGBS for vsHINet
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="limited", range_s="full")
# denoising/debluring/deraining using HINet
clip = HINet(clip, device_index=0) #  call 0
clip = core.std.Crop(clip=clip, left=0, right=0, top=2, bottom=2) # removing added borders from mod requirement (vsHINet) -  640x356

How Hybrid calls it

  • The clip is padded to mod 8 and converted to 32-bit float RGB, the only format HINet takes, and cropped and converted back afterwards.
  • The device is always written; mode, FP16 and the tile values only when they differ from the defaults (tiles only when both sizes are above 0).
  • With Scratch the network is wrapped: resize, levels adjustment, invert, HINet, invert, reverse levels adjustment, resize back.
  • HINet also offers the common options (Masked, ApplyOnlyTo, Merge, OnRegion, Gimmick).

Background

HINet

HINet (Megvii, 2021) builds on Half Instance Normalization blocks in a two-stage network and won the NTIRE 2021 deblurring challenge (track 2). The models were trained on SIDD (denoising), GoPro and REDS (deblurring) and Rain13k (deraining).

vs-hinet

HolyWu's vs-hinet (archived) wraps HINet for VapourSynth with PyTorch and only accepts 32-bit float RGB. Frames that are not divisible by 16 are padded internally.

Sources