Vapoursynth SwinIR Resize (mlrt): Difference between revisions
New filter page (via create-page on MediaWiki MCP Server) |
Update to current Hybrid behaviour; code as <pre> (via update-page on MediaWiki MCP Server) |
||
| Line 21: | Line 21: | ||
| Backend || backend || ONNX (CUDA), ONNX (CPU), OpenVINO (GPU/CPU), TensorRT, TensorRT RTX (macOS: ONNX CoreML/CPU) || ONNX (CPU) (macOS: CoreML) || Runtime that executes the network; only installed backends are listed. | | Backend || backend || ONNX (CUDA), ONNX (CPU), OpenVINO (GPU/CPU), TensorRT, TensorRT RTX (macOS: ONNX CoreML/CPU) || ONNX (CPU) (macOS: CoreML) || Runtime that executes the network; only installed backends are listed. | ||
|- | |- | ||
| 32bit || fp16 || on/off || off || Full precision; off runs in half precision. | | 32bit || fp16 || on/off || off || Full precision; off runs in half precision. The CPU backends always run in full precision (forced on and greyed out); TensorRT RTX starts with it on; OpenVINO (GPU) also allows half precision. | ||
|- | |- | ||
| BF16 || bf16 || on/off || off || bfloat16; TensorRT and OpenVINO (CPU) only. Switches 32bit on. | | BF16 || bf16 || on/off || off || bfloat16; TensorRT and OpenVINO (CPU) only. Switches 32bit on. | ||
| Line 36: | Line 36: | ||
|- | |- | ||
| Tile (wxh) || tilesize || 1 – 8192 || off, 256×256 || Splits the frame to save memory; off = whole frame. | | Tile (wxh) || tilesize || 1 – 8192 || off, 256×256 || Splits the frame to save memory; off = whole frame. | ||
|- | |||
| Overlap || overlap || 2 – 624 || 16 || Overlap of the tiles; only with tiling. | |||
|- | |- | ||
| Dynamic Shapes (opt/min/max) || opt_shapes, min_shapes, max_shapes || 0 – 8192 || unset || TensorRT engine sizes; with a minimum size the engine becomes dynamic. | | Dynamic Shapes (opt/min/max) || opt_shapes, min_shapes, max_shapes || 0 – 8192 || unset || TensorRT engine sizes; with a minimum size the engine becomes dynamic. | ||
| Line 44: | Line 46: | ||
== Example == | == Example == | ||
Default settings on a 10-bit YUV 4:2:0 source (640x356), resized to 1280x712: | Default settings on a 10-bit YUV 4:2:0 source (640x356), resized to 1280x712: | ||
< | <pre> | ||
# changing range from limited to full range for vsSwinIRmlrtResize | # changing range from limited to full range for vsSwinIRmlrtResize | ||
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full") | clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full") | ||
| Line 50: | Line 52: | ||
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange' | prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange' | ||
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL}) | clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL}) | ||
# adjusting color space from YUV420P10 to | # adjusting color space from YUV420P10 to RGBS for vsSwinIRmlrtResize | ||
clip = core.resize.Bicubic(clip=clip, format=vs. | clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="full", range_s="full") | ||
# resizing using SwinIR (mlrt) | # resizing using SwinIR (mlrt) | ||
from vsmlrt import Backend | from vsmlrt import Backend | ||
clip = vsmlrt.SwinIR(clip=clip, model=0, backend=Backend.ORT_CPU(fp16= | clip = vsmlrt.SwinIR(clip=clip, model=0, backend=Backend.ORT_CPU(fp16=False,output_format=0)) | ||
# adjusting output color from | # adjusting output color from RGBS to YUV420P10 for NVEncModel | ||
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_in_s="full", range_s="full", dither_type="error_diffusion") | clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_in_s="full", range_s="full", dither_type="error_diffusion") | ||
# changing range from full to limited range for matching filter target luma scale | # changing range from full to limited range for matching filter target luma scale | ||
| Line 62: | Line 64: | ||
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange' | prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange' | ||
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED}) | clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED}) | ||
</ | </pre> | ||
== How Hybrid calls it == | == How Hybrid calls it == | ||
* Selected with Resize enabled and Resizer set to SwinIR (mlrt) (see [[Vapoursynth Resizer]]). | * Selected with Resize enabled and Resizer set to SwinIR (mlrt) (see [[Vapoursynth Resizer]]). | ||
* The clip is converted to 32-bit float RGB, or 16-bit float RGB with 32bit off, and treated as full range; there is no size multiple. | * The clip is converted to 32-bit float RGB, or 16-bit float RGB with TensorRT and 32bit off, and treated as full range; there is no size multiple. | ||
* The output is input × the model's factor. If that differs from the target, fmtconv resizes it with the ''Resizer'' kernel to the exact size (half-float clips are converted to 32-bit float first). With separately filtered fields the target height is halved. | * The output is input × the model's factor. If that differs from the target, fmtconv resizes it with the ''Resizer'' kernel to the exact size (half-float clips are converted to 32-bit float first). With separately filtered fields the target height is halved. | ||
* Model and backend options are always written; | * Model and backend options are always written; tile size and overlap only when tiling is enabled. | ||
* TensorRT builds an engine on the first run for each size. TensorRT cannot be used in vs-mlrt filters and torch filters in the same script. | * TensorRT builds an engine on the first run for each size. TensorRT cannot be used in vs-mlrt filters and torch filters in the same script. | ||
* The PyTorch version is [[Vapoursynth SwinIR Resize|SwinIR (torch)]]. | * The PyTorch version is [[Vapoursynth SwinIR Resize|SwinIR (torch)]]. | ||
Latest revision as of 09:53, 16 November 2026
SwinIR (mlrt) upscales with the super-resolution models of SwinIR, a Swin Transformer restoration network, run as ONNX models through vs-mlrt. It is one of the resizers of Vapoursynth Resizer and works on NVIDIA and Intel hardware and on the CPU.
Filtering → Vapoursynth → Frame → Resize

What it does
Every frame is enlarged by the factor of the chosen model (2, 3, 4 or 8). Lightweight and classical models were trained on clean downscaled images; the realSR (BSRGAN) models on real-world degradations, where GAN models look sharper and PSNR models smoother. Hybrid resizes the result to the exact output size afterwards.
Requirements
- Hybrid's vs-mlrt add-on with the SwinIR (mlrt) models and at least one backend.
- GPU backends: TensorRT and ONNX (CUDA) for NVIDIA, TensorRT RTX for RTX cards, OpenVINO (GPU) for Intel; on macOS ONNX (CoreML).
Settings
| Control | Parameter | Range / values | Hybrid default | Meaning |
|---|---|---|---|---|
| Model | model | lightweight SR x2/x3/x4, classical SR x2/x3/x4/x8, realSR BSRGAN L GAN x4, L PSNR x4, M GAN x2, M PSNR x2, M GAN x4, M PSNR x4 | lightweight SR x2 | Network; the "xN" in the name is the upscale factor. |
| Backend | backend | ONNX (CUDA), ONNX (CPU), OpenVINO (GPU/CPU), TensorRT, TensorRT RTX (macOS: ONNX CoreML/CPU) | ONNX (CPU) (macOS: CoreML) | Runtime that executes the network; only installed backends are listed. |
| 32bit | fp16 | on/off | off | Full precision; off runs in half precision. The CPU backends always run in full precision (forced on and greyed out); TensorRT RTX starts with it on; OpenVINO (GPU) also allows half precision. |
| BF16 | bf16 | on/off | off | bfloat16; TensorRT and OpenVINO (CPU) only. Switches 32bit on. |
| Device | device_id | 0 – 4 | 0 | GPU to use; not for CPU and CoreML backends. |
| Streams | num_streams | 1 – 16 | 1 | Parallel streams; TensorRT, TensorRT RTX, OpenVINO (GPU) and CoreML only. |
| Cuda Graph | use_cuda_graph | on/off | off | TensorRT, TensorRT RTX and ONNX (CUDA) only. |
| TRT backend optimization | builder_optimization_level | 0 – 5 | 3 | TensorRT engine build effort. |
| TRT max workspace | workspace | 0 – 1048576 MiB | 0 | TensorRT workspace limit; 0 = not set. |
| Tile (wxh) | tilesize | 1 – 8192 | off, 256×256 | Splits the frame to save memory; off = whole frame. |
| Overlap | overlap | 2 – 624 | 16 | Overlap of the tiles; only with tiling. |
| Dynamic Shapes (opt/min/max) | opt_shapes, min_shapes, max_shapes | 0 – 8192 | unset | TensorRT engine sizes; with a minimum size the engine becomes dynamic. |
| Resizer | – | point, rect, linear, cubic, lanczos, blackman, blackmanminlobe, spline36, spline64, spline144, spline, gauss, sinc | spline64 | Kernel that brings the result to the exact output size. |
Example
Default settings on a 10-bit YUV 4:2:0 source (640x356), resized to 1280x712:
# changing range from limited to full range for vsSwinIRmlrtResize
clip = core.resize.Bicubic(clip, format=vs.YUV420P10, range_in_s="limited", range_s="full")
# setting color range to PC (full) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_FULL})
# adjusting color space from YUV420P10 to RGBS for vsSwinIRmlrtResize
clip = core.resize.Bicubic(clip=clip, format=vs.RGBS, matrix_in_s="470bg", range_in_s="full", range_s="full")
# resizing using SwinIR (mlrt)
from vsmlrt import Backend
clip = vsmlrt.SwinIR(clip=clip, model=0, backend=Backend.ORT_CPU(fp16=False,output_format=0))
# adjusting output color from RGBS to YUV420P10 for NVEncModel
clip = core.resize.Bicubic(clip=clip, format=vs.YUV420P10, matrix_s="470bg", range_in_s="full", range_s="full", dither_type="error_diffusion")
# changing range from full to limited range for matching filter target luma scale
clip = core.resize.Bicubic(clip, format=vs.YUV420P10,range_in_s="full", range_s="limited")
# setting color range to TV (limited) range.
prop_name = '_Range' if core.core_version.release_major >= 74 else '_ColorRange'
clip = core.std.SetFrameProps(clip=clip, **{prop_name: vs.RANGE_LIMITED})
How Hybrid calls it
- Selected with Resize enabled and Resizer set to SwinIR (mlrt) (see Vapoursynth Resizer).
- The clip is converted to 32-bit float RGB, or 16-bit float RGB with TensorRT and 32bit off, and treated as full range; there is no size multiple.
- The output is input × the model's factor. If that differs from the target, fmtconv resizes it with the Resizer kernel to the exact size (half-float clips are converted to 32-bit float first). With separately filtered fields the target height is halved.
- Model and backend options are always written; tile size and overlap only when tiling is enabled.
- TensorRT builds an engine on the first run for each size. TensorRT cannot be used in vs-mlrt filters and torch filters in the same script.
- The PyTorch version is SwinIR (torch).
Background
SwinIR
SwinIR (Liang et al., ICCV workshops 2021) is an image restoration network built from Swin Transformer blocks. The classical models are the medium-size SwinIR-M trained on DF2K, the lightweight ones the small SwinIR-S trained on DIV2K; the realSR models use the BSRGAN degradation model. SwinIR also has denoise and JPEG models, which this resizer does not offer.
vs-mlrt
vs-mlrt runs ONNX models in VapourSynth with several backends. Its SwinIR wrapper expects 16- or 32-bit float RGB and uses the whole frame as one tile unless a tile size is given.
Sources
- vs-mlrt README
- SwinIR
- Hybrid's tooltips