Skip to content

Inference

Stem separation with pykeio/ort and single-file ONNX models: execution-provider cascade, model cache, and split pipeline.

Stem separation runs with pykeio/ort (ONNX Runtime) on a single-file ONNX. Two models are built in and selectable in Settings → Library → Stem model: HTDemucs (default) and SCNet int8. Mixar prefers the most capable execution provider available, then falls back through slower EPs to CPU. Separated PCM is muxed to NI .stem.mp4 (see NI .stem.mp4).

Burn wgpu was too slow for DJ workflow (~1× realtime on a 3060 Ti). StemSplit’s ORT-oriented ONNX exports are much faster on ORT (export writeup, htdemucs-onnx).

Execution-provider cascade

Register EPs in preference order; failed register / unsupported ops → next; always end on CPU. Log the EP that actually ran. Optional force: MIXAR_STEMS_ORT_EP=<name>.

Platform Order
Linux TensorRT RTX → TensorRT → CUDA → WebGPU → (OpenVINO if enabled on Intel builds) → CPU
Windows TensorRT RTX → TensorRT → CUDA → DirectML → WebGPU → CPU
macOS CoreML → WebGPU → CPU (Apple Silicon only)

Notes:

  • macOS ships arm64-only: ort-sys prebuilts exist for aarch64-apple-darwin alone, so there is no x86_64-apple-darwin dist. ARCHS = arm64 is pinned in apps/gui-flutter/macos/Runner/Configs/AppInfo.xcconfig and in the host_flutter podspec (cargokit reads $ARCHS); both must stay in sync.
  • WebGPU EP is experimental upstream: best-effort, never hard-fail the app.
  • CUDA/TensorRT only when libs are present; no install gate for launching Mixar.
  • CI / headless: CPU EP only (Cargo features).

See ort execution providers.

Module and API

Lives in crates/analyzer-stems: resolve/download ONNX → cached ort::Session per model path → chunked overlap-add infer → hand interleaved PCM to codec for .stem.mp4 mux.

Public API (stable shape)

StemSplitRequest, StemSplitResult, STEM_NAMES, ensure_model, resolve_model, split_interleaved_stereo.

  • Default model id follows the Mixxx / StemSplit artifact Mixar pins (htdemucs_mixxx_v1 / equivalent ORT artifact).
  • StemSplitResult.backend → {model}/{ep} (e.g. htdemucs_mixxx_v1/cuda).
  • Library cache match uses model-id-before-/ so EP changes do not force regen.

Models

HTDemucs

  • Artifact: HTDemucs ONNX under Mixar {app_support}/models/ (download + sha verify via manifest). See Storage & models.
  • I/O: stereo mix in → four stereo stems out in NI order: drums, bass, other, vocals (same as engine / .stem.mp4 streams 1–4).
  • Old Burn / SSC backend ids miss on model id and regenerate.

Inference pipeline:

  1. Resample interleaved stereo → model rate (typically 44.1 kHz).
  2. Chunk with overlap / triangular fade (Demucs apply_model math; segment length per model).
  3. For each chunk: ORT run → accumulate stems → progress (done, total).
  4. Library / codec muxes mixdown + four stems into {stems_root}/{fnv64(track_id)}.stem.mp4.

SCNet (int8)

scnet_int8_v1 (12.9 MB, MIT) is a spectrogram model, so it does not share HTDemucs’s waveform chunking. analyzer-stems/src/scnet.rs mirrors MSST’s demix():

  • whole-track mean/std normalize, optional reflect border (BORDER = 363825),
  • 11 s segments every 2.75 s (75 % overlap), 10 % linear fade, per-sample counter normalization,
  • rectangular centered STFT (4096/1024, 1/sqrt(4096)), model input mix_spec [1,4,2049,476], output stems_spec [1,16,2049,476],
  • iSTFT with frame-count normalization and per-stem denormalization.

The output order is drums, bass, other, vocals (NI order). The graph includes LSTM nodes, so WebGPU may partition parts of it to CPU. Measured on a 3060 Ti, one 11 s segment is ~1.0 s. The same card takes ~60 s for 120 s of audio (2.0× RT), because of the 75 % overlap.

HTDemucs remains the faster offline model; SCNet is the quality option with a much smaller download. The SCNet code and weights are MIT; its MUSDB18-HQ training corpus is educational-use, the same unresolved caveat that applies to other MUSDB-trained artifacts. The adapter is covered by STFT round-trip and padding tests, an ignored real-model smoke test, and a frame-level comparison against an independent Python/MSST reference.

Live pipeline

start_live_separator runs the same chunk/window loop ahead of a deck playhead. It publishes each finalized stride window to audio_core::LiveStemRing instead of accumulating a full-track buffer. The window finalizer is shared with the offline path, so output is bit-identical; tests/live_split_matches_offline.rs (ignored, needs the model) asserts that frame by frame.

  • Producer: 10 ms pacing poll, two-stride lookahead, re-epoch on backward/forward seeks, closes on track unload.
  • Consumer: Deck::mix_at_position reads up to four covered frames lock-free and crossfades from the main mix over 5 ms. Outside coverage it returns the main mix verbatim.
  • No disk I/O, no per-load model download, and no second decode: the producer borrows the deck’s decoded LoadedAudio.