Skip to content

Overview

How Mixar prepares, caches, and plays NI-compatible stems, and how the pipeline's modules divide the work.

Mixar plays four stem layers (drums, bass, other, vocals) plus a mixdown. Two sources feed that one deck contract. A user NI Stem file (.stem.mp4) is opened in place. A generated NI-compatible cache comes from HTDemucs ONNX (via ORT) and lives under app support.

Prepare resolves the source into a StemBundle. The deck receives the mixdown as its main buffer and up to four stems in NI index order. The UI reads deck state to decide whether pads are live. The ownership table names the crate behind each step.

User-facing docs: Stems (native NI, generate, storage) and Stems pad mode. This topic covers architecture for contributors.

Load path

Cached stem ensure runs only from the per-track Generate stems action (library worker). Deck load never blocks on separation. It attaches a cache when one exists. Otherwise it starts the live producer below, and keeps playing the main mix until the first window lands.

Prepare / load track
        │
        ├─ source is .stem.mp4? ──► StemBundle (in place) ──► deck load
        │
        ├─ valid stem cache? ──► StemBundle(cache) ──► deck load
        │         (main = mixdown; stems attached)
        │
        └─ else ──► decode original ──► deck load + live producer
              (main mix until first window; no disk writes)

Live separation

When Realtime Stems is enabled in Settings and a track has no cache, deck load starts a streaming separation model on a background CPU thread. It reads the decoded PCM, then runs the causal 128-sample-hop ONNX graph with its 13 recurrent states. Four-stem hops go into a 2^20-frame (≈23.8 s) LiveStemRing that the deck reads lock-free. The four stems are summed at the deck output, before the channel strip gain/EQ/filter chain. EQ and FX therefore process the stem mix as one signal.

  • The first model use downloads the 32 MB ONNX in the background; the deck plays the main mix until coverage exists. Cached stems remain authoritative.
  • No .stem.mp4 is written: live playback adds no file duplication.
  • The deck crossfades from the main mix over 5 ms when a hop is covered; pads become ready when coverage exists.
  • Seeks outside the ring re-epoch the producer with a short warm-up. The model is stateful, so a seek starts from reconstructed context rather than an offline-parity cache. The deck falls back to the main mix until it catches up.
  • The live filter runs on the CPU at roughly 0.9× real time on a Ryzen 7 5700X. It is independent of the GPU EP used by offline HTDemucs generation. The explicit Generate stems path is unchanged.
  • The live model is separate from offline stem generation and is a fallback for tracks without cached stems. Tracks with offline stems always take priority. Not recommended over offline separation: quality is lower and it uses a CPU core.

App-support layout

{app_support}/
  library.db
  settings.json
  stems/{fnv64(track_id)}.stem.mp4   ← generated NI Stem cache
  models/{name}-{sha8}.onnx          ← HTDemucs ONNX (Mixxx htdemucs_mixxx_v1)

Ownership

Layer Responsibility
analyzer-stems Model ensure/download, ORT session + EP cascade, HTDemucs split
codec Detect/open .stem.mp4; mux/demux five consistent streams + STEM atom
library track_stem rows, stems_root / models_root, clear + usage
Prepare / host Resolve native Stem or cache → StemBundle; never enqueue ensure on deck load
engine-dsp Main buffer + optional four stems in NI index order
gui-flutter Stems pads, generating banner, Settings → Storage

Use LibraryTransport for storage usage/clear and generate actions. No raw FRB from widgets.

StemBundle

Conceptual type used by prepare/host:

  • mixdown: main LoadedAudio (or PCM + rate)
  • stems: [LoadedAudio; 4]: drums, bass, other, vocals
  • optional STEM atom labels/colors
  • source_path

Demucs model output may use a different slot order; permute once when packing the bundle/file. Load does not remap.

Pages in this topic

References

Resource Path / URL
User guide Stems, Stems pads
Codec STEM mux crates/codec/src/stem_mp4.rs
Library stems crates/library/src/stems.rs
Issues #46, #380