On-Device Audio Transcription with Speaker Diarization

Best-in-class local speech processing on Apple Silicon — zero cloud, zero privacy compromise.

This project benchmarks and implements the optimal pipeline for transcribing multi-speaker audio entirely on-device, combining automatic speech recognition (ASR) with speaker diarization via the speech-swift library.

Recommended Pipeline

Based on a full comparative study on real German multi-speaker recordings:

# Transcribe a file with the recommended pipeline
swift run hybrid-transcribe meeting.wav -o transcript.txt --json transcript.json

Output:

[0:00.3–0:12.1] Speaker 1: Ich würde mich sehr freuen, wenn wir die Kochplatte nach rechts versetzen.
[0:12.4–0:14.2] Speaker 2: Welche Hochplatte meinst du? An der Wand?
[0:14.5–0:55.0] Speaker 1: Das finde ich auch. Du hast ja weniger gekocht ...

Architecture

Audio file (WAV/M4A/FLAC)
       │
       ├──▶  SortformerDiarizer (CoreML, Neural Engine)
       │         NVIDIA Sortformer — end-to-end diarization
       │         → who spoke when  (speaker segments)
       │
       └──▶  Apple SpeechTranscriber (de-DE, Speech API)
                 word-level timestamps
                 → what was said
                        │
                        ▼
               Segment merger (time overlap)
                        │
                        ▼
               Speaker-attributed transcript

Both pipelines run on the same audio. Results are merged by time overlap: each word is assigned to the speaker segment that covers its midpoint.

Benchmark Results

Measured on two real German legal office recordings (Kanzlei planning sessions), 10-minute clips, against Gemini 3-flash-preview ground-truth transcripts.

Diarizer Comparison

Pyannote over-estimates speaker count on German conversational audio — its default clustering threshold is tuned on English benchmarks. Sortformer's end-to-end CoreML approach is both more accurate and 1.5–2× faster.

ASR Comparison (Apple SpeechTranscriber + Sortformer)

Vocabulary recall is measured against the full-recording Gemini GT (10-min clip ≈ 25% of total), so recall is underestimated. Qualitative review confirms transcription accuracy is high for German.

Speaker-Switch Quality

ASR Engine Comparison

Three engines are available. All run fully on-device on Apple Silicon.

# Run with Qwen3-ASR instead of Apple (first run downloads ~380 MB)
swift run diarization-study --asr qwen3-asr --diarizers sortformer --file both

# Compare all three ASR engines
swift run diarization-study --asr all --diarizers sortformer --file both

Requirements

• macOS 27 (sequoia 27+), Xcode 27 beta
• Apple Intelligence speech models installed (de-DE for German)
• ffmpeg — for audio conversion (brew install ffmpeg)
• Internet on first run — speech-swift downloads model weights to ~/Library/Caches/

Model Downloads (first run only)

Quick Start

cd ~/ai-mac-air/development/hybrid-transcribe

# Transcribe a meeting recording
swift run hybrid-transcribe recording.m4a -o transcript.txt --json transcript.json -v

# Use Neural Engine instead of GPU for speaker embeddings (pyannote only)
swift run hybrid-transcribe recording.wav --embedding-engine coreml

# Run the full diarization + ASR comparative study
./scripts/prepare-study-clips.sh              # prepares 10-min WAV clips
swift run diarization-study --file both       # runs all engines, writes reports

CLI Reference

hybrid-transcribe

hybrid-transcribe <audio> [options]

Options:
  --locale de-DE              ASR language (default: de-DE)
  --embedding-engine mlx      WeSpeaker on GPU (default)
  --embedding-engine coreml   WeSpeaker on Neural Engine
  -o, --output PATH           Write plain-text transcript
  --json PATH                 Write JSON with speaker/start/end/text
  -v, --verbose               Show diarization progress
  -h, --help

diarization-study

diarization-study [options]

Diarization engines:  pyannote-mlx | pyannote-coreml | sortformer
ASR engines:          apple-speech | qwen3-asr | parakeet

Options:
  --file A|B|both             Audio clips to process (default: both)
  --diarizers all|a,b,...     Diarizer IDs (default: all)
  --asr all|a,b,...           ASR engine IDs (default: all)
  --root PATH                 Project root (default: auto-detected)
  -v, --verbose               Show per-engine progress
  -h, --help

Output:
  output/study.csv            Raw metrics
  output/study.md             Markdown report
  output/study.typ            Typst source (compile to PDF with typst 0.15)

# Compile the PDF report
typst compile output/study.typ output/study.pdf

Ultra-Sortformer (Python/NeMo reference)

Ultra-Sortformer (mago-research) is a NeMo-based benchmark extending NVIDIA Sortformer to 8 speakers. Its published results strongly validate the Sortformer approach for German:

Ultra-Sortformer requires Python + NVIDIA NeMo and cannot run on-device. The Hugging Face model (devsy0117/ultra_diar_streaming_sortformer_8spk_v1) is in principle convertible to CoreML for on-device use. Speech-swift's SortformerDiarizer uses a related NVIDIA Sortformer model (CoreML-converted) and showed 100% speaker count accuracy on both test files.

Note on Qwen3-ASR / MLX: The MLX backend failed in this benchmark run with a Metal library error (Failed to load the default metallib). Based on the PersonaPlexDemo benchmarks (M2 Max), Qwen3-ASR achieves ~0.2 s transcription latency and supports 52 languages including German. Results from the Qwen3-ASR engine will be available once the Metal environment is restored.

PersonaPlexDemo

PersonaPlexDemo demonstrates a full-duplex speech-to-speech voice assistant using:

• Qwen3-ASR (same 0.6B model as above) — user speech → text
• PersonaPlex 7B (NVIDIA, Moshi architecture, 4-bit, ~5.5 GB) — generates spoken response

Important: PersonaPlex is English-only — trained on English conversational data only. It is not suitable for German transcription or response. The Qwen3-ASR front-end, however, works for German independently.

For a German-capable voice assistant, the recommended path is: Qwen3-ASR (German input) + Sortformer (diarization) + future German speech-to-speech model.

Why Sortformer over Pyannote?

Pyannote's agglomerative clustering pipeline uses a cosine similarity threshold tuned on English VoxCeleb/CALLHOME data. German conversational speech has different prosodic patterns and overlapping speech segments, causing the default threshold to split single speakers into multiple clusters.

Sortformer uses an end-to-end neural architecture (trained by NVIDIA) that jointly learns segmentation and speaker identity — no clustering threshold to tune. On the German legal recordings tested here it achieved 100% speaker count accuracy vs. 0% for both Pyannote variants.

If you need Pyannote for specific reasons (e.g., partial overlap handling), lower the clustering threshold:

// In MLXDiarizer.swift — tune for German conversational audio
let pipeline = try await DiarizationPipeline.fromPretrained(
    embeddingEngine: .mlx,
    minClusterSize: 0.15   // default ~0.25; lower = fewer false splits
)

Study Report

Full benchmark report (226 KB PDF, Typst 0.15, Oxford Blue / Academic Gold):

• output/study.pdf — Compiled PDF (open directly)
• output/study.typ — Typst source (editable)
• output/study.md — Markdown summary
• output/study.csv — Raw metrics CSV

The report includes: - ASR quality comparison (word count, WER vs Gemini GT, vocabulary recall, RTFx) - Inter-ASR agreement (cross-engine WER) - Diarization metrics (speaker count accuracy, RTFx, false speaker switches per minute) - Combined ASR × Diarizer pipeline table - PersonaPlexDemo architecture and Qwen3-ASR context - Ultra-Sortformer published benchmarks (German DER 6.70%) - Full resources & links section (papers, models, datasets) - Recommendations and resource budget table

# Re-run and regenerate PDF
swift run -c release diarization-study --asr apple-speech,parakeet --diarizers sortformer --file both
typst compile output/study.typ output/study.pdf