Offline STT Guide

A comprehensive reference for deploying speech-to-text locally — covering Whisper variants, NVIDIA NeMo models, Moonshine, and edge STT options with DGX Spark compatibility notes.

Overview

The STT (speech-to-text) stage of a voice pipeline converts spoken audio into text for the LLM to process. While TTS quality gets more attention, STT latency and accuracy directly gate the entire pipeline — you can't respond to what you haven't transcribed.

The 2026 offline STT landscape is dominated by three families:

• Whisper ecosystem (OpenAI): The default starting point. 9 model sizes, 100 languages, MIT license, massive community tooling
• NVIDIA NeMo models (Parakeet, Canary, Nemotron Speech): State-of-the-art accuracy and speed, open-weight, native DGX Spark support
• Edge-optimized models (Moonshine, SenseVoice, Vosk): Purpose-built for low-power and embedded deployment

The Whisper Family

OpenAI's Whisper (MIT license) spans nine model sizes from 39M to 1.55B parameters, supporting 100 languages.

Model Sizes

large-v3 (Nov 2023) added 128 Mel filter banks (vs 80 in v2), improving accuracy 10–20%. large-v3-turbo (Oct 2024) shrunk the decoder from 32 to 4 layers, delivering 8× speedup at near-identical accuracy. MLCommons selected Whisper Large-v3 as the MLPerf Inference ASR benchmark in September 2025.

Distil-Whisper

HuggingFace's distilled Whisper variants compress the decoder to 2 layers:

distil-large-v3.5 (trained on 98K hours) is the current best distilled option. All distil variants are English-only. For multilingual, use large-v3-turbo directly.

Whisper Inference Engines

whisper.cpp

The CPU-first C/C++ Whisper implementation by ggml-org. MIT license, v1.8.3 (January 2026).

Key features:

• GGML quantization (Q4_0 through Q8_0) — up to 65% disk reduction
• Vulkan iGPU support (12× speedup on AMD/Intel integrated graphics)
• Native aarch64 build — confirmed working on DGX Spark Grace CPU
• Streaming via whisper-stream (0.5s sampling intervals)
• No Python dependency

git clone https://github.com/ggml-org/whisper.cpp.git
cd whisper.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=121
cmake --build build -j4
./build/bin/whisper-cli -m models/ggml-large-v3-turbo-q8_0.bin -f audio.wav

faster-whisper

SYSTRAN's CTranslate2-based implementation. MIT license. 4× faster than original Whisper with identical accuracy, plus int8 quantization halving VRAM usage.

Benchmarks (RTX 3070 Ti, large-v3):

• Original Whisper: 2m23s
• faster-whisper FP16: 52s
• faster-whisper int8: ~45s
• distil-large-v3 int8: 22.5s

DGX Spark: Confirmed working via NVIDIA's official Arm Learning Paths tutorial (CPU int8 mode). Community Docker images available for CUDA 13 on Blackwell.

pip install faster-whisper

# CPU inference (confirmed on DGX Spark):
from faster_whisper import WhisperModel
model = WhisperModel("large-v3-turbo", device="cpu", compute_type="int8")
segments, info = model.transcribe("audio.wav", beam_size=5)
for segment in segments:
    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")

NVIDIA NeMo Speech Models

All models below are open-weight under CC-BY-4.0 (exception noted) and available on HuggingFace and NGC.

Parakeet-TDT-0.6B

Parakeet-TDT uses a FastConformer encoder with Token-and-Duration Transducer decoder. The v3 variant adds automatic language detection across 25 European languages and supports up to 3 hours of audio via local attention.

Canary Flash

Canary models combine ASR with speech translation. The 180M Flash variant is small enough for edge deployment. Note: the original Canary-1B used CC-BY-NC-4.0, but all Flash and v2 variants use permissive CC-BY-4.0.

Nemotron Speech Streaming 0.6B

Released January 2026 under NVIDIA Open Model License. Purpose-built for low-latency voice agent STT with cache-aware FastConformer-RNNT architecture.

Key specs:

• Each audio frame encoded exactly once with cached context
• Configurable chunk sizes: 80ms to 1,120ms
• Median time-to-final-transcription: ~24ms
• Handles 560 concurrent streams on H100
• Confirmed on DGX Spark (demonstrated in pipecat-ai project)

Edge and Specialized STT

SenseVoice Small (Alibaba FunAudioLLM)

234M parameters. Unique for combining ASR + speech emotion recognition + audio event detection (applause, laughter, coughing) in a single model.

• Languages: Chinese, English, Cantonese, Japanese, Korean
• Speed: 10 seconds of audio in 70ms on A800 (5× faster than Whisper-Small)
• Chinese/Cantonese accuracy: >50% better than Whisper
• ONNX export supported, integrated into sherpa-onnx
• License: Apache 2.0 (model), MIT (code)

Moonshine (Moonshine AI)

The standout edge STT model. MIT license.

Moonshine v2 (February 2026) introduced sliding-window self-attention for bounded-latency streaming. All models ship in ONNX format, run on Raspberry Pi and mobile, and can operate in as little as 8 MB RAM.

Language-specialized tiny models available for: Arabic, Chinese, Japanese, Korean, Ukrainian, Vietnamese.

Vosk (Alpha Cephei)

Apache 2.0 license. Ultra-lightweight with 50 MB portable models across 20+ languages and true zero-latency streaming. Accuracy (~10–15% WER on general English) is far behind Whisper-class models but excellent for command-and-control applications.

sherpa-onnx STT

Not a model but a cross-platform runtime (Apache 2.0, v1.12.29, March 2026) supporting 12+ STT architectures: Zipformer, Paraformer, Whisper, Moonshine, SenseVoice, Parakeet CTC, Canary Flash, FunASR-Nano. Runs on x86, ARM, RISC-V, Rockchip NPU, Ascend NPU, Jetson with 12 programming language bindings.

Mozilla DeepSpeech — Discontinued

STT Model Comparison Table

DGX Spark Compatibility

Choosing an STT Model

References