TTS Model Catalog & Optimization

Complete TTS Model Catalog

4A. LLM-Backbone Models with Voice Cloning

These are the current state-of-the-art for quality, expressiveness, and zero-shot voice cloning. All require GPU acceleration for real-time performance.

4B. Models Without Voice Cloning (Preset Voices)

4C. Classical / Formant Engines

4D. Closed-Source Offline-Capable Options

4E. Corrections Log

Seven material errors were identified in our initial research and corrected:

DGX Spark Compatibility Status

Confirmed working on GPU (March 2026)

Confirmed broken on DGX Spark GPU ("Blackwell Noise")

Workarounds for broken models

• CPU fallback: Many models work correctly on the Grace ARM CPU (which is quite capable at 4 GHz). Use device="cpu" and accept slower generation
• Replace torchaudio with soundfile for audio I/O
• Build Flash Attention 2 from source (~15 min with MAX_JOBS=4)
• Use SDPA fallback instead of Flash Attention (VibeVoice works with SDPA and may actually be faster on Blackwell)

Not yet tested but likely compatible

• TADA-1B/3B (raw PyTorch, Llama backbone — should work with cu130 wheels)
• Fun-CosyVoice 3 (PyTorch-based, similar architecture to working Qwen3-TTS)
• Orpheus 3B via llama.cpp GGUF (llama.cpp has excellent DGX Spark support)
• OuteTTS 1.0 via llama.cpp GGUF (same inference path as Orpheus)
• NeuTTS Air/Nano via llama.cpp GGUF (CPU-first architecture)
• Sesame CSM-1B (HuggingFace Transformers, should work with BF16)

Key DGX Spark deployment resources

Optimization Techniques Catalog

6A. Model-Level Optimizations

KV Cache Optimization

PagedAttention (vLLM) splits KV cache into fixed-size blocks, reducing waste from 60–80% to under 4% and enabling 2–4× throughput. SGLang's RadixAttention stores KV tensors in a radix tree — when the same voice prompt is reused, cached computation yields 75–95% cache hit rates. Fish Audio S2 Pro achieves 86.4% average prefix-cache hit rate (>90% at peak) via RadixAttention, making repeated voice cloning nearly free. On DGX Spark, NVFP4 KV cache quantization cuts KV memory by ~50%.

Speculative Decoding

A small draft model proposes token sequences that the target model verifies in parallel. On DGX Spark, EAGLE3 speculative decoding delivers ~2× speedup and is supported by SGLang's official Spark Docker image. Speech-specific variants: VADUSA applies Medusa-style draft heads to TTS; PredGen overlaps speculative text/audio generation with user speech for up to 3× perceived latency reduction.

Quantization

On DGX Spark, quantization directly reduces bytes-per-token through the 273 GB/s bandwidth wall. INT8 achieves 2–3× speedup with <3% MOS drop. INT4 (W4A16) with Marlin kernels outperforms NVFP4 for decode on DGX Spark — FP8→FP4 disappoints because the bottleneck is memory reads, not compute. Quality hierarchy: BF16 > FP16 > FP8 > INT8 >> INT4 (INT4 can cause prosody artifacts).

torch.compile

mode="reduce-overhead" uses CUDA graphs and delivers 30–50% latency reduction. However, on DGX Spark sm_121a, CUDA graph capture via Triton frequently fails — vLLM requires enforce_eager=True. Raw CUDA graphs via SGLang work better (17 ms first-request overhead vs torch.compile's ~4 second warm-up).

Flash Attention

FA2 reduces attention memory from O(n²) to O(n). For Qwen3-TTS specifically, without FlashAttention, inference is 10× slower even on RTX 5090. FA3 is incompatible with Blackwell; use FA2 built from source or SDPA fallback.

Multi-Token Prediction (MTP)

Qwen3-TTS-12Hz predicts all 16 codebook layers simultaneously per frame via hierarchical MTP. Self-distilled MTP achieves >3× faster decoding with <5% accuracy drop. This is a game-changer for multi-codebook architectures.

Continuous Batching

Orca-style iteration-level scheduling lets requests join/leave at each generation step. On DGX Spark, throughput scales nearly linearly: Llama 8B goes from 20.5 tok/s at BS=1 to 368 tok/s at BS=32 via SGLang.

Model Distillation

Chatterbox Turbo's 10→1 step diffusion distillation eliminates the decoder bottleneck. Fast F5-TTS reduces diffusion steps from 32 to 7, achieving RTF 0.030 (4× speedup).

6B. Pipeline and Streaming Optimizations

Sentence-Level Chunking

Split LLM output at punctuation; submit first sentence to TTS while LLM generates the rest. Deepgram explicitly warns against splitting mid-sentence (produces choppy prosody).

Streaming Text Input

True streaming TTS processes text character-by-character and starts synthesizing 6.5× faster than batch mode. Supported by: Qwen3-TTS, CosyVoice 2/3, VibeVoice-Realtime, Dia2.

Pipeline Stage Overlap

Start TTS once first few LLM tokens are available. Pipecat automates this with sentence-boundary aggregation. The Cresta production approach uses hedging (multiple parallel LLM calls, use whichever returns first).

Voice Prompt Caching

Pre-encode reference audio and reuse across requests. Implementation: SGLang RadixAttention (automatic for S2 Pro), CosyVoice LRU timbre manager, manual dict cache for TADA/others.

Warm-Up Passes

Non-negotiable for production. torch.compile requires ~3 min warm-up; CUDA graphs need sub-second. Pre-warm vocoder/codec separately — often the cold-start bottleneck.

6C. Audio Codec Frame Rate Impact

The single most impactful architectural choice for TTS speed:

6D. Serving Infrastructure Comparison

6E. Conversational "Feel" Optimizations

Filler Word Injection

Reduces perceived latency by 50–70% (from 1.5–2s to 0.5–0.7s). Instruct LLM to start responses with "umm," "so," or "well." Send tiny filler chunk to TTS instantly, stream the rest behind it.

Turn-Taking Prediction

Beyond VAD silence detection. NVIDIA ACE Agent uses two-pass endpointing: send interim transcript at 240 ms pause, final at 800 ms — saving 500–600 ms overall.

Barge-in Support

Stop TTS playback within 200 ms when user interrupts. Pipecat uses Silero VAD + SmartTurn, flushing downstream buffers instantly.

Backchanneling

Filter short utterances ("yeah," "okay") from triggering full responses. NVIDIA ACE Agent filters words under 3 characters.

6F. DGX Spark Specific Optimizations

Model Optimization Compatibility Matrix

Serving Framework Support

Quantization and Optimization Availability

Voice Pipeline Architecture

STT → LLM → TTS Pipeline

The standard real-time voice agent pipeline has three stages, with the LLM consistently dominating total latency:

User speaks → [VAD] → STT (~50ms) → LLM (~300-700ms) → TTS (~100-600ms) → Audio playback
                                      ↑ DOMINANT        ↑ YOU ARE HERE

Latency budget (conversational threshold: <500ms perceived):

Pipecat Integration

Pipecat is the leading open-source Python framework for real-time voice AI, supporting 25+ TTS backends. Key integration patterns:

• TextAggregationMode.SENTENCE (default): Natural prosody, sends complete sentences to TTS
• TextAggregationMode.TOKEN: Lowest latency, may produce unnatural prosody
• Interruption handling: Silero VAD + SmartTurn → flush all downstream buffers
• OpenAI-compatible servers bridge any local TTS to Pipecat via OpenAITTSService(base_url=...)

Community TTS API servers for Pipecat integration:

DGX Spark Memory Budget for Full Pipeline

Configuration Reference

Model-Specific Deployment Settings

TADA-1B:

# Ultra-low frame rate is the optimization itself
model = TadaForCausalLM.from_pretrained("HumeAI/tada-1b").to("cuda").to(torch.bfloat16)
# Flow matching steps configurable for quality/speed tradeoff
# ~700 seconds continuous audio with 2048 token context

Qwen3-TTS-12Hz (0.6B/1.7B):

# BF16 is MANDATORY — FP16 causes NaN/overflow in MTP sampling
# vLLM-Omni on DGX Spark:
vllm serve Qwen/Qwen3-TTS-12Hz-0.6B-Base --enforce-eager --trust-remote-code \
  --gpu-memory-utilization 0.9 --dtype bfloat16
# Reference audio: 10-15s optimal; append 0.5s silence to end
# 3s minimum for cloning

Fun-CosyVoice 3:

# LightTTS framework for optimized serving:
# github.com/ModelTC/LightTTS
# TRT-LLM gives 4× speedup on LLM component
# Shared memory timbre manager with LRU for repeated speakers

Fish S2 Pro (via SGLang):

# Build sgl-kernel for Spark:
TORCH_CUDA_ARCH_LIST="12.1a" MAX_JOBS=4 CMAKE_BUILD_PARALLEL_LEVEL=1 \
  python -m build --wheel --no-isolation
# CRITICAL: BF16 RoPE truncation patch required
# --mem-fraction-static 0.15 (keeps allocation small for shared memory)
# --max-running-requests 2 (bandwidth-limited on Spark)

Orpheus 3B (via llama.cpp):

# Build for DGX Spark:
cmake -B build -DGGML_CUDA=ON -DGGML_CUDA_F16=ON -DCMAKE_CUDA_ARCHITECTURES=121
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
# Serving:
./llama-server -m orpheus-3b-q4_k_m.gguf -c 8192 -ngl 99 \
  --cache-type-k q8_0 --cache-type-v q8_0 -fa --mlock
# repetition_penalty >= 1.1 REQUIRED for stable output
# Stream every 7 SNAC tokens (= 1 audio frame)

Chatterbox Turbo:

model = ChatterboxTurboTTS.from_pretrained(device="cuda")
# cfg_weight ~0.3 for slower, more natural rhythm
# exaggeration 0.5 default; higher = more expressive + faster speaking
# ONNX export available for cross-platform

OuteTTS 1.0 (via llama.cpp):

# GGUF Q4-Q8 available
# Windowed repetition penalty (64-token window) is CRITICAL
# Full context penalty breaks output
# llama.cpp llama-tts binary (PR #12794 pending for 1.0)

Essential Environment Variables

# DGX Spark general
export PATH=/usr/local/cuda/bin:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
export CUDA_HOME=/usr/local/cuda
export TORCH_CUDA_ARCH_LIST="12.1a"
 
# PyTorch optimization
export PYTORCH_CUDA_ALLOC_CONF="expandable_segments:True"
 
# llama.cpp unified memory
export GGML_CUDA_ENABLE_UNIFIED_MEMORY=1
 
# Disable GUI for headless serving
sudo systemctl set-default multi-user.target

# PyTorch inference optimization (all models)
torch.set_grad_enabled(False)
torch.set_float32_matmul_precision('high')  # TF32 on Blackwell
torch.backends.cudnn.benchmark = True  # for vocoder conv layers
torch.backends.cuda.matmul.allow_tf32 = True

Canonical TTSConfig Schema

This portable, offline-first configuration schema can drive multiple engines. It separates static capabilities from request-time controls and deployment settings:

{
  "engine": "qwen3_tts | cosyvoice3 | orpheus | tada | chatterbox | fish_s2 | ...",
  "voice": {
    "locale": "en-US",
    "voice_id": "engine-specific voice name or voicepack ID",
    "model_id": "HuggingFace model hub identifier",
    "model_path": "/path/for/air-gapped/deployment",
    "speaker_ref_wav": "/path/to/reference.wav",
    "speaker_index": 0
  },
  "audio": {
    "format": "wav | raw_pcm_s16le | flac | opus",
    "sample_rate_hz": 24000,
    "channels": 1,
    "normalize": true,
    "volume_gain": 1.0,
    "postfx": { "lufs_target": -18, "limiter": true, "resample_quality": "high" }
  },
  "controls": {
    "rate": 1.0,
    "pitch": 0.0,
    "temperature": 0.7,
    "noise_scale": 0.667,
    "style": "warm and conversational",
    "seed": 42
  },
  "segmentation": {
    "mode": "sentence",
    "max_chars": 500,
    "split_pattern": "\\n+",
    "pause_ms_between_segments": 200
  },
  "execution": {
    "device": "cuda",
    "precision": "bf16",
    "threads": 4,
    "streaming": {
      "enabled": true,
      "chunk_ms": 40,
      "ttfa_budget_ms": 300,
      "buffer_strategy": "low_latency"
    },
    "cache": {
      "model_cache_dir": "/models",
      "warm_start": true,
      "max_models_loaded": 3
    }
  }
}

Validation and Testing

Metrics to measure

Test suite structure

Unit tests: Text normalization determinism, phoneme override verification, SSML parsing for supported engines.

Integration tests: Golden-audio spectral fingerprinting (not raw waveform — lossy variability exists), cross-engine output duration bounds, architecture smoke tests (x86_64, aarch64).

Perceptual tests: MOS listening panels, ABX preference tests, accessibility evaluation at high speech rates.

Benchmark harness

Capture cold-start vs warm-start TTFA, steady-state RTF at 1/2/4/8 concurrent requests, CPU thread scaling, quantization impact (BF16 vs INT8 vs INT4), and output sample rate normalization. sherpa-onnx publishes per-model RTF tables including Raspberry Pi measurements, providing a useful cross-platform reference.

References and Links

Model repositories

Classical engines

Frameworks and tools

DGX Spark resources

Key papers