15 — Latency Budget and Tuning

The system targets < 2.5 s end-to-end speech-to-speech latency on the reference RTX 3060 12 GB workstation, with < 1.5 s for OBS subtitles.

Budget Breakdown

Stage	Typical	Driven by
Mic capture chunk	≤ 200 ms	stt.chunk_duration
STT partial	300–800 ms	stt.window_seconds, stt.beam_size, model size
STT confirmed	+ 300–700 ms	stt.confirm_threshold, stt.max_segment_age
LLM translation	100–250 ms	translation.max_tokens, n_ctx, prompt length
TTS TTFA	400–1500 ms	Backend, tts.streaming_mode, batch / chunk size
Audio playback start	~50 ms	audio_output.buffer_size, crossfade_ms

Latencies are sampled live by LatencyTracker and shown in Section A2.

Knobs

STT

• compute_type — int8_float16 is the default. Switching to int8 drops VRAM and is a touch faster on Ampere; float16 is slower but marginally more accurate.
• beam_size — 1 (greedy) is the default. Higher beams add 100–300 ms with minor quality gains.
• window_seconds — shorter windows reduce partial latency but hurt long-segment coherence. 6.5 s is a good trade.
• max_segment_age — was raised from 2.0 s to 4.0 s (commit 53e3cbe) to avoid mid-sentence force-flushes; lower it again if you prefer twitchier confirmations.

Translation

• n_ctx — kept at 512 to keep TTFB low. Increase only if your prompts plus context plus glossary need more headroom.
• max_tokens — capped at 64 to bound the worst case. Translations longer than this are rare and indicate a too-large source batch.
• temperature / top_k — held at 0 / 1 for determinism.
• context_window_size — 2 pairs is enough for pronoun stability without inflating prompt length.
• Backend — switch to api and point at LM Studio if you have a bigger card; the controller switches transparently.

TTS

• streaming_mode — 3 is the SoVITS chunk-streaming mode. Lower values produce one big chunk (higher TTFA, easier on the player).
• speed_factor — slight speed-up (1.2–1.3) cuts perceived latency because audio finishes sooner relative to the next translation.
• tts_queue_max_pending — drop-oldest cap. Kept low (3) so a long speech burst does not back-pressure the queue.
• Two-tier playback — already on by default since Phase 5 (e9d4556), with crossfade defaulting to 50 ms.

AEC

AEC adds negligible latency in the audio path (a single per-chunk NLMS pass), but it removes the need to mute the mic during TTS, which would otherwise add hundreds of milliseconds of silence to every reply. Leave it on unless you are using a hardware-cancelling headset.

Diagnostic Workflow

1. Watch Section A2 metrics during a known-length utterance.
2. If STT partial is high: lower window_seconds or check that compute_type is int8_float16.
3. If translation is high: shorten the system prompt, drop context_window_size, or move to the API backend.
4. If TTS TTFA is high: confirm warm-up succeeded (logs); check streaming_mode; for Qwen3-TTS confirm the synth-latency preset (commit 123b65e).
5. If the audio device under-runs: raise audio_output.buffer_size.