Speaches STT

WebUI speech-to-text provider for Agent Zero backed by an external OpenAI-compatible speaches server (faster-whisper / CTranslate2). One shared STT server can serve Agent Zero and other clients (e.g. Open WebUI) at the same time.

What it does

• Injects a microphone button into the chat input with the same interaction model as the built-in Whisper STT: RMS silence detection, a listening/recording/waiting/processing state machine, and send/draft message modes.
• Records browser audio via MediaRecorder and sends it to the configured speaches server (POST /v1/audio/transcriptions) using the recorder's real MIME type (typically audio/webm).
• Registers itself as the WebUI sttService provider; no local model is loaded, so the plugin adds no weight to the Agent Zero container.
• The status API redacts the API key; the settings modal is the only surface that reads the stored credential.

Configuration

Managed from the plugin settings UI (Agent tab):

Key	Default	Description
server	""	speaches host:port over http, no protocol prefix (e.g. localhost:9000).
api_key	""	Bearer token for the server; empty = no auth header.
model	deepdml/faster-whisper-large-v3-turbo-ct2	Model ID as configured on the server (see its /v1/registry).
language	auto	ISO 639-1 hint or auto for server-side detection.
message_mode	send	send transcriptions immediately, draft keeps them in the composer.
silence_threshold	0.3	Signal level before listening switches to recording.
silence_duration	1000	Milliseconds of silence before the waiting phase.
waiting_timeout	2000	Milliseconds to wait after silence before transcription.
timeout	60	Seconds per transcription request (5-300).

While a spoken answer is playing, the microphone never starts a new recording: the frontend silence gate blocks the listening -> recording transition while ttsService.isSpeaking() is true (echo protection).

The config option stop_mic_during_tts (default true) controls what happens to an active microphone session when a spoken answer starts:

• true: the session is stopped and the microphone button returns to standby (one click to listen again).
• false: the session stays active and keeps listening; a new recording starts right after playback ends.

With false, a recording that is already running is not cut when TTS starts, so loudspeaker output can end up in the transcript. That is the deliberate trade-off of keeping the session alive. Changes apply after the next status refresh; every microphone click forces one.

Provider conflict

The WebUI sttService uses the first registered STT provider. Disable the built-in _whisper_stt plugin when enabling this one - otherwise the active provider depends on plugin load order.

Known limitation: the browser annotation voice input (_browser) is hard-wired to the built-in Whisper store in framework code and does not work through this plugin.

Server setup (example)

services:
  speaches:
    image: ghcr.io/speaches-ai/speaches:latest-cuda-12.6.3
    container_name: speaches-stt
    restart: unless-stopped
    ports:
      - "9000:8000"
    environment:
      - PRELOAD_MODELS=["deepdml/faster-whisper-large-v3-turbo-ct2"]
      - WHISPER__INFERENCE_DEVICE=cuda
      - WHISPER__COMPUTE_TYPE=int8_float16
      - STT_MODEL_TTL=-1
      - API_KEY=change-me

Verify the model ID against the running server's /v1/registry; the OpenAI-style alias whisper-1 resolves to the full large-v3 model, not the turbo variant.

Testing

# Offline checks (manifest, handlers, units, status redaction, errors):
cd /a0/usr/projects/agent_zero_-_plugin_forge && \
  /opt/venv-a0/bin/python plugins/speaches_stt/tests/e2e_predeploy.py

# Optional live checks (from the same directory):
SPEACHES_STT_TEST_SERVER=localhost:9000 \
SPEACHES_STT_TEST_API_KEY=change-me \
SPEACHES_STT_TEST_TTS_SERVER=localhost:8123 \
/opt/venv-a0/bin/python plugins/speaches_stt/tests/e2e_predeploy.py

Deployment

./deploy.sh validate && ./deploy.sh deploy

Deployed Python changes go live without a framework restart (the deploy touches hooks.py and fires the plugin watchdog). Hard-reload the browser for WebUI JS changes.

License

MIT (SPDX: MIT) - see LICENSE.