OpenParable
Skip to content
← Back to skills

Tts Integration

Development
tts-integration

Tts Integration is an Agent Skill for development published in the rapidaai/voice-ai repository on GitHub: a SKILL.md instruction file an AI agent can load to do this task.

Add or modify text-to-speech providers in assistant-api with transport-aware behavior (WS/SSE/SDK/HTTP), correct packet lifecycle, and UI/provider wiring.

View on GitHub ↗
QUICK START

How to use this skill

Bring this guide into your coding agent with a prompt tailored to the tool you use.

  1. Open your project in Codex.
  2. Copy and paste the prompt below.
  3. Review files and risks before approving any action.
Prompt to paste
Inspect the Agent Skill at https://github.com/rapidaai/voice-ai/blob/HEAD/.claude/skills/tts-integration/SKILL.md and all supporting files. Report provenance, permissions, scripts, dependencies, and security risks. Propose an exact installation into .agents/skills/tts-integration, but wait for my approval before writing files or running scripts.

Copying this prompt does not install or run the skill. Review third-party files before use.

TTS Integration Skill

Mission

Integrate TTS providers with correct streaming/flush/interruption semantics and stable tts_latency_ms metrics.

Inputs expected from user

  1. Transport model: bidirectional WebSocket, SSE/chunked HTTP, SDK callback, or request-response flush.
  2. Voice/model parameter requirements.
  3. Output audio format from provider.

If user does not answer:

  • Select nearest baseline transport and preserve existing packet semantics.
  • Keep internal output format compatibility with existing streamer path.

Hard boundaries

In scope:

  • api/assistant-api/internal/transformer/<provider>/tts.go (+ provider option/normalizer helpers)
  • api/assistant-api/internal/transformer/transformer.go (factory case)
  • optional contract updates: api/assistant-api/internal/type/tts_transformer.go, packet.go
  • provider config JSON + UI component files for TTS

Out of scope:

  • STT-only logic changes unrelated to shared provider setup
  • telephony transport internals
  • EOS/VAD internals

Transport mapping

  • WS streaming baseline: deepgram, rime, sarvam
  • SSE/chunked baseline: minimax
  • SDK/API callback baseline: azure, google
  • HTTP flush baseline: aws, resembleai

Packet contract

Input packets:

  • LLMResponseDeltaPacket
  • LLMResponseDonePacket
  • InterruptionPacket

Required outputs:

  • TextToSpeechAudioPacket
  • TextToSpeechEndPacket
  • ConversationEventPacket{Name:"tts", ...}
  • MessageMetricPacket{Name:"tts_latency_ms"} once per utterance

Implementation workflow

  1. Pick transport baseline and clone lifecycle shape.
  2. Implement provider under transformer/<provider>/.
  3. Add factory registration in transformer.go.
  4. Wire UI metadata (voices, languages, text-to-speech-models) and config form.
  5. Verify interruption behavior clears output state without deadlocks.
  6. Add provider tests (init, streaming, completion, interruption, metric-on-first-audio).

Validation commands

  • go test ./api/assistant-api/internal/transformer/... -run TestTTS
  • go test ./api/assistant-api/internal/transformer/<provider>/...
  • cd ui && yarn test providers
  • ./.claude/skills/tts-integration/scripts/validate.sh --check-diff --provider <provider>