stt

AssemblyAI speech-to-text service implementation.

This module provides integration with AssemblyAI’s real-time speech-to-text WebSocket API for streaming audio transcription.

pipecat.services.assemblyai.stt.is_u3_pro_model(model: str | None | _NotGiven) → bool[source]

Return whether a model name is a Universal-3 Pro streaming variant.

Matches the u3-rt-pro family (u3-rt-pro, u3-rt-pro-beta-1, and any u3-rt-pro-* variant) and universal-3-5-pro (and any universal-3-5-pro-* variant) so U3 Pro-only features are gated on the whole family rather than a single exact string.

Parameters:: model – The model identifier. Accepts the Settings.model union (str | None | _NotGiven); anything that is not a matching string returns False.
Returns:: True if model is a U3 Pro family model.

pipecat.services.assemblyai.stt.map_language_from_assemblyai(language_code: str) → Language[source]

Map AssemblyAI language codes to Pipecat Language enum.

AssemblyAI returns simple language codes like “es”, “fr”, etc. This function maps them to the corresponding Language enum values.

Parameters:: language_code – AssemblyAI language code (e.g., “es”, “fr”, “de”)
Returns:: Corresponding Language enum value, defaulting to Language.EN if not found.

Bases: STTSettings

Settings for AssemblyAISTTService.

Parameters:

formatted_finals – Whether to enable transcript formatting.
word_finalization_max_wait_time – Maximum time to wait for word finalization in milliseconds.
end_of_turn_confidence_threshold – Confidence threshold for end-of-turn detection.
min_turn_silence – Minimum silence duration when confident about end-of-turn.
max_turn_silence – Maximum silence duration before forcing end-of-turn.
keyterms_prompt – List of key terms to guide transcription.
prompt – Optional text prompt to guide the transcription. Only used when model is “u3-rt-pro”.
language_detection – Enable automatic language detection.
language_code – Customer-declared audio language as an ISO code (e.g. “en”, “es”, “fr”). On U3 Pro models, a tier-1 code (“en”/”es”/”fr”/”de”/”it”/”pt”) steers transcription toward that language; other supported codes are “de”, “tr”, “nl”, “sv”, “no”, “da”, “fi”, “hi”, “vi”, “ar”, “he”, “ja”, “ur”, “zh”. Mutually exclusive with language_detection. Defaults to None (not sent; no steering).
format_turns – Whether to format transcript turns.
speaker_labels – Enable speaker diarization.
vad_threshold – VAD confidence threshold (0.0–1.0) for classifying audio frames as silence. Only applicable to u3-rt-pro.
domain – Optional domain for specialized recognition modes. For example, set to “medical-v1” to enable Medical Mode for healthcare transcription.
continuous_partials – Emit partial transcripts at a steady cadence during long turns, rather than only one early partial near the turn start. Only applicable to u3-rt-pro; not sent for other models. Defaults to True in this plugin so voice agents receive continuous interim updates.
interruption_delay – Override, in milliseconds (0–1000), for how soon the first partial is emitted. The server adds 256ms (MIN_TURN_DURATION_MS) on top, so 0 → 256ms effective. Only applicable to u3-rt-pro. Defaults to None (use the server default).
agent_context – Context carryover seed — the agent’s most recent spoken reply, used to improve transcription of the user’s next turn (short answers, spelled-out entities, disambiguation). Only applicable to u3-rt-pro; clipped to ~1500 characters and reset on reconnect. Set this for a known opening line, or call AssemblyAISTTService.update_agent_context() to update it mid-session. Defaults to None (not sent).
previous_context_n_turns – Maximum number of prior conversation entries (user transcripts and any agent_context values) carried forward as context for each transcription. Integer in [0, 100]; set to 0 to disable automatic context carryover entirely. Only applicable to u3-rt-pro. Most integrations should leave this at the default. Defaults to None (use the server default, which is 3).
voice_focus – Isolate the primary voice and suppress background noise. Set to “near-field” for close-talking mics (headsets, handsets) or “far-field” for distant capture (conference rooms, laptop mics). Only applicable to U3 Pro models. Defaults to None (not sent).
voice_focus_threshold – How aggressively background audio is suppressed. Float in [0.0, 1.0]; higher values suppress more. Only takes effect when voice_focus is set. Only applicable to U3 Pro models. Defaults to None (use the server default).
mode – Latency/accuracy preset, trading transcription accuracy against turn-finalization latency. One of “min_latency”, “balanced”, or “max_accuracy”. Only applicable to U3 Pro models; the server defaults to “balanced” and rejects this parameter for other models. Defaults to None (not sent).

formatted_finals: bool | _NotGiven

word_finalization_max_wait_time: int | None | _NotGiven

end_of_turn_confidence_threshold: float | None | _NotGiven

min_turn_silence: int | None | _NotGiven

max_turn_silence: int | None | _NotGiven

keyterms_prompt: list[str] | None | _NotGiven

prompt: str | None | _NotGiven

language_detection: bool | None | _NotGiven

language_code: str | None | _NotGiven

format_turns: bool | _NotGiven

speaker_labels: bool | None | _NotGiven

vad_threshold: float | None | _NotGiven

domain: str | None | _NotGiven

continuous_partials: bool | _NotGiven

interruption_delay: int | None | _NotGiven

agent_context: str | None | _NotGiven

previous_context_n_turns: int | None | _NotGiven

voice_focus: Literal['near-field', 'far-field'] | None | _NotGiven

voice_focus_threshold: float | None | _NotGiven

mode: Literal['min_latency', 'balanced', 'max_accuracy'] | None | _NotGiven

class pipecat.services.assemblyai.stt.AssemblyAISTTService(*, api_key: str, language: Language | None = None, api_endpoint_base_url: str = 'wss://streaming.assemblyai.com/v3/ws', sample_rate: int = 16000, encoding: str = 'pcm_s16le', connection_params: AssemblyAIConnectionParams | None = None, vad_force_turn_endpoint: bool = True, should_interrupt: bool = True, speaker_format: str | None = None, settings: AssemblyAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.42, **kwargs)[source]

Bases: WebsocketSTTService

AssemblyAI real-time speech-to-text service.

Provides real-time speech transcription using AssemblyAI’s WebSocket API. Supports both interim and final transcriptions with configurable parameters for audio processing and connection management.

Event handlers available (in addition to WebsocketSTTService events):

on_end_of_turn(service, transcript): Called when AssemblyAI detects end of turn.

Example:

@service.event_handler("on_end_of_turn")
async def on_end_of_turn(service, transcript):
    ...

Settings: alias of AssemblyAISTTSettings

__init__(*, api_key: str, language: Language | None = None, api_endpoint_base_url: str = 'wss://streaming.assemblyai.com/v3/ws', sample_rate: int = 16000, encoding: str = 'pcm_s16le', connection_params: AssemblyAIConnectionParams | None = None, vad_force_turn_endpoint: bool = True, should_interrupt: bool = True, speaker_format: str | None = None, settings: AssemblyAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.42, **kwargs)[source]

Initialize the AssemblyAI STT service.

Parameters:

api_key – AssemblyAI API key for authentication.
language –
Language code for transcription. Defaults to English (Language.EN).

Deprecated since version 0.0.105: Use settings=AssemblyAISTTService.Settings(language=...) instead. Will be removed in 2.0.0.
api_endpoint_base_url – WebSocket endpoint URL. Defaults to AssemblyAI’s streaming endpoint.
sample_rate – Audio sample rate in Hz. Defaults to 16000.
encoding – Audio encoding format. Defaults to “pcm_s16le”.
connection_params –
Connection configuration parameters.

Deprecated since version 0.0.105: Use settings=AssemblyAISTTService.Settings(...) instead. Will be removed in 2.0.0.
vad_force_turn_endpoint – Controls turn detection mode. When True (Pipecat mode, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. - min_turn_silence defaults to 100ms (user can override) - max_turn_silence is ALWAYS set equal to min_turn_silence - VAD stop sends ForceEndpoint as ceiling - No UserStarted/StoppedSpeakingFrame emitted from STT When False (AssemblyAI turn detection mode, u3-rt-pro only): AssemblyAI’s model controls turn endings using built-in turn detection. - Uses AssemblyAI API defaults for all parameters (unless user explicitly sets them) - Emits UserStarted/StoppedSpeakingFrame from STT - No ForceEndpoint on VAD stop
should_interrupt – Whether to interrupt the bot when the user starts speaking in AssemblyAI turn detection mode (vad_force_turn_endpoint=False). Only applies when using AssemblyAI’s built-in turn detection. Defaults to True.
speaker_format – Optional format string for speaker labels when diarization is enabled. Use {speaker} for speaker label and {text} for transcript text. Example: “<{speaker}>{text}</{speaker}>” or “{speaker}: {text}” If None, transcript text is not modified. Defaults to None.
settings – Runtime-updatable settings. When provided alongside deprecated parameters, settings values take precedence.
ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark
**kwargs – Additional arguments passed to parent STTService class.

can_generate_metrics() → bool[source]

Check if the service can generate metrics.

Returns:: True if metrics generation is supported.

service_metadata_frame() → STTMetadataFrame[source]

Request external turn strategies in AssemblyAI’s turn-detection mode.

With vad_force_turn_endpoint=False AssemblyAI’s model decides turn endings and emits UserStarted/StoppedSpeakingFrame, so the user aggregator defers to those rather than running local VAD/smart-turn. In the default Pipecat mode (vad_force_turn_endpoint=True) the STT emits no turn frames, so the defaults are left in place. Applied unless the user passed their own user_turn_strategies.

async start(frame: StartFrame)[source]

Start the speech-to-text service.

Parameters:: frame – Start frame to begin processing.

async stop(frame: EndFrame)[source]

Stop the speech-to-text service.

Parameters:: frame – End frame to stop processing.

async cancel(frame: CancelFrame)[source]

Cancel the speech-to-text service.

Parameters:: frame – Cancel frame to abort processing.

async run_stt(audio: bytes) → AsyncGenerator[Frame | None, None][source]

Process audio data for speech-to-text conversion.

Parameters:: audio – Raw audio bytes to process.
Yields:: None (processing handled via WebSocket messages).

async process_frame(frame: Frame, direction: FrameDirection)[source]

Process frames for VAD and metrics handling.

Parameters:

frame – Frame to process.
direction – Direction of frame processing.

async update_agent_context(text: str)[source]

Send the agent’s latest spoken reply to AssemblyAI as carryover context.

Context carryover (u3-rt-pro only) gives the model a short memory of what the agent just said so it can better transcribe the user’s reply — short answers, spelled-out entities (emails, IDs), and similar-sounding words. No-op for non-u3-rt-pro models.

Parameters:: text – The agent’s spoken reply text. Clipped to ~1500 characters.

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.

Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.

Parameters:

frame – The frame to push.
direction – The direction to push the frame.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:: end_time – Optional timestamp to use as the end time. If None, uses the current time.