stt
AssemblyAI speech-to-text service implementation.
This module provides integration with AssemblyAI’s real-time speech-to-text WebSocket API for streaming audio transcription.
- pipecat.services.assemblyai.stt.is_u3_pro_model(model: str | None | NotGiven) bool[source]
Return whether a model name is a Universal-3 Pro streaming variant.
Matches the
u3-rt-profamily (u3-rt-pro,u3-rt-pro-beta-1, and anyu3-rt-pro-*variant) and theuniversal-3-5-pro/universal-3-6-proreleases (and anyuniversal-3-5-pro-*/universal-3-6-pro-*variant) so U3 Pro-only features are gated on the whole family rather than a single exact string.- Parameters:
model – The model identifier. Accepts the
Settings.modelunion (str | None | NotGiven); anything that is not a matching string returns False.- Returns:
True if
modelis a U3 Pro family model.
- pipecat.services.assemblyai.stt.map_language_from_assemblyai(language_code: str) Language[source]
Map AssemblyAI language codes to Pipecat Language enum.
AssemblyAI returns simple language codes like “es”, “fr”, etc. This function maps them to the corresponding Language enum values.
- Parameters:
language_code – AssemblyAI language code (e.g., “es”, “fr”, “de”)
- Returns:
Corresponding Language enum value, defaulting to Language.EN if not found.
- pipecat.services.assemblyai.stt.language_to_assemblyai_language(language: Language) str[source]
Convert a Pipecat Language to an AssemblyAI language code.
AssemblyAI declares languages as base ISO codes, so regional variants (
Language.ES_MX) resolve to their base code ("es").- Parameters:
language – The Language enum value to convert.
- Returns:
The AssemblyAI language code.
- class pipecat.services.assemblyai.stt.AssemblyAISTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, formatted_finals: bool | NotGiven = <factory>, word_finalization_max_wait_time: int | None | NotGiven = <factory>, end_of_turn_confidence_threshold: float | None | NotGiven = <factory>, min_turn_silence: int | None | NotGiven = <factory>, max_turn_silence: int | None | NotGiven = <factory>, keyterms_prompt: list[str] | None | NotGiven = <factory>, prompt: str | None | NotGiven = <factory>, language_detection: bool | None | NotGiven = <factory>, language_code: str | None | NotGiven = <factory>, language_codes: list[Language] | None | NotGiven = <factory>, format_turns: bool | NotGiven = <factory>, speaker_labels: bool | None | NotGiven = <factory>, max_speakers: int | None | NotGiven = <factory>, speaker_format: str | None | NotGiven = <factory>, filter_profanity: bool | None | NotGiven = <factory>, vad_threshold: float | None | NotGiven = <factory>, domain: str | None | NotGiven = <factory>, continuous_partials: bool | NotGiven = <factory>, interruption_delay: int | None | NotGiven = <factory>, agent_context: str | None | NotGiven = <factory>, previous_context_n_turns: int | None | NotGiven = <factory>, voice_focus: Literal['near-field', 'far-field'] | None | ~pipecat.utils.types.NotGiven=<factory>, voice_focus_threshold: float | None | NotGiven = <factory>, mode: Literal['min_latency', 'balanced', 'max_accuracy'] | None | ~pipecat.utils.types.NotGiven=<factory>)[source]
Bases:
STTSettingsSettings for AssemblyAISTTService.
- Parameters:
formatted_finals – Whether to enable transcript formatting.
word_finalization_max_wait_time – Maximum time to wait for word finalization in milliseconds.
end_of_turn_confidence_threshold – Confidence threshold for end-of-turn detection.
min_turn_silence – Minimum silence duration when confident about end-of-turn.
max_turn_silence – Maximum silence duration before forcing end-of-turn.
keyterms_prompt – List of key terms to guide transcription.
prompt – Optional text prompt to guide the transcription. Only applicable to U3 Pro models; may be combined with
keyterms_prompton those models.language_detection – Enable automatic language detection.
language_code – Customer-declared audio language as an ISO code (e.g. “en”, “es”, “fr”). On U3 Pro models, a tier-1 code (“en”/”es”/”fr”/”de”/”it”/”pt”) steers transcription toward that language; other supported codes are “tr”, “nl”, “sv”, “no”, “da”, “fi”, “hi”, “vi”, “ar”, “he”, “ja”, “ur”, “zh”, “ru”, “ko”, “ca”, “gl”, “ro”, “et”, “fa”, “yue”, “af”, “mr”, “zu”, “xh”, “nn”. This is one of the names AssemblyAI accepts for its declared-language parameter, alongside
language_codes, which covers the same languages asLanguageenums and is bound in preference to this one when both are set. Preferlanguage_codes. Defaults to None (not sent; no steering).language_codes – Customer-declared audio languages. A single language (e.g.
[Language.ES]) pins transcription to that language; several (e.g.[Language.EN, Language.ES]) steer toward that subset while keeping code-switching among them. Order is significant — the steering prompt follows the declared order. Regional variants resolve to their base code, so at most 10 distinct languages. Steering is prompt-based, so it applies to U3 Pro models only and is not sent for other models — includinguniversal-streaming-multilingual, which transcribes multilingual audio without steering. Unlike most settings, a change applies to a live session without reconnecting; pass an empty list to clear steering back to the model default. Defaults to None (not sent; no steering).format_turns – Whether to format transcript turns.
speaker_labels – Enable speaker diarization. Final transcripts carry AssemblyAI’s speaker label (e.g.
"A","B") asspeaker.label; interim transcripts, and turns whose speaker is not assigned yet, have none. Labels AssemblyAI later revises are not applied to frames already sent.max_speakers – Hard cap on the number of speaker labels, an integer in [1, 10]. Once the cap is reached, further speakers are merged into the closest existing label rather than given a new one, so leave a little headroom above the number of speakers expected; a cap set too high can over-split a single speaker. Only used when
speaker_labelsis enabled. Defaults to None (not sent).speaker_format – Optional format string for diarized transcripts. Use
{speaker}for the speaker label,{text}for the transcript text, and{speaker_number}for the speaker’s order of first appearance in the session (1,2, …), e.g."{speaker}: {text}". If None, transcript text is not modified. Applied locally, so changing it does not reconnect.filter_profanity – Replace profanity in transcribed text with asterisks. Defaults to None (not sent; the server leaves text unfiltered).
vad_threshold – VAD confidence threshold (0.0–1.0) for classifying audio frames as silence. Only applicable to U3 Pro models.
domain – Optional domain for specialized recognition modes. For example, set to “medical-v1” to enable Medical Mode for healthcare transcription.
continuous_partials – Emit partial transcripts at a steady cadence during long turns, rather than only one early partial near the turn start. Only applicable to U3 Pro models; not sent for other models. Defaults to True in this plugin so voice agents receive continuous interim updates.
interruption_delay – Override, in milliseconds (0–1000), for how soon the first partial is emitted. The server adds 256ms (MIN_TURN_DURATION_MS) on top, so 0 → 256ms effective. Only applicable to U3 Pro models. Defaults to None (use the server default).
agent_context – Context carryover seed — the agent’s most recent spoken reply, used to improve transcription of the user’s next turn (short answers, spelled-out entities, disambiguation). Only applicable to U3 Pro models; clipped to ~1500 characters and reset on reconnect. Set this for a known opening line; the service sends each of the agent’s replies as context on its own. To replace it mid-session, push an
STTUpdateSettingsFramewithagent_context. Defaults to None (not sent).previous_context_n_turns – Maximum number of prior conversation entries (user transcripts and any
agent_contextvalues) carried forward as context for each transcription. Integer in [0, 100]; set to 0 to disable automatic context carryover entirely. Only applicable to U3 Pro models. Most integrations should leave this at the default. Defaults to None (use the server default, which is 3).voice_focus – Isolate the primary voice and suppress background noise. Set to “near-field” for close-talking mics (headsets, handsets) or “far-field” for distant capture (conference rooms, laptop mics). Only applicable to U3 Pro models. Defaults to None (not sent).
voice_focus_threshold – How aggressively background audio is suppressed. Float in [0.0, 1.0]; higher values suppress more. Only takes effect when
voice_focusis set. Only applicable to U3 Pro models. Defaults to None (use the server default).mode – Latency/accuracy preset, trading transcription accuracy against turn-finalization latency. One of “min_latency”, “balanced”, or “max_accuracy”. Only applicable to U3 Pro models; the server defaults to “balanced” and rejects this parameter for other models. Defaults to None (not sent).
- class pipecat.services.assemblyai.stt.AssemblyAISTTService(*, api_key: str, language: Language | None = None, api_endpoint_base_url: str = 'wss://streaming.assemblyai.com/v3/ws', sample_rate: int | None = None, encoding: str = 'pcm_s16le', connection_params: AssemblyAIConnectionParams | None = None, vad_force_turn_endpoint: bool = True, should_interrupt: bool = True, speaker_format: str | None = None, settings: AssemblyAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.42, **kwargs)[source]
Bases:
WebsocketSTTServiceAssemblyAI real-time speech-to-text service.
Provides real-time speech transcription using AssemblyAI’s WebSocket API. Supports both interim and final transcriptions with configurable parameters for audio processing and connection management.
Event handlers available (in addition to WebsocketSTTService events):
on_end_of_turn(service, transcript): Called when AssemblyAI detects end of turn.
Example:
@service.event_handler("on_end_of_turn") async def on_end_of_turn(service, transcript): ...
- Settings
alias of
AssemblyAISTTSettings
- __init__(*, api_key: str, language: Language | None = None, api_endpoint_base_url: str = 'wss://streaming.assemblyai.com/v3/ws', sample_rate: int | None = None, encoding: str = 'pcm_s16le', connection_params: AssemblyAIConnectionParams | None = None, vad_force_turn_endpoint: bool = True, should_interrupt: bool = True, speaker_format: str | None = None, settings: AssemblyAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.42, **kwargs)[source]
Initialize the AssemblyAI STT service.
- Parameters:
api_key – AssemblyAI API key for authentication.
language –
Language code for transcription. Defaults to English (Language.EN).
Deprecated since version 0.0.105: Use
settings=AssemblyAISTTService.Settings(language=...)instead. Will be removed in 2.0.0.api_endpoint_base_url – WebSocket endpoint URL. Defaults to AssemblyAI’s streaming endpoint.
sample_rate – Audio sample rate in Hz. If None, uses the input sample rate from the start frame.
encoding – Audio encoding format. Defaults to “pcm_s16le”.
connection_params –
Connection configuration parameters.
Deprecated since version 0.0.105: Use
settings=AssemblyAISTTService.Settings(...)instead. Will be removed in 2.0.0.vad_force_turn_endpoint – Controls turn detection mode. When True (Pipecat mode, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. - min_turn_silence defaults to 100ms (user can override) - max_turn_silence is ALWAYS set equal to min_turn_silence - VAD stop sends ForceEndpoint as ceiling - No UserStarted/StoppedSpeakingFrame emitted from STT When False (AssemblyAI turn detection mode, U3 Pro models only): AssemblyAI’s model controls turn endings using built-in turn detection. - Uses AssemblyAI API defaults for all parameters (unless user explicitly sets them) - Emits UserStarted/StoppedSpeakingFrame from STT - No ForceEndpoint on VAD stop
should_interrupt – Whether to interrupt the bot when the user starts speaking in AssemblyAI turn detection mode (vad_force_turn_endpoint=False). Only applies when using AssemblyAI’s built-in turn detection. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied
user_turn_strategiesoverrides the recommendation and this setting with it. Defaults to True.speaker_format –
Optional format string for speaker labels when diarization is enabled.
Deprecated since version 1.16.0: Use
settings=AssemblyAISTTService.Settings(speaker_format=...)instead. Will be removed in 2.0.0.settings – Runtime-updatable settings. When provided alongside deprecated parameters,
settingsvalues take precedence.ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark
**kwargs – Additional arguments passed to parent STTService class.
- can_generate_metrics() bool[source]
Check if the service can generate metrics.
- Returns:
True if metrics generation is supported.
- service_metadata_frame() STTMetadataFrame[source]
Request external turn strategies in AssemblyAI’s turn-detection mode.
With
vad_force_turn_endpoint=FalseAssemblyAI’s model decides turn endings and emitsProposedUserStarted/StoppedSpeakingFrame, so the user aggregator resolves those rather than running local VAD/smart-turn. In the default Pipecat mode (vad_force_turn_endpoint=True) the STT proposes no turns, so the defaults are left in place. Applied unless the user passed their ownuser_turn_strategies.
- async setup(setup: FrameProcessorSetup)[source]
Set up the service and connect.
- Parameters:
setup – Configuration object containing setup parameters.
- async stop(frame: EndFrame)[source]
Stop the speech-to-text service.
- Parameters:
frame – End frame to stop processing.
- async cancel(frame: CancelFrame)[source]
Cancel the speech-to-text service.
- Parameters:
frame – Cancel frame to abort processing.
- async run_stt(audio: bytes) AsyncGenerator[Frame | None, None][source]
Process audio data for speech-to-text conversion.
- Parameters:
audio – Raw audio bytes to process.
- Yields:
None (processing handled via WebSocket messages).
- async process_frame(frame: Frame, direction: FrameDirection)[source]
Forward a Pipecat-detected turn end to AssemblyAI as a ForceEndpoint.
- Parameters:
frame – Frame to process.
direction – Direction of frame processing.
- async update_agent_context(text: str)[source]
Send the agent’s latest spoken reply to AssemblyAI as carryover context.
Context carryover (U3 Pro models only) gives the model a short memory of what the agent just said so it can better transcribe the user’s reply — short answers, spelled-out entities (emails, IDs), and similar-sounding words. No-op for non-U3 Pro models.
- Parameters:
text – The agent’s spoken reply text. Clipped to ~1500 characters.
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)
Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.
Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.
- Parameters:
frame – The frame to push.
direction – The direction to push the frame.
- class pipecat.services.assemblyai.stt.AssemblyAISyncSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, prompt: str | None | NotGiven = <factory>, keyterms_prompt: list[str] | None | NotGiven = <factory>, conversation_context: str | list[str] | None | NotGiven = <factory>, language_codes: list[Language] | None | NotGiven = <factory>, timestamps: bool | None | NotGiven = <factory>)[source]
Bases:
STTSettingsSettings for
AssemblyAISyncSTTService.- Parameters:
prompt – A natural-language description of what the audio is about: the domain, the scenario, or details of the conversation. Prepended to the model’s system prompt to guide transcription.
keyterms_prompt – Key terms or phrases to bias the decoder toward.
conversation_context – Prior turns from the same conversation, oldest first, giving the model the surrounding dialogue for better continuity and proper-noun consistency across a multi-turn conversation. A single string is treated as one turn. Setting this explicitly turns off the service’s automatic context buffer and sends exactly this value; leave it unset to let the service manage context (see
max_context_turns).language_codes – Declared audio languages for multilingual or code-switching audio (e.g.
[Language.EN, Language.ES]). A single-element list pins one language. Regional variants resolve to their base code and duplicates are dropped, preserving declaration order. Bound in preference to the singlelanguagesetting when both are set. Defaults to unset (the singlelanguageis used).timestamps – Whether to compute per-word
start/endtimestamps, returned on thewordsof the transcription result, at a small added latency. Left unset by default, so the API default ofFalse(no timestamps) applies.
- class pipecat.services.assemblyai.stt.AssemblyAISyncSTTService(*, api_key: str, aiohttp_session: ClientSession, base_url: str = 'https://sync.assemblyai.com', sample_rate: int | None = None, enable_prewarming: bool = True, max_context_turns: int = 5, max_context_chars: int = 1500, settings: AssemblyAISyncSTTSettings | None = None, ttfs_p99_latency: float | None = 0.65, **kwargs)[source]
Bases:
SegmentedSTTServiceAssemblyAI Sync speech-to-text service.
Transcribes a complete speech segment in a single request/response against AssemblyAI’s Sync API, rather than holding open a streaming WebSocket. It inherits from
SegmentedSTTService, which uses VAD events to buffer each utterance and callsrun_stt()once per segment with the audio as a WAV container. Each segment is POSTed asmultipart/form-dataand the transcript is returned directly — no upload step, no polling, no session to manage.Audio segments must be at most 120 seconds; the API rejects longer ones. This suits dictation, scribe, and voice-agent turns where the client detects speech locally and transcribes each turn on demand.
When pre-warming is enabled (the default), the service opens the connection as soon as the user starts speaking, so the transcription request skips the DNS, TCP, and TLS handshake once the utterance ends. Call
warm()directly to warm the connection at any other time.Because the Sync API is stateless, the service also carries recent conversation context automatically: it keeps a rolling buffer of the most recent turns — user transcripts and the agent’s replies together, in the order spoken — and sends them as
conversation_contexton each request, so the model transcribes each turn with the surrounding dialogue. Agent replies are captured from the pipeline’s assistant-turn frame, so this needs no wiring in a standard voice-agent pipeline. Setmax_context_turns=0to disable it, or setconversation_contextexplicitly insettingsto supply the context yourself (that value is honored as-is).- Settings
alias of
AssemblyAISyncSTTSettings
- __init__(*, api_key: str, aiohttp_session: ClientSession, base_url: str = 'https://sync.assemblyai.com', sample_rate: int | None = None, enable_prewarming: bool = True, max_context_turns: int = 5, max_context_chars: int = 1500, settings: AssemblyAISyncSTTSettings | None = None, ttfs_p99_latency: float | None = 0.65, **kwargs)[source]
Initialize the AssemblyAI Sync STT service.
- Parameters:
api_key – AssemblyAI API key for authentication.
aiohttp_session – aiohttp ClientSession for HTTP requests. Pre-warming only helps when the warm and transcribe requests share this session’s connection pool, so keep one session for the service.
base_url – Base URL for the Sync API. Override for a data-residency endpoint (e.g.
https://sync.us.assemblyai.comorhttps://sync.eu.assemblyai.com).sample_rate – Audio sample rate in Hz. If not provided, uses the pipeline’s rate.
enable_prewarming – Whether to open the connection when the user starts speaking so the transcription request avoids the connection handshake. Defaults to True.
max_context_turns – Number of prior conversation turns — user transcripts and agent replies together, in one chronological buffer — automatically carried as
conversation_contexton each request, so the model transcribes each turn with the surrounding dialogue. Set to 0 to disable automatic context. Ignored whenconversation_contextis set explicitly insettings(that value is honored as-is). Defaults to 5.max_context_chars – Character budget for the same buffer; the oldest turns are dropped first once either cap is exceeded. Defaults to 1500.
settings – Runtime-updatable settings.
ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Broadcast at pipeline start for downstream turn timing; set it to your measured value.
**kwargs – Additional arguments passed to SegmentedSTTService.
- can_generate_metrics() bool[source]
Whether this service generates processing metrics.
- Returns:
True. The Sync API issues a discrete request per segment, so its duration is measured and reported.
- language_to_service_language(language: Language) str | None[source]
Convert a Pipecat Language to an AssemblyAI language code.
- Parameters:
language – The language to convert.
- Returns:
The AssemblyAI language code (base ISO code).
- async start(frame: StartFrame)[source]
Start the service on an empty conversation context.
The buffer holds one conversation, so an instance reused for another run starts it fresh.
- async warm()[source]
Pre-warm the connection to the Sync API.
Establishes the connection (DNS, TCP, TLS) ahead of a transcription request so the next
run_stt()starts uploading audio immediately. Best-effort: failures are logged and swallowed, since a failed warm-up only forfeits the latency saving. The warmed connection is reused only by requests that share this service’s aiohttp session and base URL.
- async run_stt(audio: bytes) AsyncGenerator[Frame | None, None][source]
Transcribe an audio segment using the AssemblyAI Sync API.
- Parameters:
audio – Raw audio bytes in WAV format (already converted by the base class).
- Yields:
Frame – A TranscriptionFrame with the transcribed text, or an ErrorFrame on failure. Only non-empty transcriptions are yielded.
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)
Push a frame, marking a TranscriptionFrame finalized when its segment is the latest.
Once the user speaks again, the transcript of an earlier segment covers only the words before they resumed, so it isn’t finalized, the same way
STTServicedrops a pending finalize when the user starts speaking. When a segment’s transcript arrives as several diarized TranscriptionFrames (one per speaker run), only the last one can be.- Parameters:
frame – The frame to push.
direction – The direction of frame flow in the pipeline.