stt

AssemblyAI speech-to-text service implementation.

This module provides integration with AssemblyAI’s real-time speech-to-text WebSocket API for streaming audio transcription.

pipecat.services.assemblyai.stt.is_u3_pro_model(model: str | None | NotGiven) → bool[source]

Return whether a model name is a Universal-3 Pro streaming variant.

Matches the u3-rt-pro family (u3-rt-pro, u3-rt-pro-beta-1, and any u3-rt-pro-* variant) and the universal-3-5-pro / universal-3-6-pro releases (and any universal-3-5-pro-* / universal-3-6-pro-* variant) so U3 Pro-only features are gated on the whole family rather than a single exact string.

Parameters:

model – The model identifier. Accepts the Settings.model union (str | None | NotGiven); anything that is not a matching string returns False.

Returns:

True if model is a U3 Pro family model.

pipecat.services.assemblyai.stt.map_language_from_assemblyai(language_code: str) → Language[source]

Map AssemblyAI language codes to Pipecat Language enum.

AssemblyAI returns simple language codes like “es”, “fr”, etc. This function maps them to the corresponding Language enum values.

Parameters:

language_code – AssemblyAI language code (e.g., “es”, “fr”, “de”)

Returns:

Corresponding Language enum value, defaulting to Language.EN if not found.

pipecat.services.assemblyai.stt.language_to_assemblyai_language(language: Language) → str[source]

Convert a Pipecat Language to an AssemblyAI language code.

AssemblyAI declares languages as base ISO codes, so regional variants (Language.ES_MX) resolve to their base code ("es").

Parameters:

language – The Language enum value to convert.

Returns:

The AssemblyAI language code.

class pipecat.services.assemblyai.stt.AssemblyAISTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, formatted_finals: bool | NotGiven = <factory>, word_finalization_max_wait_time: int | None | NotGiven = <factory>, end_of_turn_confidence_threshold: float | None | NotGiven = <factory>, min_turn_silence: int | None | NotGiven = <factory>, max_turn_silence: int | None | NotGiven = <factory>, keyterms_prompt: list[str] | None | NotGiven = <factory>, prompt: str | None | NotGiven = <factory>, language_detection: bool | None | NotGiven = <factory>, language_code: str | None | NotGiven = <factory>, language_codes: list[Language] | None | NotGiven = <factory>, format_turns: bool | NotGiven = <factory>, speaker_labels: bool | None | NotGiven = <factory>, max_speakers: int | None | NotGiven = <factory>, speaker_format: str | None | NotGiven = <factory>, filter_profanity: bool | None | NotGiven = <factory>, vad_threshold: float | None | NotGiven = <factory>, domain: str | None | NotGiven = <factory>, continuous_partials: bool | NotGiven = <factory>, interruption_delay: int | None | NotGiven = <factory>, agent_context: str | None | NotGiven = <factory>, previous_context_n_turns: int | None | NotGiven = <factory>, voice_focus: Literal['near-field', 'far-field'] | None | ~pipecat.utils.types.NotGiven=<factory>, voice_focus_threshold: float | None | NotGiven = <factory>, mode: Literal['min_latency', 'balanced', 'max_accuracy'] | None | ~pipecat.utils.types.NotGiven=<factory>)[source]

Bases: STTSettings

Settings for AssemblyAISTTService.

Parameters:
  • formatted_finals – Whether to enable transcript formatting.

  • word_finalization_max_wait_time – Maximum time to wait for word finalization in milliseconds.

  • end_of_turn_confidence_threshold – Confidence threshold for end-of-turn detection.

  • min_turn_silence – Minimum silence duration when confident about end-of-turn.

  • max_turn_silence – Maximum silence duration before forcing end-of-turn.

  • keyterms_prompt – List of key terms to guide transcription.

  • prompt – Optional text prompt to guide the transcription. Only applicable to U3 Pro models; may be combined with keyterms_prompt on those models.

  • language_detection – Enable automatic language detection.

  • language_code – Customer-declared audio language as an ISO code (e.g. “en”, “es”, “fr”). On U3 Pro models, a tier-1 code (“en”/”es”/”fr”/”de”/”it”/”pt”) steers transcription toward that language; other supported codes are “tr”, “nl”, “sv”, “no”, “da”, “fi”, “hi”, “vi”, “ar”, “he”, “ja”, “ur”, “zh”, “ru”, “ko”, “ca”, “gl”, “ro”, “et”, “fa”, “yue”, “af”, “mr”, “zu”, “xh”, “nn”. This is one of the names AssemblyAI accepts for its declared-language parameter, alongside language_codes, which covers the same languages as Language enums and is bound in preference to this one when both are set. Prefer language_codes. Defaults to None (not sent; no steering).

  • language_codes – Customer-declared audio languages. A single language (e.g. [Language.ES]) pins transcription to that language; several (e.g. [Language.EN, Language.ES]) steer toward that subset while keeping code-switching among them. Order is significant — the steering prompt follows the declared order. Regional variants resolve to their base code, so at most 10 distinct languages. Steering is prompt-based, so it applies to U3 Pro models only and is not sent for other models — including universal-streaming-multilingual, which transcribes multilingual audio without steering. Unlike most settings, a change applies to a live session without reconnecting; pass an empty list to clear steering back to the model default. Defaults to None (not sent; no steering).

  • format_turns – Whether to format transcript turns.

  • speaker_labels – Enable speaker diarization. Final transcripts carry AssemblyAI’s speaker label (e.g. "A", "B") as speaker.label; interim transcripts, and turns whose speaker is not assigned yet, have none. Labels AssemblyAI later revises are not applied to frames already sent.

  • max_speakers – Hard cap on the number of speaker labels, an integer in [1, 10]. Once the cap is reached, further speakers are merged into the closest existing label rather than given a new one, so leave a little headroom above the number of speakers expected; a cap set too high can over-split a single speaker. Only used when speaker_labels is enabled. Defaults to None (not sent).

  • speaker_format – Optional format string for diarized transcripts. Use {speaker} for the speaker label, {text} for the transcript text, and {speaker_number} for the speaker’s order of first appearance in the session (1, 2, …), e.g. "{speaker}: {text}". If None, transcript text is not modified. Applied locally, so changing it does not reconnect.

  • filter_profanity – Replace profanity in transcribed text with asterisks. Defaults to None (not sent; the server leaves text unfiltered).

  • vad_threshold – VAD confidence threshold (0.0–1.0) for classifying audio frames as silence. Only applicable to U3 Pro models.

  • domain – Optional domain for specialized recognition modes. For example, set to “medical-v1” to enable Medical Mode for healthcare transcription.

  • continuous_partials – Emit partial transcripts at a steady cadence during long turns, rather than only one early partial near the turn start. Only applicable to U3 Pro models; not sent for other models. Defaults to True in this plugin so voice agents receive continuous interim updates.

  • interruption_delay – Override, in milliseconds (0–1000), for how soon the first partial is emitted. The server adds 256ms (MIN_TURN_DURATION_MS) on top, so 0 → 256ms effective. Only applicable to U3 Pro models. Defaults to None (use the server default).

  • agent_context – Context carryover seed — the agent’s most recent spoken reply, used to improve transcription of the user’s next turn (short answers, spelled-out entities, disambiguation). Only applicable to U3 Pro models; clipped to ~1500 characters and reset on reconnect. Set this for a known opening line; the service sends each of the agent’s replies as context on its own. To replace it mid-session, push an STTUpdateSettingsFrame with agent_context. Defaults to None (not sent).

  • previous_context_n_turns – Maximum number of prior conversation entries (user transcripts and any agent_context values) carried forward as context for each transcription. Integer in [0, 100]; set to 0 to disable automatic context carryover entirely. Only applicable to U3 Pro models. Most integrations should leave this at the default. Defaults to None (use the server default, which is 3).

  • voice_focus – Isolate the primary voice and suppress background noise. Set to “near-field” for close-talking mics (headsets, handsets) or “far-field” for distant capture (conference rooms, laptop mics). Only applicable to U3 Pro models. Defaults to None (not sent).

  • voice_focus_threshold – How aggressively background audio is suppressed. Float in [0.0, 1.0]; higher values suppress more. Only takes effect when voice_focus is set. Only applicable to U3 Pro models. Defaults to None (use the server default).

  • mode – Latency/accuracy preset, trading transcription accuracy against turn-finalization latency. One of “min_latency”, “balanced”, or “max_accuracy”. Only applicable to U3 Pro models; the server defaults to “balanced” and rejects this parameter for other models. Defaults to None (not sent).

class pipecat.services.assemblyai.stt.AssemblyAISTTService(*, api_key: str, language: Language | None = None, api_endpoint_base_url: str = 'wss://streaming.assemblyai.com/v3/ws', sample_rate: int | None = None, encoding: str = 'pcm_s16le', connection_params: AssemblyAIConnectionParams | None = None, vad_force_turn_endpoint: bool = True, should_interrupt: bool = True, speaker_format: str | None = None, settings: AssemblyAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.42, **kwargs)[source]

Bases: WebsocketSTTService

AssemblyAI real-time speech-to-text service.

Provides real-time speech transcription using AssemblyAI’s WebSocket API. Supports both interim and final transcriptions with configurable parameters for audio processing and connection management.

Event handlers available (in addition to WebsocketSTTService events):

  • on_end_of_turn(service, transcript): Called when AssemblyAI detects end of turn.

Example:

@service.event_handler("on_end_of_turn")
async def on_end_of_turn(service, transcript):
    ...
Settings

alias of AssemblyAISTTSettings

__init__(*, api_key: str, language: Language | None = None, api_endpoint_base_url: str = 'wss://streaming.assemblyai.com/v3/ws', sample_rate: int | None = None, encoding: str = 'pcm_s16le', connection_params: AssemblyAIConnectionParams | None = None, vad_force_turn_endpoint: bool = True, should_interrupt: bool = True, speaker_format: str | None = None, settings: AssemblyAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.42, **kwargs)[source]

Initialize the AssemblyAI STT service.

Parameters:
  • api_key – AssemblyAI API key for authentication.

  • language –

    Language code for transcription. Defaults to English (Language.EN).

    Deprecated since version 0.0.105: Use settings=AssemblyAISTTService.Settings(language=...) instead. Will be removed in 2.0.0.

  • api_endpoint_base_url – WebSocket endpoint URL. Defaults to AssemblyAI’s streaming endpoint.

  • sample_rate – Audio sample rate in Hz. If None, uses the input sample rate from the start frame.

  • encoding – Audio encoding format. Defaults to “pcm_s16le”.

  • connection_params –

    Connection configuration parameters.

    Deprecated since version 0.0.105: Use settings=AssemblyAISTTService.Settings(...) instead. Will be removed in 2.0.0.

  • vad_force_turn_endpoint – Controls turn detection mode. When True (Pipecat mode, default): Forces AssemblyAI to return finals ASAP so Pipecat’s turn detection (e.g., Smart Turn) decides when the user is done. - min_turn_silence defaults to 100ms (user can override) - max_turn_silence is ALWAYS set equal to min_turn_silence - VAD stop sends ForceEndpoint as ceiling - No UserStarted/StoppedSpeakingFrame emitted from STT When False (AssemblyAI turn detection mode, U3 Pro models only): AssemblyAI’s model controls turn endings using built-in turn detection. - Uses AssemblyAI API defaults for all parameters (unless user explicitly sets them) - Emits UserStarted/StoppedSpeakingFrame from STT - No ForceEndpoint on VAD stop

  • should_interrupt – Whether to interrupt the bot when the user starts speaking in AssemblyAI turn detection mode (vad_force_turn_endpoint=False). Only applies when using AssemblyAI’s built-in turn detection. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied user_turn_strategies overrides the recommendation and this setting with it. Defaults to True.

  • speaker_format –

    Optional format string for speaker labels when diarization is enabled.

    Deprecated since version 1.16.0: Use settings=AssemblyAISTTService.Settings(speaker_format=...) instead. Will be removed in 2.0.0.

  • settings – Runtime-updatable settings. When provided alongside deprecated parameters, settings values take precedence.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark

  • **kwargs – Additional arguments passed to parent STTService class.

can_generate_metrics() → bool[source]

Check if the service can generate metrics.

Returns:

True if metrics generation is supported.

service_metadata_frame() → STTMetadataFrame[source]

Request external turn strategies in AssemblyAI’s turn-detection mode.

With vad_force_turn_endpoint=False AssemblyAI’s model decides turn endings and emits ProposedUserStarted/StoppedSpeakingFrame, so the user aggregator resolves those rather than running local VAD/smart-turn. In the default Pipecat mode (vad_force_turn_endpoint=True) the STT proposes no turns, so the defaults are left in place. Applied unless the user passed their own user_turn_strategies.

async setup(setup: FrameProcessorSetup)[source]

Set up the service and connect.

Parameters:

setup – Configuration object containing setup parameters.

async stop(frame: EndFrame)[source]

Stop the speech-to-text service.

Parameters:

frame – End frame to stop processing.

async cancel(frame: CancelFrame)[source]

Cancel the speech-to-text service.

Parameters:

frame – Cancel frame to abort processing.

async run_stt(audio: bytes) → AsyncGenerator[Frame | None, None][source]

Process audio data for speech-to-text conversion.

Parameters:

audio – Raw audio bytes to process.

Yields:

None (processing handled via WebSocket messages).

async process_frame(frame: Frame, direction: FrameDirection)[source]

Forward a Pipecat-detected turn end to AssemblyAI as a ForceEndpoint.

Parameters:
  • frame – Frame to process.

  • direction – Direction of frame processing.

async update_agent_context(text: str)[source]

Send the agent’s latest spoken reply to AssemblyAI as carryover context.

Context carryover (U3 Pro models only) gives the model a short memory of what the agent just said so it can better transcribe the user’s reply — short answers, spelled-out entities (emails, IDs), and similar-sounding words. No-op for non-U3 Pro models.

Parameters:

text – The agent’s spoken reply text. Clipped to ~1500 characters.

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.

Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.

Parameters:
  • frame – The frame to push.

  • direction – The direction to push the frame.

async start_stt_usage_metrics(usage: STTUsage)

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.

class pipecat.services.assemblyai.stt.AssemblyAISyncSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, prompt: str | None | NotGiven = <factory>, keyterms_prompt: list[str] | None | NotGiven = <factory>, conversation_context: str | list[str] | None | NotGiven = <factory>, language_codes: list[Language] | None | NotGiven = <factory>, timestamps: bool | None | NotGiven = <factory>)[source]

Bases: STTSettings

Settings for AssemblyAISyncSTTService.

Parameters:
  • prompt – A natural-language description of what the audio is about: the domain, the scenario, or details of the conversation. Prepended to the model’s system prompt to guide transcription.

  • keyterms_prompt – Key terms or phrases to bias the decoder toward.

  • conversation_context – Prior turns from the same conversation, oldest first, giving the model the surrounding dialogue for better continuity and proper-noun consistency across a multi-turn conversation. A single string is treated as one turn. Setting this explicitly turns off the service’s automatic context buffer and sends exactly this value; leave it unset to let the service manage context (see max_context_turns).

  • language_codes – Declared audio languages for multilingual or code-switching audio (e.g. [Language.EN, Language.ES]). A single-element list pins one language. Regional variants resolve to their base code and duplicates are dropped, preserving declaration order. Bound in preference to the single language setting when both are set. Defaults to unset (the single language is used).

  • timestamps – Whether to compute per-word start/end timestamps, returned on the words of the transcription result, at a small added latency. Left unset by default, so the API default of False (no timestamps) applies.

class pipecat.services.assemblyai.stt.AssemblyAISyncSTTService(*, api_key: str, aiohttp_session: ClientSession, base_url: str = 'https://sync.assemblyai.com', sample_rate: int | None = None, enable_prewarming: bool = True, max_context_turns: int = 5, max_context_chars: int = 1500, settings: AssemblyAISyncSTTSettings | None = None, ttfs_p99_latency: float | None = 0.65, **kwargs)[source]

Bases: SegmentedSTTService

AssemblyAI Sync speech-to-text service.

Transcribes a complete speech segment in a single request/response against AssemblyAI’s Sync API, rather than holding open a streaming WebSocket. It inherits from SegmentedSTTService, which uses VAD events to buffer each utterance and calls run_stt() once per segment with the audio as a WAV container. Each segment is POSTed as multipart/form-data and the transcript is returned directly — no upload step, no polling, no session to manage.

Audio segments must be at most 120 seconds; the API rejects longer ones. This suits dictation, scribe, and voice-agent turns where the client detects speech locally and transcribes each turn on demand.

When pre-warming is enabled (the default), the service opens the connection as soon as the user starts speaking, so the transcription request skips the DNS, TCP, and TLS handshake once the utterance ends. Call warm() directly to warm the connection at any other time.

Because the Sync API is stateless, the service also carries recent conversation context automatically: it keeps a rolling buffer of the most recent turns — user transcripts and the agent’s replies together, in the order spoken — and sends them as conversation_context on each request, so the model transcribes each turn with the surrounding dialogue. Agent replies are captured from the pipeline’s assistant-turn frame, so this needs no wiring in a standard voice-agent pipeline. Set max_context_turns=0 to disable it, or set conversation_context explicitly in settings to supply the context yourself (that value is honored as-is).

Settings

alias of AssemblyAISyncSTTSettings

__init__(*, api_key: str, aiohttp_session: ClientSession, base_url: str = 'https://sync.assemblyai.com', sample_rate: int | None = None, enable_prewarming: bool = True, max_context_turns: int = 5, max_context_chars: int = 1500, settings: AssemblyAISyncSTTSettings | None = None, ttfs_p99_latency: float | None = 0.65, **kwargs)[source]

Initialize the AssemblyAI Sync STT service.

Parameters:
  • api_key – AssemblyAI API key for authentication.

  • aiohttp_session – aiohttp ClientSession for HTTP requests. Pre-warming only helps when the warm and transcribe requests share this session’s connection pool, so keep one session for the service.

  • base_url – Base URL for the Sync API. Override for a data-residency endpoint (e.g. https://sync.us.assemblyai.com or https://sync.eu.assemblyai.com).

  • sample_rate – Audio sample rate in Hz. If not provided, uses the pipeline’s rate.

  • enable_prewarming – Whether to open the connection when the user starts speaking so the transcription request avoids the connection handshake. Defaults to True.

  • max_context_turns – Number of prior conversation turns — user transcripts and agent replies together, in one chronological buffer — automatically carried as conversation_context on each request, so the model transcribes each turn with the surrounding dialogue. Set to 0 to disable automatic context. Ignored when conversation_context is set explicitly in settings (that value is honored as-is). Defaults to 5.

  • max_context_chars – Character budget for the same buffer; the oldest turns are dropped first once either cap is exceeded. Defaults to 1500.

  • settings – Runtime-updatable settings.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Broadcast at pipeline start for downstream turn timing; set it to your measured value.

  • **kwargs – Additional arguments passed to SegmentedSTTService.

can_generate_metrics() → bool[source]

Whether this service generates processing metrics.

Returns:

True. The Sync API issues a discrete request per segment, so its duration is measured and reported.

language_to_service_language(language: Language) → str | None[source]

Convert a Pipecat Language to an AssemblyAI language code.

Parameters:

language – The language to convert.

Returns:

The AssemblyAI language code (base ISO code).

async start(frame: StartFrame)[source]

Start the service on an empty conversation context.

The buffer holds one conversation, so an instance reused for another run starts it fresh.

async warm()[source]

Pre-warm the connection to the Sync API.

Establishes the connection (DNS, TCP, TLS) ahead of a transcription request so the next run_stt() starts uploading audio immediately. Best-effort: failures are logged and swallowed, since a failed warm-up only forfeits the latency saving. The warmed connection is reused only by requests that share this service’s aiohttp session and base URL.

async run_stt(audio: bytes) → AsyncGenerator[Frame | None, None][source]

Transcribe an audio segment using the AssemblyAI Sync API.

Parameters:

audio – Raw audio bytes in WAV format (already converted by the base class).

Yields:

Frame – A TranscriptionFrame with the transcribed text, or an ErrorFrame on failure. Only non-empty transcriptions are yielded.

async cleanup()[source]

Cancel any in-flight pre-warm task and clean up.

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame, marking a TranscriptionFrame finalized when its segment is the latest.

Once the user speaks again, the transcript of an earlier segment covers only the words before they resumed, so it isn’t finalized, the same way STTService drops a pending finalize when the user starts speaking. When a segment’s transcript arrives as several diarized TranscriptionFrames (one per speaker run), only the last one can be.

Parameters:
  • frame – The frame to push.

  • direction – The direction of frame flow in the pipeline.

async start_stt_usage_metrics(usage: STTUsage)

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.