stt

Zoom Scribe speech-to-text service.

ZoomAISTTService streams audio to Scribe Live over a WebSocket. By default Scribe’s own voice activity detection ends each turn and returns its final transcript. With vad_force_turn_endpoint=True, Pipecat’s VAD and turn strategies decide when the user stops speaking instead, and the service asks Scribe for the final transcript at each VAD stop.

Settings map to the Scribe API fields of the same name. For what each feature does, and its availability, pricing and enablement, see https://developers.zoom.us/docs/ai-services/scribe/ and https://www.zoom.ai/

pipecat.services.zoom_ai.stt.language_to_zoom_ai_language(language: Language) → str[source]

Convert a Language to a Scribe locale, e.g. Language.EN to "en-US".

A regional variant Scribe doesn’t list resolves to the locale of its base language, e.g. Language.EN_GB to "en-US". For the supported languages of each model, see https://developers.zoom.us/docs/ai-services/scribe/speech-understanding/

Parameters:

language – The language to convert.

Returns:

The Scribe locale.

class pipecat.services.zoom_ai.stt.ZoomAISTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, silence_duration_ms: int | None | NotGiven = <factory>, diarization: bool | None | NotGiven = <factory>, language_detection: str | None | NotGiven = <factory>, vocabulary: list[str] | None | NotGiven = <factory>, text_polishing: bool | None | NotGiven = <factory>)[source]

Bases: STTSettings

Settings for ZoomAISTTService.

Changing a setting reconnects with a new Scribe session.

Parameters:
  • silence_duration_ms – Milliseconds of silence that end a turn in Scribe’s voice activity detection. Defaults to 350, tuned for voice agents; lower is faster, higher keeps sentences with longer pauses together. For transcription without an agent, a longer value such as 600 can improve accuracy. None uses Scribe’s default (see the Scribe docs). Not used with vad_force_turn_endpoint=True, where Scribe’s voice activity detection is off.

  • diarization – Whether Scribe labels each completed transcript with a speaker. Off by default. The labels are in the Scribe event attached as TranscriptionFrame.result: speaker, plus speaker_segments with each one’s span when several speakers spoke. Scribe’s speaker tags are removed from the transcript text.

  • language_detection – Scribe language detection mode (config.language_detection.mode). None turns it off. A detected language Scribe reports is in TranscriptionFrame.language; when Scribe reports none, it is None.

  • vocabulary – Words and phrases that customize recognition (config.vocabulary.phrases).

  • text_polishing – Scribe’s text polishing (outputs.text_polishing). The polished text is the transcript, and the recognized text is in the Scribe event attached as TranscriptionFrame.result (transcript); if polishing fails, the recognized text is used. Off by default.

extra holds Scribe request fields this class has no setting for, shaped like the request (e.g. {"config": {"profanity_filter": True}}). It is merged into the session.update over the settings above, except that audio.format stays pcm16.

class pipecat.services.zoom_ai.stt.ZoomAISTTService(*, api_key: str, url: str = 'wss://api.zoom.us/v2/aiservices/scribe/live', sample_rate: int | None = None, should_interrupt: bool = True, vad_force_turn_endpoint: bool = False, settings: ZoomAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.53, **kwargs)[source]

Bases: WebsocketSTTService

Zoom Scribe Live streaming speech-to-text service.

Streams 16 kHz PCM16 audio to Scribe Live over a WebSocket. It has two turn detection modes, chosen with vad_force_turn_endpoint.

Scribe turn detection (vad_force_turn_endpoint=False, the default). Scribe’s own voice activity detection decides when the user starts and stops speaking: its speech_started event broadcasts a ProposedUserStartedSpeakingFrame, and each completed transcript is pushed as a TranscriptionFrame followed by a ProposedUserStoppedSpeakingFrame. service_metadata_frame() recommends ExternalUserTurnStrategies so the user aggregator resolves those proposals instead of running local VAD. The service pushes final transcripts only. A transcript that arrives without a preceding speech_started (for example right after a reconnect) opens its own turn, so it is never dropped; a turn whose transcript comes back empty ends with no transcription.

Pipecat turn detection (vad_force_turn_endpoint=True). Scribe’s voice activity detection is turned off (turn_detection: {"type": "none"}), and Pipecat’s VAD and user turn strategies (e.g. Smart Turn) decide when the user is done. On each VADUserStoppedSpeakingFrame the service sends input_audio_buffer.commit, and Scribe returns that turn’s final transcript. The service proposes no turns, so the default user turn strategies apply, and it reports a TTFS P99 latency. A local VAD analyzer is required in this mode.

Example:

stt = ZoomAISTTService(
    api_key=os.getenv("ZOOM_AI_API_KEY"),
    settings=ZoomAISTTService.Settings(language=Language.EN_US),
)
Settings

alias of ZoomAISTTSettings

__init__(*, api_key: str, url: str = 'wss://api.zoom.us/v2/aiservices/scribe/live', sample_rate: int | None = None, should_interrupt: bool = True, vad_force_turn_endpoint: bool = False, settings: ZoomAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.53, **kwargs)[source]

Initialize the Zoom Scribe Live STT service.

Parameters:
  • api_key – Zoom AI Services API key or JWT. It is sent on every request and reconnect, so it must stay valid for as long as the service runs.

  • url – Scribe Live WebSocket URL. A plaintext ws:// URL to a host other than localhost logs a warning.

  • sample_rate – Input audio sample rate. Audio is resampled to 16 kHz.

  • should_interrupt – Whether a user turn started by Scribe interrupts the bot. Only applies with Scribe turn detection (vad_force_turn_endpoint=False). Passed along to the user turn strategies this service recommends; a user-supplied user_turn_strategies overrides it.

  • vad_force_turn_endpoint – Controls turn detection mode. When False (the default), Scribe’s voice activity detection ends each turn (silence_duration_ms) and the service proposes turn boundaries. When True (Pipecat mode), Scribe’s voice activity detection is off and each VADUserStoppedSpeakingFrame commits the turn, so Pipecat’s VAD and turn strategies decide when the user is done. Requires a local VAD analyzer.

  • settings – Runtime-updatable settings.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds, used with vad_force_turn_endpoint=True. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark

  • **kwargs – Additional arguments passed to the parent WebsocketSTTService.

can_generate_metrics() → bool[source]

Check if this service can generate processing metrics.

Returns:

True, as this service supports metrics.

language_to_service_language(language: Language) → str | None[source]

Convert a Language to a Scribe locale.

Parameters:

language – The language to convert.

Returns:

The Scribe locale. Regional variants Scribe doesn’t list resolve to the locale of their base language.

property supports_ttfs: bool

Whether there is a TTFS to report.

With Scribe turn detection, Scribe defines the turn boundary, so there is none. With vad_force_turn_endpoint=True, the final transcript follows the local VAD stop, so there is.

service_metadata_frame() → STTMetadataFrame[source]

Let Scribe’s turn detection drive user turns in Scribe turn detection mode.

With vad_force_turn_endpoint=False, Scribe decides when the user starts and stops speaking, and this service proposes those turns. It recommends ExternalUserTurnStrategies so the user aggregator follows the proposals instead of running its own VAD and Smart Turn (“external” means turn detection outside the aggregator, here Scribe’s). With vad_force_turn_endpoint=True the service proposes no turns, so the default turn strategies (local VAD and Smart Turn) are left in place.

Returns:

The STT metadata frame.

async process_frame(frame: Frame, direction: FrameDirection)[source]

Commit the Scribe turn on a local VAD stop, in Pipecat turn detection mode.

Parameters:
  • frame – The frame to process.

  • direction – The direction of frame processing.

async setup(setup: FrameProcessorSetup)[source]

Set up the service and connect to Scribe Live.

Parameters:

setup – Configuration object containing setup parameters.

async stop(frame: EndFrame)[source]

Close the Scribe session, then stop the service.

Parameters:

frame – The end frame.

async run_stt(audio: bytes) → AsyncGenerator[Frame | None, None][source]

Send audio to Scribe Live.

Transcriptions are pushed from the receive task, not yielded here.

Parameters:

audio – Raw PCM16 mono audio at the pipeline sample rate.

Yields:

None.

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.

Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.

Parameters:
  • frame – The frame to push.

  • direction – The direction to push the frame.

async start_stt_usage_metrics(usage: STTUsage)

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.