stt
Zoom Scribe speech-to-text service.
ZoomAISTTService streams audio to Scribe Live over a WebSocket. By default
Scribe’s own voice activity detection ends each turn and returns its final transcript.
With vad_force_turn_endpoint=True, Pipecat’s VAD and turn strategies decide when
the user stops speaking instead, and the service asks Scribe for the final transcript
at each VAD stop.
Settings map to the Scribe API fields of the same name. For what each feature does, and its availability, pricing and enablement, see https://developers.zoom.us/docs/ai-services/scribe/ and https://www.zoom.ai/
- pipecat.services.zoom_ai.stt.language_to_zoom_ai_language(language: Language) str[source]
Convert a Language to a Scribe locale, e.g.
Language.ENto"en-US".A regional variant Scribe doesn’t list resolves to the locale of its base language, e.g.
Language.EN_GBto"en-US". For the supported languages of each model, see https://developers.zoom.us/docs/ai-services/scribe/speech-understanding/- Parameters:
language – The language to convert.
- Returns:
The Scribe locale.
- class pipecat.services.zoom_ai.stt.ZoomAISTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, silence_duration_ms: int | None | NotGiven = <factory>, diarization: bool | None | NotGiven = <factory>, language_detection: str | None | NotGiven = <factory>, vocabulary: list[str] | None | NotGiven = <factory>, text_polishing: bool | None | NotGiven = <factory>)[source]
Bases:
STTSettingsSettings for
ZoomAISTTService.Changing a setting reconnects with a new Scribe session.
- Parameters:
silence_duration_ms – Milliseconds of silence that end a turn in Scribe’s voice activity detection. Defaults to 350, tuned for voice agents; lower is faster, higher keeps sentences with longer pauses together. For transcription without an agent, a longer value such as 600 can improve accuracy.
Noneuses Scribe’s default (see the Scribe docs). Not used withvad_force_turn_endpoint=True, where Scribe’s voice activity detection is off.diarization – Whether Scribe labels each completed transcript with a speaker. Off by default. The labels are in the Scribe event attached as
TranscriptionFrame.result:speaker, plusspeaker_segmentswith each one’s span when several speakers spoke. Scribe’s speaker tags are removed from the transcript text.language_detection – Scribe language detection mode (
config.language_detection.mode).Noneturns it off. A detected language Scribe reports is inTranscriptionFrame.language; when Scribe reports none, it isNone.vocabulary – Words and phrases that customize recognition (
config.vocabulary.phrases).text_polishing – Scribe’s text polishing (
outputs.text_polishing). The polished text is the transcript, and the recognized text is in the Scribe event attached asTranscriptionFrame.result(transcript); if polishing fails, the recognized text is used. Off by default.
extraholds Scribe request fields this class has no setting for, shaped like the request (e.g.{"config": {"profanity_filter": True}}). It is merged into thesession.updateover the settings above, except thataudio.formatstayspcm16.
- class pipecat.services.zoom_ai.stt.ZoomAISTTService(*, api_key: str, url: str = 'wss://api.zoom.us/v2/aiservices/scribe/live', sample_rate: int | None = None, should_interrupt: bool = True, vad_force_turn_endpoint: bool = False, settings: ZoomAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.53, **kwargs)[source]
Bases:
WebsocketSTTServiceZoom Scribe Live streaming speech-to-text service.
Streams 16 kHz PCM16 audio to Scribe Live over a WebSocket. It has two turn detection modes, chosen with
vad_force_turn_endpoint.Scribe turn detection (
vad_force_turn_endpoint=False, the default). Scribe’s own voice activity detection decides when the user starts and stops speaking: itsspeech_startedevent broadcasts aProposedUserStartedSpeakingFrame, and each completed transcript is pushed as aTranscriptionFramefollowed by aProposedUserStoppedSpeakingFrame.service_metadata_frame()recommendsExternalUserTurnStrategiesso the user aggregator resolves those proposals instead of running local VAD. The service pushes final transcripts only. A transcript that arrives without a precedingspeech_started(for example right after a reconnect) opens its own turn, so it is never dropped; a turn whose transcript comes back empty ends with no transcription.Pipecat turn detection (
vad_force_turn_endpoint=True). Scribe’s voice activity detection is turned off (turn_detection: {"type": "none"}), and Pipecat’s VAD and user turn strategies (e.g. Smart Turn) decide when the user is done. On eachVADUserStoppedSpeakingFramethe service sendsinput_audio_buffer.commit, and Scribe returns that turn’s final transcript. The service proposes no turns, so the default user turn strategies apply, and it reports a TTFS P99 latency. A local VAD analyzer is required in this mode.Example:
stt = ZoomAISTTService( api_key=os.getenv("ZOOM_AI_API_KEY"), settings=ZoomAISTTService.Settings(language=Language.EN_US), )
- Settings
alias of
ZoomAISTTSettings
- __init__(*, api_key: str, url: str = 'wss://api.zoom.us/v2/aiservices/scribe/live', sample_rate: int | None = None, should_interrupt: bool = True, vad_force_turn_endpoint: bool = False, settings: ZoomAISTTSettings | None = None, ttfs_p99_latency: float | None = 0.53, **kwargs)[source]
Initialize the Zoom Scribe Live STT service.
- Parameters:
api_key – Zoom AI Services API key or JWT. It is sent on every request and reconnect, so it must stay valid for as long as the service runs.
url – Scribe Live WebSocket URL. A plaintext
ws://URL to a host other than localhost logs a warning.sample_rate – Input audio sample rate. Audio is resampled to 16 kHz.
should_interrupt – Whether a user turn started by Scribe interrupts the bot. Only applies with Scribe turn detection (
vad_force_turn_endpoint=False). Passed along to the user turn strategies this service recommends; a user-supplieduser_turn_strategiesoverrides it.vad_force_turn_endpoint – Controls turn detection mode. When False (the default), Scribe’s voice activity detection ends each turn (
silence_duration_ms) and the service proposes turn boundaries. When True (Pipecat mode), Scribe’s voice activity detection is off and eachVADUserStoppedSpeakingFramecommits the turn, so Pipecat’s VAD and turn strategies decide when the user is done. Requires a local VAD analyzer.settings – Runtime-updatable settings.
ttfs_p99_latency – P99 latency from speech end to final transcript in seconds, used with
vad_force_turn_endpoint=True. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark**kwargs – Additional arguments passed to the parent WebsocketSTTService.
- can_generate_metrics() bool[source]
Check if this service can generate processing metrics.
- Returns:
True, as this service supports metrics.
- language_to_service_language(language: Language) str | None[source]
Convert a Language to a Scribe locale.
- Parameters:
language – The language to convert.
- Returns:
The Scribe locale. Regional variants Scribe doesn’t list resolve to the locale of their base language.
- property supports_ttfs: bool
Whether there is a TTFS to report.
With Scribe turn detection, Scribe defines the turn boundary, so there is none. With
vad_force_turn_endpoint=True, the final transcript follows the local VAD stop, so there is.
- service_metadata_frame() STTMetadataFrame[source]
Let Scribe’s turn detection drive user turns in Scribe turn detection mode.
With
vad_force_turn_endpoint=False, Scribe decides when the user starts and stops speaking, and this service proposes those turns. It recommendsExternalUserTurnStrategiesso the user aggregator follows the proposals instead of running its own VAD and Smart Turn (“external” means turn detection outside the aggregator, here Scribe’s). Withvad_force_turn_endpoint=Truethe service proposes no turns, so the default turn strategies (local VAD and Smart Turn) are left in place.- Returns:
The STT metadata frame.
- async process_frame(frame: Frame, direction: FrameDirection)[source]
Commit the Scribe turn on a local VAD stop, in Pipecat turn detection mode.
- Parameters:
frame – The frame to process.
direction – The direction of frame processing.
- async setup(setup: FrameProcessorSetup)[source]
Set up the service and connect to Scribe Live.
- Parameters:
setup – Configuration object containing setup parameters.
- async stop(frame: EndFrame)[source]
Close the Scribe session, then stop the service.
- Parameters:
frame – The end frame.
- async run_stt(audio: bytes) AsyncGenerator[Frame | None, None][source]
Send audio to Scribe Live.
Transcriptions are pushed from the receive task, not yielded here.
- Parameters:
audio – Raw PCM16 mono audio at the pipeline sample rate.
- Yields:
None.
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)
Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.
Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.
- Parameters:
frame – The frame to push.
direction – The direction to push the frame.