stt
OpenAI Speech-to-Text service implementations.
Provides two STT services:
OpenAISTTService: REST-based transcription using the Audio API (Whisper / GPT transcription models).OpenAIRealtimeSTTService: WebSocket-based streaming transcription using the Realtime API in transcription-only mode.
- class pipecat.services.openai.stt.OpenAISTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, prompt: str | None | NotGiven = <factory>, temperature: float | None | NotGiven = <factory>, keywords: list[str] | None | NotGiven = <factory>, speaker_format: str | None | NotGiven = <factory>)[source]
Bases:
BaseWhisperSTTSettingsSettings for the OpenAI STT service.
- Parameters:
keywords – Words or phrases that guide transcription of the input audio, such as product names or acronyms. Supported by
"gpt-transcribe".speaker_format – Optional format string for diarized speaker runs. Use
{speaker}for the speaker label,{text}for the transcript text, and{speaker_number}for the speaker’s order of first appearance in the session (1,2, …), e.g."{speaker}: {text}". IfNone, transcript text is not modified.
- class pipecat.services.openai.stt.OpenAISTTService(*, model: str | None = None, api_key: str | None = None, base_url: str | None = None, language: Language | None = Language.EN, prompt: str | None = None, temperature: float | None = None, settings: OpenAISTTSettings | None = None, ttfs_p99_latency: float | None = 2.01, **kwargs)[source]
Bases:
BaseWhisperSTTServiceOpenAI Speech-to-Text service that generates text from audio.
Uses OpenAI’s transcription API to convert audio to text. Requires an OpenAI API key set via the api_key parameter or OPENAI_API_KEY environment variable.
With
include_prob_metrics=True, GPT transcription models report per-token logprobs and Whisper models report per-segment logprobs. Diarization models support neither, so they report no probabilities.Diarization models (e.g.
"gpt-4o-transcribe-diarize") label each speaker: every contiguous speaker run of a segment is emitted as its own frame with OpenAI’s speaker label (e.g."A") asspeaker.label.- Settings
alias of
OpenAISTTSettings
- __init__(*, model: str | None = None, api_key: str | None = None, base_url: str | None = None, language: Language | None = Language.EN, prompt: str | None = None, temperature: float | None = None, settings: OpenAISTTSettings | None = None, ttfs_p99_latency: float | None = 2.01, **kwargs)[source]
Initialize OpenAI STT service.
- Parameters:
model –
Transcription model to use. Defaults to
"gpt-transcribe".Deprecated since version 0.0.105: Use
settings=OpenAISTTService.Settings(model=...)instead. Will be removed in 2.0.0.api_key – OpenAI API key. Defaults to None.
base_url – API base URL. Defaults to None.
language –
Language of the audio input. Defaults to English.
Deprecated since version 0.0.105: Use
settings=OpenAISTTService.Settings(language=...)instead. Will be removed in 2.0.0.prompt –
Optional text to guide the model’s style or continue a previous segment.
Deprecated since version 0.0.105: Use
settings=OpenAISTTService.Settings(prompt=...)instead. Will be removed in 2.0.0.temperature –
Optional sampling temperature between 0 and 1. Defaults to 0.0.
Deprecated since version 0.0.105: Use
settings=OpenAISTTService.Settings(temperature=...)instead. Will be removed in 2.0.0.settings – Runtime-updatable settings. When provided alongside deprecated parameters,
settingsvalues take precedence.ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark
**kwargs – Additional arguments passed to BaseWhisperSTTService.
- class pipecat.services.openai.stt.OpenAIRealtimeSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, prompt: str | None | NotGiven = <factory>, noise_reduction: Literal['near_field', 'far_field'] | None | ~pipecat.utils.types.NotGiven=<factory>, delay: Literal['minimal', 'low', 'medium', 'high', 'xhigh'] | None | ~pipecat.utils.types.NotGiven=<factory>)[source]
Bases:
STTSettingsSettings for OpenAIRealtimeSTTService.
- Parameters:
prompt – Optional prompt text to guide transcription style. Not supported by
"gpt-realtime-whisper".noise_reduction – Noise reduction mode.
"near_field"for close microphones,"far_field"for distant microphones, orNoneto disable.delay – How long the model waits before emitting transcription text, trading latency for accuracy. Only supported by
"gpt-realtime-whisper".
- class pipecat.services.openai.stt.OpenAIRealtimeSTTService(*, api_key: str, model: str | None = None, base_url: str = 'wss://api.openai.com/v1/realtime', language: Language | None = Language.EN, prompt: str | None = None, turn_detection: dict | Literal[False] | None = False, noise_reduction: Literal['near_field', 'far_field'] | None = None, should_interrupt: bool = True, settings: OpenAIRealtimeSTTSettings | None = None, ttfs_p99_latency: float | None = 1.66, **kwargs)[source]
Bases:
WebsocketSTTServiceOpenAI Realtime Speech-to-Text service using WebSocket transcription sessions.
Uses OpenAI’s Realtime API in transcription-only mode for real-time streaming speech recognition with optional server-side VAD and noise reduction. The model does not generate conversational responses — only transcription output.
This service supports two VAD modes:
Local VAD (default): Disable server-side VAD and use a local VAD processor in the pipeline instead. When a
VADUserStoppedSpeakingFrameis received, the service commits the audio buffer so that the server begins transcription for the completed speech segment.Server-side VAD (
turn_detection={"type": "server_vad"}): The OpenAI server performs voice-activity detection. The service proposes turn boundaries when the server detects them, and recommends the external user turn strategies that resolve those proposals into turn frames. Do not use a separate VAD processor in the pipeline in this mode. Requires a model that supports turn detection —gpt-transcribedoes, the defaultgpt-realtime-whisperdoes not.Audio is sent as 24 kHz 16-bit mono PCM as required by the OpenAI Realtime API. If the pipeline runs at a different sample rate (e.g. 16 kHz for Silero VAD compatibility), audio is automatically upsampled before sending.
Example:
stt = OpenAIRealtimeSTTService( api_key="sk-...", settings=OpenAIRealtimeSTTService.Settings( model="gpt-realtime-whisper", noise_reduction="near_field", ), )
- Settings
alias of
OpenAIRealtimeSTTSettings
- __init__(*, api_key: str, model: str | None = None, base_url: str = 'wss://api.openai.com/v1/realtime', language: Language | None = Language.EN, prompt: str | None = None, turn_detection: dict | Literal[False] | None = False, noise_reduction: Literal['near_field', 'far_field'] | None = None, should_interrupt: bool = True, settings: OpenAIRealtimeSTTSettings | None = None, ttfs_p99_latency: float | None = 1.66, **kwargs)[source]
Initialize the OpenAI Realtime STT service.
- Parameters:
api_key – OpenAI API key for authentication.
model –
Transcription model. For low-latency streaming transcription, use
"gpt-realtime-whisper". Other supported transcription models include"gpt-transcribe"and"gpt-live-transcribe".Deprecated since version 0.0.105: Use
settings=OpenAIRealtimeSTTService.Settings(model=...)instead. Will be removed in 2.0.0.base_url – WebSocket base URL for the Realtime API. Defaults to
"wss://api.openai.com/v1/realtime".language –
Language of the audio input. Defaults to English.
Deprecated since version 0.0.105: Use
settings=OpenAIRealtimeSTTService.Settings(language=...)instead. Will be removed in 2.0.0.prompt –
Optional prompt text to guide transcription style or provide keyword hints. Not supported by
"gpt-realtime-whisper".Deprecated since version 0.0.105: Use
settings=OpenAIRealtimeSTTService.Settings(prompt=...)instead. Will be removed in 2.0.0.turn_detection – Server-side VAD configuration. Defaults to
False(disabled), which relies on a local VAD processor in the pipeline. Pass a dict to enable it — at minimum{"type": "server_vad"}, optionally with custom settings (e.g.{"type": "server_vad", "threshold": 0.5}). Requires a model that supports turn detection; seemodel.Noneomits the field entirely, leaving the session’s own default in place.noise_reduction –
Noise reduction mode.
"near_field"for close microphones,"far_field"for distant microphones, orNoneto disable.Deprecated since version 0.0.106: Use
settings=OpenAIRealtimeSTTService.Settings(noise_reduction=...)instead. Will be removed in 2.0.0.should_interrupt – Whether to interrupt bot output when speech is detected by server-side VAD. Only applies when turn detection is enabled. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied
user_turn_strategiesoverrides the recommendation and this setting with it. Defaults to True.settings – Runtime-updatable settings. When provided alongside deprecated parameters,
settingsvalues take precedence.ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark
**kwargs – Additional arguments passed to parent WebsocketSTTService.
- can_generate_metrics() bool[source]
Check if the service can generate processing metrics.
- Returns:
True, as this service supports metrics generation.
- property send_sample_rate: int
Get the rate audio is sent at, the 24 kHz the Realtime API takes.
- Returns:
The sample rate in Hz.
- service_metadata_frame() STTMetadataFrame[source]
Request external turn strategies when server-side VAD drives turns.
With server-side VAD enabled the OpenAI server detects speech boundaries and this service proposes turns from them, so the user aggregator resolves those rather than running local VAD/smart-turn. With
turn_detection=False(the default) the server emits no VAD events and the defaults are left in place. Applied unless the user passed their ownuser_turn_strategies.
- async setup(setup: FrameProcessorSetup)[source]
Set up the service and connect.
- Parameters:
setup – Configuration object containing setup parameters.
- async stop(frame: EndFrame)[source]
Stop the service and close WebSocket connection.
- Parameters:
frame – The end frame triggering service shutdown.
- async cancel(frame: CancelFrame)[source]
Cancel the service and close WebSocket connection.
- Parameters:
frame – The cancel frame triggering service cancellation.
- async run_stt(audio: bytes) AsyncGenerator[Frame | None, None][source]
Send audio data to the transcription session.
Audio is streamed over the WebSocket. Transcription results arrive asynchronously via the receive task and are pushed as
InterimTranscriptionFrameorTranscriptionFrame.- Parameters:
audio – Raw audio bytes (16-bit mono PCM at the pipeline sample rate). Automatically resampled to 24 kHz.
- Yields:
None — results are delivered via the WebSocket receive task.
- async process_frame(frame: Frame, direction: FrameDirection)[source]
Process frames from the pipeline.
Extends the base STT service to handle local VAD events when server-side VAD is disabled. On
VADUserStoppedSpeakingFrame, commits the audio buffer so the server begins transcription for the completed speech segment.- Parameters:
frame – The frame to process.
direction – The direction of frame flow in the pipeline.
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)
Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.
Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.
- Parameters:
frame – The frame to push.
direction – The direction to push the frame.