stt

OpenAI Speech-to-Text service implementations.

Provides two STT services:

  • OpenAISTTService: REST-based transcription using the Audio API (Whisper / GPT transcription models).

  • OpenAIRealtimeSTTService: WebSocket-based streaming transcription using the Realtime API in transcription-only mode.

class pipecat.services.openai.stt.OpenAISTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, prompt: str | None | NotGiven = <factory>, temperature: float | None | NotGiven = <factory>, keywords: list[str] | None | NotGiven = <factory>, speaker_format: str | None | NotGiven = <factory>)[source]

Bases: BaseWhisperSTTSettings

Settings for the OpenAI STT service.

Parameters:
  • keywords – Words or phrases that guide transcription of the input audio, such as product names or acronyms. Supported by "gpt-transcribe".

  • speaker_format – Optional format string for diarized speaker runs. Use {speaker} for the speaker label, {text} for the transcript text, and {speaker_number} for the speaker’s order of first appearance in the session (1, 2, …), e.g. "{speaker}: {text}". If None, transcript text is not modified.

class pipecat.services.openai.stt.OpenAISTTService(*, model: str | None = None, api_key: str | None = None, base_url: str | None = None, language: Language | None = Language.EN, prompt: str | None = None, temperature: float | None = None, settings: OpenAISTTSettings | None = None, ttfs_p99_latency: float | None = 2.01, **kwargs)[source]

Bases: BaseWhisperSTTService

OpenAI Speech-to-Text service that generates text from audio.

Uses OpenAI’s transcription API to convert audio to text. Requires an OpenAI API key set via the api_key parameter or OPENAI_API_KEY environment variable.

With include_prob_metrics=True, GPT transcription models report per-token logprobs and Whisper models report per-segment logprobs. Diarization models support neither, so they report no probabilities.

Diarization models (e.g. "gpt-4o-transcribe-diarize") label each speaker: every contiguous speaker run of a segment is emitted as its own frame with OpenAI’s speaker label (e.g. "A") as speaker.label.

Settings

alias of OpenAISTTSettings

__init__(*, model: str | None = None, api_key: str | None = None, base_url: str | None = None, language: Language | None = Language.EN, prompt: str | None = None, temperature: float | None = None, settings: OpenAISTTSettings | None = None, ttfs_p99_latency: float | None = 2.01, **kwargs)[source]

Initialize OpenAI STT service.

Parameters:
  • model –

    Transcription model to use. Defaults to "gpt-transcribe".

    Deprecated since version 0.0.105: Use settings=OpenAISTTService.Settings(model=...) instead. Will be removed in 2.0.0.

  • api_key – OpenAI API key. Defaults to None.

  • base_url – API base URL. Defaults to None.

  • language –

    Language of the audio input. Defaults to English.

    Deprecated since version 0.0.105: Use settings=OpenAISTTService.Settings(language=...) instead. Will be removed in 2.0.0.

  • prompt –

    Optional text to guide the model’s style or continue a previous segment.

    Deprecated since version 0.0.105: Use settings=OpenAISTTService.Settings(prompt=...) instead. Will be removed in 2.0.0.

  • temperature –

    Optional sampling temperature between 0 and 1. Defaults to 0.0.

    Deprecated since version 0.0.105: Use settings=OpenAISTTService.Settings(temperature=...) instead. Will be removed in 2.0.0.

  • settings – Runtime-updatable settings. When provided alongside deprecated parameters, settings values take precedence.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark

  • **kwargs – Additional arguments passed to BaseWhisperSTTService.

class pipecat.services.openai.stt.OpenAIRealtimeSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, prompt: str | None | NotGiven = <factory>, noise_reduction: Literal['near_field', 'far_field'] | None | ~pipecat.utils.types.NotGiven=<factory>, delay: Literal['minimal', 'low', 'medium', 'high', 'xhigh'] | None | ~pipecat.utils.types.NotGiven=<factory>)[source]

Bases: STTSettings

Settings for OpenAIRealtimeSTTService.

Parameters:
  • prompt – Optional prompt text to guide transcription style. Not supported by "gpt-realtime-whisper".

  • noise_reduction – Noise reduction mode. "near_field" for close microphones, "far_field" for distant microphones, or None to disable.

  • delay – How long the model waits before emitting transcription text, trading latency for accuracy. Only supported by "gpt-realtime-whisper".

class pipecat.services.openai.stt.OpenAIRealtimeSTTService(*, api_key: str, model: str | None = None, base_url: str = 'wss://api.openai.com/v1/realtime', language: Language | None = Language.EN, prompt: str | None = None, turn_detection: dict | Literal[False] | None = False, noise_reduction: Literal['near_field', 'far_field'] | None = None, should_interrupt: bool = True, settings: OpenAIRealtimeSTTSettings | None = None, ttfs_p99_latency: float | None = 1.66, **kwargs)[source]

Bases: WebsocketSTTService

OpenAI Realtime Speech-to-Text service using WebSocket transcription sessions.

Uses OpenAI’s Realtime API in transcription-only mode for real-time streaming speech recognition with optional server-side VAD and noise reduction. The model does not generate conversational responses — only transcription output.

This service supports two VAD modes:

Local VAD (default): Disable server-side VAD and use a local VAD processor in the pipeline instead. When a VADUserStoppedSpeakingFrame is received, the service commits the audio buffer so that the server begins transcription for the completed speech segment.

Server-side VAD (turn_detection={"type": "server_vad"}): The OpenAI server performs voice-activity detection. The service proposes turn boundaries when the server detects them, and recommends the external user turn strategies that resolve those proposals into turn frames. Do not use a separate VAD processor in the pipeline in this mode. Requires a model that supports turn detection — gpt-transcribe does, the default gpt-realtime-whisper does not.

Audio is sent as 24 kHz 16-bit mono PCM as required by the OpenAI Realtime API. If the pipeline runs at a different sample rate (e.g. 16 kHz for Silero VAD compatibility), audio is automatically upsampled before sending.

Example:

stt = OpenAIRealtimeSTTService(
    api_key="sk-...",
    settings=OpenAIRealtimeSTTService.Settings(
        model="gpt-realtime-whisper",
        noise_reduction="near_field",
    ),
)
Settings

alias of OpenAIRealtimeSTTSettings

__init__(*, api_key: str, model: str | None = None, base_url: str = 'wss://api.openai.com/v1/realtime', language: Language | None = Language.EN, prompt: str | None = None, turn_detection: dict | Literal[False] | None = False, noise_reduction: Literal['near_field', 'far_field'] | None = None, should_interrupt: bool = True, settings: OpenAIRealtimeSTTSettings | None = None, ttfs_p99_latency: float | None = 1.66, **kwargs)[source]

Initialize the OpenAI Realtime STT service.

Parameters:
  • api_key – OpenAI API key for authentication.

  • model –

    Transcription model. For low-latency streaming transcription, use "gpt-realtime-whisper". Other supported transcription models include "gpt-transcribe" and "gpt-live-transcribe".

    Deprecated since version 0.0.105: Use settings=OpenAIRealtimeSTTService.Settings(model=...) instead. Will be removed in 2.0.0.

  • base_url – WebSocket base URL for the Realtime API. Defaults to "wss://api.openai.com/v1/realtime".

  • language –

    Language of the audio input. Defaults to English.

    Deprecated since version 0.0.105: Use settings=OpenAIRealtimeSTTService.Settings(language=...) instead. Will be removed in 2.0.0.

  • prompt –

    Optional prompt text to guide transcription style or provide keyword hints. Not supported by "gpt-realtime-whisper".

    Deprecated since version 0.0.105: Use settings=OpenAIRealtimeSTTService.Settings(prompt=...) instead. Will be removed in 2.0.0.

  • turn_detection – Server-side VAD configuration. Defaults to False (disabled), which relies on a local VAD processor in the pipeline. Pass a dict to enable it — at minimum {"type": "server_vad"}, optionally with custom settings (e.g. {"type": "server_vad", "threshold": 0.5}). Requires a model that supports turn detection; see model. None omits the field entirely, leaving the session’s own default in place.

  • noise_reduction –

    Noise reduction mode. "near_field" for close microphones, "far_field" for distant microphones, or None to disable.

    Deprecated since version 0.0.106: Use settings=OpenAIRealtimeSTTService.Settings(noise_reduction=...) instead. Will be removed in 2.0.0.

  • should_interrupt – Whether to interrupt bot output when speech is detected by server-side VAD. Only applies when turn detection is enabled. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied user_turn_strategies overrides the recommendation and this setting with it. Defaults to True.

  • settings – Runtime-updatable settings. When provided alongside deprecated parameters, settings values take precedence.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark

  • **kwargs – Additional arguments passed to parent WebsocketSTTService.

can_generate_metrics() → bool[source]

Check if the service can generate processing metrics.

Returns:

True, as this service supports metrics generation.

property send_sample_rate: int

Get the rate audio is sent at, the 24 kHz the Realtime API takes.

Returns:

The sample rate in Hz.

service_metadata_frame() → STTMetadataFrame[source]

Request external turn strategies when server-side VAD drives turns.

With server-side VAD enabled the OpenAI server detects speech boundaries and this service proposes turns from them, so the user aggregator resolves those rather than running local VAD/smart-turn. With turn_detection=False (the default) the server emits no VAD events and the defaults are left in place. Applied unless the user passed their own user_turn_strategies.

async setup(setup: FrameProcessorSetup)[source]

Set up the service and connect.

Parameters:

setup – Configuration object containing setup parameters.

async stop(frame: EndFrame)[source]

Stop the service and close WebSocket connection.

Parameters:

frame – The end frame triggering service shutdown.

async cancel(frame: CancelFrame)[source]

Cancel the service and close WebSocket connection.

Parameters:

frame – The cancel frame triggering service cancellation.

async run_stt(audio: bytes) → AsyncGenerator[Frame | None, None][source]

Send audio data to the transcription session.

Audio is streamed over the WebSocket. Transcription results arrive asynchronously via the receive task and are pushed as InterimTranscriptionFrame or TranscriptionFrame.

Parameters:

audio – Raw audio bytes (16-bit mono PCM at the pipeline sample rate). Automatically resampled to 24 kHz.

Yields:

None — results are delivered via the WebSocket receive task.

async process_frame(frame: Frame, direction: FrameDirection)[source]

Process frames from the pipeline.

Extends the base STT service to handle local VAD events when server-side VAD is disabled. On VADUserStoppedSpeakingFrame, commits the audio buffer so the server begins transcription for the completed speech segment.

Parameters:
  • frame – The frame to process.

  • direction – The direction of frame flow in the pipeline.

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.

Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.

Parameters:
  • frame – The frame to push.

  • direction – The direction to push the frame.

async start_stt_usage_metrics(usage: STTUsage)

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.