stt

Sarvam AI Speech-to-Text service implementations.

SarvamSTTService and SarvamRealtimeSTTService stream audio to Sarvam’s WebSocket APIs for Indian language speech recognition. The former covers the transcription endpoint, with Voice Activity Detection and a choice of audio formats; the latter targets the realtime endpoint, which adds server-side endpointing and in-band configuration updates.

SarvamHttpSTTService instead sends each VAD-detected speech segment to Sarvam’s synchronous REST API and yields the transcription inline, so it also suits callers that use run_stt() directly.

pipecat.services.sarvam.stt.language_to_sarvam_language(language: Language) → str[source]

Convert a Language enum to Sarvam’s language code format.

Parameters:

language – The Language enum value to convert.

Returns:

The Sarvam language code string.

class pipecat.services.sarvam.stt.ModelConfig(supports_mode: bool, supports_language: bool, default_language: str | None, default_mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None)[source]

Bases: object

Immutable configuration for a Sarvam STT model.

Parameters:
  • supports_mode – Whether the model accepts mode parameter.

  • supports_language – Whether the model accepts language parameter.

  • default_language – Default language code (None = auto-detect).

  • default_mode – Default mode (None = not applicable).

class pipecat.services.sarvam.stt.SarvamSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, vad_signals: bool | None | NotGiven = <factory>, high_vad_sensitivity: bool | None | NotGiven = <factory>, positive_speech_threshold: float | None | NotGiven = <factory>, negative_speech_threshold: float | None | NotGiven = <factory>, min_speech_frames: int | None | NotGiven = <factory>, first_turn_min_speech_frames: int | None | NotGiven = <factory>, negative_frames_count: int | None | NotGiven = <factory>, negative_frames_window: int | None | NotGiven = <factory>, start_speech_volume_threshold: float | None | NotGiven = <factory>, interrupt_min_speech_frames: int | None | NotGiven = <factory>, pre_speech_pad_frames: int | None | NotGiven = <factory>, num_initial_ignored_frames: int | None | NotGiven = <factory>)[source]

Bases: STTSettings

Settings for SarvamSTTService.

Parameters:
  • vad_signals – Enable VAD signals in response.

  • high_vad_sensitivity – Enable high VAD sensitivity.

  • positive_speech_threshold – VAD probability threshold (0.0-1.0) above which a frame is considered speech.

  • negative_speech_threshold – VAD probability threshold (0.0-1.0) below which a frame is considered silence.

  • min_speech_frames – Minimum consecutive speech frames to start a speech segment.

  • first_turn_min_speech_frames – Minimum speech frames for the first user turn.

  • negative_frames_count – Number of silence frames within the window to end a speech segment.

  • negative_frames_window – Sliding window size (in frames) for counting negative frames.

  • start_speech_volume_threshold – Volume level (dB) below which audio is too quiet to be speech.

  • interrupt_min_speech_frames – Minimum speech frames to register a barge-in/interruption.

  • pre_speech_pad_frames – Number of audio frames to prepend before detected speech onset.

  • num_initial_ignored_frames – Number of leading audio frames to skip at connection start.

class pipecat.services.sarvam.stt.SarvamSTTService(*, api_key: str, model: str | None = None, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, sample_rate: int | None = None, input_audio_codec: str = 'wav', keyterms: list[str] | None = None, params: InputParams | None = None, settings: SarvamSTTSettings | None = None, ttfs_p99_latency: float | None = 1.17, keepalive_timeout: float | None = None, keepalive_interval: float = 5.0, **kwargs)[source]

Bases: STTService

Sarvam speech-to-text service.

Provides real-time speech recognition using Sarvam’s WebSocket API.

Event handlers available (in addition to STTService events):

  • on_connected(service): Connected to Sarvam WebSocket

  • on_disconnected(service): Disconnected from Sarvam WebSocket

  • on_connection_error(service, error): Connection error occurred

Example:

@stt.event_handler("on_connected")
async def on_connected(service):
    ...
Settings

alias of SarvamSTTSettings

class InputParams(*, language: Language | None = None, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, vad_signals: bool | None = None, high_vad_sensitivity: bool | None = None)[source]

Bases: BaseModel

Configuration parameters for Sarvam STT service.

Deprecated since version 0.0.105: Use settings=SarvamSTTService.Settings(...) instead. Will be removed in 2.0.0.

Parameters:
  • language – Target language for transcription. - saaras:v3: Defaults to “unknown” (auto-detect supported) - saaras:v4: Defaults to “unknown” (auto-detect supported)

  • mode – Mode of operation for models that support it. Options: transcribe, translate, verbatim, translit, codemix. Defaults to “transcribe”.

  • vad_signals – Enable VAD signals in response. Defaults to None.

  • high_vad_sensitivity – Enable high VAD sensitivity. Defaults to None.

language: Language | None
mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None
vad_signals: bool | None
high_vad_sensitivity: bool | None
__init__(*, api_key: str, model: str | None = None, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, sample_rate: int | None = None, input_audio_codec: str = 'wav', keyterms: list[str] | None = None, params: InputParams | None = None, settings: SarvamSTTSettings | None = None, ttfs_p99_latency: float | None = 1.17, keepalive_timeout: float | None = None, keepalive_interval: float = 5.0, **kwargs)[source]

Initialize the Sarvam STT service.

Parameters:
  • api_key – Sarvam API key for authentication.

  • model –

    Sarvam model to use for transcription.

    Deprecated since version 0.0.105: Use settings=SarvamSTTService.Settings(model=...) instead. Will be removed in 2.0.0.

  • mode – Mode of operation. Options: transcribe, translate, verbatim, translit, codemix. Only applicable to models that support it. Defaults to the model’s default mode.

  • sample_rate – Audio sample rate. Defaults to 16000 if not specified.

  • input_audio_codec – Audio codec/format of the input file. Defaults to “wav”.

  • keyterms – Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item.

  • params –

    Configuration parameters for Sarvam STT service.

    Deprecated since version 0.0.105: Use settings=SarvamSTTService.Settings(...) instead. Will be removed in 2.0.0.

  • settings – Runtime-updatable settings. When provided alongside deprecated parameters, settings values take precedence.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark

  • keepalive_timeout – Seconds of no audio before sending silence to keep the connection alive. None disables keepalive.

  • keepalive_interval – Seconds between idle checks when keepalive is enabled.

  • **kwargs – Additional arguments passed to the parent STTService.

language_to_service_language(language: Language) → str[source]

Convert pipecat Language enum to Sarvam’s language code.

Parameters:

language – The Language enum value to convert.

Returns:

The Sarvam language code string.

can_generate_metrics() → bool[source]

Check if this service can generate processing metrics.

Returns:

True, as Sarvam service supports metrics generation.

service_metadata_frame() → STTMetadataFrame[source]

Request external turn strategies when Sarvam’s VAD signals drive turns.

With vad_signals enabled Sarvam detects speech boundaries server-side and this service proposes turns from them, so the user aggregator resolves those rather than running local VAD/smart-turn. Without it the defaults are left in place. Applied unless the user passed their own user_turn_strategies.

async process_frame(frame: Frame, direction: FrameDirection)[source]

Process incoming frames.

Flushes on Pipecat’s VAD turn end when Sarvam’s built-in VAD signals aren’t driving the turn.

async setup(setup: FrameProcessorSetup)[source]

Set up the service and connect.

Parameters:

setup – Configuration object containing setup parameters.

async stop(frame: EndFrame)[source]

Stop the Sarvam STT service.

Parameters:

frame – The end frame.

async cancel(frame: CancelFrame)[source]

Cancel the Sarvam STT service.

Parameters:

frame – The cancel frame.

async run_stt(audio: bytes) → AsyncGenerator[Frame | None, None][source]

Send audio data to Sarvam for transcription.

Parameters:

audio – Raw audio bytes to transcribe.

Yields:

Frame – None (transcription results come via WebSocket callbacks).

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.

Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.

Parameters:
  • frame – The frame to push.

  • direction – The direction to push the frame.

async start_stt_usage_metrics(usage: STTUsage)

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.

pipecat.services.sarvam.stt.language_to_sarvam_realtime_language(language: Language) → str[source]

Convert a Language enum to Sarvam realtime’s language code.

class pipecat.services.sarvam.stt.SarvamRealtimeSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, language_code: str | None | NotGiven = <factory>, stream_type: Literal['fast', 'balanced', 'simulated'] | ~pipecat.utils.types.NotGiven=<factory>, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | ~pipecat.utils.types.NotGiven=<factory>, prompt: str | None | NotGiven = <factory>, threshold: float | None | NotGiven = <factory>, silence_duration_ms: int | None | NotGiven = <factory>, min_speech_duration_ms: int | None | NotGiven = <factory>)[source]

Bases: STTSettings

Settings for SarvamRealtimeSTTService.

Sarvam reads these from the connection query string but also accepts them in a config.update. The values it only reads at connection time are constructor arguments on the service instead. language=None asks Sarvam to detect the language of each utterance.

Parameters:
  • language_code –

    Sarvam realtime language code or auto.

    Deprecated since version 1.13.0: Use language instead, with None for auto. Will be removed in 2.0.0.

  • stream_type – Streaming cadence: fast, balanced, or simulated.

  • mode – Realtime STT task mode.

  • prompt – Optional decoding prompt.

  • threshold – Optional VAD sensitivity threshold.

  • silence_duration_ms – Optional silence duration for end-of-speech.

  • min_speech_duration_ms – Optional minimum speech duration.

class pipecat.services.sarvam.stt.SarvamRealtimeSTTService(*, api_key: str, base_url: str = 'wss://api.sarvam.ai/speech-to-text-realtime/ws', endpointing: Literal['vad', 'manual'] = 'vad', sample_rate: int | None = None, return_timestamps: bool = False, keyterms: list[str] | None = None, prefix_padding_ms: int | None = None, settings: SarvamRealtimeSTTSettings | None = None, should_interrupt: bool = True, ttfs_p99_latency: float | None = 1.0, **kwargs)[source]

Bases: WebsocketSTTService

Sarvam realtime Speech-to-Text service.

Streams raw audio bytes to Sarvam’s realtime websocket endpoint and maps provider VAD and transcript events into Pipecat frames.

Serves saaras:v3-realtime and saaras:v4, selected with settings=SarvamRealtimeSTTService.Settings(model=...) and defaulting to saaras:v3-realtime, Sarvam’s own default.

With the default endpointing="vad" the server drives turn boundaries. With endpointing="manual" the pipeline drives them instead, so the pipeline needs a turn strategy that emits VADUserStartedSpeakingFrame and VADUserStoppedSpeakingFrame; without one, Sarvam never receives a boundary and never emits a final transcript. The mode is a constructor argument rather than a setting, since it decides which turn strategies the user aggregator runs and those are announced once, at startup.

A VAD analyzer is required in either endpointing mode. Under vad it times transcription latency: TTFB is measured from VADUserStoppedSpeakingFrame, which carries the stop delay needed to place the real end of speech, where Sarvam’s own vad.speech_end arrives only after the server’s silence window and would time a shorter interval than every other STT service reports. Under manual those same frames also mark the turn for Sarvam, reaching it as speech_start and speech_end.

A dropped socket is reopened with backoff in the background unless reconnect_on_error=False; audio that arrives while it is down isn’t transcribed. An utterance lives in one connection, so one cut short by a drop ends with the transcript received so far, and speech that continues on the new connection is transcribed as a new utterance. Settings Sarvam rejects are a permanent invalid-request error rather than retried. Once the service stops trying it is no longer usable, so a ServiceSwitcher can move on.

Settings

alias of SarvamRealtimeSTTSettings

__init__(*, api_key: str, base_url: str = 'wss://api.sarvam.ai/speech-to-text-realtime/ws', endpointing: Literal['vad', 'manual'] = 'vad', sample_rate: int | None = None, return_timestamps: bool = False, keyterms: list[str] | None = None, prefix_padding_ms: int | None = None, settings: SarvamRealtimeSTTSettings | None = None, should_interrupt: bool = True, ttfs_p99_latency: float | None = 1.0, **kwargs)[source]

Initialize Sarvam realtime STT.

Parameters:
  • api_key – Sarvam API key.

  • base_url – Realtime STT websocket endpoint.

  • endpointing – Which side detects turn boundaries: vad for Sarvam’s own detection, or manual for the pipeline’s. Decides the turn strategies this service asks the user aggregator to run, so it is fixed for the life of the service. Defaults to vad.

  • sample_rate – Declared input audio sample rate, 8000 or 16000. None adopts the pipeline’s input rate.

  • return_timestamps – Whether final transcripts should include segment offsets. Defaults to False.

  • keyterms – Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item.

  • prefix_padding_ms – Optional VAD prefix padding, used only under endpointing="vad".

  • settings – Runtime-updatable realtime settings.

  • should_interrupt – Determine whether the bot should be interrupted when Sarvam detects user speech. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied user_turn_strategies overrides the recommendation and this setting with it. Defaults to True.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds.

  • **kwargs – Additional arguments passed to WebsocketSTTService.

can_generate_metrics() → bool[source]

Check if this service can generate processing and usage metrics.

service_metadata_frame() → STTMetadataFrame[source]

Request external turn strategies when Sarvam endpoints server-side.

With endpointing="vad" (the default) Sarvam’s VAD decides turn boundaries and this service proposes them via ProposedUserStarted/StoppedSpeakingFrame, so the user aggregator resolves those rather than running local VAD/smart-turn. In endpointing="manual" the pipeline supplies the boundaries, so the defaults are left in place. Applied unless the user passed their own user_turn_strategies.

language_to_service_language(language: Language) → str[source]

Convert a Language enum to Sarvam realtime’s language code.

async setup(setup: FrameProcessorSetup)[source]

Set up the service and connect the websocket.

Parameters:

setup – Configuration object containing setup parameters.

async stop(frame: EndFrame)[source]

Stop the service and close the websocket.

async cancel(frame: CancelFrame)[source]

Cancel the service and close the websocket.

async process_frame(frame: Frame, direction: FrameDirection)[source]

Process frames and send manual endpointing boundaries when configured.

async run_stt(audio: bytes) → AsyncGenerator[Frame | None, None][source]

Buffer and send raw audio bytes to Sarvam as base64-encoded JSON messages.

async update_config(**fields: Any)[source]

Send a live Sarvam config.update message without reconnecting.

Parameters:

fields – Runtime-updatable Sarvam config values. Connection-only values are rejected; they are fixed for the life of the stream.

Raises:

ValueError – If a connection-only or invalid value is supplied.

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.

Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.

Parameters:
  • frame – The frame to push.

  • direction – The direction to push the frame.

async start_stt_usage_metrics(usage: STTUsage)

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.

class pipecat.services.sarvam.stt.SarvamHttpSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>)[source]

Bases: STTSettings

Settings for SarvamHttpSTTService.

class pipecat.services.sarvam.stt.SarvamHttpSTTService(*, api_key: str, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, keyterms: list[str] | None = None, sample_rate: int | None = None, settings: SarvamHttpSTTSettings | None = None, ttfs_p99_latency: float | None = 1.0, **kwargs)[source]

Bases: SegmentedSTTService

Sarvam speech-to-text service using the synchronous REST API.

The HTTP sibling of the WebSocket SarvamSTTService: each VAD-detected speech segment is sent to Sarvam’s REST speech-to-text endpoint and the transcription is yielded inline from run_stt(), so it suits pipelines that prefer per-segment HTTP requests over a long-lived socket, and callers that use run_stt() directly.

The REST endpoint accepts at most 30 seconds of audio per request, so a longer segment is sent as consecutive 30-second pieces whose transcripts are joined into one TranscriptionFrame. A word spoken across a piece boundary may be transcribed less accurately.

Requires VAD to be enabled in the pipeline.

Settings

alias of SarvamHttpSTTSettings

__init__(*, api_key: str, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, keyterms: list[str] | None = None, sample_rate: int | None = None, settings: SarvamHttpSTTSettings | None = None, ttfs_p99_latency: float | None = 1.0, **kwargs)[source]

Initialize the Sarvam HTTP STT service.

Parameters:
  • api_key – Sarvam API key for authentication.

  • mode – Mode of operation. Options: transcribe, translate, verbatim, translit, codemix. Only applicable to models that support it. Defaults to the model’s default mode.

  • keyterms – Recognition hints for saaras:v4, sent with every request. At most 50 terms of 64 characters each; put each phrase in a separate item.

  • sample_rate – Audio sample rate. If None, uses the pipeline’s rate. Sarvam’s REST API performs best with 16 kHz audio.

  • settings – Runtime-updatable settings. Defaults to the saaras:v4 model with automatic language detection.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark

  • **kwargs – Additional arguments passed to SegmentedSTTService.

can_generate_metrics() → bool[source]

Check if this service can generate processing metrics.

Returns:

True, as Sarvam HTTP service supports metrics generation.

language_to_service_language(language: Language) → str[source]

Convert pipecat Language enum to Sarvam’s language code.

Parameters:

language – The Language enum value to convert.

Returns:

The Sarvam language code string.

async run_stt(audio: bytes) → AsyncGenerator[Frame, None][source]

Transcribe an audio segment using Sarvam’s REST API.

A segment longer than the REST endpoint’s 30-second limit is sent as consecutive pieces, and their transcripts are joined.

Parameters:

audio – Audio segment in WAV format (already wrapped by the SegmentedSTTService base class).

Yields:

Frame – TranscriptionFrame containing the transcribed text, or ErrorFrame on failure. Empty transcriptions are not yielded.

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame, marking a TranscriptionFrame finalized when its segment is the latest.

Once the user speaks again, the transcript of an earlier segment covers only the words before they resumed, so it isn’t finalized, the same way STTService drops a pending finalize when the user starts speaking. When a segment’s transcript arrives as several diarized TranscriptionFrames (one per speaker run), only the last one can be.

Parameters:
  • frame – The frame to push.

  • direction – The direction of frame flow in the pipeline.

async start_stt_usage_metrics(usage: STTUsage)

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.