stt
Sarvam AI Speech-to-Text service implementations.
SarvamSTTService and SarvamRealtimeSTTService stream audio to
Sarvam’s WebSocket APIs for Indian language speech recognition. The former
covers the transcription endpoint, with Voice Activity Detection and a choice
of audio formats; the latter targets the realtime endpoint, which adds
server-side endpointing and in-band configuration updates.
SarvamHttpSTTService instead sends each VAD-detected speech segment to
Sarvam’s synchronous REST API and yields the transcription inline, so it also
suits callers that use run_stt() directly.
- pipecat.services.sarvam.stt.language_to_sarvam_language(language: Language) str[source]
Convert a Language enum to Sarvam’s language code format.
- Parameters:
language – The Language enum value to convert.
- Returns:
The Sarvam language code string.
- class pipecat.services.sarvam.stt.ModelConfig(supports_mode: bool, supports_language: bool, default_language: str | None, default_mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None)[source]
Bases:
objectImmutable configuration for a Sarvam STT model.
- Parameters:
supports_mode – Whether the model accepts mode parameter.
supports_language – Whether the model accepts language parameter.
default_language – Default language code (None = auto-detect).
default_mode – Default mode (None = not applicable).
- class pipecat.services.sarvam.stt.SarvamSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, vad_signals: bool | None | NotGiven = <factory>, high_vad_sensitivity: bool | None | NotGiven = <factory>, positive_speech_threshold: float | None | NotGiven = <factory>, negative_speech_threshold: float | None | NotGiven = <factory>, min_speech_frames: int | None | NotGiven = <factory>, first_turn_min_speech_frames: int | None | NotGiven = <factory>, negative_frames_count: int | None | NotGiven = <factory>, negative_frames_window: int | None | NotGiven = <factory>, start_speech_volume_threshold: float | None | NotGiven = <factory>, interrupt_min_speech_frames: int | None | NotGiven = <factory>, pre_speech_pad_frames: int | None | NotGiven = <factory>, num_initial_ignored_frames: int | None | NotGiven = <factory>)[source]
Bases:
STTSettingsSettings for SarvamSTTService.
- Parameters:
vad_signals – Enable VAD signals in response.
high_vad_sensitivity – Enable high VAD sensitivity.
positive_speech_threshold – VAD probability threshold (0.0-1.0) above which a frame is considered speech.
negative_speech_threshold – VAD probability threshold (0.0-1.0) below which a frame is considered silence.
min_speech_frames – Minimum consecutive speech frames to start a speech segment.
first_turn_min_speech_frames – Minimum speech frames for the first user turn.
negative_frames_count – Number of silence frames within the window to end a speech segment.
negative_frames_window – Sliding window size (in frames) for counting negative frames.
start_speech_volume_threshold – Volume level (dB) below which audio is too quiet to be speech.
interrupt_min_speech_frames – Minimum speech frames to register a barge-in/interruption.
pre_speech_pad_frames – Number of audio frames to prepend before detected speech onset.
num_initial_ignored_frames – Number of leading audio frames to skip at connection start.
- class pipecat.services.sarvam.stt.SarvamSTTService(*, api_key: str, model: str | None = None, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, sample_rate: int | None = None, input_audio_codec: str = 'wav', keyterms: list[str] | None = None, params: InputParams | None = None, settings: SarvamSTTSettings | None = None, ttfs_p99_latency: float | None = 1.17, keepalive_timeout: float | None = None, keepalive_interval: float = 5.0, **kwargs)[source]
Bases:
STTServiceSarvam speech-to-text service.
Provides real-time speech recognition using Sarvam’s WebSocket API.
Event handlers available (in addition to STTService events):
on_connected(service): Connected to Sarvam WebSocket
on_disconnected(service): Disconnected from Sarvam WebSocket
on_connection_error(service, error): Connection error occurred
Example:
@stt.event_handler("on_connected") async def on_connected(service): ...
- Settings
alias of
SarvamSTTSettings
- class InputParams(*, language: Language | None = None, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, vad_signals: bool | None = None, high_vad_sensitivity: bool | None = None)[source]
Bases:
BaseModelConfiguration parameters for Sarvam STT service.
Deprecated since version 0.0.105: Use
settings=SarvamSTTService.Settings(...)instead. Will be removed in 2.0.0.- Parameters:
language – Target language for transcription. - saaras:v3: Defaults to “unknown” (auto-detect supported) - saaras:v4: Defaults to “unknown” (auto-detect supported)
mode – Mode of operation for models that support it. Options: transcribe, translate, verbatim, translit, codemix. Defaults to “transcribe”.
vad_signals – Enable VAD signals in response. Defaults to None.
high_vad_sensitivity – Enable high VAD sensitivity. Defaults to None.
- __init__(*, api_key: str, model: str | None = None, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, sample_rate: int | None = None, input_audio_codec: str = 'wav', keyterms: list[str] | None = None, params: InputParams | None = None, settings: SarvamSTTSettings | None = None, ttfs_p99_latency: float | None = 1.17, keepalive_timeout: float | None = None, keepalive_interval: float = 5.0, **kwargs)[source]
Initialize the Sarvam STT service.
- Parameters:
api_key – Sarvam API key for authentication.
model –
Sarvam model to use for transcription.
Deprecated since version 0.0.105: Use
settings=SarvamSTTService.Settings(model=...)instead. Will be removed in 2.0.0.mode – Mode of operation. Options: transcribe, translate, verbatim, translit, codemix. Only applicable to models that support it. Defaults to the model’s default mode.
sample_rate – Audio sample rate. Defaults to 16000 if not specified.
input_audio_codec – Audio codec/format of the input file. Defaults to “wav”.
keyterms – Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item.
params –
Configuration parameters for Sarvam STT service.
Deprecated since version 0.0.105: Use
settings=SarvamSTTService.Settings(...)instead. Will be removed in 2.0.0.settings – Runtime-updatable settings. When provided alongside deprecated parameters,
settingsvalues take precedence.ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark
keepalive_timeout – Seconds of no audio before sending silence to keep the connection alive. None disables keepalive.
keepalive_interval – Seconds between idle checks when keepalive is enabled.
**kwargs – Additional arguments passed to the parent STTService.
- language_to_service_language(language: Language) str[source]
Convert pipecat Language enum to Sarvam’s language code.
- Parameters:
language – The Language enum value to convert.
- Returns:
The Sarvam language code string.
- can_generate_metrics() bool[source]
Check if this service can generate processing metrics.
- Returns:
True, as Sarvam service supports metrics generation.
- service_metadata_frame() STTMetadataFrame[source]
Request external turn strategies when Sarvam’s VAD signals drive turns.
With
vad_signalsenabled Sarvam detects speech boundaries server-side and this service proposes turns from them, so the user aggregator resolves those rather than running local VAD/smart-turn. Without it the defaults are left in place. Applied unless the user passed their ownuser_turn_strategies.
- async process_frame(frame: Frame, direction: FrameDirection)[source]
Process incoming frames.
Flushes on Pipecat’s VAD turn end when Sarvam’s built-in VAD signals aren’t driving the turn.
- async setup(setup: FrameProcessorSetup)[source]
Set up the service and connect.
- Parameters:
setup – Configuration object containing setup parameters.
- async stop(frame: EndFrame)[source]
Stop the Sarvam STT service.
- Parameters:
frame – The end frame.
- async cancel(frame: CancelFrame)[source]
Cancel the Sarvam STT service.
- Parameters:
frame – The cancel frame.
- async run_stt(audio: bytes) AsyncGenerator[Frame | None, None][source]
Send audio data to Sarvam for transcription.
- Parameters:
audio – Raw audio bytes to transcribe.
- Yields:
Frame – None (transcription results come via WebSocket callbacks).
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)
Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.
Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.
- Parameters:
frame – The frame to push.
direction – The direction to push the frame.
- pipecat.services.sarvam.stt.language_to_sarvam_realtime_language(language: Language) str[source]
Convert a Language enum to Sarvam realtime’s language code.
- class pipecat.services.sarvam.stt.SarvamRealtimeSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, language_code: str | None | NotGiven = <factory>, stream_type: Literal['fast', 'balanced', 'simulated'] | ~pipecat.utils.types.NotGiven=<factory>, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | ~pipecat.utils.types.NotGiven=<factory>, prompt: str | None | NotGiven = <factory>, threshold: float | None | NotGiven = <factory>, silence_duration_ms: int | None | NotGiven = <factory>, min_speech_duration_ms: int | None | NotGiven = <factory>)[source]
Bases:
STTSettingsSettings for SarvamRealtimeSTTService.
Sarvam reads these from the connection query string but also accepts them in a
config.update. The values it only reads at connection time are constructor arguments on the service instead.language=Noneasks Sarvam to detect the language of each utterance.- Parameters:
language_code –
Sarvam realtime language code or
auto.Deprecated since version 1.13.0: Use
languageinstead, withNoneforauto. Will be removed in 2.0.0.stream_type – Streaming cadence:
fast,balanced, orsimulated.mode – Realtime STT task mode.
prompt – Optional decoding prompt.
threshold – Optional VAD sensitivity threshold.
silence_duration_ms – Optional silence duration for end-of-speech.
min_speech_duration_ms – Optional minimum speech duration.
- class pipecat.services.sarvam.stt.SarvamRealtimeSTTService(*, api_key: str, base_url: str = 'wss://api.sarvam.ai/speech-to-text-realtime/ws', endpointing: Literal['vad', 'manual'] = 'vad', sample_rate: int | None = None, return_timestamps: bool = False, keyterms: list[str] | None = None, prefix_padding_ms: int | None = None, settings: SarvamRealtimeSTTSettings | None = None, should_interrupt: bool = True, ttfs_p99_latency: float | None = 1.0, **kwargs)[source]
Bases:
WebsocketSTTServiceSarvam realtime Speech-to-Text service.
Streams raw audio bytes to Sarvam’s realtime websocket endpoint and maps provider VAD and transcript events into Pipecat frames.
Serves
saaras:v3-realtimeandsaaras:v4, selected withsettings=SarvamRealtimeSTTService.Settings(model=...)and defaulting tosaaras:v3-realtime, Sarvam’s own default.With the default
endpointing="vad"the server drives turn boundaries. Withendpointing="manual"the pipeline drives them instead, so the pipeline needs a turn strategy that emitsVADUserStartedSpeakingFrameandVADUserStoppedSpeakingFrame; without one, Sarvam never receives a boundary and never emits a final transcript. The mode is a constructor argument rather than a setting, since it decides which turn strategies the user aggregator runs and those are announced once, at startup.A VAD analyzer is required in either
endpointingmode. Undervadit times transcription latency: TTFB is measured fromVADUserStoppedSpeakingFrame, which carries the stop delay needed to place the real end of speech, where Sarvam’s ownvad.speech_endarrives only after the server’s silence window and would time a shorter interval than every other STT service reports. Undermanualthose same frames also mark the turn for Sarvam, reaching it asspeech_startandspeech_end.A dropped socket is reopened with backoff in the background unless
reconnect_on_error=False; audio that arrives while it is down isn’t transcribed. An utterance lives in one connection, so one cut short by a drop ends with the transcript received so far, and speech that continues on the new connection is transcribed as a new utterance. Settings Sarvam rejects are a permanent invalid-request error rather than retried. Once the service stops trying it is no longer usable, so aServiceSwitchercan move on.- Settings
alias of
SarvamRealtimeSTTSettings
- __init__(*, api_key: str, base_url: str = 'wss://api.sarvam.ai/speech-to-text-realtime/ws', endpointing: Literal['vad', 'manual'] = 'vad', sample_rate: int | None = None, return_timestamps: bool = False, keyterms: list[str] | None = None, prefix_padding_ms: int | None = None, settings: SarvamRealtimeSTTSettings | None = None, should_interrupt: bool = True, ttfs_p99_latency: float | None = 1.0, **kwargs)[source]
Initialize Sarvam realtime STT.
- Parameters:
api_key – Sarvam API key.
base_url – Realtime STT websocket endpoint.
endpointing – Which side detects turn boundaries:
vadfor Sarvam’s own detection, ormanualfor the pipeline’s. Decides the turn strategies this service asks the user aggregator to run, so it is fixed for the life of the service. Defaults tovad.sample_rate – Declared input audio sample rate, 8000 or 16000.
Noneadopts the pipeline’s input rate.return_timestamps – Whether final transcripts should include segment offsets. Defaults to False.
keyterms – Connection-time recognition hints for saaras:v4. At most 50 terms of 64 characters each; put each phrase in a separate item.
prefix_padding_ms – Optional VAD prefix padding, used only under
endpointing="vad".settings – Runtime-updatable realtime settings.
should_interrupt – Determine whether the bot should be interrupted when Sarvam detects user speech. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied
user_turn_strategiesoverrides the recommendation and this setting with it. Defaults to True.ttfs_p99_latency – P99 latency from speech end to final transcript in seconds.
**kwargs – Additional arguments passed to
WebsocketSTTService.
- can_generate_metrics() bool[source]
Check if this service can generate processing and usage metrics.
- service_metadata_frame() STTMetadataFrame[source]
Request external turn strategies when Sarvam endpoints server-side.
With
endpointing="vad"(the default) Sarvam’s VAD decides turn boundaries and this service proposes them viaProposedUserStarted/StoppedSpeakingFrame, so the user aggregator resolves those rather than running local VAD/smart-turn. Inendpointing="manual"the pipeline supplies the boundaries, so the defaults are left in place. Applied unless the user passed their ownuser_turn_strategies.
- language_to_service_language(language: Language) str[source]
Convert a Language enum to Sarvam realtime’s language code.
- async setup(setup: FrameProcessorSetup)[source]
Set up the service and connect the websocket.
- Parameters:
setup – Configuration object containing setup parameters.
- async cancel(frame: CancelFrame)[source]
Cancel the service and close the websocket.
- async process_frame(frame: Frame, direction: FrameDirection)[source]
Process frames and send manual endpointing boundaries when configured.
- async run_stt(audio: bytes) AsyncGenerator[Frame | None, None][source]
Buffer and send raw audio bytes to Sarvam as base64-encoded JSON messages.
- async update_config(**fields: Any)[source]
Send a live Sarvam
config.updatemessage without reconnecting.- Parameters:
fields – Runtime-updatable Sarvam config values. Connection-only values are rejected; they are fixed for the life of the stream.
- Raises:
ValueError – If a connection-only or invalid value is supplied.
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)
Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.
Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.
- Parameters:
frame – The frame to push.
direction – The direction to push the frame.
- class pipecat.services.sarvam.stt.SarvamHttpSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>)[source]
Bases:
STTSettingsSettings for SarvamHttpSTTService.
- class pipecat.services.sarvam.stt.SarvamHttpSTTService(*, api_key: str, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, keyterms: list[str] | None = None, sample_rate: int | None = None, settings: SarvamHttpSTTSettings | None = None, ttfs_p99_latency: float | None = 1.0, **kwargs)[source]
Bases:
SegmentedSTTServiceSarvam speech-to-text service using the synchronous REST API.
The HTTP sibling of the WebSocket
SarvamSTTService: each VAD-detected speech segment is sent to Sarvam’s REST speech-to-text endpoint and the transcription is yielded inline fromrun_stt(), so it suits pipelines that prefer per-segment HTTP requests over a long-lived socket, and callers that userun_stt()directly.The REST endpoint accepts at most 30 seconds of audio per request, so a longer segment is sent as consecutive 30-second pieces whose transcripts are joined into one
TranscriptionFrame. A word spoken across a piece boundary may be transcribed less accurately.Requires VAD to be enabled in the pipeline.
- Settings
alias of
SarvamHttpSTTSettings
- __init__(*, api_key: str, mode: Literal['transcribe', 'translate', 'verbatim', 'translit', 'codemix'] | None = None, keyterms: list[str] | None = None, sample_rate: int | None = None, settings: SarvamHttpSTTSettings | None = None, ttfs_p99_latency: float | None = 1.0, **kwargs)[source]
Initialize the Sarvam HTTP STT service.
- Parameters:
api_key – Sarvam API key for authentication.
mode – Mode of operation. Options: transcribe, translate, verbatim, translit, codemix. Only applicable to models that support it. Defaults to the model’s default mode.
keyterms – Recognition hints for saaras:v4, sent with every request. At most 50 terms of 64 characters each; put each phrase in a separate item.
sample_rate – Audio sample rate. If None, uses the pipeline’s rate. Sarvam’s REST API performs best with 16 kHz audio.
settings – Runtime-updatable settings. Defaults to the
saaras:v4model with automatic language detection.ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark
**kwargs – Additional arguments passed to SegmentedSTTService.
- can_generate_metrics() bool[source]
Check if this service can generate processing metrics.
- Returns:
True, as Sarvam HTTP service supports metrics generation.
- language_to_service_language(language: Language) str[source]
Convert pipecat Language enum to Sarvam’s language code.
- Parameters:
language – The Language enum value to convert.
- Returns:
The Sarvam language code string.
- async run_stt(audio: bytes) AsyncGenerator[Frame, None][source]
Transcribe an audio segment using Sarvam’s REST API.
A segment longer than the REST endpoint’s 30-second limit is sent as consecutive pieces, and their transcripts are joined.
- Parameters:
audio – Audio segment in WAV format (already wrapped by the SegmentedSTTService base class).
- Yields:
Frame – TranscriptionFrame containing the transcribed text, or ErrorFrame on failure. Empty transcriptions are not yielded.
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)
Push a frame, marking a TranscriptionFrame finalized when its segment is the latest.
Once the user speaks again, the transcript of an earlier segment covers only the words before they resumed, so it isn’t finalized, the same way
STTServicedrops a pending finalize when the user starts speaking. When a segment’s transcript arrives as several diarized TranscriptionFrames (one per speaker run), only the last one can be.- Parameters:
frame – The frame to push.
direction – The direction of frame flow in the pipeline.