stt
Smallest AI speech-to-text service implementation.
This module provides a STT service using Smallest AI’s Waves API:
SmallestSTTService: WebSocket-based real-time STT. Streams audio continuously and receives interim/final transcripts with low latency.
- pipecat.services.smallest.stt.language_to_smallest_stt_language(language: Language) str[source]
Convert a Language enum to Smallest STT language code.
- Parameters:
language – The Language enum value to convert.
- Returns:
The Smallest language code string.
- class pipecat.services.smallest.stt.SmallestSTTModel(*values)[source]
Bases:
StrEnumAvailable Smallest AI STT models.
- PULSE = 'pulse'
- PULSE_2 = 'pulse-2'
- class pipecat.services.smallest.stt.SmallestSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, word_timestamps: bool | NotGiven = <factory>, full_transcript: bool | NotGiven = <factory>, sentence_timestamps: bool | NotGiven = <factory>, redact_pii: bool | NotGiven = <factory>, redact_pci: bool | NotGiven = <factory>, numerals: str | NotGiven = <factory>, diarize: bool | NotGiven = <factory>, endpointing: bool | NotGiven = <factory>, eou_timeout_ms: int | None | NotGiven = <factory>, keywords: str | NotGiven = <factory>, format: bool | NotGiven = <factory>, emotion_detection: bool | NotGiven = <factory>, gender_detection: bool | NotGiven = <factory>, itn_normalize: bool | NotGiven = <factory>, no_format_delay: bool | NotGiven = <factory>, speaker_format: str | None | NotGiven = <factory>)[source]
Bases:
STTSettingsSettings for SmallestSTTService.
- Parameters:
word_timestamps – Include word-level timestamps.
full_transcript – Include cumulative transcript.
sentence_timestamps – Include sentence-level timestamps.
redact_pii – Redact personally identifiable information.
redact_pci – Redact payment card information.
numerals – Convert spoken numerals to digits.
diarize – Enable speaker diarization. Each contiguous speaker run of a final transcript is emitted as its own frame with Smallest’s speaker number as
speaker.label. Requests word timestamps, which carry the speakers. Interim transcripts are not labelled.endpointing – Finalize promptly on trailing silence.
eou_timeout_ms – Trailing-silence duration, in milliseconds, before Pulse finalizes the transcript on its own. Ranges 100-10000; defaults to 600 when
endpointingisTrueand 800 whenFalse. Governs mid-utterance segmentation (e.g. a natural pause) since the VAD-triggeredfinalizethis service sends onVADUserStoppedSpeakingFramealready finalizes the turn’s last segment immediately, ahead of this timer. No effect onmodel="pulse-2", which uses its own built-in end-of-turn detection.keywords – Comma-separated
KEYWORD:INTENSIFIERpairs to boost recognition of domain-specific words/phrases (e.g."NVIDIA:2").format – Apply punctuation and capitalization to transcripts.
emotion_detection –
model="pulse-2"only. Adds a six-class per-utteranceemotionobject to every final transcript. No effect onmodel="pulse".gender_detection –
model="pulse-2"only. Adds an acoustic per-utterancegenderestimate to every final transcript. No effect onmodel="pulse".itn_normalize – Run inverse text normalization so spoken numbers come back as digits (e.g. “twenty five” -> “25”).
no_format_delay – With
itn_normalizeon, a number still being dictated at the end of a final is held back and merged into the next final so it normalizes whole. Set toTrueto emit every final as soon as it is ready instead. No effect unlessitn_normalizeisTrue; text is never dropped either way.speaker_format – Optional format string for diarized speaker runs. Use
{speaker}for the speaker label,{text}for the transcript text, and{speaker_number}for the speaker’s order of first appearance in the session (1,2, …), e.g."{speaker}: {text}". IfNone, transcript text is not modified. Applied locally, so changing it does not reconnect.
- class pipecat.services.smallest.stt.SmallestSTTService(*, api_key: str, base_url: str = 'wss://api.smallest.ai', encoding: str = 'linear16', sample_rate: int | None = None, settings: SmallestSTTSettings | None = None, ttfs_p99_latency: float | None = 1.59, **kwargs)[source]
Bases:
WebsocketSTTServiceSmallest AI real-time speech-to-text service using the Pulse WebSocket API.
Streams audio continuously over a WebSocket connection and receives interim and final transcription results with low latency. Best suited for real-time voice applications where immediate feedback is needed.
Uses Pipecat’s VAD to detect when the user stops speaking and sends a
finalizemessage to flush the final transcript while keeping the session alive for the next utterance.Connects to
wss://api.smallest.ai/waves/v1/stt/live?model=pulse. Setmodel=SmallestSTTModel.PULSE_2in the settings for Pulse 2.0, an English-only model that adds per-utteranceemotionandgenderto final transcripts (seeemotion_detection/gender_detection, on by default). Both appear in theTranscriptionFrameresultand are omitted for very short utterances. Pulse 2.0 also has built-in end-of-turn detection, but Pipecat’s VAD and user turn strategies still decide when the user’s turn ends, and the VAD-triggeredfinalizeflushes the final transcript as with Pulse.endpointinghas no effect on this model.Example:
stt = SmallestSTTService( api_key="your-api-key", settings=SmallestSTTService.Settings( language=Language.EN, word_timestamps=True, ), )
- Settings
alias of
SmallestSTTSettings
- __init__(*, api_key: str, base_url: str = 'wss://api.smallest.ai', encoding: str = 'linear16', sample_rate: int | None = None, settings: SmallestSTTSettings | None = None, ttfs_p99_latency: float | None = 1.59, **kwargs)[source]
Initialize the Smallest AI STT service.
- Parameters:
api_key – Smallest AI API key for authentication.
base_url – Base WebSocket URL for the Smallest API.
encoding – Audio encoding format. Defaults to “linear16”.
sample_rate – Audio sample rate in Hz. If None, uses the pipeline’s rate.
settings – Runtime-updatable settings for the STT service.
ttfs_p99_latency – P99 latency from speech end to final transcript in seconds.
**kwargs – Additional arguments passed to WebsocketSTTService.
- language_to_service_language(language: Language) str | None[source]
Convert a Language enum to Smallest service language format.
- Parameters:
language – The language to convert.
- Returns:
The Smallest-specific language code, or None if not supported.
- async setup(setup: FrameProcessorSetup)[source]
Set up the service and connect.
- Parameters:
setup – Configuration object containing setup parameters.
- async cancel(frame: CancelFrame)[source]
Cancel the service and disconnect from the WebSocket.
- async process_frame(frame: Frame, direction: FrameDirection)[source]
Process frames, handling VAD events for finalization.
- async run_stt(audio: bytes) AsyncGenerator[Frame | None, None][source]
Send audio to the Smallest Pulse WebSocket for transcription.
- Parameters:
audio – Raw PCM audio bytes.
- Yields:
None – transcription results arrive via WebSocket messages.
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)
Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.
Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.
- Parameters:
frame – The frame to push.
direction – The direction to push the frame.