stt

Smallest AI speech-to-text service implementation.

This module provides a STT service using Smallest AI’s Waves API:

  • SmallestSTTService: WebSocket-based real-time STT. Streams audio continuously and receives interim/final transcripts with low latency.

pipecat.services.smallest.stt.language_to_smallest_stt_language(language: Language) → str[source]

Convert a Language enum to Smallest STT language code.

Parameters:

language – The Language enum value to convert.

Returns:

The Smallest language code string.

class pipecat.services.smallest.stt.SmallestSTTModel(*values)[source]

Bases: StrEnum

Available Smallest AI STT models.

PULSE = 'pulse'
PULSE_2 = 'pulse-2'
class pipecat.services.smallest.stt.SmallestSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, word_timestamps: bool | NotGiven = <factory>, full_transcript: bool | NotGiven = <factory>, sentence_timestamps: bool | NotGiven = <factory>, redact_pii: bool | NotGiven = <factory>, redact_pci: bool | NotGiven = <factory>, numerals: str | NotGiven = <factory>, diarize: bool | NotGiven = <factory>, endpointing: bool | NotGiven = <factory>, eou_timeout_ms: int | None | NotGiven = <factory>, keywords: str | NotGiven = <factory>, format: bool | NotGiven = <factory>, emotion_detection: bool | NotGiven = <factory>, gender_detection: bool | NotGiven = <factory>, itn_normalize: bool | NotGiven = <factory>, no_format_delay: bool | NotGiven = <factory>, speaker_format: str | None | NotGiven = <factory>)[source]

Bases: STTSettings

Settings for SmallestSTTService.

Parameters:
  • word_timestamps – Include word-level timestamps.

  • full_transcript – Include cumulative transcript.

  • sentence_timestamps – Include sentence-level timestamps.

  • redact_pii – Redact personally identifiable information.

  • redact_pci – Redact payment card information.

  • numerals – Convert spoken numerals to digits.

  • diarize – Enable speaker diarization. Each contiguous speaker run of a final transcript is emitted as its own frame with Smallest’s speaker number as speaker.label. Requests word timestamps, which carry the speakers. Interim transcripts are not labelled.

  • endpointing – Finalize promptly on trailing silence.

  • eou_timeout_ms – Trailing-silence duration, in milliseconds, before Pulse finalizes the transcript on its own. Ranges 100-10000; defaults to 600 when endpointing is True and 800 when False. Governs mid-utterance segmentation (e.g. a natural pause) since the VAD-triggered finalize this service sends on VADUserStoppedSpeakingFrame already finalizes the turn’s last segment immediately, ahead of this timer. No effect on model="pulse-2", which uses its own built-in end-of-turn detection.

  • keywords – Comma-separated KEYWORD:INTENSIFIER pairs to boost recognition of domain-specific words/phrases (e.g. "NVIDIA:2").

  • format – Apply punctuation and capitalization to transcripts.

  • emotion_detection – model="pulse-2" only. Adds a six-class per-utterance emotion object to every final transcript. No effect on model="pulse".

  • gender_detection – model="pulse-2" only. Adds an acoustic per-utterance gender estimate to every final transcript. No effect on model="pulse".

  • itn_normalize – Run inverse text normalization so spoken numbers come back as digits (e.g. “twenty five” -> “25”).

  • no_format_delay – With itn_normalize on, a number still being dictated at the end of a final is held back and merged into the next final so it normalizes whole. Set to True to emit every final as soon as it is ready instead. No effect unless itn_normalize is True; text is never dropped either way.

  • speaker_format – Optional format string for diarized speaker runs. Use {speaker} for the speaker label, {text} for the transcript text, and {speaker_number} for the speaker’s order of first appearance in the session (1, 2, …), e.g. "{speaker}: {text}". If None, transcript text is not modified. Applied locally, so changing it does not reconnect.

LOCAL_FIELDS: ClassVar[frozenset[str]] = frozenset({'speaker_format'})

Fields applied locally to transcripts — changing them needs no reconnect.

class pipecat.services.smallest.stt.SmallestSTTService(*, api_key: str, base_url: str = 'wss://api.smallest.ai', encoding: str = 'linear16', sample_rate: int | None = None, settings: SmallestSTTSettings | None = None, ttfs_p99_latency: float | None = 1.59, **kwargs)[source]

Bases: WebsocketSTTService

Smallest AI real-time speech-to-text service using the Pulse WebSocket API.

Streams audio continuously over a WebSocket connection and receives interim and final transcription results with low latency. Best suited for real-time voice applications where immediate feedback is needed.

Uses Pipecat’s VAD to detect when the user stops speaking and sends a finalize message to flush the final transcript while keeping the session alive for the next utterance.

Connects to wss://api.smallest.ai/waves/v1/stt/live?model=pulse. Set model=SmallestSTTModel.PULSE_2 in the settings for Pulse 2.0, an English-only model that adds per-utterance emotion and gender to final transcripts (see emotion_detection/gender_detection, on by default). Both appear in the TranscriptionFrame result and are omitted for very short utterances. Pulse 2.0 also has built-in end-of-turn detection, but Pipecat’s VAD and user turn strategies still decide when the user’s turn ends, and the VAD-triggered finalize flushes the final transcript as with Pulse. endpointing has no effect on this model.

Example:

stt = SmallestSTTService(
    api_key="your-api-key",
    settings=SmallestSTTService.Settings(
        language=Language.EN,
        word_timestamps=True,
    ),
)
Settings

alias of SmallestSTTSettings

__init__(*, api_key: str, base_url: str = 'wss://api.smallest.ai', encoding: str = 'linear16', sample_rate: int | None = None, settings: SmallestSTTSettings | None = None, ttfs_p99_latency: float | None = 1.59, **kwargs)[source]

Initialize the Smallest AI STT service.

Parameters:
  • api_key – Smallest AI API key for authentication.

  • base_url – Base WebSocket URL for the Smallest API.

  • encoding – Audio encoding format. Defaults to “linear16”.

  • sample_rate – Audio sample rate in Hz. If None, uses the pipeline’s rate.

  • settings – Runtime-updatable settings for the STT service.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds.

  • **kwargs – Additional arguments passed to WebsocketSTTService.

can_generate_metrics() → bool[source]

Check if this service can generate processing metrics.

language_to_service_language(language: Language) → str | None[source]

Convert a Language enum to Smallest service language format.

Parameters:

language – The language to convert.

Returns:

The Smallest-specific language code, or None if not supported.

async setup(setup: FrameProcessorSetup)[source]

Set up the service and connect.

Parameters:

setup – Configuration object containing setup parameters.

async stop(frame: EndFrame)[source]

Stop the service and disconnect from the WebSocket.

async cancel(frame: CancelFrame)[source]

Cancel the service and disconnect from the WebSocket.

async process_frame(frame: Frame, direction: FrameDirection)[source]

Process frames, handling VAD events for finalization.

async run_stt(audio: bytes) → AsyncGenerator[Frame | None, None][source]

Send audio to the Smallest Pulse WebSocket for transcription.

Parameters:

audio – Raw PCM audio bytes.

Yields:

None – transcription results arrive via WebSocket messages.

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.

Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.

Parameters:
  • frame – The frame to push.

  • direction – The direction to push the frame.

async start_stt_usage_metrics(usage: STTUsage)

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.