stt

Speechmatics STT service integration.

class pipecat.services.speechmatics.stt.TurnDetectionMode(*values)[source]

Bases: StrEnum

How turn boundaries (end of speech) are detected.

VAD: the STT service runs its own VAD and closes turns itself.

EXTERNAL: turn boundaries are controlled by the pipeline — the service does not endpoint on its own, and closes a turn when a VADUserStoppedSpeakingFrame arrives (for example from Pipecat’s own VAD).

The values mirror the Agent STT SDK’s own turn-detection modes so the two never drift.

VAD = 'vad'
EXTERNAL = 'external'
class pipecat.services.speechmatics.stt.SpeechmaticsSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, domain: str | None | NotGiven = <factory>, turn_detection_mode: TurnDetectionMode | NotGiven = <factory>, speaker_format: str | None | NotGiven = <factory>, speaker_active_format: str | None | NotGiven = <factory>, known_speakers: list[SpeakerIdentifier] | NotGiven = <factory>, additional_vocab: list[AdditionalVocabEntry] | NotGiven = <factory>, operating_point: Model | str | None | NotGiven = <factory>, enable_partials: bool | None | NotGiven = <factory>, emit_sentences: bool | None | NotGiven = <factory>, punctuation_overrides: dict[str, ~typing.Any] | None | ~pipecat.utils.types.NotGiven=<factory>, transcript_filtering_config: dict[str, ~typing.Any] | None | ~pipecat.utils.types.NotGiven=<factory>, enable_diarization: bool | None | NotGiven = <factory>, speaker_sensitivity: float | None | NotGiven = <factory>, max_speakers: int | None | NotGiven = <factory>, prefer_current_speaker: bool | None | NotGiven = <factory>)[source]

Bases: STTSettings

Settings for SpeechmaticsSTTService.

See SpeechmaticsSTTService.InputParams for detailed descriptions of each field.

Parameters:
  • domain – Domain for Speechmatics API.

  • turn_detection_mode – Endpoint handling mode.

  • speaker_format – Optional format string for diarized transcripts. Use {speaker} for the speaker label, {text} for the transcript text, and {speaker_number} for the speaker’s order of first appearance in the session (1, 2, …), e.g. "{speaker}: {text}". If None, transcript text is not modified. Applied locally, so changing it does not reconnect.

  • speaker_active_format –

    Formatter for speaker ID, with the text, speaker_id, ts and lang attributes. Takes precedence over speaker_format.

    Deprecated since version 1.16.0: Use speaker_format instead. Will be removed in 2.0.0.

  • known_speakers – List of known speaker labels and identifiers.

  • additional_vocab – List of additional vocabulary entries.

  • model – Resolved transcription model (operating point). See _resolve_model.

  • operating_point –

    Alias for model.

    Deprecated since version 1.10.0: Use model instead. Will be removed in 2.0.0.

  • enable_partials – Include partial segment fragments.

  • emit_sentences – Close a segment at every end of sentence, not only at the end of the turn, so each sentence is emitted as its own final transcription.

  • punctuation_overrides – Punctuation overrides.

  • transcript_filtering_config – Post-processing applied to the transcript.

  • enable_diarization – Enable speaker diarization. Transcripts carry the Speechmatics speaker label (e.g. "S1", or the label you assigned in known_speakers) as speaker.label.

  • speaker_sensitivity – Diarization sensitivity.

  • max_speakers – Maximum number of speakers to detect.

  • prefer_current_speaker – Prefer current speaker ID.

LOCAL_FIELDS: ClassVar[frozenset[str]] = frozenset({'speaker_active_format', 'speaker_format'})

Fields that are purely local (formatting templates) — no reconnect and no API call needed.

class pipecat.services.speechmatics.stt.SpeechmaticsSTTService(*, api_key: str | None = None, base_url: str | None = None, sample_rate: int | None = None, encoding: AudioEncoding = AudioEncoding.PCM_S16LE, params: InputParams | None = None, should_interrupt: bool = True, settings: SpeechmaticsSTTSettings | None = None, ttfs_p99_latency: float | None = 0.74, **kwargs)[source]

Bases: STTService

Speechmatics STT service implementation.

This service provides real-time speech-to-text transcription using the Speechmatics API. It supports partial and final transcriptions, multiple languages, various audio formats, and speaker diarization. With diarization enabled, each transcript carries its speaker as speaker.label; speaker_format optionally labels the text.

Event handlers available (in addition to STTService events):

  • on_speakers_result(service, speakers): Speaker diarization results received

Example:

@stt.event_handler("on_speakers_result")
async def on_speakers_result(service, speakers):
    ...
Settings

alias of SpeechmaticsSTTSettings

class TurnDetectionMode(*values)

Bases: StrEnum

How turn boundaries (end of speech) are detected.

VAD: the STT service runs its own VAD and closes turns itself.

EXTERNAL: turn boundaries are controlled by the pipeline — the service does not endpoint on its own, and closes a turn when a VADUserStoppedSpeakingFrame arrives (for example from Pipecat’s own VAD).

The values mirror the Agent STT SDK’s own turn-detection modes so the two never drift.

VAD = 'vad'
EXTERNAL = 'external'
class AudioEncoding(*values)

Bases: str, Enum

Supported audio encoding formats for real-time transcription.

The Speechmatics RT API supports several audio encoding formats for optimal compatibility with different audio sources and quality requirements.

PCM_F32LE

32-bit float PCM used in the WAV audio format, little-endian architecture. 4 bytes per sample.

PCM_S16LE

16-bit signed integer PCM used in the WAV audio format, little-endian architecture. 2 bytes per sample.

MULAW

8 bit μ-law (mu-law) encoding. 1 byte per sample.

Examples

>>> encoding = AudioEncoding.PCM_S16LE
PCM_F32LE = 'pcm_f32le'
PCM_S16LE = 'pcm_s16le'
MULAW = 'mulaw'
class Model(*values)

Bases: str, Enum

Models available on the Agent STT service.

LINDEN_1

The first Agent STT model.

LINDEN_1 = 'linden-1'
class SpeakerIdentifier(label: str = '', speaker_identifiers: list[str] = <factory>)

Bases: object

Labeled speaker identifier for guided speaker diarization.

Use this to map one or more known speaker identifiers to a human-readable label. When provided in SpeakerDiarizationConfig.speakers, the engine can use these identifiers as hints to consistently assign the specified label.

label

Human-readable label to assign to this speaker or group (e.g., “Agent”, “Customer”, “Alice”).

Type:

str

speaker_identifiers

A list of string identifiers associated with this speaker. These can be any stable identifiers relevant to your application (for example device IDs, prior session speaker IDs, channel tags, etc.).

Type:

list[str]

Examples

>>> config = SpeakerDiarizationConfig(
...     max_speakers=2,
...     speakers=[
...         SpeakerIdentifier(label="Agent", speaker_identifiers=["agent_1"]),
...         SpeakerIdentifier(label="Customer", speaker_identifiers=["cust_1"]),
...     ],
... )
label: str = ''
speaker_identifiers: list[str]
class AdditionalVocabEntry(content: str, sounds_like: list[str] | None = None)

Bases: object

A word to bias the engine towards, optionally with pronunciation hints.

content

The word or phrase.

Type:

str

sounds_like

Alternative pronunciations, written as they sound.

Type:

list[str] | None

Examples

>>> AdditionalVocabEntry(content="Speechmatics", sounds_like=["speech matics"])
sounds_like: list[str] | None = None
content: str
RECONNECT_MAX_ATTEMPTS: ClassVar[int] = 3
class InputParams(*, domain: str | None = None, language: Language | str = Language.EN, turn_detection_mode: TurnDetectionMode = TurnDetectionMode.EXTERNAL, speaker_active_format: str | None = None, known_speakers: list[SpeakerIdentifier] = [], additional_vocab: list[AdditionalVocabEntry] = [], audio_encoding: AudioEncoding = AudioEncoding.PCM_S16LE, model: Model | str | None = None, operating_point: Model | str | None = None, enable_partials: bool | None = None, punctuation_overrides: dict | None = None, transcript_filtering_config: dict | None = None, enable_diarization: bool | None = None, speaker_sensitivity: float | None = None, max_speakers: int | None = None, prefer_current_speaker: bool | None = None)[source]

Bases: BaseModel

Configuration parameters for Speechmatics STT service.

Parameters:
  • domain – Domain for Speechmatics API. Defaults to None.

  • language – Language code for transcription. Defaults to Language.EN.

  • turn_detection_mode – How turns are closed. TurnDetectionMode.EXTERNAL closes a turn when a VADUserStoppedSpeakingFrame arrives (e.g. from Pipecat’s own VAD); TurnDetectionMode.VAD lets the STT service run its own VAD and close turns itself. Defaults to DEFAULT_TURN_DETECTION_MODE.

  • speaker_active_format – Formatter for the speaker ID. This formatter is used to format the text output for individual speakers and ensures that the context is clear for language models further down the pipeline. The attributes text and speaker_id are available. The system instructions for the language model may need to include any necessary instructions to handle the formatting. Example: @{speaker_id}: {text}. Defaults to None.

  • known_speakers – List of known speaker labels and identifiers. If you supply a list of labels and identifiers for speakers, then the STT engine will use them to attribute any spoken words to that speaker. This is useful when you want to attribute words to a specific speaker, such as the assistant or a specific user. Labels and identifiers can be obtained from a running STT session and then used in subsequent sessions. Identifiers are unique to each Speechmatics account and cannot be used across accounts. Refer to our examples on the format of the known_speakers parameter. Defaults to [].

  • additional_vocab – List of additional vocabulary entries. If you supply a list of additional vocabulary entries, the this will increase the weight of the words in the vocabulary and help the STT engine to better transcribe the words. Defaults to [].

  • audio_encoding – Audio encoding format. Defaults to AudioEncoding.PCM_S16LE.

  • model – The transcription model (operating point) to use, e.g. “linden-1”. Defaults to Model.LINDEN_1, the SDK’s default model. Preferred over operating_point.

  • operating_point –

    Alias for model. If both are given they must name the same value, otherwise a ValueError is raised. Optional.

    Deprecated since version 1.10.0: Use model instead. Will be removed in 2.0.0.

  • enable_partials – Include partial segment fragments (words) in the output of AddPartialSegment messages. Partial fragments from the STT will always be used for speaker activity detection. This setting is used only for the formatted text output of individual segments.

  • punctuation_overrides – Punctuation overrides. This allows you to override the punctuation in the STT engine. This is useful for languages that use different punctuation than English. See documentation for more information.

  • transcript_filtering_config – Post-processing applied to the transcript, as Speechmatics’ transcript_filtering_config object. remove_disfluencies drops fillers such as “um” and “uh”; replacements rewrites matched words. Defaults to None, which leaves the transcript as the engine produced it.

  • enable_diarization – Enable speaker diarization. When enabled, the STT engine will determine and attribute words to unique speakers. The speaker_sensitivity parameter can be used to adjust the sensitivity of diarization.

  • speaker_sensitivity – Diarization sensitivity. A higher value increases the sensitivity of diarization and helps when two or more speakers have similar voices.

  • max_speakers – Maximum number of speakers to detect. This forces the STT engine to cluster words into a fixed number of speakers. It should not be used to limit the number of speakers, unless it is clear that there will only be a known number of speakers.

  • prefer_current_speaker – Prefer current speaker ID. When set to true, groups of words close together are given extra weight to be identified as the same speaker.

domain: str | None
language: Language | str
turn_detection_mode: TurnDetectionMode
speaker_active_format: str | None
known_speakers: list[SpeakerIdentifier]
additional_vocab: list[AdditionalVocabEntry]
audio_encoding: AudioEncoding
model: Model | str | None
operating_point: Model | str | None
enable_partials: bool | None
punctuation_overrides: dict | None
transcript_filtering_config: dict | None
enable_diarization: bool | None
speaker_sensitivity: float | None
max_speakers: int | None
prefer_current_speaker: bool | None
__init__(*, api_key: str | None = None, base_url: str | None = None, sample_rate: int | None = None, encoding: AudioEncoding = AudioEncoding.PCM_S16LE, params: InputParams | None = None, should_interrupt: bool = True, settings: SpeechmaticsSTTSettings | None = None, ttfs_p99_latency: float | None = 0.74, **kwargs)[source]

Initialize the Speechmatics STT service.

Parameters:
  • api_key – Speechmatics API key for authentication. Uses environment variable SPEECHMATICS_API_KEY if not provided.

  • base_url – Base URL for Speechmatics API. Uses environment variable SPEECHMATICS_RT_URL or defaults to wss://eu2.rt.speechmatics.com/v2/agent.

  • sample_rate – Optional audio sample rate in Hz. Speechmatics accepts 8000 and 16000. Defaults to the pipeline’s input sample rate.

  • encoding – Audio encoding format. Defaults to AudioEncoding.PCM_S16LE.

  • params –

    Input parameters for the service.

    Deprecated since version 0.0.105: Use settings=SpeechmaticsSTTService.Settings(...) instead. Will be removed in 2.0.0.

  • should_interrupt – Determine whether the bot should be interrupted when Speechmatics turn_detection_mode is configured to detect user speech. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied user_turn_strategies overrides the recommendation and this setting with it.

  • settings – Runtime-updatable settings. When provided alongside deprecated params, settings values take precedence.

  • ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark

  • **kwargs – Additional arguments passed to STTService.

service_metadata_frame() → STTMetadataFrame[source]

Request external turn strategies when Speechmatics endpoints server-side.

Every mode other than EXTERNAL (which uses Pipecat’s own endpointing) has Speechmatics detect turns and propose the boundaries, so the user aggregator resolves those. Applied unless the user passed their own user_turn_strategies.

property session_id: str | None

The Agent STT session id, set once RecognitionStarted arrives (else None).

async start(frame: StartFrame)[source]

Called when the new session starts.

async stop(frame: EndFrame)[source]

Called when the session ends.

async cancel(frame: CancelFrame)[source]

Called when the session is cancelled.

async cleanup()[source]

Release Speechmatics resources at pipeline teardown.

async process_frame(frame: Frame, direction: FrameDirection)[source]

Process frames for VAD and metrics handling.

Parameters:
  • frame – Frame to process.

  • direction – Direction of frame processing.

async send_message(message: ClientMessageType | str, **kwargs: Any) → None[source]

Send a message to the STT service.

This sends a message to the STT service via the underlying transport. If the session is not running, this will raise an exception. Messages in the wrong format will also cause an error.

Parameters:
  • message – Message to send to the STT service.

  • **kwargs – Additional arguments passed to the underlying transport.

Raises:

RuntimeError – If the session is not connected, or the message could not be sent (e.g. a malformed payload).

can_generate_metrics() → bool[source]

Check if this service can generate processing metrics.

Returns:

True, as Speechmatics STT supports generation of metrics.

async run_stt(audio: bytes) → AsyncGenerator[Frame | None, None][source]

Adds audio to the audio buffer and yields None.

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)

Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.

Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.

Parameters:
  • frame – The frame to push.

  • direction – The direction to push the frame.

async start_stt_usage_metrics(usage: STTUsage)

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async stop_ttfb_metrics(*, end_time: float | None = None)

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.