stt
Speechmatics STT service integration.
- class pipecat.services.speechmatics.stt.TurnDetectionMode(*values)[source]
Bases:
StrEnumHow turn boundaries (end of speech) are detected.
VAD: the STT service runs its own VAD and closes turns itself.
EXTERNAL: turn boundaries are controlled by the pipeline — the service does not endpoint on its own, and closes a turn when a VADUserStoppedSpeakingFrame arrives (for example from Pipecat’s own VAD).
The values mirror the Agent STT SDK’s own turn-detection modes so the two never drift.
- VAD = 'vad'
- EXTERNAL = 'external'
- class pipecat.services.speechmatics.stt.SpeechmaticsSTTSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, language: Language | str | None | NotGiven = <factory>, domain: str | None | NotGiven = <factory>, turn_detection_mode: TurnDetectionMode | NotGiven = <factory>, speaker_format: str | None | NotGiven = <factory>, speaker_active_format: str | None | NotGiven = <factory>, known_speakers: list[SpeakerIdentifier] | NotGiven = <factory>, additional_vocab: list[AdditionalVocabEntry] | NotGiven = <factory>, operating_point: Model | str | None | NotGiven = <factory>, enable_partials: bool | None | NotGiven = <factory>, emit_sentences: bool | None | NotGiven = <factory>, punctuation_overrides: dict[str, ~typing.Any] | None | ~pipecat.utils.types.NotGiven=<factory>, transcript_filtering_config: dict[str, ~typing.Any] | None | ~pipecat.utils.types.NotGiven=<factory>, enable_diarization: bool | None | NotGiven = <factory>, speaker_sensitivity: float | None | NotGiven = <factory>, max_speakers: int | None | NotGiven = <factory>, prefer_current_speaker: bool | None | NotGiven = <factory>)[source]
Bases:
STTSettingsSettings for SpeechmaticsSTTService.
See
SpeechmaticsSTTService.InputParamsfor detailed descriptions of each field.- Parameters:
domain – Domain for Speechmatics API.
turn_detection_mode – Endpoint handling mode.
speaker_format – Optional format string for diarized transcripts. Use
{speaker}for the speaker label,{text}for the transcript text, and{speaker_number}for the speaker’s order of first appearance in the session (1,2, …), e.g."{speaker}: {text}". If None, transcript text is not modified. Applied locally, so changing it does not reconnect.speaker_active_format –
Formatter for speaker ID, with the
text,speaker_id,tsandlangattributes. Takes precedence overspeaker_format.Deprecated since version 1.16.0: Use
speaker_formatinstead. Will be removed in 2.0.0.known_speakers – List of known speaker labels and identifiers.
additional_vocab – List of additional vocabulary entries.
model – Resolved transcription model (operating point). See
_resolve_model.operating_point –
Alias for
model.Deprecated since version 1.10.0: Use
modelinstead. Will be removed in 2.0.0.enable_partials – Include partial segment fragments.
emit_sentences – Close a segment at every end of sentence, not only at the end of the turn, so each sentence is emitted as its own final transcription.
punctuation_overrides – Punctuation overrides.
transcript_filtering_config – Post-processing applied to the transcript.
enable_diarization – Enable speaker diarization. Transcripts carry the Speechmatics speaker label (e.g.
"S1", or the label you assigned inknown_speakers) asspeaker.label.speaker_sensitivity – Diarization sensitivity.
max_speakers – Maximum number of speakers to detect.
prefer_current_speaker – Prefer current speaker ID.
- class pipecat.services.speechmatics.stt.SpeechmaticsSTTService(*, api_key: str | None = None, base_url: str | None = None, sample_rate: int | None = None, encoding: AudioEncoding = AudioEncoding.PCM_S16LE, params: InputParams | None = None, should_interrupt: bool = True, settings: SpeechmaticsSTTSettings | None = None, ttfs_p99_latency: float | None = 0.74, **kwargs)[source]
Bases:
STTServiceSpeechmatics STT service implementation.
This service provides real-time speech-to-text transcription using the Speechmatics API. It supports partial and final transcriptions, multiple languages, various audio formats, and speaker diarization. With diarization enabled, each transcript carries its speaker as
speaker.label;speaker_formatoptionally labels the text.Event handlers available (in addition to STTService events):
on_speakers_result(service, speakers): Speaker diarization results received
Example:
@stt.event_handler("on_speakers_result") async def on_speakers_result(service, speakers): ...
- Settings
alias of
SpeechmaticsSTTSettings
- class TurnDetectionMode(*values)
Bases:
StrEnumHow turn boundaries (end of speech) are detected.
VAD: the STT service runs its own VAD and closes turns itself.
EXTERNAL: turn boundaries are controlled by the pipeline — the service does not endpoint on its own, and closes a turn when a VADUserStoppedSpeakingFrame arrives (for example from Pipecat’s own VAD).
The values mirror the Agent STT SDK’s own turn-detection modes so the two never drift.
- VAD = 'vad'
- EXTERNAL = 'external'
- class AudioEncoding(*values)
-
Supported audio encoding formats for real-time transcription.
The Speechmatics RT API supports several audio encoding formats for optimal compatibility with different audio sources and quality requirements.
- PCM_F32LE
32-bit float PCM used in the WAV audio format, little-endian architecture. 4 bytes per sample.
- PCM_S16LE
16-bit signed integer PCM used in the WAV audio format, little-endian architecture. 2 bytes per sample.
- MULAW
8 bit μ-law (mu-law) encoding. 1 byte per sample.
Examples
>>> encoding = AudioEncoding.PCM_S16LE
- PCM_F32LE = 'pcm_f32le'
- PCM_S16LE = 'pcm_s16le'
- MULAW = 'mulaw'
- class Model(*values)
-
Models available on the Agent STT service.
- LINDEN_1
The first Agent STT model.
- LINDEN_1 = 'linden-1'
- class SpeakerIdentifier(label: str = '', speaker_identifiers: list[str] = <factory>)
Bases:
objectLabeled speaker identifier for guided speaker diarization.
Use this to map one or more known speaker identifiers to a human-readable label. When provided in SpeakerDiarizationConfig.speakers, the engine can use these identifiers as hints to consistently assign the specified label.
- label
Human-readable label to assign to this speaker or group (e.g., “Agent”, “Customer”, “Alice”).
- Type:
- speaker_identifiers
A list of string identifiers associated with this speaker. These can be any stable identifiers relevant to your application (for example device IDs, prior session speaker IDs, channel tags, etc.).
Examples
>>> config = SpeakerDiarizationConfig( ... max_speakers=2, ... speakers=[ ... SpeakerIdentifier(label="Agent", speaker_identifiers=["agent_1"]), ... SpeakerIdentifier(label="Customer", speaker_identifiers=["cust_1"]), ... ], ... )
- class AdditionalVocabEntry(content: str, sounds_like: list[str] | None = None)
Bases:
objectA word to bias the engine towards, optionally with pronunciation hints.
Examples
>>> AdditionalVocabEntry(content="Speechmatics", sounds_like=["speech matics"])
- class InputParams(*, domain: str | None = None, language: Language | str = Language.EN, turn_detection_mode: TurnDetectionMode = TurnDetectionMode.EXTERNAL, speaker_active_format: str | None = None, known_speakers: list[SpeakerIdentifier] = [], additional_vocab: list[AdditionalVocabEntry] = [], audio_encoding: AudioEncoding = AudioEncoding.PCM_S16LE, model: Model | str | None = None, operating_point: Model | str | None = None, enable_partials: bool | None = None, punctuation_overrides: dict | None = None, transcript_filtering_config: dict | None = None, enable_diarization: bool | None = None, speaker_sensitivity: float | None = None, max_speakers: int | None = None, prefer_current_speaker: bool | None = None)[source]
Bases:
BaseModelConfiguration parameters for Speechmatics STT service.
- Parameters:
domain – Domain for Speechmatics API. Defaults to None.
language – Language code for transcription. Defaults to Language.EN.
turn_detection_mode – How turns are closed. TurnDetectionMode.EXTERNAL closes a turn when a VADUserStoppedSpeakingFrame arrives (e.g. from Pipecat’s own VAD); TurnDetectionMode.VAD lets the STT service run its own VAD and close turns itself. Defaults to DEFAULT_TURN_DETECTION_MODE.
speaker_active_format – Formatter for the speaker ID. This formatter is used to format the text output for individual speakers and ensures that the context is clear for language models further down the pipeline. The attributes text and speaker_id are available. The system instructions for the language model may need to include any necessary instructions to handle the formatting. Example: @{speaker_id}: {text}. Defaults to None.
known_speakers – List of known speaker labels and identifiers. If you supply a list of labels and identifiers for speakers, then the STT engine will use them to attribute any spoken words to that speaker. This is useful when you want to attribute words to a specific speaker, such as the assistant or a specific user. Labels and identifiers can be obtained from a running STT session and then used in subsequent sessions. Identifiers are unique to each Speechmatics account and cannot be used across accounts. Refer to our examples on the format of the known_speakers parameter. Defaults to [].
additional_vocab – List of additional vocabulary entries. If you supply a list of additional vocabulary entries, the this will increase the weight of the words in the vocabulary and help the STT engine to better transcribe the words. Defaults to [].
audio_encoding – Audio encoding format. Defaults to AudioEncoding.PCM_S16LE.
model – The transcription model (operating point) to use, e.g. “linden-1”. Defaults to Model.LINDEN_1, the SDK’s default model. Preferred over operating_point.
operating_point –
Alias for model. If both are given they must name the same value, otherwise a ValueError is raised. Optional.
Deprecated since version 1.10.0: Use
modelinstead. Will be removed in 2.0.0.enable_partials – Include partial segment fragments (words) in the output of AddPartialSegment messages. Partial fragments from the STT will always be used for speaker activity detection. This setting is used only for the formatted text output of individual segments.
punctuation_overrides – Punctuation overrides. This allows you to override the punctuation in the STT engine. This is useful for languages that use different punctuation than English. See documentation for more information.
transcript_filtering_config – Post-processing applied to the transcript, as Speechmatics’ transcript_filtering_config object. remove_disfluencies drops fillers such as “um” and “uh”; replacements rewrites matched words. Defaults to None, which leaves the transcript as the engine produced it.
enable_diarization – Enable speaker diarization. When enabled, the STT engine will determine and attribute words to unique speakers. The speaker_sensitivity parameter can be used to adjust the sensitivity of diarization.
speaker_sensitivity – Diarization sensitivity. A higher value increases the sensitivity of diarization and helps when two or more speakers have similar voices.
max_speakers – Maximum number of speakers to detect. This forces the STT engine to cluster words into a fixed number of speakers. It should not be used to limit the number of speakers, unless it is clear that there will only be a known number of speakers.
prefer_current_speaker – Prefer current speaker ID. When set to true, groups of words close together are given extra weight to be identified as the same speaker.
- turn_detection_mode: TurnDetectionMode
- known_speakers: list[SpeakerIdentifier]
- additional_vocab: list[AdditionalVocabEntry]
- audio_encoding: AudioEncoding
- __init__(*, api_key: str | None = None, base_url: str | None = None, sample_rate: int | None = None, encoding: AudioEncoding = AudioEncoding.PCM_S16LE, params: InputParams | None = None, should_interrupt: bool = True, settings: SpeechmaticsSTTSettings | None = None, ttfs_p99_latency: float | None = 0.74, **kwargs)[source]
Initialize the Speechmatics STT service.
- Parameters:
api_key – Speechmatics API key for authentication. Uses environment variable SPEECHMATICS_API_KEY if not provided.
base_url – Base URL for Speechmatics API. Uses environment variable SPEECHMATICS_RT_URL or defaults to wss://eu2.rt.speechmatics.com/v2/agent.
sample_rate – Optional audio sample rate in Hz. Speechmatics accepts 8000 and 16000. Defaults to the pipeline’s input sample rate.
encoding – Audio encoding format. Defaults to
AudioEncoding.PCM_S16LE.params –
Input parameters for the service.
Deprecated since version 0.0.105: Use
settings=SpeechmaticsSTTService.Settings(...)instead. Will be removed in 2.0.0.should_interrupt – Determine whether the bot should be interrupted when Speechmatics turn_detection_mode is configured to detect user speech. Passed along to the user turn strategies this service recommends, which own the interruption; a user-supplied
user_turn_strategiesoverrides the recommendation and this setting with it.settings – Runtime-updatable settings. When provided alongside deprecated
params,settingsvalues take precedence.ttfs_p99_latency – P99 latency from speech end to final transcript in seconds. Override for your deployment. See https://github.com/pipecat-ai/stt-benchmark
**kwargs – Additional arguments passed to STTService.
- service_metadata_frame() STTMetadataFrame[source]
Request external turn strategies when Speechmatics endpoints server-side.
Every mode other than
EXTERNAL(which uses Pipecat’s own endpointing) has Speechmatics detect turns and propose the boundaries, so the user aggregator resolves those. Applied unless the user passed their ownuser_turn_strategies.
- property session_id: str | None
The Agent STT session id, set once
RecognitionStartedarrives (else None).
- async start(frame: StartFrame)[source]
Called when the new session starts.
- async cancel(frame: CancelFrame)[source]
Called when the session is cancelled.
- async process_frame(frame: Frame, direction: FrameDirection)[source]
Process frames for VAD and metrics handling.
- Parameters:
frame – Frame to process.
direction – Direction of frame processing.
- async send_message(message: ClientMessageType | str, **kwargs: Any) None[source]
Send a message to the STT service.
This sends a message to the STT service via the underlying transport. If the session is not running, this will raise an exception. Messages in the wrong format will also cause an error.
- Parameters:
message – Message to send to the STT service.
**kwargs – Additional arguments passed to the underlying transport.
- Raises:
RuntimeError – If the session is not connected, or the message could not be sent (e.g. a malformed payload).
- can_generate_metrics() bool[source]
Check if this service can generate processing metrics.
- Returns:
True, as Speechmatics STT supports generation of metrics.
- async run_stt(audio: bytes) AsyncGenerator[Frame | None, None][source]
Adds audio to the audio buffer and yields None.
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)
Push a frame downstream, tracking TranscriptionFrame timestamps for TTFB.
Stores the timestamp of each TranscriptionFrame for TTFB calculation. If the frame is marked as finalized (via request_finalize/confirm_finalize), reports TTFB immediately and cancels any pending timeout. Otherwise, TTFB is reported after a timeout.
- Parameters:
frame – The frame to push.
direction – The direction to push the frame.