tts
NVIDIA text-to-speech service implementation.
This module provides integration with NVIDIA TTS through gRPC API for high-quality speech synthesis.
Refer to the NVIDIA TTS NIM documentation for usage, customization, and local deployment steps: https://docs.nvidia.com/nim/speech/latest/tts/
- class pipecat.services.nvidia.tts.NvidiaTTSSynthesisMode(*values)[source]
Bases:
StrEnumControls how text is sent to NVIDIA TTS.
- Parameters:
PER_SENTENCE – Open a separate
SynthesizeOnlinecall for eachrun_ttsinvocation. This is the default mode and can be used with all supported NVIDIA TTS NIMs, including Chatterbox, Magpie multilingual, and Magpie zero-shot. Text may still be chunked within that call to satisfy model request length limits.STITCHED – Reuse one
SynthesizeOnlinestream across multiplerun_ttscalls within the same LLM response. Enable this only for models with cross-sentence stitching support, such as Magpie multilingual and Magpie zero-shot v1.7.0 or later. Compatible models can still useper_sentence, butstitchedcan improve multi-sentence synthesis quality. Individualrun_ttsinputs may still be chunked within the shared stream to satisfy model request length limits.
- PER_SENTENCE = 'per_sentence'
- STITCHED = 'stitched'
- class pipecat.services.nvidia.tts.NvidiaTTSSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, voice: str | None | NotGiven = <factory>, language: Language | str | None | NotGiven = <factory>, quality: int | NotGiven = <factory>, synthesis_mode: NvidiaTTSSynthesisMode | NotGiven = <factory>, custom_configuration: dict[str, str] | None | ~pipecat.utils.types.NotGiven=<factory>)[source]
Bases:
TTSSettingsSettings for NvidiaTTSService.
- Parameters:
quality – Audio quality setting (0-100). For Magpie zero-shot, NVIDIA expects values in the range
1to40.synthesis_mode – Whether to synthesize one sentence per request or stitch multiple sentences across a single streaming request.
custom_configuration – Model-specific key/value parameters forwarded to the synthesizer, e.g. Magpie’s streaming controls
flush,chunk_len_thresholdandmax_chunk_threshold, or Chatterbox’sexaggeration_factor. See https://docs.nvidia.com/nim/speech/latest/tts/customization.html
- class pipecat.services.nvidia.tts.NvidiaTTSService(*, api_key: str | None = None, server: str = 'grpc.nvcf.nvidia.com:443', voice_id: str | None = None, sample_rate: int | None = None, model_function_map: Mapping[str, str] = {'function_id': '877104f7-e885-42b9-8de8-f6e4c6303969', 'model_name': 'magpie-tts-multilingual'}, params: InputParams | None = None, settings: NvidiaTTSSettings | None = None, use_ssl: bool = True, custom_dictionary: dict | None = None, zero_shot_audio_prompt_file: str | PathLike[str] | None = None, audio_prompt_encoding: EnumTypeWrapper | None = 0, encoding: EnumTypeWrapper | None = 1, **kwargs)[source]
Bases:
TTSServiceNVIDIA TTS service.
Provides high-quality text-to-speech synthesis using both locally deployed and cloud-based NVIDIA TTS models. Supports multiple voices, languages, and configurable quality settings.
- Settings
alias of
NvidiaTTSSettings
- class InputParams(*, language: Language | None = Language.EN_US, quality: int | None = 20)[source]
Bases:
BaseModelInput parameters for NVIDIA TTS configuration.
Deprecated since version 0.0.105: Use
NvidiaTTSService.Settingsdirectly via thesettingsparameter instead. Will be removed in 2.0.0.- Parameters:
language – Language code for synthesis. Defaults to US English.
quality – Audio quality setting (0-100). For Magpie zero-shot, NVIDIA expects values in the range
1to40. Defaults to 20.
- __init__(*, api_key: str | None = None, server: str = 'grpc.nvcf.nvidia.com:443', voice_id: str | None = None, sample_rate: int | None = None, model_function_map: Mapping[str, str] = {'function_id': '877104f7-e885-42b9-8de8-f6e4c6303969', 'model_name': 'magpie-tts-multilingual'}, params: InputParams | None = None, settings: NvidiaTTSSettings | None = None, use_ssl: bool = True, custom_dictionary: dict | None = None, zero_shot_audio_prompt_file: str | PathLike[str] | None = None, audio_prompt_encoding: EnumTypeWrapper | None = 0, encoding: EnumTypeWrapper | None = 1, **kwargs)[source]
Initialize the NVIDIA TTS service.
- Parameters:
api_key – NVIDIA API key for authentication. Required when using the cloud endpoint. Not needed for local deployments.
server – gRPC server endpoint. Defaults to NVIDIA’s cloud endpoint. For local deployments, pass the local address (e.g.
localhost:50051).voice_id –
Voice model identifier. Defaults to multilingual Aria voice.
Deprecated since version 0.0.105: Use
settings=NvidiaTTSService.Settings(voice=...)instead. Will be removed in 2.0.0.sample_rate – Audio sample rate. If None, uses service default.
model_function_map – Dictionary containing function_id and model_name for the TTS model.
params –
Additional configuration parameters for TTS synthesis.
Deprecated since version 0.0.105: Use
settings=NvidiaTTSService.Settings(...)instead. Will be removed in 2.0.0.settings – Runtime-updatable settings. When provided alongside deprecated parameters,
settingsvalues take precedence.use_ssl – Whether to use SSL for the gRPC connection. Defaults to True for the NVIDIA cloud endpoint. Set to False for local deployments.
custom_dictionary –
Custom pronunciation dictionary mapping words (graphemes) to IPA phonetic representations (phonemes), e.g.
{"NVIDIA": "ɛn.vɪ.diː.ʌ"}. See https://docs.nvidia.com/nim/speech/latest/tts/phoneme-support.html for the list of supported IPA phonemes.Another way to guide pronunciation is to pass
replace_text()find-and-replace rules through thetext_transformsparameter. Pipecat applies them before the text reaches the provider, so the same rules can work across TTS services and depend less on what each provider supports. They’re worth trying if this setting isn’t giving you the results you want.zero_shot_audio_prompt_file – Optional audio prompt file for Magpie zero-shot voice cloning. NVIDIA recommends a 16-bit mono WAV prompt, sample rate 22.05 kHz or higher, and duration 3 to 10 seconds. Access to NVIDIA’s hosted zero-shot models requires approval through: https://developer.nvidia.com/riva-tts-zeroshot-models
audio_prompt_encoding – Audio encoding for
zero_shot_audio_prompt_file. Use this when the server expects a specific prompt encoding for Magpie zero-shot voice cloning.encoding – Output audio encoding format. Defaults to
AudioEncoding.LINEAR_PCM.**kwargs – Additional arguments passed to parent TTSService.
- can_generate_metrics() bool[source]
Check if this service can generate metrics.
- Returns:
True as this service supports metric generation.
- async set_model(model: str)[source]
Set the TTS model.
Deprecated since version 0.0.104: No replacement. Will be removed in 2.0.0.
Example:
NvidiaTTSService( api_key=..., model_function_map={"function_id": "<UUID>", "model_name": "<model_name>"}, )
- Parameters:
model – The model name to set.
- async setup(setup: FrameProcessorSetup)[source]
Set up the service.
- Parameters:
setup – Configuration object containing setup parameters.
- async stop(frame: EndFrame)[source]
Stop the NVIDIA TTS service.
- Parameters:
frame – The end frame.
- async cancel(frame: CancelFrame)[source]
Cancel the NVIDIA TTS service.
- Parameters:
frame – The cancel frame.
- async flush_audio(context_id: str | None = None)[source]
Flush any pending audio and finalize the current context.
- Parameters:
context_id – The specific context to flush. If None, falls back to the currently active context.
- async on_audio_context_interrupted(context_id: str)[source]
Cancel the active gRPC synthesis stream when the bot is interrupted.
- Parameters:
context_id – The ID of the audio context that was interrupted.
- async run_tts(text: str, context_id: str) AsyncGenerator[Frame | None, None][source]
Generate speech from text using NVIDIA TTS.
On the first call for a turn, starts a persistent
synthesize_onlinegRPC stream. Subsequent calls within the same turn feed text into the existing stream, enabling Magpie’s cross-sentence stitching.Text is split into chunks respecting Magpie’s per-request limits. Each chunk becomes a separate request in the gRPC stream, stitched seamlessly by Magpie.
In
per_sentencemode, opens a freshSynthesizeOnlinecall per invocation and yields audio directly before returning.- Parameters:
text – The text to synthesize into speech.
context_id – The context ID for tracking audio frames.
- Yields:
- None on success in
stitchedmode.TTSAudioRawFrameobjects in
per_sentencemode.ErrorFrameon failure.
- None on success in