tts

NVIDIA text-to-speech service implementation.

This module provides integration with NVIDIA TTS through gRPC API for high-quality speech synthesis.

Refer to the NVIDIA TTS NIM documentation for usage, customization, and local deployment steps: https://docs.nvidia.com/nim/speech/latest/tts/

class pipecat.services.nvidia.tts.NvidiaTTSSynthesisMode(*values)[source]

Bases: StrEnum

Controls how text is sent to NVIDIA TTS.

Parameters:
  • PER_SENTENCE – Open a separate SynthesizeOnline call for each run_tts invocation. This is the default mode and can be used with all supported NVIDIA TTS NIMs, including Chatterbox, Magpie multilingual, and Magpie zero-shot. Text may still be chunked within that call to satisfy model request length limits.

  • STITCHED – Reuse one SynthesizeOnline stream across multiple run_tts calls within the same LLM response. Enable this only for models with cross-sentence stitching support, such as Magpie multilingual and Magpie zero-shot v1.7.0 or later. Compatible models can still use per_sentence, but stitched can improve multi-sentence synthesis quality. Individual run_tts inputs may still be chunked within the shared stream to satisfy model request length limits.

PER_SENTENCE = 'per_sentence'
STITCHED = 'stitched'
class pipecat.services.nvidia.tts.NvidiaTTSSettings(model: str | None | NotGiven = <factory>, extra: dict[str, ~typing.Any]=<factory>, voice: str | None | NotGiven = <factory>, language: Language | str | None | NotGiven = <factory>, quality: int | NotGiven = <factory>, synthesis_mode: NvidiaTTSSynthesisMode | NotGiven = <factory>, custom_configuration: dict[str, str] | None | ~pipecat.utils.types.NotGiven=<factory>)[source]

Bases: TTSSettings

Settings for NvidiaTTSService.

Parameters:
  • quality – Audio quality setting (0-100). For Magpie zero-shot, NVIDIA expects values in the range 1 to 40.

  • synthesis_mode – Whether to synthesize one sentence per request or stitch multiple sentences across a single streaming request.

  • custom_configuration – Model-specific key/value parameters forwarded to the synthesizer, e.g. Magpie’s streaming controls flush, chunk_len_threshold and max_chunk_threshold, or Chatterbox’s exaggeration_factor. See https://docs.nvidia.com/nim/speech/latest/tts/customization.html

class pipecat.services.nvidia.tts.NvidiaTTSService(*, api_key: str | None = None, server: str = 'grpc.nvcf.nvidia.com:443', voice_id: str | None = None, sample_rate: int | None = None, model_function_map: Mapping[str, str] = {'function_id': '877104f7-e885-42b9-8de8-f6e4c6303969', 'model_name': 'magpie-tts-multilingual'}, params: InputParams | None = None, settings: NvidiaTTSSettings | None = None, use_ssl: bool = True, custom_dictionary: dict | None = None, zero_shot_audio_prompt_file: str | PathLike[str] | None = None, audio_prompt_encoding: EnumTypeWrapper | None = 0, encoding: EnumTypeWrapper | None = 1, **kwargs)[source]

Bases: TTSService

NVIDIA TTS service.

Provides high-quality text-to-speech synthesis using both locally deployed and cloud-based NVIDIA TTS models. Supports multiple voices, languages, and configurable quality settings.

Settings

alias of NvidiaTTSSettings

class InputParams(*, language: Language | None = Language.EN_US, quality: int | None = 20)[source]

Bases: BaseModel

Input parameters for NVIDIA TTS configuration.

Deprecated since version 0.0.105: Use NvidiaTTSService.Settings directly via the settings parameter instead. Will be removed in 2.0.0.

Parameters:
  • language – Language code for synthesis. Defaults to US English.

  • quality – Audio quality setting (0-100). For Magpie zero-shot, NVIDIA expects values in the range 1 to 40. Defaults to 20.

language: Language | None
quality: int | None
__init__(*, api_key: str | None = None, server: str = 'grpc.nvcf.nvidia.com:443', voice_id: str | None = None, sample_rate: int | None = None, model_function_map: Mapping[str, str] = {'function_id': '877104f7-e885-42b9-8de8-f6e4c6303969', 'model_name': 'magpie-tts-multilingual'}, params: InputParams | None = None, settings: NvidiaTTSSettings | None = None, use_ssl: bool = True, custom_dictionary: dict | None = None, zero_shot_audio_prompt_file: str | PathLike[str] | None = None, audio_prompt_encoding: EnumTypeWrapper | None = 0, encoding: EnumTypeWrapper | None = 1, **kwargs)[source]

Initialize the NVIDIA TTS service.

Parameters:
  • api_key – NVIDIA API key for authentication. Required when using the cloud endpoint. Not needed for local deployments.

  • server – gRPC server endpoint. Defaults to NVIDIA’s cloud endpoint. For local deployments, pass the local address (e.g. localhost:50051).

  • voice_id –

    Voice model identifier. Defaults to multilingual Aria voice.

    Deprecated since version 0.0.105: Use settings=NvidiaTTSService.Settings(voice=...) instead. Will be removed in 2.0.0.

  • sample_rate – Audio sample rate. If None, uses service default.

  • model_function_map – Dictionary containing function_id and model_name for the TTS model.

  • params –

    Additional configuration parameters for TTS synthesis.

    Deprecated since version 0.0.105: Use settings=NvidiaTTSService.Settings(...) instead. Will be removed in 2.0.0.

  • settings – Runtime-updatable settings. When provided alongside deprecated parameters, settings values take precedence.

  • use_ssl – Whether to use SSL for the gRPC connection. Defaults to True for the NVIDIA cloud endpoint. Set to False for local deployments.

  • custom_dictionary –

    Custom pronunciation dictionary mapping words (graphemes) to IPA phonetic representations (phonemes), e.g. {"NVIDIA": "ɛn.vɪ.diː.ʌ"}. See https://docs.nvidia.com/nim/speech/latest/tts/phoneme-support.html for the list of supported IPA phonemes.

    Another way to guide pronunciation is to pass replace_text() find-and-replace rules through the text_transforms parameter. Pipecat applies them before the text reaches the provider, so the same rules can work across TTS services and depend less on what each provider supports. They’re worth trying if this setting isn’t giving you the results you want.

  • zero_shot_audio_prompt_file – Optional audio prompt file for Magpie zero-shot voice cloning. NVIDIA recommends a 16-bit mono WAV prompt, sample rate 22.05 kHz or higher, and duration 3 to 10 seconds. Access to NVIDIA’s hosted zero-shot models requires approval through: https://developer.nvidia.com/riva-tts-zeroshot-models

  • audio_prompt_encoding – Audio encoding for zero_shot_audio_prompt_file. Use this when the server expects a specific prompt encoding for Magpie zero-shot voice cloning.

  • encoding – Output audio encoding format. Defaults to AudioEncoding.LINEAR_PCM.

  • **kwargs – Additional arguments passed to parent TTSService.

can_generate_metrics() → bool[source]

Check if this service can generate metrics.

Returns:

True as this service supports metric generation.

async set_model(model: str)[source]

Set the TTS model.

Deprecated since version 0.0.104: No replacement. Will be removed in 2.0.0.

Example:

NvidiaTTSService(
    api_key=...,
    model_function_map={"function_id": "<UUID>", "model_name": "<model_name>"},
)
Parameters:

model – The model name to set.

async setup(setup: FrameProcessorSetup)[source]

Set up the service.

Parameters:

setup – Configuration object containing setup parameters.

async stop(frame: EndFrame)[source]

Stop the NVIDIA TTS service.

Parameters:

frame – The end frame.

async cancel(frame: CancelFrame)[source]

Cancel the NVIDIA TTS service.

Parameters:

frame – The cancel frame.

async cleanup()[source]

Release all resources held by the service.

async flush_audio(context_id: str | None = None)[source]

Flush any pending audio and finalize the current context.

Parameters:

context_id – The specific context to flush. If None, falls back to the currently active context.

async on_audio_context_interrupted(context_id: str)[source]

Cancel the active gRPC synthesis stream when the bot is interrupted.

Parameters:

context_id – The ID of the audio context that was interrupted.

async run_tts(text: str, context_id: str) → AsyncGenerator[Frame | None, None][source]

Generate speech from text using NVIDIA TTS.

On the first call for a turn, starts a persistent synthesize_online gRPC stream. Subsequent calls within the same turn feed text into the existing stream, enabling Magpie’s cross-sentence stitching.

Text is split into chunks respecting Magpie’s per-request limits. Each chunk becomes a separate request in the gRPC stream, stitched seamlessly by Magpie.

In per_sentence mode, opens a fresh SynthesizeOnline call per invocation and yields audio directly before returning.

Parameters:
  • text – The text to synthesize into speech.

  • context_id – The context ID for tracking audio frames.

Yields:
None on success in stitched mode. TTSAudioRawFrame objects

in per_sentence mode. ErrorFrame on failure.