services

Service constructors for the eval harness.

Each function builds a concrete pipecat service (TTS, STT, or judge LLM) from a scenario’s config mapping. They are the dispatch targets behind the service: name in pipecat.evals.speech.EvalSpeech.from_config(), pipecat.evals.transcribe.EvalTranscriber.from_config(), and pipecat.evals.judge.EvalJudge.from_config(). The heavy provider imports stay lazy inside each function so importing this module stays cheap.

pipecat.evals.services.kokoro_service(voice_cfg: dict, sample_rate: int) → TTSService

Build a local Kokoro TTS service from the user_audio config.

Kokoro runs an ONNX model locally (no API key, no per-run cost), so the eval suite synthesizes user audio for free. The model files are downloaded once on first use and cached under ~/.cache/kokoro-onnx.

pipecat.evals.services.cartesia_service(voice_cfg: dict, sample_rate: int) → TTSService

Build a Cartesia TTS service from the user_audio config.

pipecat.evals.services.whisper_service(config: dict) → STTService

Build a local Whisper STT service from the bot_audio config.

Runs on the CPU by default (device: cpu): the GPU is reserved for the judge LLM and the per-run audio models, and bot-speech transcription happens once per turn off the hot path, so the extra latency is fine. This frees enough GPU memory to run a larger, more accurate model (e.g. distil-medium or large-v3-turbo) at higher concurrency. Override with device: cuda (and compute_type) in the transcription config if you have GPU headroom.

The eval transcribes audio it already knows is the bot speaking (the harness captures it between bot-started-speaking and bot-stopped-speaking), so Whisper’s non-speech filter is counterproductive here: the default no_speech_prob=0.4 drops correct transcriptions of synthetic/TTS speech, whose no_speech_prob jitters across ~0.4-0.6 run to run (a dropped segment yields no TranscriptionFrame, so the harness then waits out the whole transcription timeout). Disable the filter with a permissive threshold.

pipecat.evals.services.moonshine_service(config: dict) → STTService

Build a local Moonshine STT service from the bot_audio config.

Moonshine runs on the CPU via ONNX Runtime (no GPU, no API key) and is small and fast. On the short, isolated bot-answer segments the harness transcribes, it tends to keep the answer where Whisper sometimes drops it. model selects the architecture (a Model value or string; default Model.SMALL_STREAMING).

pipecat.evals.services.ollama_service(config: dict) → LLMService[Any]

Build a local Ollama LLM service from the judge: config.

pipecat.evals.services.openai_service(config: dict) → LLMService[Any]

Build an OpenAI LLM service from the judge: config.