suite

Multi-bot eval suite runner.

An EvalManifest lists bots to spawn and the scenarios to run against each; an EvalSuite spawns each bot with its eval transport on its own port, drives it with the harness (pipecat.evals.harness.EvalSession.from_scenario()), and runs several concurrently. pipecat eval suite is the CLI in front of it; the release evals are just a manifest plus that command.

Manifest format (YAML):

concurrency: 4
runs_dir: test-runs           # logs + recordings go to <runs_dir>/<timestamp>/
record: false                 # record conversation audio
cache_dir: null               # optional
scenarios_dir: scenarios      # resolved relative to this manifest file
# {python}=interpreter (default sys.executable), {bot}=bot path,
# {port}=assigned per run by the suite runner
spawn: "{python} {bot} -t eval --port {port}"
suite:
  - bot: examples/voice/voice-cartesia.py
    scenarios: [simple_math, greeting]
  - bot: examples/voice/voice-openai.py
    scenarios: [simple_math, interruption]
  - bot: examples/vision/vision-openai.py
    runner_body: scenarios/vision-cat.json   # passed to the bot as --runner-body
    scenarios: [vision_describe]

An optional runner_body: (a JSON file, resolved relative to the manifest) is passed to the bot as --runner-body, supplying runner-args data it would normally receive in a /start request body (e.g. a vision bot’s image path). The bot is spawned with the body file’s directory as its working directory, so relative paths inside the body (like an image) resolve next to the file.

Manifest-relative paths (bot/bots_dir, scenarios_dir, runs_dir) resolve relative to the manifest file, so a manifest is portable; the same values passed as CLI overrides resolve against the working directory.

pipecat.evals.suite.capture_pipeline_logs(logs_dir: Path, prefix: str, *, name: str, enabled: bool) → Iterator[None][source]

Capture the harness’s logs for one run and write a single <prefix>.debug.log.

Rather than one file per sub-pipeline, the harness’s logs are buffered in memory (bounded: one scenario’s worth) and written on exit as one file with a ===== <label>: <name> ===== section per pipeline (see PIPELINE_LOG_LABELS). The run is tagged with prefix as its eval_run id and the sink filters on it, so the suite’s concurrent runs never mix into each other’s file. A no-op (and writes nothing) when enabled is False, so the debug log only appears under --debug.

Parameters:

logs_dir – Directory the <prefix>.debug.log is written to.
prefix – Filename stem; also the eval_run id the sink filters on.
name – Human test name shown in each section heading.
enabled – When False, do nothing and write no file.

class pipecat.evals.suite.EvalRun(bot: str, scenario: str, scenario_path: Path, bot_path: Path | None = None, bot_url: str | None = None, runner_body_path: Path | None = None, status: str = 'pending', result: EvalResult | None = None, error: str | None = None, started_at: float | None = None, duration_ms: int | None = None)[source]

Bases: object

Mutable per-(bot, scenario) state, updated in place so a live display can read it.

Parameters:

bot – Display name — the manifest’s bot: (suite) or the bot URL (run).
scenario – Display name (the scenario, without .yaml).
scenario_path – Path to the scenario file.
bot_path – The bot to spawn (suite); None when connecting to bot_url.
bot_url – Connect here instead of spawning (used by pipecat eval run).
runner_body_path – Optional --runner-body JSON for the bot’s runner args.
status – pending, running, or done.
result – The outcome, once the run is done.
error – Spawn/connection error message, if the run failed before producing a result.
started_at – Monotonic start time, for the live elapsed counter.
duration_ms – Wall-clock time the run took, in milliseconds.

bot: str

scenario: str

scenario_path: Path

bot_path: Path | None = None

bot_url: str | None = None

runner_body_path: Path | None = None

status: str = 'pending'

result: EvalResult | None = None

error: str | None = None

started_at: float | None = None

duration_ms: int | None = None

class pipecat.evals.suite.EvalManifest(runs: list[EvalRun], spawn: str, python: str, concurrency: int, base_port: int, runs_dir: Path | None, record: bool, cache_dir: str | None)[source]

Bases: object

A parsed eval-suite manifest.

Parameters:

runs – The (bot, scenario) runs to execute.
spawn – Spawn command template ({python}/{bot}/{port} substituted).
python – Interpreter used to spawn each bot.
concurrency – How many runs to execute at once.
base_port – First port to assign; each run gets base_port + index.
runs_dir – Base for run output (a <name>/ subdir is added), or None.
record – Whether to record conversation audio.
cache_dir – Directory for cached synthesized user audio, or None.

runs: list[EvalRun]

spawn: str

python: str

concurrency: int

base_port: int

runs_dir: Path | None

record: bool

cache_dir: str | None

Parse a manifest YAML into an EvalManifest, with optional overrides.

Any keyword that is not None overrides the corresponding manifest value (so the CLI wins), which means a manifest can be just a suite: list with everything else supplied on the command line. Manifest-relative paths resolve against the manifest’s directory; path overrides resolve against the current working directory.

Parameters:

path – Path to the manifest YAML.
bots_dir – Override for the manifest’s bots_dir (bot paths are relative to it).
scenarios_dir – Override for the manifest’s scenarios_dir.
runs_dir – Override for the manifest’s runs_dir (base for run output).
spawn – Override for the spawn command template.
python – Override for the interpreter used to spawn bots.
concurrency – Override for how many runs execute at once.
base_port – Override for the first port assigned.
record – Override for whether to record conversation audio.
cache_dir – Override for the synthesized-audio cache directory.

Returns:

The parsed EvalManifest.

class pipecat.evals.suite.EvalSuite(manifest: EvalManifest)[source]

Bases: object

Runs the (bot, scenario) runs of an EvalManifest, spawning each bot.

Spawns each bot with its eval transport on its own port, drives it with the harness (pipecat.evals.harness.EvalSession.from_scenario()), and runs several concurrently (up to the manifest’s concurrency). The runs are mutated in place as they execute so a live display can read their progress.

Example:

manifest = EvalManifest.load("manifest.yaml")
suite = EvalSuite(manifest)
suite.filter(pattern="voice")
await suite.run(Path("logs"))

__init__(manifest: EvalManifest)[source]

Initialize the suite from a parsed manifest.

Parameters:: manifest – The parsed EvalManifest; its runs become the suite’s working set (narrowed by filter(), executed by run()).

filter(*, pattern: str | None = None, scenario: str | None = None) → list[EvalRun][source]

Subset the suite’s runs by bot-name substring and/or scenario name.

Narrows runs in place (and returns it) so only matching runs are executed and displayed.

Parameters:

pattern – Keep only runs whose bot name contains this substring.
scenario – Keep only runs for this exact scenario name.

Returns:

The matching runs, in their original order.

async run(logs_dir: Path, *, record_dir: Path | None = None, on_update: Callable[[EvalRun], None] | None = None, debug: bool = False, use_cache: bool = True, default_timeout_ms: int = 60000) → None[source]

Run all of the suite’s runs with the manifest’s concurrency, in place.

Each run is spawned on its own port (base_port + index).

Parameters:

logs_dir – Directory for per-run logs.
record_dir – Directory for per-run conversation recordings, or None.
on_update – Called whenever a run changes status, for live display.
debug – When True, save each run’s combined <run>.debug.log.
use_cache – When False, ignore cached user audio and force fresh synthesis.
default_timeout_ms – Per-expectation budget for expectations without their own within_ms. Defaults to 60s.