eval

pipecat eval typer commands.

Thin CLI wrappers over pipecat.evals — all the load/run logic lives in that module. Mounted as the eval subcommand of the pipecat CLI (see pipecat.cli.main) and also reachable as python -m pipecat.evals.

pipecat.cli.commands.eval.run(scenarios: list[Path] = <typer.models.ArgumentInfo object>, bot_url: str = <typer.models.OptionInfo object>, verbose: bool = <typer.models.OptionInfo object>, audio: bool = <typer.models.OptionInfo object>, record_dir: str = <typer.models.OptionInfo object>, cache_dir: str = <typer.models.OptionInfo object>, no_cache: bool = <typer.models.OptionInfo object>, timeout: int = <typer.models.OptionInfo object>, logs_dir: str = <typer.models.OptionInfo object>, debug: bool = <typer.models.OptionInfo object>, stop_bot: bool = <typer.models.OptionInfo object>, trigger_disconnect: bool = <typer.models.OptionInfo object>) → None

Run one or more evals against an already-running bot.

A list of scenarios is treated as a one-bot suite: configs are printed up front, then a live dashboard (or streamed lines when piped / --verbose) shows each scenario’s status and timing, the running tally, and the total time, sharing the display with pipecat eval suite.

pipecat.cli.commands.eval.suite(manifest_path: Path = <typer.models.ArgumentInfo object>, pattern: str = <typer.models.OptionInfo object>, scenario: str = <typer.models.OptionInfo object>, name: str = <typer.models.OptionInfo object>, runs_dir: Path = <typer.models.OptionInfo object>, bots_dir: Path = <typer.models.OptionInfo object>, scenarios_dir: Path = <typer.models.OptionInfo object>, concurrency: int = <typer.models.OptionInfo object>, base_port: int = <typer.models.OptionInfo object>, cache_dir: str = <typer.models.OptionInfo object>, no_cache: bool = <typer.models.OptionInfo object>, timeout: int = <typer.models.OptionInfo object>, spawn: str = <typer.models.OptionInfo object>, python: str = <typer.models.OptionInfo object>, audio: bool = <typer.models.OptionInfo object>, debug: bool = <typer.models.OptionInfo object>) → None

Spawn the bots in a manifest and run their scenarios concurrently.

Everything except the suite: list can be set in the manifest or overridden here (the command line wins), so a manifest can be just a suite: list.