scenario

Scenario file format for Pipecat behavioral evaluations.

A scenario is a YAML file describing a scripted conversation and the semantic events expected to flow back from the bot. Simple example:

name: simple_user_input
turns:
  - user: "hello world"
    expect:
      - event: user_started_speaking
      - event: user_transcription
        text_contains: "hello world"

The runner (see pipecat.evals.harness) loads the scenario, connects to the bot’s eval transport over RTVI, drives each turn, collects the RTVI events the bot emits, and asserts on them in order.

Event names are the friendly names the harness maps RTVI server messages onto: user_started_speaking, user_stopped_speaking, vad_user_started_speaking, vad_user_stopped_speaking, user_transcription, llm_started, response, llm_response, tts_response, function_call. The vad_* events are the raw VAD signal, useful as a timing anchor when a turn-detection strategy gates or defers the turn-level user_stopped_speaking (e.g. filtering incomplete turns).

The bot’s reply can be asserted three ways:

response the transcription of the bot’s actual synthesized audio (a: local STT — Moonshine or Whisper — run by the harness) in audio modality, or the LLM text in text modality. The real end-to-end check — prefer this.
llm_response the LLM’s text output (bot-llm-text). Available in both: modalities.
tts_response the text the TTS reports speaking (bot-tts-text, with: word timing). Audio modality only.

Supported expectation fields (per event):

event: <name> required — event type name within_ms: <int> latency budget from the most recent anchor

(optional; defaults to 60s when omitted)

text_contains: <str> substring check on the event’s text content calls: for function_call — the set of calls the turn

should make, matched by name in any order; the expectation passes only when all are found:

- event: function_call
  calls:
    - name: get_current_weather
      args: { location: San Francisco }
    - name: get_restaurant_recommendation

eval: <str> natural-language criterion the event’s text content

must satisfy, evaluated by a judge LLM (see pipecat.evals.judge).

absent: true invert the expectation: assert that NO event of

this type arrives before the within_ms budget expires (default 60s — set within_ms explicitly to keep the quiet-window wait short). Matches on event type only, so it cannot be combined with text_contains, eval:, or calls:. Used for duplicate-output regressions:

- event: response
  eval: "answers the question"
- event: response
  absent: true
  within_ms: 30000

Instead of user:, a turn may press DTMF keys with dtmf: (the two are mutually exclusive — you press keys or you talk):

turns:
  - dtmf: "123#"            # quote it: an unquoted # starts a YAML comment
    expect:
      - event: user_transcription
        text_contains: "DTMF: 123#"
      - event: response
        eval: "confirms the entered digits"

Each character is sent as one InputDTMFFrame (0-9, *, #), regardless of the scenario’s user/judge modality. A bot running a DTMFAggregator accumulates them and flushes — on the # terminator or its idle timeout — into a DTMF: ... transcription it reacts to.

A turn may also include send_after: to schedule its user/dtmf send relative to a prior event (used for interruption / barge-in tests), or image: (a path, relative to the scenario file) to register an image for the turn — when a function-calling-video bot requests a user image, the eval transport serves it. send_after with only delay_ms (no event) is a pure time delay relative to the previous send — handy for pacing keypresses across dtmf turns to exercise the aggregator’s idle-timeout flush.

expect: is optional; omit it for a turn that only sends input or only waits.

Top-level optional fields:

context: LLM messages the bot’s context should start from. When given, the: harness sends them before driving turns (replacing the bot’s context); omit to leave the bot’s own context untouched.

user: how user turns are delivered:

    user:
      modality: audio          # audio | text (default text)
      speech:                  # required when modality is audio
        service: kokoro        # local TTS that synthesizes the user turns
        voice: af_heart
        sample_rate: 16000     # optional

``audio`` streams synthesized user audio to the bot (exercising its
STT for real); ``text`` (the default) sends RTVI ``send-text``.

judge: what the judge evaluates, and with which LLM:

    judge:
      modality: audio          # audio | text (default text)
      eval:                    # the judge LLM (default ollama)
        service: openai
        model: gpt-4o-mini
      transcription:           # required when modality is audio
        service: moonshine     # STT for the bot's audio (or whisper)
        model: small-streaming # optional
        padding_secs: 0        # optional; silence padded around the
                               # segment (default: 2)

``audio`` makes the bot speak and judges the transcription of its
actual audio (``tts_response``); ``text`` (the default) skips TTS and
judges the LLM text (``llm_response``), which is faster and silent.

Any value can be pulled from a separate file with !include, resolved relative to the scenario file’s directory. This is handy for sharing the judge: and user: blocks across scenarios:

user: !include user_audio.yaml
judge: !include judge_audio.yaml

class pipecat.evals.scenario.EvalFunctionCall(name: str | None = None, args: dict | None = None)[source]

Bases: object

One expected function call within a function_call expectation.

Parameters:

name – The function name to match. None matches any call (used by a bare function_call expectation that just asserts a call happened).
args – Optional subset check on the call’s arguments (every listed key/value must be present; extra arguments are ignored).

name: str | None = None

args: dict | None = None

class pipecat.evals.scenario.EvalExpectation(event: str, within_ms: int | None = None, text_contains: str | None = None, calls: list[EvalFunctionCall] | None = None, eval: str | None = None, absent: bool = False)[source]

Bases: object

A single expected event in a scenario turn.

Parameters:

event – Required — the semantic event name (e.g. user_stopped_speaking).
within_ms – Optional latency budget, measured from the turn’s user send — all of a turn’s expectations share that one anchor, so a stalled turn fails within a single budget rather than one per expectation. For audio turns the anchor is when the utterance was sent, not when it finishes playing out of the transport’s virtual mic. Defaults to 60s when omitted, so timing isn’t asserted unless set explicitly.
text_contains – Optional substring check on the event’s text content (llm_response.text or user_transcription.transcript).
calls – For a function_call event, the set of calls expected in the turn. They are matched by name in any order and the expectation passes only when all of them are found. Built from calls: in the YAML, or from the single name:/args: shorthand.
eval – Optional natural-language criterion the event’s text content must satisfy. Evaluated by a judge LLM. Only meaningful on the bot-generated text events: response, llm_response, and tts_response.
absent – When True, the expectation is inverted: it passes only when NO event of this type arrives before the within_ms budget expires, and fails as soon as one does. Matches on event type only; text_contains, eval, and calls are not allowed alongside it.

event: str

within_ms: int | None = None

text_contains: str | None = None

calls: list[EvalFunctionCall] | None = None

eval: str | None = None

absent: bool = False

class pipecat.evals.scenario.EvalSendAfter(event: str | None, delay_ms: int)[source]

Bases: object

Scheduling for when a turn’s input (user or dtmf) is sent.

When set on a EvalTurn, the harness waits for event to have been seen (either earlier in the run or arriving now), then waits an additional delay_ms before sending the turn’s input. Used for barge-in tests: send_after: {event: llm_started, delay_ms: 500} means “interrupt 500ms after the bot started responding.”

event is optional: a bare send_after: {delay_ms: 500} is a pure time delay with no event anchor (500ms after the previous turn’s send). Handy for pacing keypresses across dtmf turns, where there is no per-key event to anchor on.

Parameters:

event – Name of the event to schedule from, or None for a pure delay_ms time delay with no event anchor.
delay_ms – Additional delay in milliseconds after the event was received (or, when event is None, after the previous turn’s send).

event: str | None

delay_ms: int

class pipecat.evals.scenario.EvalTurn(user: str | None, expect: list[EvalExpectation] = <factory>, dtmf: str | None = None, send_after: EvalSendAfter | None = None, image: str | None = None)[source]

Bases: object

One turn in a scenario.

A turn drives the bot one of three ways: the harness sends a user utterance (the person speaks), it sends a dtmf keypress sequence (the person presses keys), or it is observation-only (neither field — useful for bot-first scenarios like opening greetings). user and dtmf are mutually exclusive: a turn is one or the other.

Parameters:

user – Optional text the harness sends as the user’s turn — an RTVI send-text in text modality, or synthesized speech (raw-audio) in audio modality. If absent, the turn just waits for and asserts on expected events.
dtmf – Optional DTMF keypad sequence the harness sends, one InputDTMFFrame per character (e.g. "123#"). Each character must be a valid KeypadEntry (0-9, *, #). Mutually exclusive with user. The keys are injected the same way regardless of the scenario’s user/judge modality; a bot with a DTMFAggregator turns them into a transcription it reacts to. Quote the value in YAML (dtmf: "123#") — an unquoted # starts a comment.
expect – Expected events, in the order they should arrive. Optional — omit it for a pure pacing/observation turn (e.g. a dtmf turn that only presses keys, with the assertion on a later turn).
send_after – Optional schedule for when the turn’s input should fire. Only meaningful when user or dtmf is set.
image – Optional path to an image to register for this turn (resolved relative to the scenario file). When a function-calling-video bot requests a user image during the turn, the eval transport serves this one. Stays registered until a later turn provides a different image.

user: str | None

expect: list[EvalExpectation]

dtmf: str | None = None

send_after: EvalSendAfter | None = None

image: str | None = None

class pipecat.evals.scenario.EvalScenario(name: str, turns: list[~pipecat.evals.scenario.EvalTurn], context: list[dict] = <factory>, judge: dict = <factory>, bot_audio: bool = False, transcriber: dict | None = None, user_audio: dict | None = None, trigger_disconnect: bool = False, source_path: ~pathlib.Path | None = None)[source]

Bases: object

A parsed scenario file.

Parameters:

name – The eval name (from name:).
turns – Ordered list of turns.
context – LLM messages the bot’s context should start from for this eval. When non-empty, the harness sends them as an eval-context client message right after the bot-ready handshake (the eval serializer turns it into an LLMMessagesUpdateFrame, which replaces the context); bots without an LLM context aggregator ignore the frame. Omitted or empty (the default): the harness sends nothing and the bot keeps the context it set up itself.
judge – Judge LLM configuration dict with keys service, model, and optional endpoint. Defaults to {"service": "ollama", "model": "gemma2:9b"}.
bot_audio – Whether the bot produces speech, derived from judge.modality. False (text, the default): the bot skips TTS — the harness configures skip-TTS at connect, so even an on-connect greeting is silent. True (audio): the bot speaks, and the judge evaluates the transcription of its actual audio.
transcriber – Parsed from the judge.transcription: block; the STT config (service defaults to moonshine, plus model) used to transcribe the bot’s audio for the response event (None in text modality).
user_audio – TTS config the harness uses to generate user audio. When present, the harness streams RTVI raw-audio (not send-text) to the bot, exercising its STT for real. Mapping with service, voice, and optional model / sample_rate / api_key. Omit for text-only evals (default).
trigger_disconnect – Whether the harness fires the bot’s on_client_disconnected handler when this scenario’s connection ends. Bots often cancel their pipeline there, so this is False by default to avoid that between scenarios; set True to exercise the bot’s disconnect path. Independent of --stop-bot, which tears the bot down via eval-cancel regardless of the handler.
source_path – Path the scenario was loaded from, for error messages.

name: str

turns: list[EvalTurn]

context: list[dict]

judge: dict

bot_audio: bool = False

transcriber: dict | None = None

user_audio: dict | None = None

trigger_disconnect: bool = False

source_path: Path | None = None

classmethod load(path: str | Path) → EvalScenario[source]

Parse a scenario YAML file into an EvalScenario.

Parameters:

path – Path to a YAML file with the scenario schema.

Returns:

The parsed scenario.

Raises:

ValueError – If the file structure is invalid.
FileNotFoundError – If the path doesn’t exist.

pipecat.evals.scenario.describe_config(scenario: EvalScenario, *, color: bool = False) → str[source]

Two-line summary of a scenario’s user + judge config, for pre-run logs.

Parameters:

scenario – The parsed scenario to summarize.
color – When True, ANSI-color each segment’s keyword by category (modality, service, judge LLM) so they’re easy to tell apart.

Returns:

A user line and a judge line, each a set of key: value segments separated by |, e.g.:

user  -> modality: audio | speech: kokoro/af_heart
judge -> modality: audio | transcription: moonshine/small-streaming | eval: ollama/gemma2:9b