script_session

The scripted session: runs a scripted scenario against a bot.

Example:

for scenario in EvalScenarioFile.load("scenarios/greeting.yaml"):
    result = await EvalScriptSession.from_scenario(scenario, "ws://localhost:7860").run()
    if result.passed:
        print(scenario.name, "PASS")
    else:
        for f in result.failures:
            print(f"  {f}")

# Per-turn outcomes, for a scenario scored a turn at a time.
scored = [t for t in result.turns if t.status != "not_run"]
print(f"{sum(1 for t in scored if t.status == 'passed')}/{len(scored)} turns")
class pipecat.evals.script_session.EvalScriptSession(scenario: EvalScriptScenario, bot_url: str, *, params: EvalSessionParams | None = None, on_progress: Callable[[EvalScriptTurnProgress], None] | None = None, judge: EvalJudge | None = None, user_tts: CachingTTSService | None = None, bot_stt: STTService | None = None)[source]

Bases: EvalSession[EvalScriptResult]

Runs one EvalScriptScenario against a bot.

Build one with from_scenario(), which constructs the judge, the user TTS, and the STT the scenario needs, then await run().

Example:

@session.event_handler("on_progress")
async def on_progress(session, progress):
    print(progress.event_name, progress.status)
__init__(scenario: EvalScriptScenario, bot_url: str, *, params: EvalSessionParams | None = None, on_progress: Callable[[EvalScriptTurnProgress], None] | None = None, judge: EvalJudge | None = None, user_tts: CachingTTSService | None = None, bot_stt: STTService | None = None)[source]

Initialize the eval session.

The services come pre-built, or None where the scenario has no use for them; from_scenario() constructs the ones a scenario needs.

Parameters:

  • scenario – The parsed scenario to run.

  • bot_url – WebSocket URL of the bot’s eval transport.

  • params – How the run behaves; None for the defaults.

  • on_progress –

    Optional callback invoked with a EvalScriptTurnProgress as each turn and expectation resolves (used for verbose output).

    Deprecated since version 1.9.0: Use the on_progress event handler instead. Will be removed in 2.0.0.

  • judge – The EvalJudge for eval: assertions, or None if the scenario has none.

  • user_tts – The user-audio TTS, or None for text mode.

  • bot_stt – The bot-audio STT for the response transcription, or None when the scenario has none.

classmethod from_scenario(scenario: EvalScriptScenario, bot_url: str, *, params: EvalSessionParams | None = None, on_progress: Callable[[EvalScriptTurnProgress], None] | None = None, judge: EvalJudge | None = None, user_tts: CachingTTSService | None = None, bot_stt: STTService | None = None, connect_timeout_s: float | None = None, default_timeout_ms: int | None = None, record_path: str | None = None, cache_dir: str | None = None, use_cache: bool | None = None, stop_bot: bool | None = None, trigger_disconnect: bool | None = None) → EvalScriptSession[source]

Build a ready-to-run session from a scenario, constructing the services it needs.

Pass judge, user_tts, or bot_stt to use your own. Then await run():

session = EvalScriptSession.from_scenario(scenario, "ws://localhost:7860")
result = await session.run()
Parameters:

  • scenario – The parsed scenario to run.

  • bot_url – WebSocket URL of the bot’s eval transport.

  • params – How the run behaves; None for the defaults.

  • on_progress –

    Optional per-turn/expectation progress callback (verbose).

    Deprecated since version 1.9.0: Use the on_progress event handler instead. Will be removed in 2.0.0.

  • judge – Override the judge (default: built from scenario.judge when the scenario has eval: assertions).

  • user_tts – Override the user-audio TTS (default: built from scenario.user_speech in audio mode).

  • bot_stt – Override the bot-audio STT (default: built from scenario.transcriber when the scenario asserts response).

  • connect_timeout_s –

    The params field of the same name.

    Deprecated since version 1.9.0: Use params instead. Will be removed in 2.0.0.

  • default_timeout_ms –

    The params field of the same name.

    Deprecated since version 1.9.0: Use params instead. Will be removed in 2.0.0.

  • record_path –

    The params field of the same name.

    Deprecated since version 1.9.0: Use params instead. Will be removed in 2.0.0.

  • cache_dir –

    The params field of the same name.

    Deprecated since version 1.9.0: Use params instead. Will be removed in 2.0.0.

  • use_cache –

    The params field of the same name.

    Deprecated since version 1.9.0: Use params instead. Will be removed in 2.0.0.

  • stop_bot –

    The params field of the same name.

    Deprecated since version 1.9.0: Use params instead. Will be removed in 2.0.0.

  • trigger_disconnect –

    The params field of the same name.

    Deprecated since version 1.9.0: Use params instead. Will be removed in 2.0.0.

Returns:

A configured session, ready for run().