judge

EvalJudge LLM for content assertions in behavioral evaluations.

Uses an LLM to decide whether a bot’s response satisfies a natural-language criterion (judge: "describes the bot's capabilities"). The judge runs as a one-shot, out-of-pipeline inference via pipecat.services.openai.base_llm.BaseOpenAILLMService.run_inference(), so it works with any pipecat LLM service backed by an OpenAI-compatible API (OpenAI, Ollama, Together, etc.).

The judge keeps the conversation as an LLMContext: the harness feeds it the user turns and the bot’s reply segments, and evaluate judges the most recent reply against the criterion in that context. This lets it resolve a terse or ambiguous reply (e.g. “That’s four”, which an STT pass might render as “That’s for”) that wouldn’t make sense in isolation.

Verdicts are cached by (criterion, conversation) hash so that re-runs are stable and so that a single scenario doesn’t pay multiple judge round-trips for the same assertion.

Example:

from pipecat.services.ollama.llm import OLLamaLLMService

service = OLLamaLLMService(settings=OLLamaLLMService.Settings(model="gemma2:9b"))
judge = EvalJudge(service)
judge.add_user_message("What can you help me with?")
judge.add_assistant_message("I can answer questions, set reminders, and look things up.")
verdict = await judge.evaluate("describes the bot's capabilities")
if not verdict.passed:
    print(f"judge said no: {verdict.reason}")

class pipecat.evals.judge.JudgeVerdict(verdict: str, reason: str, raw_response: str)

Bases: object

Outcome of a single judge call.

Parameters:

• verdict – "yes" (satisfies), "no" (substantive answer that fails), or "continue" (interim/filler/incomplete — re-judge once more text arrives).

• reason – One-sentence justification.

• raw_response – The judge LLM’s raw text, for diagnostics.

verdict: str

reason: str

raw_response: str

property passed: bool

True only when the verdict is a definite "yes".

class pipecat.evals.judge.EvalJudge(service: LLMService[Any], *, max_tokens: int = 200)

Bases: object

Wraps a pipecat LLM service and runs single-shot evaluations.

Parameters:

• service – A pipecat LLM service with a run_inference() method (i.e. BaseOpenAILLMService or any subclass: OpenAI, Ollama, etc.).

• max_tokens – Cap on the judge’s response length. Default 200 — enough for a JSON verdict + short reason.

__init__(service: LLMService[Any], *, max_tokens: int = 200)

Initialize the judge with a configured pipecat LLM service.

Parameters:

• service – A pipecat LLM service exposing run_inference().

• max_tokens – Cap on the judge’s response length.

classmethod from_config(judge_config: dict | None) → EvalJudge

Build an EvalJudge from a scenario’s judge.eval: config block.

Honors a custom factory (dotted path to a callable taking (config) and returning a pipecat LLM service with run_inference()); otherwise dispatches on the service name (default "ollama"). Add providers by extending this. To use a fully custom judge, construct EvalJudge directly and pass it to pipecat.evals.harness.EvalSession.from_scenario().

Parameters:

judge_config – Mapping with keys service (default "ollama"), model (default "gemma2:9b"), and optional endpoint (service-specific default if omitted). None uses all defaults.

Returns:

A configured EvalJudge.

Raises:

ValueError – If service is unknown (matching pipecat.evals.speech.EvalSpeech.from_config() and pipecat.evals.transcribe.EvalTranscriber.from_config()).

Example:

# In the scenario: judge.eval.factory: "my_pkg.make_judge_llm"
def make_judge_llm(config):
    return TogetherLLMService(...)  # any service exposing run_inference()

add_user_message(text: str | None) → None

Record a user turn in the conversation the judge evaluates against.

Called by the harness when it sends a user turn, so a later reply can be judged in context (e.g. a terse “That’s four” after “What is two plus two?”).

Parameters:

text – The user’s utterance, or None for a bot-first turn (ignored).

add_assistant_message(text: str | None) → None

Append a streamed segment of the bot’s current reply to the conversation.

The bot’s reply may arrive in several segments; each is added as its own assistant message, so the accumulated conversation is exactly what the judge sees — there is no separate “commit” step.

Parameters:

text – The new reply segment; empty or None is ignored.

async evaluate(criterion: str) → JudgeVerdict

Judge whether the bot’s most recent reply satisfies criterion.

Evaluates the conversation built up via add_user_message() and add_assistant_message(). The judge’s own answer is never written back into that conversation.

Parameters:

criterion – Natural-language description of what the reply should express.

Returns:

A JudgeVerdict with the pass/fail decision and a one-sentence justification. Cached by (criterion, conversation) so the same assertion over the same conversation hits the judge only once.