frame_processor

Frame processing pipeline infrastructure for Pipecat.

This module provides the core frame processing system that enables building audio/video processing pipelines. It includes frame processors, pipeline management, and frame flow control mechanisms.

class pipecat.processors.frame_processor.FrameDirection(*values)[source]

Bases: Enum

Direction of frame flow in the processing pipeline.

Parameters:
  • DOWNSTREAM – Frames flowing from input to output.

  • UPSTREAM – Frames flowing back from output to input.

DOWNSTREAM = 1
UPSTREAM = 2
class pipecat.processors.frame_processor.FrameProcessorSetup(clock: BaseClock, task_manager: BaseTaskManager, pipeline_worker: PipelineWorker, audio_in_sample_rate: int = 16000, audio_out_sample_rate: int = 24000, enable_expressive_conversation: bool = False, enable_metrics: bool = False, enable_tracing: bool = False, enable_usage_metrics: bool = False, expression_policy: ExpressionPolicy | None = None, observer: BaseObserver | None = None, report_only_initial_ttfb: bool = False, tracing_context: TracingContext | None = None, tool_resources: Any = None)[source]

Bases: object

Configuration parameters for frame processor initialization.

Parameters:
  • audio_in_sample_rate – Input audio sample rate in Hz.

  • audio_out_sample_rate – Output audio sample rate in Hz.

  • clock – The clock instance for timing operations.

  • enable_expressive_conversation – Whether the agent takes part in an expressive conversation (alpha). See enable_expressive_conversation.

  • enable_metrics – Whether to enable performance metrics collection.

  • enable_tracing – Whether to enable OpenTelemetry tracing.

  • enable_usage_metrics – Whether to enable usage metrics collection.

  • expression_policy – Which expression tags the LLMs are offered and when to use them (alpha). See expression_policy.

  • pipeline_worker – The PipelineWorker running this pipeline. Stored on each processor as self.pipeline_worker so processors can reach task-scoped state (e.g. self.pipeline_worker.app_resources).

  • observer – Optional observer for monitoring frame processing events.

  • task_manager – The task manager for handling async operations.

  • tool_resources –

    Deprecated. PipelineWorker continues to populate this with app_resources so that custom FrameProcessor subclasses whose setup() overrides read setup.tool_resources keep working. New code should read setup.pipeline_worker.app_resources instead.

    Deprecated since version 1.2.0: Read setup.pipeline_worker.app_resources instead. Will be removed in 2.0.0.

  • tracing_context – Pipeline-scoped tracing context for span hierarchy.

audio_in_sample_rate: int = 16000
audio_out_sample_rate: int = 24000
enable_expressive_conversation: bool = False
enable_metrics: bool = False
enable_tracing: bool = False
enable_usage_metrics: bool = False
expression_policy: ExpressionPolicy | None = None
observer: BaseObserver | None = None
report_only_initial_ttfb: bool = False
tracing_context: TracingContext | None = None
tool_resources: Any = None
class pipecat.processors.frame_processor.FrameProcessorQueue[source]

Bases: PriorityQueue

A priority queue for the frames arriving at a frame processor.

Frames are dequeued in three tiers: the StartFrame first, then SystemFrame, then data and control frames. Frames of the same tier keep their arrival order.

START_PRIORITY = 1
SYSTEM_PRIORITY = 10
DEFAULT_PRIORITY = 20
__init__()[source]

Initialize the FrameProcessorQueue.

async put(item: tuple[Frame, FrameDirection, Callable[[FrameProcessor, Frame, FrameDirection], Awaitable[None]] | None])[source]

Put an item into the priority queue.

The StartFrame outranks every other frame and SystemFrame frames outrank data and control frames.

Parameters:

item – The frame to enqueue, with its direction and callback.

async get() → Any[source]

Retrieve the next item from the queue.

Waits until an item is available.

Returns:

The highest priority item in the queue.

Return type:

Any

class pipecat.processors.frame_processor.FrameProcessor(*, name: str | None = None, enable_direct_mode: bool = False, metrics: FrameProcessorMetrics | None = None, **kwargs)[source]

Bases: BaseObject

Base class for all frame processors in the pipeline.

Frame processors are the building blocks of Pipecat pipelines, they can be linked to form complex processing pipelines. They receive frames, process them, and pass them to the next or previous processor in the chain. Each frame processor guarantees frame ordering and processes frames in its own task. System frames are also processed in a separate task which guarantees frame priority.

Event handlers available:

  • on_before_process_frame: Called before a frame is processed

  • on_after_process_frame: Called after a frame is processed

  • on_before_push_frame: Called before a frame is pushed

  • on_after_push_frame: Called after a frame is pushed

  • on_error: Called when an error is raised in the frame processing.

  • on_usable_changed: Called with the new value of is_usable when the processor stops or starts being able to do its job.

Example:

@processor.event_handler("on_usable_changed")
async def on_usable_changed(processor, is_usable):
    ...
__init__(*, name: str | None = None, enable_direct_mode: bool = False, metrics: FrameProcessorMetrics | None = None, **kwargs)[source]

Initialize the frame processor.

Parameters:
  • name – Optional name for this processor instance.

  • enable_direct_mode – Whether to process frames immediately or use internal queues.

  • metrics – Optional metrics collector for this processor.

  • **kwargs – Additional arguments passed to parent class.

property id: int

Get the unique identifier for this processor.

Returns:

The unique integer ID of this processor.

property name: str

Get the name of this processor.

Returns:

The name of this processor instance.

property is_usable: bool

Whether this processor can still do its job.

A processor stays usable through failures it might recover from, and becomes unusable once its work can no longer succeed: a provider has rejected its API key, model or voice, or it has failed enough times to stop trying. Sending it more work would only produce more of the same error, so services stop accepting work and stop reconnecting once this is False.

Errors set this as they are reported, so an error handler reading frame.processor.is_usable sees the verdict that came with the error it is handling.

Returns:

True while the processor can still be given work.

async set_usable(is_usable: bool)[source]

Set whether this processor can be given work.

Call this to bring back a processor that became unusable, once whatever stopped it working has been dealt with — new credentials, or a provider that has come back up. Services also do this for themselves when their settings change, since new settings may be the fix.

Parameters:

is_usable – Whether the processor can be given work.

property processors: list[FrameProcessor]

Return the list of sub-processors contained within this processor.

Only compound processors (e.g. pipelines and parallel pipelines) have sub-processors. Non-compound processors will return an empty list.

Returns:

The list of sub-processors if this is a compound processor.

property entry_processors: list[FrameProcessor]

Return the list of entry processors for this processor.

Entry processors are the first processors in a compound processor (e.g. pipelines, parallel pipelines). Note that pipelines can also be an entry processor as pipelines are processors themselves. Non-compound processors will simply return an empty list.

Returns:

The list of entry processors.

property next: FrameProcessor | None

Get the next processor.

Returns:

The next processor, or None if there’s no next processor.

property previous: FrameProcessor | None

Get the previous processor.

Returns:

The previous processor, or None if there’s no previous processor.

property processor_setup: FrameProcessorSetup

Get the configuration this processor was set up with.

Returns:

The FrameProcessorSetup given to setup().

property metrics_enabled: bool

Check if metrics collection is enabled.

Returns:

True if metrics collection is enabled.

property usage_metrics_enabled: bool

Check if usage metrics collection is enabled.

Returns:

True if usage metrics collection is enabled.

property report_only_initial_ttfb: bool

Check if only initial TTFB should be reported.

Returns:

True if only initial time-to-first-byte should be reported.

property pipeline_worker: PipelineWorker

Get the PipelineWorker this processor is running in.

Provides access to worker-scoped state from inside a processor — most notably self.pipeline_worker.app_resources for the application’s shared bag of resources (DB handles, clients, feature flags, etc.).

Returns:

The PipelineWorker instance that set up this processor.

property worker_runner: WorkerRunner

Get the WorkerRunner hosting this processor’s worker.

Use it to reach another worker on the runner by name, e.g. self.worker_runner.get_worker("ui-jobs").

Returns:

The runner this processor’s PipelineWorker is attached to.

property pipeline_task: PipelineWorker

Deprecated alias for pipeline_worker.

Deprecated since version 1.3.0: Use pipeline_worker instead. Will be removed in 2.0.0.

processors_with_metrics()[source]

Return processors that can generate metrics.

Recursively collects all processors that support metrics generation, including those from nested processors.

Returns:

List of frame processors that can generate metrics.

can_generate_metrics() → bool[source]

Check if this processor can generate metrics.

Returns:

True if this processor can generate metrics.

set_core_metrics_data(data: MetricsData)[source]

Set core metrics data for this processor.

Parameters:

data – The metrics data to set.

async start_ttfb_metrics(*, start_time: float | None = None)[source]

Start time-to-first-byte metrics collection.

Parameters:

start_time – Optional timestamp to use as the start time. If None, uses the current time.

async cancel_ttfb_metrics()[source]

Abandon the current time-to-first-byte measurement without reporting it.

async stop_ttfb_metrics(*, end_time: float | None = None)[source]

Stop time-to-first-byte metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.

async process_ttfa_metrics(frame: TTSAudioRawFrame)[source]

Scan a TTS audio frame for the first audible sample and push TTFA.

Should be called for every audio frame until a measurement is produced; the metrics collector tracks leading silence across chunks internally.

Parameters:

frame – The TTS audio frame to inspect.

async stop_ttfat_metrics(*, end_time: float | None = None)[source]

Stop time-to-first-answer-token metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.

async start_processing_metrics(*, start_time: float | None = None)[source]

Start processing metrics collection.

Parameters:

start_time – Optional timestamp to use as the start time. If None, uses the current time.

async stop_processing_metrics(*, end_time: float | None = None)[source]

Stop processing metrics collection and push results.

Parameters:

end_time – Optional timestamp to use as the end time. If None, uses the current time.

async start_llm_usage_metrics(tokens: LLMTokenUsage)[source]

Start LLM usage metrics collection.

Parameters:

tokens – Token usage information for the LLM.

async start_stt_usage_metrics(usage: STTUsage)[source]

Start STT usage metrics collection.

Parameters:

usage – Usage information for the STT operation.

async start_tts_usage_metrics(text: str)[source]

Start TTS usage metrics collection.

Parameters:

text – The text being processed by TTS.

async start_text_aggregation_metrics()[source]

Start text aggregation time metrics collection.

async stop_text_aggregation_metrics()[source]

Stop text aggregation time metrics collection and push results.

async stop_all_metrics()[source]

Stop all active metrics collection.

async setup(setup: FrameProcessorSetup)[source]

Set up the processor with required components.

This is where a processor connects and does its other slow start-up work, so that the pipeline pays for the slowest processor rather than all of them: a pipeline sets its processors up concurrently, so this runs alongside every other processor’s. A resource shared with another processor therefore needs guarding, which acquires() does.

Parameters:

setup – Configuration object containing setup parameters.

async cleanup()[source]

Release this processor’s resources at teardown.

This base implementation cancels only the processor’s internal input/process tasks; tasks created via create_task() are released by an override. Like setup(), this runs concurrently with every other processor’s, so a resource shared with another processor is released with releases().

Link this processor to the next processor in the pipeline.

Parameters:

processor – The processor to link to.

get_clock() → BaseClock[source]

Get the clock used by this processor.

Returns:

The clock instance.

Raises:

Exception – If the clock is not initialized.

get_event_loop() → AbstractEventLoop[source]

Get the event loop used by this processor.

Returns:

The asyncio event loop.

async queue_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM, callback: Callable[[FrameProcessor, Frame, FrameDirection], Awaitable[None]] | None = None)[source]

Queue a frame for processing.

Parameters:
  • frame – The frame to queue.

  • direction – The direction of frame flow.

  • callback – Optional callback to call after processing.

async pause_processing_frames()[source]

Pause processing of queued frames.

async pause_processing_system_frames()[source]

Pause processing of queued system frames.

async resume_processing_frames()[source]

Resume processing of queued frames.

async resume_processing_system_frames()[source]

Resume processing of queued system frames.

async pause_processing_all_frames_until(ready: Callable[[], Awaitable[Any]], *, timeout: float = 5.0)[source]

Hold frames arriving at this processor until ready resolves.

Useful for a processor that cannot act on frames until some condition holds, such as one that establishes a connection in the background. Frames wait in the processor’s queues and are delivered in order once the condition resolves, so nothing is lost.

The frame being processed when this is called is unaffected: the pause takes hold from the next frame onwards. A processor pausing while it handles its StartFrame, for instance, still passes that frame on downstream, so pipeline startup is not delayed.

Both queues are held, so a processor left paused could not process the frames that shut it down. The pause is therefore always lifted: when ready resolves, when timeout elapses, or at teardown, whichever comes first.

Parameters:
  • ready – Awaited to learn when frames can be acted on, e.g. some_event.wait.

  • timeout – Seconds to hold frames before giving up and resuming.

async process_frame(frame: Frame, direction: FrameDirection)[source]

Process a frame.

Parameters:
  • frame – The frame to process.

  • direction – The direction of frame flow.

async push_error(error_msg: str, exception: Exception | None = None, fatal: bool = False, category: ErrorCategory | None = None, force_treat_as_permanent: bool = False)[source]

Creates and pushes an ErrorFrame upstream.

Creates and pushes an ErrorFrame upstream to notify other processors in the pipeline about an error condition. The error frame will include context about which processor generated the error.

Parameters:
  • error_msg – Descriptive message explaining the error condition.

  • exception – Optional exception object that caused the error, if available. This provides additional context for debugging and error handling.

  • fatal –

    Whether this error should be considered fatal to the pipeline. Fatal errors typically cause the entire pipeline to stop processing. Defaults to False for non-fatal errors.

    Deprecated since version 1.8.0: Use force_treat_as_permanent=True instead, when the error leaves its originating processor unable to do its job: the pipeline worker then applies its ProcessorUnusablePolicy, of which CANCEL matches what fatal=True did. For an error that isn’t about that processor’s state, push an EndWorkerFrame after the error to end the pipeline. Will be removed in 2.0.0.

  • category – Why the error occurred, when the caller knows. Leave it unset to let the category be worked out from the exception, or pass ErrorCategory.UNKNOWN to report an error whose cause can’t be attributed — an unexpected one caught by a broad except, say, which may not have come from this processor at all.

  • force_treat_as_permanent – Whether to treat this error as one that will keep recurring, leaving the processor unable to do any more work — having failed too many times to keep trying, say. Only needed for a failure the category doesn’t already convey: leaving it False doesn’t keep the processor usable, since a permanent category costs it its usability on its own. Either way the processor stops being given work, and the pipeline worker decides what to do about it through its ProcessorUnusablePolicy: report the error and keep running (the default), end the pipeline, or cancel it.

Example:

```python
# An error this processor can recover from
await self.push_error("Failed to process audio chunk, skipping")

# An error that leaves this processor unable to do any more work
try:
    result = some_critical_operation()
except Exception as e:
    await self.push_error(
        "Critical operation failed", exception=e, force_treat_as_permanent=True
    )
```
async push_error_frame(error: ErrorFrame, force_treat_as_permanent: bool = False)[source]

Push an error frame upstream.

Parameters:
  • error – The error frame to push. Its deprecated fatal flag still cancels the pipeline; force_treat_as_permanent is the replacement.

  • force_treat_as_permanent – Whether to treat this error as one that will keep recurring, leaving the processor unable to do any more work. Leaving it False doesn’t keep the processor usable — a permanent category costs it its usability either way. See push_error().

async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)[source]

Push a frame to the next processor in the pipeline.

Parameters:
  • frame – The frame to push.

  • direction – The direction to push the frame.

async broadcast_interruption()[source]

Broadcast an InterruptionFrame both upstream and downstream.

async push_interruption_task_frame_and_wait(*, timeout: float = 5.0)[source]

Push an interruption task frame upstream and wait for the interruption.

Deprecated since version 0.0.104: Use broadcast_interruption() instead. This method now delegates to broadcast_interruption() and ignores timeout. Will be removed in 2.0.0.

async broadcast_frame(frame_cls: type[Frame], **kwargs)[source]

Broadcasts a frame of the specified class upstream and downstream.

This method creates two instances of the given frame class using the provided keyword arguments (without deep-copying them) and pushes them upstream and downstream.

Parameters:
  • frame_cls – The class of the frame to be broadcasted.

  • **kwargs – Keyword arguments to be passed to the frame’s constructor.

async broadcast_frame_instance(frame: Frame)[source]

Broadcasts a frame instance upstream and downstream.

This method creates two new frame instances shallow-copying all fields from the original frame except id and name, which get fresh values.

Parameters:

frame – The frame instance to broadcast.

Note

Prefer using broadcast_frame() when possible, as it is more efficient. This method should only be used when you are not the creator of the frame and need to broadcast an existing instance.

has_queued_frame(frame_type: type[Frame]) → bool[source]

Return True if a frame of the given type is waiting in the processing queue.

Delegates to FrameQueue.has_frame() so the check is O(distinct enqueued types) with no queue scanning.

Parameters:

frame_type – The frame class to look for.

Returns:

True if at least one matching frame is queued.