frame_processor
Frame processing pipeline infrastructure for Pipecat.
This module provides the core frame processing system that enables building audio/video processing pipelines. It includes frame processors, pipeline management, and frame flow control mechanisms.
- class pipecat.processors.frame_processor.FrameDirection(*values)[source]
Bases:
EnumDirection of frame flow in the processing pipeline.
- Parameters:
DOWNSTREAM – Frames flowing from input to output.
UPSTREAM – Frames flowing back from output to input.
- DOWNSTREAM = 1
- UPSTREAM = 2
- class pipecat.processors.frame_processor.FrameProcessorSetup(clock: BaseClock, task_manager: BaseTaskManager, pipeline_worker: PipelineWorker, audio_in_sample_rate: int = 16000, audio_out_sample_rate: int = 24000, enable_expressive_conversation: bool = False, enable_metrics: bool = False, enable_tracing: bool = False, enable_usage_metrics: bool = False, expression_policy: ExpressionPolicy | None = None, observer: BaseObserver | None = None, report_only_initial_ttfb: bool = False, tracing_context: TracingContext | None = None, tool_resources: Any = None)[source]
Bases:
objectConfiguration parameters for frame processor initialization.
- Parameters:
audio_in_sample_rate – Input audio sample rate in Hz.
audio_out_sample_rate – Output audio sample rate in Hz.
clock – The clock instance for timing operations.
enable_expressive_conversation – Whether the agent takes part in an expressive conversation (alpha). See
enable_expressive_conversation.enable_metrics – Whether to enable performance metrics collection.
enable_tracing – Whether to enable OpenTelemetry tracing.
enable_usage_metrics – Whether to enable usage metrics collection.
expression_policy – Which expression tags the LLMs are offered and when to use them (alpha). See
expression_policy.pipeline_worker – The
PipelineWorkerrunning this pipeline. Stored on each processor asself.pipeline_workerso processors can reach task-scoped state (e.g.self.pipeline_worker.app_resources).observer – Optional observer for monitoring frame processing events.
task_manager – The task manager for handling async operations.
tool_resources –
Deprecated.
PipelineWorkercontinues to populate this withapp_resourcesso that customFrameProcessorsubclasses whosesetup()overrides readsetup.tool_resourceskeep working. New code should readsetup.pipeline_worker.app_resourcesinstead.Deprecated since version 1.2.0: Read
setup.pipeline_worker.app_resourcesinstead. Will be removed in 2.0.0.tracing_context – Pipeline-scoped tracing context for span hierarchy.
- expression_policy: ExpressionPolicy | None = None
- observer: BaseObserver | None = None
- tracing_context: TracingContext | None = None
- tool_resources: Any = None
- class pipecat.processors.frame_processor.FrameProcessorQueue[source]
Bases:
PriorityQueueA priority queue for the frames arriving at a frame processor.
Frames are dequeued in three tiers: the StartFrame first, then SystemFrame, then data and control frames. Frames of the same tier keep their arrival order.
- START_PRIORITY = 1
- SYSTEM_PRIORITY = 10
- DEFAULT_PRIORITY = 20
- async put(item: tuple[Frame, FrameDirection, Callable[[FrameProcessor, Frame, FrameDirection], Awaitable[None]] | None])[source]
Put an item into the priority queue.
The StartFrame outranks every other frame and SystemFrame frames outrank data and control frames.
- Parameters:
item – The frame to enqueue, with its direction and callback.
- class pipecat.processors.frame_processor.FrameProcessor(*, name: str | None = None, enable_direct_mode: bool = False, metrics: FrameProcessorMetrics | None = None, **kwargs)[source]
Bases:
BaseObjectBase class for all frame processors in the pipeline.
Frame processors are the building blocks of Pipecat pipelines, they can be linked to form complex processing pipelines. They receive frames, process them, and pass them to the next or previous processor in the chain. Each frame processor guarantees frame ordering and processes frames in its own task. System frames are also processed in a separate task which guarantees frame priority.
Event handlers available:
on_before_process_frame: Called before a frame is processed
on_after_process_frame: Called after a frame is processed
on_before_push_frame: Called before a frame is pushed
on_after_push_frame: Called after a frame is pushed
on_error: Called when an error is raised in the frame processing.
on_usable_changed: Called with the new value of is_usable when the processor stops or starts being able to do its job.
Example:
@processor.event_handler("on_usable_changed") async def on_usable_changed(processor, is_usable): ...
- __init__(*, name: str | None = None, enable_direct_mode: bool = False, metrics: FrameProcessorMetrics | None = None, **kwargs)[source]
Initialize the frame processor.
- Parameters:
name – Optional name for this processor instance.
enable_direct_mode – Whether to process frames immediately or use internal queues.
metrics – Optional metrics collector for this processor.
**kwargs – Additional arguments passed to parent class.
- property id: int
Get the unique identifier for this processor.
- Returns:
The unique integer ID of this processor.
- property is_usable: bool
Whether this processor can still do its job.
A processor stays usable through failures it might recover from, and becomes unusable once its work can no longer succeed: a provider has rejected its API key, model or voice, or it has failed enough times to stop trying. Sending it more work would only produce more of the same error, so services stop accepting work and stop reconnecting once this is False.
Errors set this as they are reported, so an error handler reading
frame.processor.is_usablesees the verdict that came with the error it is handling.- Returns:
True while the processor can still be given work.
- async set_usable(is_usable: bool)[source]
Set whether this processor can be given work.
Call this to bring back a processor that became unusable, once whatever stopped it working has been dealt with — new credentials, or a provider that has come back up. Services also do this for themselves when their settings change, since new settings may be the fix.
- Parameters:
is_usable – Whether the processor can be given work.
- property processors: list[FrameProcessor]
Return the list of sub-processors contained within this processor.
Only compound processors (e.g. pipelines and parallel pipelines) have sub-processors. Non-compound processors will return an empty list.
- Returns:
The list of sub-processors if this is a compound processor.
- property entry_processors: list[FrameProcessor]
Return the list of entry processors for this processor.
Entry processors are the first processors in a compound processor (e.g. pipelines, parallel pipelines). Note that pipelines can also be an entry processor as pipelines are processors themselves. Non-compound processors will simply return an empty list.
- Returns:
The list of entry processors.
- property next: FrameProcessor | None
Get the next processor.
- Returns:
The next processor, or None if there’s no next processor.
- property previous: FrameProcessor | None
Get the previous processor.
- Returns:
The previous processor, or None if there’s no previous processor.
- property processor_setup: FrameProcessorSetup
Get the configuration this processor was set up with.
- Returns:
The
FrameProcessorSetupgiven tosetup().
- property metrics_enabled: bool
Check if metrics collection is enabled.
- Returns:
True if metrics collection is enabled.
- property usage_metrics_enabled: bool
Check if usage metrics collection is enabled.
- Returns:
True if usage metrics collection is enabled.
- property report_only_initial_ttfb: bool
Check if only initial TTFB should be reported.
- Returns:
True if only initial time-to-first-byte should be reported.
- property pipeline_worker: PipelineWorker
Get the
PipelineWorkerthis processor is running in.Provides access to worker-scoped state from inside a processor — most notably
self.pipeline_worker.app_resourcesfor the application’s shared bag of resources (DB handles, clients, feature flags, etc.).- Returns:
The
PipelineWorkerinstance that set up this processor.
- property worker_runner: WorkerRunner
Get the
WorkerRunnerhosting this processor’s worker.Use it to reach another worker on the runner by name, e.g.
self.worker_runner.get_worker("ui-jobs").- Returns:
The runner this processor’s
PipelineWorkeris attached to.
- property pipeline_task: PipelineWorker
Deprecated alias for
pipeline_worker.Deprecated since version 1.3.0: Use
pipeline_workerinstead. Will be removed in 2.0.0.
- processors_with_metrics()[source]
Return processors that can generate metrics.
Recursively collects all processors that support metrics generation, including those from nested processors.
- Returns:
List of frame processors that can generate metrics.
- can_generate_metrics() bool[source]
Check if this processor can generate metrics.
- Returns:
True if this processor can generate metrics.
- set_core_metrics_data(data: MetricsData)[source]
Set core metrics data for this processor.
- Parameters:
data – The metrics data to set.
- async start_ttfb_metrics(*, start_time: float | None = None)[source]
Start time-to-first-byte metrics collection.
- Parameters:
start_time – Optional timestamp to use as the start time. If None, uses the current time.
- async cancel_ttfb_metrics()[source]
Abandon the current time-to-first-byte measurement without reporting it.
- async stop_ttfb_metrics(*, end_time: float | None = None)[source]
Stop time-to-first-byte metrics collection and push results.
- Parameters:
end_time – Optional timestamp to use as the end time. If None, uses the current time.
- async process_ttfa_metrics(frame: TTSAudioRawFrame)[source]
Scan a TTS audio frame for the first audible sample and push TTFA.
Should be called for every audio frame until a measurement is produced; the metrics collector tracks leading silence across chunks internally.
- Parameters:
frame – The TTS audio frame to inspect.
- async stop_ttfat_metrics(*, end_time: float | None = None)[source]
Stop time-to-first-answer-token metrics collection and push results.
- Parameters:
end_time – Optional timestamp to use as the end time. If None, uses the current time.
- async start_processing_metrics(*, start_time: float | None = None)[source]
Start processing metrics collection.
- Parameters:
start_time – Optional timestamp to use as the start time. If None, uses the current time.
- async stop_processing_metrics(*, end_time: float | None = None)[source]
Stop processing metrics collection and push results.
- Parameters:
end_time – Optional timestamp to use as the end time. If None, uses the current time.
- async start_llm_usage_metrics(tokens: LLMTokenUsage)[source]
Start LLM usage metrics collection.
- Parameters:
tokens – Token usage information for the LLM.
- async start_stt_usage_metrics(usage: STTUsage)[source]
Start STT usage metrics collection.
- Parameters:
usage – Usage information for the STT operation.
- async start_tts_usage_metrics(text: str)[source]
Start TTS usage metrics collection.
- Parameters:
text – The text being processed by TTS.
- async stop_text_aggregation_metrics()[source]
Stop text aggregation time metrics collection and push results.
- async setup(setup: FrameProcessorSetup)[source]
Set up the processor with required components.
This is where a processor connects and does its other slow start-up work, so that the pipeline pays for the slowest processor rather than all of them: a pipeline sets its processors up concurrently, so this runs alongside every other processor’s. A resource shared with another processor therefore needs guarding, which
acquires()does.- Parameters:
setup – Configuration object containing setup parameters.
- async cleanup()[source]
Release this processor’s resources at teardown.
This base implementation cancels only the processor’s internal input/process tasks; tasks created via
create_task()are released by an override. Likesetup(), this runs concurrently with every other processor’s, so a resource shared with another processor is released withreleases().
- link(processor: FrameProcessor)[source]
Link this processor to the next processor in the pipeline.
- Parameters:
processor – The processor to link to.
- get_clock() BaseClock[source]
Get the clock used by this processor.
- Returns:
The clock instance.
- Raises:
Exception – If the clock is not initialized.
- get_event_loop() AbstractEventLoop[source]
Get the event loop used by this processor.
- Returns:
The asyncio event loop.
- async queue_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM, callback: Callable[[FrameProcessor, Frame, FrameDirection], Awaitable[None]] | None = None)[source]
Queue a frame for processing.
- Parameters:
frame – The frame to queue.
direction – The direction of frame flow.
callback – Optional callback to call after processing.
- async pause_processing_all_frames_until(ready: Callable[[], Awaitable[Any]], *, timeout: float = 5.0)[source]
Hold frames arriving at this processor until
readyresolves.Useful for a processor that cannot act on frames until some condition holds, such as one that establishes a connection in the background. Frames wait in the processor’s queues and are delivered in order once the condition resolves, so nothing is lost.
The frame being processed when this is called is unaffected: the pause takes hold from the next frame onwards. A processor pausing while it handles its
StartFrame, for instance, still passes that frame on downstream, so pipeline startup is not delayed.Both queues are held, so a processor left paused could not process the frames that shut it down. The pause is therefore always lifted: when
readyresolves, whentimeoutelapses, or at teardown, whichever comes first.- Parameters:
ready – Awaited to learn when frames can be acted on, e.g.
some_event.wait.timeout – Seconds to hold frames before giving up and resuming.
- async process_frame(frame: Frame, direction: FrameDirection)[source]
Process a frame.
- Parameters:
frame – The frame to process.
direction – The direction of frame flow.
- async push_error(error_msg: str, exception: Exception | None = None, fatal: bool = False, category: ErrorCategory | None = None, force_treat_as_permanent: bool = False)[source]
Creates and pushes an ErrorFrame upstream.
Creates and pushes an ErrorFrame upstream to notify other processors in the pipeline about an error condition. The error frame will include context about which processor generated the error.
- Parameters:
error_msg – Descriptive message explaining the error condition.
exception – Optional exception object that caused the error, if available. This provides additional context for debugging and error handling.
fatal –
Whether this error should be considered fatal to the pipeline. Fatal errors typically cause the entire pipeline to stop processing. Defaults to False for non-fatal errors.
Deprecated since version 1.8.0: Use
force_treat_as_permanent=Trueinstead, when the error leaves its originating processor unable to do its job: the pipeline worker then applies itsProcessorUnusablePolicy, of whichCANCELmatches whatfatal=Truedid. For an error that isn’t about that processor’s state, push anEndWorkerFrameafter the error to end the pipeline. Will be removed in 2.0.0.category – Why the error occurred, when the caller knows. Leave it unset to let the category be worked out from the exception, or pass ErrorCategory.UNKNOWN to report an error whose cause can’t be attributed — an unexpected one caught by a broad
except, say, which may not have come from this processor at all.force_treat_as_permanent – Whether to treat this error as one that will keep recurring, leaving the processor unable to do any more work — having failed too many times to keep trying, say. Only needed for a failure the category doesn’t already convey: leaving it False doesn’t keep the processor usable, since a permanent category costs it its usability on its own. Either way the processor stops being given work, and the pipeline worker decides what to do about it through its
ProcessorUnusablePolicy: report the error and keep running (the default), end the pipeline, or cancel it.
Example:
```python # An error this processor can recover from await self.push_error("Failed to process audio chunk, skipping") # An error that leaves this processor unable to do any more work try: result = some_critical_operation() except Exception as e: await self.push_error( "Critical operation failed", exception=e, force_treat_as_permanent=True ) ```
- async push_error_frame(error: ErrorFrame, force_treat_as_permanent: bool = False)[source]
Push an error frame upstream.
- Parameters:
error – The error frame to push. Its deprecated
fatalflag still cancels the pipeline;force_treat_as_permanentis the replacement.force_treat_as_permanent – Whether to treat this error as one that will keep recurring, leaving the processor unable to do any more work. Leaving it False doesn’t keep the processor usable — a permanent category costs it its usability either way. See
push_error().
- async push_frame(frame: Frame, direction: FrameDirection = FrameDirection.DOWNSTREAM)[source]
Push a frame to the next processor in the pipeline.
- Parameters:
frame – The frame to push.
direction – The direction to push the frame.
- async broadcast_interruption()[source]
Broadcast an InterruptionFrame both upstream and downstream.
- async push_interruption_task_frame_and_wait(*, timeout: float = 5.0)[source]
Push an interruption task frame upstream and wait for the interruption.
Deprecated since version 0.0.104: Use
broadcast_interruption()instead. This method now delegates tobroadcast_interruption()and ignores timeout. Will be removed in 2.0.0.
- async broadcast_frame(frame_cls: type[Frame], **kwargs)[source]
Broadcasts a frame of the specified class upstream and downstream.
This method creates two instances of the given frame class using the provided keyword arguments (without deep-copying them) and pushes them upstream and downstream.
- Parameters:
frame_cls – The class of the frame to be broadcasted.
**kwargs – Keyword arguments to be passed to the frame’s constructor.
- async broadcast_frame_instance(frame: Frame)[source]
Broadcasts a frame instance upstream and downstream.
This method creates two new frame instances shallow-copying all fields from the original frame except id and name, which get fresh values.
- Parameters:
frame – The frame instance to broadcast.
Note
Prefer using broadcast_frame() when possible, as it is more efficient. This method should only be used when you are not the creator of the frame and need to broadcast an existing instance.
- has_queued_frame(frame_type: type[Frame]) bool[source]
Return True if a frame of the given type is waiting in the processing queue.
Delegates to
FrameQueue.has_frame()so the check is O(distinct enqueued types) with no queue scanning.- Parameters:
frame_type – The frame class to look for.
- Returns:
True if at least one matching frame is queued.