transport

Small WebRTC transport implementation for Pipecat.

This module provides a WebRTC transport implementation using aiortc for real-time audio and video communication. It supports bidirectional media streaming, application messaging, and client connection management.

class pipecat.transports.smallwebrtc.transport.SmallWebRTCCallbacks(*, on_app_message: Callable[[Any, str], Awaitable[None]], on_client_connected: Callable[[SmallWebRTCConnection], Awaitable[None]], on_client_disconnected: Callable[[SmallWebRTCConnection], Awaitable[None]])[source]

Bases: BaseModel

Callback handlers for SmallWebRTC events.

Parameters:

on_app_message – Called when an application message is received.
on_client_connected – Called when a client establishes connection.
on_client_disconnected – Called when a client disconnects.

on_app_message: Callable[[Any, str], Awaitable[None]]

on_client_connected: Callable[[SmallWebRTCConnection], Awaitable[None]]

on_client_disconnected: Callable[[SmallWebRTCConnection], Awaitable[None]]

class pipecat.transports.smallwebrtc.transport.RawAudioTrack(sample_rate: int, auto_silence: bool = True)[source]

Bases: AudioStreamTrack

Custom audio stream track for WebRTC output.

Handles audio frame generation and timing for WebRTC transmission, supporting queued audio data with proper synchronization.

__init__(sample_rate: int, auto_silence: bool = True)[source]

Initialize the raw audio track.

Parameters:

sample_rate – The audio sample rate in Hz.
auto_silence – If True, emit silence when the queue is empty. If False, wait until audio data is available.

add_audio_bytes(audio_bytes: bytes)[source]

Add audio bytes to the buffer for transmission.

Parameters:: audio_bytes – Raw audio data to queue for transmission.
Returns:: A Future that completes when the data is processed.
Raises:: ValueError – If audio bytes are not a multiple of 10ms size.

async recv()[source]

Return the next audio frame for WebRTC transmission.

Returns:: An AudioFrame containing the next audio data, or silence if the queue is empty and auto_silence is True.

class pipecat.transports.smallwebrtc.transport.RawVideoTrack(width, height)[source]

Bases: VideoStreamTrack

Custom video stream track for WebRTC output.

Handles video frame queuing and conversion for WebRTC transmission.

__init__(width, height)[source]

Initialize the raw video track.

Parameters:

width – Video frame width in pixels.
height – Video frame height in pixels.

add_video_frame(frame)[source]

Add a video frame to the transmission buffer.

Parameters:: frame – The video frame to queue for transmission.

async recv()[source]

Return the next video frame for WebRTC transmission.

Returns:: A VideoFrame ready for WebRTC transmission.

class pipecat.transports.smallwebrtc.transport.SmallWebRTCClient(webrtc_connection: SmallWebRTCConnection, callbacks: SmallWebRTCCallbacks)[source]

Bases: object

WebRTC client implementation for handling connections and media streams.

Manages WebRTC peer connections, audio/video streaming, and application messaging through the SmallWebRTCConnection interface.

__init__(webrtc_connection: SmallWebRTCConnection, callbacks: SmallWebRTCCallbacks)[source]

Initialize the WebRTC client.

Parameters:

webrtc_connection – The underlying WebRTC connection handler.
callbacks – Event callbacks for connection and message handling.

async read_video_frame(video_source: str)[source]

Read video frames from the WebRTC connection.

Reads a video frame from the given MediaStreamTrack, converts it to RGB, and creates an InputImageRawFrame.

Parameters:: video_source – Video source to capture (“camera” or “screenVideo”).
Yields:: UserImageRawFrame objects containing video data from the peer.

async read_audio_frame()[source]

Read audio frames from the WebRTC connection.

Reads 20ms of audio from the given MediaStreamTrack and creates an InputAudioRawFrame.

Yields:: InputAudioRawFrame objects containing audio data from the peer.

async write_audio_frame(frame: OutputAudioRawFrame) → bool[source]

Write an audio frame to the WebRTC connection.

Parameters:: frame – The audio frame to transmit.
Returns:: True if the audio frame was written successfully, False otherwise.

async write_video_frame(frame: OutputImageRawFrame) → bool[source]

Write a video frame to the WebRTC connection.

Parameters:: frame – The video frame to transmit.
Returns:: True if the video frame was written successfully, False otherwise.

async setup(_params: TransportParams, frame)[source]

Set up the client with transport parameters.

Parameters:

_params – Transport configuration parameters.
frame – The initialization frame containing setup data.

async connect()[source]: Establish the WebRTC connection.

async disconnect()[source]: Disconnect from the WebRTC peer.

async send_message(frame: OutputTransportMessageFrame | OutputTransportMessageUrgentFrame)[source]

Send an application message through the WebRTC connection.

Messages sent before the data channel is open (e.g. while the peer connection is still being established) are buffered by the connection and flushed, in order, once the channel opens.

Parameters:: frame – The message frame to send.

property is_connected: bool

Check if the WebRTC connection is established.

Returns:: True if connected to the peer.

property is_closing: bool

Check if the connection is in the process of closing.

Returns:: True if the connection is closing.

class pipecat.transports.smallwebrtc.transport.SmallWebRTCInputTransport(client: SmallWebRTCClient, params: TransportParams, **kwargs)[source]

Bases: BaseInputTransport

Input transport implementation for SmallWebRTC.

Handles incoming audio and video streams from WebRTC peers, including user image requests and application message handling.

__init__(client: SmallWebRTCClient, params: TransportParams, **kwargs)[source]

Initialize the WebRTC input transport.

Parameters:

client – The WebRTC client instance.
params – Transport configuration parameters.
**kwargs – Additional arguments passed to parent class.

async process_frame(frame: Frame, direction: FrameDirection)[source]

Process incoming frames including user image requests.

Parameters:

frame – The frame to process.
direction – The direction of frame flow in the pipeline.

async start(frame: StartFrame)[source]

Start the input transport and establish WebRTC connection.

Parameters:: frame – The start frame containing initialization parameters.

async stop(frame: EndFrame)[source]

Stop the input transport and disconnect from WebRTC.

Parameters:: frame – The end frame signaling transport shutdown.

async cancel(frame: CancelFrame)[source]

Cancel the input transport and disconnect immediately.

Parameters:: frame – The cancel frame signaling immediate cancellation.

async cleanup()[source]: Release resources during teardown.

async push_app_message(message: Any)[source]

Push an application message into the pipeline.

Parameters:: message – The application message to process.

async request_participant_image(frame: UserImageRequestFrame)[source]

Request an image frame from the participant’s video stream.

When a UserImageRequestFrame is received, this method will store the request and the next video frame received will be converted to a UserImageRawFrame.

Parameters:: frame – The user image request frame.

async capture_participant_media(source: str = 'camera')[source]

Capture media from a specific participant.

Parameters:: source – Media source to capture from. (“camera”, “microphone”, or “screenVideo”)

class pipecat.transports.smallwebrtc.transport.SmallWebRTCOutputTransport(client: SmallWebRTCClient, params: TransportParams, **kwargs)[source]

Bases: BaseOutputTransport

Output transport implementation for SmallWebRTC.

Handles outgoing audio and video streams to WebRTC peers, including transport message sending.

__init__(client: SmallWebRTCClient, params: TransportParams, **kwargs)[source]

Initialize the WebRTC output transport.

Parameters:

client – The WebRTC client instance.
params – Transport configuration parameters.
**kwargs – Additional arguments passed to parent class.

async start(frame: StartFrame)[source]

Start the output transport and establish WebRTC connection.

Parameters:: frame – The start frame containing initialization parameters.

async stop(frame: EndFrame)[source]

Stop the output transport and disconnect from WebRTC.

Parameters:: frame – The end frame signaling transport shutdown.

async cancel(frame: CancelFrame)[source]

Cancel the output transport and disconnect immediately.

Parameters:: frame – The cancel frame signaling immediate cancellation.

async cleanup()[source]: Release resources during teardown.

async send_message(frame: OutputTransportMessageFrame | OutputTransportMessageUrgentFrame)[source]

Send a transport message through the WebRTC connection.

Parameters:: frame – The transport message frame to send.

async write_audio_frame(frame: OutputAudioRawFrame) → bool[source]

Write an audio frame to the WebRTC connection.

Parameters:: frame – The output audio frame to transmit.
Returns:: True if the audio frame was written successfully, False otherwise.

async write_video_frame(frame: OutputImageRawFrame) → bool[source]

Write a video frame to the WebRTC connection.

Parameters:: frame – The output video frame to transmit.
Returns:: True if the video frame was written successfully, False otherwise.

class pipecat.transports.smallwebrtc.transport.SmallWebRTCTransport(webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: str | None = None, output_name: str | None = None)[source]

Bases: BaseTransport

WebRTC transport implementation for real-time communication.

Provides bidirectional audio and video streaming over WebRTC connections with support for application messaging and connection event handling.

Event handlers available:

on_client_connected(transport, client): Client connected to WebRTC session
on_client_disconnected(transport, client): Client disconnected from WebRTC session
on_client_message(transport, message, client): Received a data channel message

Example:

@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    ...

__init__(webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: str | None = None, output_name: str | None = None)[source]