llm

Baseten LLM service implementation using OpenAI-compatible interface.

class pipecat.services.baseten.llm.BasetenLLMSettings(model: str | None | _NotGiven = <factory>, extra: dict[str, Any]=<factory>, system_instruction: str | None | _NotGiven = <factory>, temperature: float | None | _NotGiven | NotGiven = <factory>, max_tokens: int | None | _NotGiven | NotGiven = <factory>, top_p: float | None | _NotGiven | NotGiven = <factory>, top_k: int | None | _NotGiven = <factory>, frequency_penalty: float | None | _NotGiven | NotGiven = <factory>, presence_penalty: float | None | _NotGiven | NotGiven = <factory>, seed: int | None | _NotGiven | NotGiven = <factory>, filter_incomplete_user_turns: bool | None | _NotGiven = <factory>, user_turn_completion_config: UserTurnCompletionConfig | None | _NotGiven = <factory>, max_completion_tokens: int | None | _NotGiven | NotGiven = <factory>)[source]

Bases: OpenAILLMSettings

Settings for BasetenLLMService.

class pipecat.services.baseten.llm.BasetenLLMService(*, api_key: str, base_url: str = 'https://inference.baseten.co/v1', settings: BasetenLLMSettings | None = None, **kwargs)[source]

Bases: OpenAILLMService

A service for interacting with Baseten’s OpenAI-compatible inference API.

Defaults to Baseten’s Model APIs, a serverless endpoint hosting open-weights models. To use a dedicated deployment running on your own GPUs, point base_url at it and set model to the deployment’s served model name:

BasetenLLMService(
    api_key=os.getenv("BASETEN_API_KEY"),
    base_url="https://model-{model_id}.api.baseten.co/environments/production/sync/v1",
    settings=BasetenLLMService.Settings(model="Qwen/Qwen2.5-3B-Instruct"),
)
supports_developer_role = False

Whether this service’s API supports the “developer” message role.

OpenAI’s native API supports it, but some OpenAI-compatible services (e.g. Cerebras) do not. Subclasses that don’t support it should set this to False, which causes the adapter to convert “developer” messages to “user” messages before sending them to the API.

Settings

alias of BasetenLLMSettings

__init__(*, api_key: str, base_url: str = 'https://inference.baseten.co/v1', settings: BasetenLLMSettings | None = None, **kwargs)[source]

Initialize the Baseten LLM service.

Parameters:

  • api_key – The API key for accessing Baseten’s inference API.

  • base_url – The base URL for the Baseten API. Defaults to "https://inference.baseten.co/v1". Set this to a dedicated deployment’s /sync/v1 URL to use your own hosted model.

  • settings – Runtime-updatable settings; values override the built-in defaults.

  • **kwargs – Additional keyword arguments passed to OpenAILLMService.

async start_llm_usage_metrics(tokens: LLMTokenUsage)[source]

Accumulate token usage metrics during processing.

Baseten reports cumulative usage on every streamed chunk rather than once at the end, so each update supersedes the previous one instead of adding to it.

Parameters:

tokens – Token usage information to accumulate.

create_client(api_key=None, base_url=None, **kwargs)[source]

Create OpenAI-compatible client for Baseten API endpoint.

Parameters:

  • api_key – The API key for authentication. If None, uses instance default.

  • base_url – The base URL for the API. If None, uses instance default.

  • **kwargs – Additional keyword arguments for client configuration.

Returns:

An OpenAI-compatible client configured for Baseten’s API.