LlamaIndex

Use Arbytra as your LLM provider in LlamaIndex. This integration is Python-only. For TypeScript, use the Vercel AI SDK integration.

Prerequisites

An Arbytra API key

Install

pip install "arbytra[llamaindex]"

Use SDK adapter

Use the ArbytraLlamaIndexLLM adapter:

from arbytra.frameworks.llamaindex import ArbytraLlamaIndexLLM

llm = ArbytraLlamaIndexLLM(model="gpt-5.4")

ArbytraLlamaIndexLLM supports chat, completion, streaming, async, per-call routing overrides, and Arbytra error mapping.

from arbytra.frameworks.llamaindex import ArbytraLlamaIndexLLM
from llama_index.core.llms import ChatMessage

llm = ArbytraLlamaIndexLLM(model="gpt-5.4")

response = llm.chat([ChatMessage(role="user", content="What is 2+2?")])
print(response.message.content)

for chunk in llm.stream_chat([ChatMessage(role="user", content="Count to 5")]):
    print(chunk.delta, end="", flush=True)

Configure options

Parameter	Type	Default	Description
`model`	`str`	(required)	Model ID
`api_key`	`str \| None`	`ARBYTRA_API_KEY` env	API key
`routing`	`RoutingOptions \| None`	`None`	Default routing configuration
`api_base`	`str`	`"https://api.arbytra.com/v1"`	API base URL
`**kwargs`			Passed through to LlamaIndex’s `OpenAI` (e.g., `temperature`, `max_tokens`)

Configure routing

Pass a RoutingOptions instance to set default routing:

from arbytra.frameworks.llamaindex import ArbytraLlamaIndexLLM
from arbytra.route_types import RoutingOptions

llm = ArbytraLlamaIndexLLM(
    model="gpt-5.4",
    routing=RoutingOptions(optimize="cost"),
)

Per-call routing overrides the instance default:

from arbytra.route_types import RoutingOptions

response = llm.chat(
    [ChatMessage(role="user", content="Hello!")],
    routing=RoutingOptions(optimize="tps-focus"),
)

Access routing metadata from the response:

response = llm.chat([ChatMessage(role="user", content="Hello!")])
metadata = response.additional_kwargs.get("routing_metadata")
if metadata:
    print(f"Provider: {metadata['provider']}")

Configure manually

Alternative: configure LlamaIndex manually

If you prefer to use LlamaIndex’s OpenAI class directly:

import os
from llama_index.llms.openai import OpenAI

llm = OpenAI(
    model="gpt-5.4",
    api_key=os.environ["ARBYTRA_API_KEY"],
    api_base="https://api.arbytra.com/v1",
)

For routing options, per-call overrides, and Arbytra error mapping, use ArbytraLlamaIndexLLM.

Use `ArbytraAsyncOpenAI` (experimental)

If your project pins a different llama-index-llms-openai version, pass ArbytraAsyncOpenAI as the async_openai_client:

from llama_index.llms.openai import OpenAI
from arbytra import ArbytraAsyncOpenAI

client = ArbytraAsyncOpenAI()
llm = OpenAI(
    model="gpt-4o",
    async_openai_client=client,
    api_key="placeholder",
)

LlamaIndex’s OpenAI requires an api_key for construction. Pass any placeholder value.

Read client.last_routing_metadata after each call. See ArbytraAsyncOpenAI for the full class reference.

Get Started

Guides

Response API

Frameworks

Integrations

Specifications

Platform

Changelog

Prerequisites

Install

Use SDK adapter

Configure options

Configure routing

Configure manually

Use `ArbytraAsyncOpenAI` (experimental)

​Prerequisites

​Install

​Use SDK adapter

​Configure options

​Configure routing

​Configure manually

​Use ArbytraAsyncOpenAI (experimental)

Prerequisites

Install

Use SDK adapter

Configure options

Configure routing

Configure manually

Use `ArbytraAsyncOpenAI` (experimental)