any-llm

https://mozilla-ai.github.io/any-llm
Admin
any-llm is a Python library providing a unified interface to interact with different LLM providers,...
Tokens:47,910
Snippets:382
Trust Score:9.7
Update:6 months ago
Show doc for...
Tokens:
Raw
### Copy Example Configuration File

Source: https://mozilla-ai.github.io/any-llm/gateway/docker-deployment

Copies the example configuration file to be edited. Ensure you have Docker and Docker Compose installed.

```bash
cp docker/config.example.yml docker/config.yml
```

--------------------------------

### Install AnyLLM SDK

Source: https://mozilla-ai.github.io/any-llm/quickstart

Install the base package with all provider support or specific providers as needed.

```bash
pip install any-llm-sdk[all]  # Install with all provider support
```

```bash
pip install any-llm-sdk[mistral]  # For Mistral provider
pip install any-llm-sdk[ollama]   # For Ollama provider
# install multiple providers
pip install any-llm-sdk[mistral,ollama]
```

```bash
pip install any-llm-sdk
```

--------------------------------

### YAML Configuration File Example

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Example `config.yml` for setting up database connection, master key, and LLM provider credentials like OpenAI, Gemini, and Vertex AI. Includes optional model pricing for cost tracking.

```yaml
#Database connection
database_url: "postgresql://gateway:gateway@localhost:5432/gateway_db"


#Master key for admin access
master_key: "your-secure-master-key"


## LLM Provider Credentials
providers:
  openai:
    api_key: "${OPENAI_API_KEY}"
  gemini:
    api_key: "${GEMINI_API_KEY}"
  vertexai:
    credentials: "/path/to/service_account.json"
    project: "your-gcp-project-id"
    location: "us-central1"


# Model pricing for cost-tracking (optional)
pricing:
  openai:gpt-4:
    input_price_per_million: 0.15
    output_price_per_million: 0.6
```

--------------------------------

### YAML Configuration Example

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Example of a `config.yml` file for setting up database connection, master key, LLM provider credentials, and optional model pricing.

```APIDOC
## YAML Configuration Example

### Description
This example demonstrates the structure of a `config.yml` file for the AnyLLM Gateway, including database connection details, a master key, credentials for various LLM providers (OpenAI, Gemini, Vertex AI), and optional model pricing for cost tracking.

### Method
File Configuration

### Endpoint
N/A

### Parameters
#### Request Body
- **database_url** (string) - Required - The connection string for the database.
- **master_key** (string) - Required - A secure master key for admin access.
- **providers** (object) - Optional - Configuration for LLM providers.
  - **openai** (object) - Optional - OpenAI provider configuration.
    - **api_key** (string) - Required - OpenAI API key, can be set via environment variable `${OPENAI_API_KEY}`.
  - **gemini** (object) - Optional - Gemini provider configuration.
    - **api_key** (string) - Required - Gemini API key, can be set via environment variable `${GEMINI_API_KEY}`.
  - **vertexai** (object) - Optional - Vertex AI provider configuration.
    - **credentials** (string) - Required - Path to the service account JSON file.
    - **project** (string) - Required - Google Cloud project ID.
    - **location** (string) - Required - Google Cloud location.
- **pricing** (object) - Optional - Model pricing for cost tracking.
  - **openai:gpt-4** (object) - Optional - Pricing for OpenAI's GPT-4 model.
    - **input_price_per_million** (number) - Optional - Cost per million input tokens.
    - **output_price_per_million** (number) - Optional - Cost per million output tokens.

### Request Example
```yaml
#Database connection
database_url: "postgresql://gateway:gateway@localhost:5432/gateway_db"

#Master key for admin access
master_key: "your-secure-master-key"

## LLM Provider Credentials
providers:
  openai:
    api_key: "${OPENAI_API_KEY}"
  gemini:
    api_key: "${GEMINI_API_KEY}"
  vertexai:
    credentials: "/path/to/service_account.json"
    project: "your-gcp-project-id"
    location: "us-central1"

# Model pricing for cost-tracking (optional)
pricing:
  openai:gpt-4:
    input_price_per_million: 0.15
    output_price_per_million: 0.6
```

### Response
N/A (This is a configuration file)

### Error Handling
N/A
```

--------------------------------

### Starting Gateway with YAML Config

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Command to start the AnyLLM Gateway using a specified YAML configuration file.

```APIDOC
## Starting Gateway with YAML Config

### Description
This command initiates the AnyLLM Gateway service, loading its configuration from the provided `config.yml` file.

### Method
Command Line

### Endpoint
N/A

### Parameters
#### Command Line Arguments
- **serve** - Command to start the gateway.
- **--config** (string) - Required - Path to the YAML configuration file.

### Request Example
```bash
any-llm-gateway serve --config config.yml
```

### Response
N/A

### Error Handling
N/A
```

--------------------------------

### Install Any-LLM SDK

Source: https://mozilla-ai.github.io/any-llm/cookbooks/any-llm-getting-started

Installs the Any-LLM SDK with support for all providers and the nest-asyncio library for asynchronous operations in environments like Jupyter notebooks.

```python
%pip install any-llm-sdk[all] nest-asyncio -q


# nest_asyncio allows us to use 'await' directly in Jupyter notebooks
# This is needed because any-llm uses async functions for API calls
import nest_asyncio


nest_asyncio.apply()
```

--------------------------------

### Start Gateway with YAML Config

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Command to launch the any-llm-gateway using a specified YAML configuration file.

```bash
any-llm-gateway serve --config config.yml
```

--------------------------------

### Environment Variables Configuration

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Example of setting up the AnyLLM Gateway using environment variables for essential settings.

```APIDOC
## Environment Variables Configuration

### Description
This example shows how to configure the AnyLLM Gateway entirely through environment variables. This method is particularly useful for containerized deployments and adhering to 12-factor app principles. Note that model pricing cannot be configured this way.

### Method
Environment Variables

### Endpoint
N/A

### Parameters
#### Environment Variables
- **DATABASE_URL** (string) - Required - The connection string for the database.
- **GATEWAY_MASTER_KEY** (string) - Required - The master key for admin access.
- **GATEWAY_HOST** (string) - Optional - The host address for the gateway (default: `0.0.0.0`).
- **GATEWAY_PORT** (integer) - Optional - The port for the gateway (default: `8000`).

### Request Example
```bash
export DATABASE_URL="postgresql://gateway:gateway@localhost:5432/gateway_db"
export GATEWAY_MASTER_KEY="your-secure-master-key"
export GATEWAY_HOST="0.0.0.0"
export GATEWAY_PORT=8000

any-llm-gateway serve
```

### Response
N/A

### Error Handling
N/A
```

--------------------------------

### Start and verify gateway

Source: https://mozilla-ai.github.io/any-llm/gateway/quickstart

Commands to launch the gateway services and check the health endpoint.

```bash
# From project root directory
docker compose up -d
```

```bash
# Verify the gateway is running
curl http://localhost:8000/health


# Expected response:
# {"status": "healthy"}
```

--------------------------------

### Batch Creation and Retrieval Usage

Source: https://mozilla-ai.github.io/any-llm/api/types/batch

Example demonstrating how to create a batch job and poll for its completion status.

```APIDOC
## Usage Example

### Description
This example shows how to use the `create_batch` and `retrieve_batch` functions to manage batch jobs.

### Code
```python
from any_llm import create_batch, retrieve_batch
import time

# Create a batch job
batch = create_batch(
    provider="openai",
    input_file_path="requests.jsonl",
    endpoint="/v1/chat/completions",
)

print(f"Batch ID: {batch.id}")
print(f"Status: {batch.status}")

# Poll for completion
while batch.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = retrieve_batch("openai", batch.id)
    print(f"Status: {batch.status}")
    if batch.request_counts:
        print(f"  Completed: {batch.request_counts.completed}/{batch.request_counts.total}")

if batch.status == "completed":
    print(f"Output file: {batch.output_file_id}")
```

### Notes
- The `Batch` and `BatchRequestCounts` types are direct re-exports from the OpenAI SDK.
- any-llm normalizes all provider batch responses into this format.
```

--------------------------------

### Provider Client Args Configuration

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Example of configuring additional arguments for provider clients, such as custom headers and timeouts.

```APIDOC
## Provider Client Args Configuration

### Description
This configuration allows you to pass additional arguments directly to the initialization of provider clients. This enables customization like setting custom headers or request timeouts for specific providers.

### Method
File Configuration

### Endpoint
N/A

### Parameters
#### Request Body (within `config.yml`)
- **providers** (object) - Configuration for LLM providers.
  - **provider-name** (object) - Configuration for a specific provider (e.g., `openai`).
    - **api_key** (string) - Required - The API key for the provider.
    - **client_args** (object) - Optional - Arguments passed to the provider's client initialization.
      - **custom_headers** (object) - Optional - Key-value pairs for custom HTTP headers.
        - **Header-Name** (string) - Value for the custom header.
      - **timeout** (number) - Optional - Connection and request timeout in seconds.

### Request Example
```yaml
providers:
  openai:
    api_key: "${OPENAI_API_KEY}"
    client_args:
      custom_headers:
        X-Custom-Header: "custom-value"
      timeout: 60
```

### Response
N/A

### Error Handling
N/A
```

--------------------------------

### Create AnyLLM Instance

Source: https://mozilla-ai.github.io/any-llm/api/any-llm

Use `AnyLLM.create()` to get a configured instance for a specific provider. Provide the provider name and optionally an API key and base URL.

```python
def create(
    provider: str | LLMProvider,
    api_key: str | None = None,
    api_base: str | None = None,
    **kwargs: Any,
) -> AnyLLM
```

```python
from any_llm import AnyLLM


llm = AnyLLM.create("openai", api_key="sk-...")


response = llm.completion(
    model="gpt-4.1-mini",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(response.choices[0].message.content)
```

--------------------------------

### List models using the AnyLLM class

Source: https://mozilla-ai.github.io/any-llm/api/list-models

Illustrates creating an `AnyLLM` instance and using its `list_models` method to get the count of available models for a provider.

```python
from any_llm import AnyLLM


llm = AnyLLM.create("openai")
models = llm.list_models()
print(f"Available models: {len(models)}")

```

--------------------------------

### Get Supported Providers

Source: https://mozilla-ai.github.io/any-llm/api/any-llm

Obtain a list of all supported provider keys (e.g., 'openai', 'anthropic') using `AnyLLM.get_supported_providers()`.

```python
def get_supported_providers() -> list[str]
```

```python
print(AnyLLM.get_supported_providers())
```

--------------------------------

### Start Services with Docker Compose

Source: https://mozilla-ai.github.io/any-llm/gateway/docker-deployment

Launches the gateway and PostgreSQL database in detached mode using Docker Compose. This command sets up networking and dependencies automatically.

```bash
docker-compose -f docker/docker-compose.yml up -d
```

--------------------------------

### Config File Model Pricing

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Example of configuring model pricing within the YAML configuration file.

```APIDOC
## Config File Model Pricing

### Description
This section illustrates how to define model pricing directly within the `config.yml` file. This allows for cost tracking of LLM usage based on input and output tokens.

### Method
File Configuration

### Endpoint
N/A

### Parameters
#### Request Body (within `config.yml`)
- **pricing** (object) - Optional - Defines pricing for models.
  - **provider:model-name** (object) - Required - Specifies pricing for a particular model (e.g., `openai:gpt-3.5-turbo`).
    - **input_price_per_million** (number) - Required - The cost per million input tokens.
    - **output_price_per_million** (number) - Required - The cost per million output tokens.

### Request Example
```yaml
pricing:
  openai:gpt-3.5-turbo:
    input_price_per_million: 0.5
    output_price_per_million: 1.5
```

### Response
N/A

### Error Handling
N/A
```

--------------------------------

### Dynamic Pricing via API

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Example of using the API to dynamically set or update model pricing.

```APIDOC
## Dynamic Pricing via API

### Description
This endpoint allows for dynamic setting or updating of model pricing without restarting the gateway. It's useful for managing pricing in production or when provider rates change.

### Method
POST

### Endpoint
`/v1/pricing`

### Parameters
#### Headers
- **X-AnyLLM-Key** (string) - Required - Bearer token for authentication, typically `${GATEWAY_MASTER_KEY}`.
- **Content-Type** (string) - Required - Must be `application/json`.

#### Request Body
- **model** (string) - Required - The model identifier (e.g., `openai:gpt-4`).
- **input_price_per_million** (number) - Required - The cost per million input tokens.
- **output_price_per_million** (number) - Required - The cost per million output tokens.

### Request Example
```bash
curl -X POST http://localhost:8000/v1/pricing \
  -H "X-AnyLLM-Key: Bearer ${GATEWAY_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai:gpt-4",
    "input_price_per_million": 30.0,
    "output_price_per_million": 60.0
  }'
```

### Response
#### Success Response (200)
- **message** (string) - Confirmation message indicating the pricing was updated.

#### Response Example
```json
{
  "message": "Model pricing updated successfully."
}
```

### Error Handling
- **400 Bad Request**: Invalid request body or parameters.
- **401 Unauthorized**: Invalid or missing `X-AnyLLM-Key`.
- **500 Internal Server Error**: Server-side issue.
```

--------------------------------

### Generate Response with Instructions

Source: https://mozilla-ai.github.io/any-llm/api/responses

Use this snippet to get a direct response from a model with specific instructions. Ensure the provider supports the Responses API.

```python
result = responses(
    model="gpt-4.1-mini",
    provider="openai",
    input_data="Translate to French: Hello, how are you?",
    instructions="You are a professional translator. Always respond with only the translation.",
)
```

--------------------------------

### Perform a basic completion with AnyLLM

Source: https://mozilla-ai.github.io/any-llm/api/completion

Executes a synchronous request to a specified model and provider. Requires the any_llm package to be installed and configured.

```python
from any_llm import completion


response = completion(
    model="mistral-small-latest",
    provider="mistral",
    messages=[{"role": "user", "content": "What is the capital of France?"}],
)
print(response.choices[0].message.content)
```

--------------------------------

### Provider Client Arguments Configuration

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Example of configuring provider-specific `client_args` in the YAML file, such as custom headers and timeouts, which are passed directly to the provider's client initialization.

```yaml
providers:
  openai:
    api_key: "${OPENAI_API_KEY}"
    client_args:
      custom_headers:
        X-Custom-Header: "custom-value"
      timeout: 60
```

--------------------------------

### Get All Provider Metadata

Source: https://mozilla-ai.github.io/any-llm/api/any-llm

Retrieve metadata for all supported LLM providers using `AnyLLM.get_all_provider_metadata()`. The results are sorted alphabetically by provider name.

```python
def get_all_provider_metadata() -> list[ProviderMetadata]
```

```python
for meta in AnyLLM.get_all_provider_metadata():
    print(f"{meta.name}: streaming={meta.streaming}, embedding={meta.embedding}")
```

--------------------------------

### Virtual API Key Creation Response

Source: https://mozilla-ai.github.io/any-llm/gateway/authentication

Example response when creating a virtual API key, showing its ID, key, name, creation timestamp, expiration, and status.

```json
{
  "id": "abc-123",
  "key": "gw-...",
  "key_name": "mobile-app",
  "created_at": "2025-10-20T10:00:00",
  "expires_at": null,
  "is_active": true,
  "metadata": {}
}
```

--------------------------------

### Generate a basic response with Any-LLM

Source: https://mozilla-ai.github.io/any-llm/api/responses

Use the responses function to send a prompt to a specified model and provider. Ensure the any_llm package is installed and configured.

```python
from any_llm import responses


result = responses(
    model="gpt-4.1-mini",
    provider="openai",
    input_data="What is the capital of France?",
)
print(result.output_text)
```

--------------------------------

### Stream Responses

Source: https://mozilla-ai.github.io/any-llm/api/responses

This example shows how to stream responses from the API, printing each event as it arrives. This is useful for real-time applications. Check provider documentation for stream support.

```python
for event in responses(
    model="gpt-4.1-mini",
    provider="openai",
    input_data="Tell me a short story.",
    stream=True,
):
    print(event)
```

--------------------------------

### Environment Variables Configuration

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Example of setting required environment variables for gateway configuration, suitable for containerized deployments. The gateway will automatically detect and use these variables.

```bash
#Required settings
export DATABASE_URL="postgresql://gateway:gateway@localhost:5432/gateway_db"
export GATEWAY_MASTER_KEY="your-secure-master-key"
export GATEWAY_HOST="0.0.0.0"
export GATEWAY_PORT=8000


any-llm-gateway serve
```

--------------------------------

### YAML Model Pricing Configuration

Source: https://mozilla-ai.github.io/any-llm/gateway/configuration

Example of how to define model pricing within the YAML configuration file for cost tracking purposes. This sets the input and output price per million tokens for a specific model.

```yaml
pricing:
  openai:gpt-3.5-turbo:
    input_price_per_million: 0.5
    output_price_per_million: 1.5
```

--------------------------------

### Get All Provider Metadata

Source: https://mozilla-ai.github.io/any-llm/api/types/provider

Iterate through all available LLM providers and print their names and supported features. This is useful for discovering available capabilities across different providers.

```python
from any_llm import AnyLLM


for meta in AnyLLM.get_all_provider_metadata():
    features = []
    if meta.streaming:
        features.append("streaming")
    if meta.embedding:
        features.append("embedding")
    if meta.reasoning:
        features.append("reasoning")
    if meta.responses:
        features.append("responses")
    print(f"{meta.name}: {', '.join(features) or 'completion only'}")

```

--------------------------------

### Production Docker Compose Configuration

Source: https://mozilla-ai.github.io/any-llm/gateway/docker-deployment

Example configuration snippet for docker-compose.yml to enhance production deployment. Includes image, restart policy, healthcheck, resource limits, and logging settings.

```yaml
services:
  gateway:
    image: ghcr.io/mozilla-ai/any-llm/gateway:latest
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://localhost:8000/health')"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    deploy:
      resources:
        limits:
          cpus: '2'
          memory: 2G
        reservations:
          cpus: '1'
          memory: 1G
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
```

--------------------------------

### Use PlatformProvider for LLM API Calls

Source: https://mozilla-ai.github.io/any-llm/platform/overview

Integrate the `PlatformProvider` into your application to make LLM API calls using a single virtual key. This setup handles authentication, decryption of provider keys client-side, and direct requests to LLM providers.

```python
from any_llm import completion
completion(provider="platform", model="openai:gpt-4", ...)
```

--------------------------------

### Create project directory

Source: https://mozilla-ai.github.io/any-llm/gateway/quickstart

Initialize the directory structure for the gateway project.

```bash
mkdir any-llm-gateway
cd any-llm-gateway
```

--------------------------------

### Get Pricing

Source: https://mozilla-ai.github.io/any-llm/openapi.json

Retrieves the pricing information for a specific model.

```APIDOC
## GET /v1/pricing/{model_key}

### Description
Retrieves the pricing information for a specific model.

### Method
GET

### Endpoint
/v1/pricing/{model_key}

### Parameters
#### Path Parameters
- **model_key** (string) - Required - The unique key of the model for which to retrieve pricing.

### Response
#### Success Response (200)
- **PricingResponse** (object) - The pricing information for the specified model.

#### Response Example
```json
{
  "model_key": "string",
  "price_per_token": 0.0,
  "currency": "string"
}
```

#### Error Response (422)
- **HTTPValidationError** (object) - Details about the validation error.
```

--------------------------------

### Generate batch embeddings

Source: https://mozilla-ai.github.io/any-llm/api/embedding

Example of embedding a list of strings in a single request.

```python

result = embedding(
    model="text-embedding-3-small",
    provider="openai",
    inputs=["First sentence", "Second sentence", "Third sentence"],
)


for item in result.data:
    print(f"Index {item.index}: {len(item.embedding)} dimensions")

```

--------------------------------

### Generate asynchronous embeddings

Source: https://mozilla-ai.github.io/any-llm/api/embedding

Example of using the asynchronous aembedding function within an event loop.

```python

import asyncio
from any_llm import aembedding


async def main():
    result = await aembedding(
        model="text-embedding-3-small",
        provider="openai",
        inputs="Hello, world!",
    )
    print(f"Dimensions: {len(result.data[0].embedding)}")


asyncio.run(main())

```

--------------------------------

### Create a User with Metadata

Source: https://mozilla-ai.github.io/any-llm/gateway/authentication

Create a user with additional metadata, such as department, team, and email, for enhanced tracking and organization.

```bash
curl -X POST http://localhost:8000/v1/users \
  -H "X-AnyLLM-Key: Bearer ${GATEWAY_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d ' {
    "user_id": "user-123",
    "alias": "Alice",
    "metadata": {
      "department": "Engineering",
      "team": "ML",
      "email": "alice@example.com"
    }
  }'
```

--------------------------------

### Troubleshoot Gateway Issues

Source: https://mozilla-ai.github.io/any-llm/gateway/docker-deployment

Commands for diagnosing container startup, database connectivity, and permission problems.

```bash
docker-compose logs gateway
```

```bash
docker-compose exec postgres psql -U gateway -c "SELECT version();"
```

```bash
chmod 644 docker/config.yml
chmod 600 docker/service_account.json
```

```bash
docker-compose -f docker/docker-compose.yml up -d --build
```

--------------------------------

### Generate single text embedding

Source: https://mozilla-ai.github.io/any-llm/api/embedding

Example of embedding a single string using the OpenAI provider.

```python

from any_llm import embedding


result = embedding(
    model="text-embedding-3-small",
    provider="openai",
    inputs="Hello, world!",
)


vector = result.data[0].embedding
print(f"Dimensions: {len(vector)}")
print(f"Tokens used: {result.usage.total_tokens}")

```

--------------------------------

### GET /v1/users/{user_id}/usage

Source: https://mozilla-ai.github.io/any-llm/openapi.json

Retrieves the usage history for a specific user, with support for pagination.

```APIDOC
## GET /v1/users/{user_id}/usage

### Description
Get usage history for a specific user.

### Method
GET

### Endpoint
/v1/users/{user_id}/usage

### Parameters
#### Path Parameters
- **user_id** (string) - Required - The unique identifier of the user.

#### Query Parameters
- **skip** (integer) - Optional - Number of records to skip (default: 0).
- **limit** (integer) - Optional - Maximum number of records to return (default: 100, max: 1000).

### Response
#### Success Response (200)
- **UsageLogResponse** (array) - A list of usage logs for the specified user.

#### Error Response (422)
- **HTTPValidationError** (object) - Validation error details.
```

--------------------------------

### Create, Retrieve, and List Batch Jobs with Any-LLM

Source: https://mozilla-ai.github.io/any-llm/api/batch

Demonstrates the core functionalities of the Any-LLM library: creating a batch job, checking its status, and listing all existing batches for a given provider. Ensure you have the necessary imports.

```python
from any_llm import create_batch, retrieve_batch, list_batches


# Create a batch job
batch = create_batch(
    provider="openai",
    input_file_path="requests.jsonl",
    endpoint="/v1/chat/completions",
)
print(f"Batch created: {batch.id}, status: {batch.status}")


# Check status
batch = retrieve_batch("openai", batch.id)
print(f"Status: {batch.status}")


# List all batches
batches = list_batches("openai")
for b in batches:
    print(f"{b.id}: {b.status}")

```

--------------------------------

### GET /health

Source: https://mozilla-ai.github.io/any-llm/openapi.json

Checks the general health status of the API. This endpoint is useful for basic monitoring.

```APIDOC
## GET /health

### Description
General health check endpoint. Returns basic health status. For infrastructure monitoring, use /health/readiness or /health/liveness instead.

### Method
GET

### Endpoint
/health

### Responses
#### Success Response (200)
- **string** (string) - Description of the health status.
```

--------------------------------

### Download Docker configuration

Source: https://mozilla-ai.github.io/any-llm/gateway/quickstart

Alternative method to fetch the docker-compose.yml file from the repository.

```bash
curl -o docker-compose.yml https://raw.githubusercontent.com/mozilla-ai/any-llm/main/docker/docker-compose.yml
```

--------------------------------

### Create a User

Source: https://mozilla-ai.github.io/any-llm/gateway/authentication

Create a new user for the gateway using a POST request. The request requires the master key for authentication and includes the user's ID and an optional alias.

```bash
curl -X POST http://localhost:8000/v1/users \
  -H "X-AnyLLM-Key: Bearer ${GATEWAY_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"user_id": "user-123", "alias": "Alice"}'
```

--------------------------------

### Get Provider Metadata

Source: https://mozilla-ai.github.io/any-llm/api/any-llm

Retrieves metadata for a specific AnyLLM provider instance, indicating its supported features.

```APIDOC
## GET /provider/metadata

### Description
Retrieves metadata for this provider instance's class, detailing its capabilities.

### Method
GET

### Endpoint
/provider/metadata

### Parameters
None

### Request Body
None

### Request Example
None

### Response
#### Success Response (200)
- **streaming** (boolean) - Indicates if the provider supports streaming responses.
- **embedding** (boolean) - Indicates if the provider supports embedding generation.
- **responses** (boolean) - Indicates if the provider supports direct response generation.

#### Response Example
```json
{
  "streaming": true,
  "embedding": false,
  "responses": true
}
```
```

--------------------------------

### GET /list_batches

Source: https://mozilla-ai.github.io/any-llm/api/batch

Retrieves a list of batch jobs for a specified provider, supporting pagination and result limiting.

```APIDOC
## GET /list_batches

### Description
Retrieves a sequence of Batch objects for a given provider.

### Method
GET

### Parameters
#### Query Parameters
- **provider** (str | LLMProvider) - Required - Provider name to use for the request (e.g., ‘openai’, ‘mistral’)
- **after** (str | None) - Optional - A cursor for pagination. Returns batches after this batch ID.
- **limit** (int | None) - Optional - Maximum number of batches to return (default: 20)
- **api_key** (str | None) - Optional - API key for the provider
- **api_base** (str | None) - Optional - Base URL for the provider API
- **client_args** (dict[str, Any] | None) - Optional - Additional provider-specific arguments for client instantiation
- **kwargs** (Any) - Required - Additional provider-specific arguments

### Response
#### Success Response (200)
- **batches** (Sequence[Batch]) - A sequence of Batch objects.
```

--------------------------------

### Build Any-LLM Gateway Docker Image from Source

Source: https://mozilla-ai.github.io/any-llm/gateway/docker-deployment

Builds a local Docker image for the any-llm-gateway from source. This is useful for testing local changes or custom builds.

```bash
docker build -t any-llm-gateway:local -f docker/Dockerfile .
```

--------------------------------

### Run Local Any-LLM Gateway Image

Source: https://mozilla-ai.github.io/any-llm/gateway/docker-deployment

Runs the any-llm-gateway using a locally built Docker image. This allows for testing custom modifications before deploying.

```bash
docker run -d \
  --name any-llm-gateway \
  -p 8000:8000 \
  -v $(pwd)/config.yml:/app/config.yml \
  -e DATABASE_URL="postgresql://user:pass@host:5432/dbname" \
  any-llm-gateway:local
```

--------------------------------

### Initialize OpenAI Client with Gateway

Source: https://mozilla-ai.github.io/any-llm/gateway/authentication

Configure the OpenAI client to use the any-llm gateway by setting the `base_url` and `api_key`. The API key is sent as a Bearer token in the `Authorization` header.

```python
from openai import OpenAI


client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-master-key-or-virtual-key",  # Sent as Authorization: Bearer ...
)
```

--------------------------------

### Get All Provider Metadata

Source: https://mozilla-ai.github.io/any-llm/api/types/provider

Retrieves metadata for all available LLM providers, allowing iteration through each provider to inspect their capabilities and features.

```APIDOC
## GET /api/providers

### Description
Retrieves metadata for all available LLM providers.

### Method
GET

### Endpoint
`/api/providers`

### Response
#### Success Response (200)
An array of `ProviderMetadata` objects, each describing a provider's capabilities.

### Request Example
```python
from any_llm import AnyLLM

for meta in AnyLLM.get_all_provider_metadata():
    features = []
    if meta.streaming:
        features.append("streaming")
    if meta.embedding:
        features.append("embedding")
    if meta.reasoning:
        features.append("reasoning")
    if meta.responses:
        features.append("responses")
    print(f"{meta.name}: {', '.join(features) or 'completion only'}")
```

### Response Example
```json
[
  {
    "name": "openai",
    "env_key": "OPENAI_API_KEY",
    "env_api_base": null,
    "doc_url": "https://platform.openai.com/docs/models",
    "streaming": true,
    "reasoning": true,
    "completion": true,
    "embedding": true,
    "responses": true,
    "image": true,
    "pdf": false,
    "class_name": "any_llm.providers.openai.OpenAIProvider",
    "list_models": true,
    "messages": true,
    "batch_completion": true
  },
  {
    "name": "anthropic",
    "env_key": "ANTHROPIC_API_KEY",
    "env_api_base": null,
    "doc_url": "https://docs.anthropic.com/claude/reference/complete-create",
    "streaming": true,
    "reasoning": true,
    "completion": true,
    "embedding": false,
    "responses": true,
    "image": false,
    "pdf": false,
    "class_name": "any_llm.providers.anthropic.AnthropicProvider",
    "list_models": false,
    "messages": true,
    "batch_completion": false
  }
]
```
```

--------------------------------

### Instance Methods (completion, messages, batch)

Source: https://mozilla-ai.github.io/any-llm/api/any-llm

Core instance methods for interacting with LLM providers.

```APIDOC
## Instance Methods

### Description
Methods called on an AnyLLM object to perform LLM operations.

### Methods
- **completion() / acompletion()**: Create a chat completion.
- **responses() / aresponses()**: Create a response using the OpenResponses API.
- **messages() / amessages()**: Create a message using the Anthropic Messages API format.
- **list_models() / alist_models()**: List available models for this provider.
- **create_batch() / acreate_batch()**: Create a batch job.
```

--------------------------------

### Get Single Provider Metadata

Source: https://mozilla-ai.github.io/any-llm/api/types/provider

Retrieve and print metadata for a single LLM provider. Ensure the provider name is valid.

```python
from any_llm import AnyLLM


llm = AnyLLM.create("openai")
meta = llm.get_provider_metadata()


print(f"Provider: {meta.name}")
print(f"API key env var: {meta.env_key}")
print(f"Supports streaming: {meta.streaming}")
print(f"Supports embedding: {meta.embedding}")
print(f"Supports responses: {meta.responses}")

```

--------------------------------

### Set Up API Keys for LLM Providers

Source: https://mozilla-ai.github.io/any-llm/cookbooks/any-llm-getting-started

Configures API keys for different LLM providers by checking environment variables and prompting the user if keys are not found. Supports OpenAI and Anthropic, with placeholders for Google and Mistral.

```python
import os
from getpass import getpass


def setup_api_key(key_name: str, provider: str) -> None:
    """Set up API key for the specified provider."""
    if key_name not in os.environ:
        print(f"🔑 {key_name} not found in environment")
        api_key = getpass(f"Enter your {provider} API key (or press Enter to skip): ")
        if api_key:
            os.environ[key_name] = api_key
            print(f"✅ {key_name} set for this session")
        else:
            print(f"⏭️  Skipping {provider}")
    else:
        print(f"✅ {key_name} found in environment")



# Set up keys for different providers
print("Setting up API keys...\n")
setup_api_key("OPENAI_API_KEY", "OpenAI")
setup_api_key("ANTHROPIC_API_KEY", "Anthropic")


#  You could add more using :
# setup_api_key("GOOGLE_API_KEY", "Google")
# setup_api_key("MISTRAL_API_KEY", "Mistral")
```

--------------------------------

### Get Provider Metadata

Source: https://mozilla-ai.github.io/any-llm/api/any-llm

Instantiate an AnyLLM model and retrieve its metadata to check for capabilities like streaming, embedding, and response types.

```python
llm = AnyLLM.create("mistral")
meta = llm.get_provider_metadata()
print(f"Supports streaming: {meta.streaming}")
print(f"Supports embedding: {meta.embedding}")
print(f"Supports responses: {meta.responses}")
```

--------------------------------

### Create a Budget

Source: https://mozilla-ai.github.io/any-llm/gateway/budget-management

Defines a new budget with a specific spending limit and duration in seconds.

```bash
# Create a budget with a $10.00 spending limit and monthly resets (30 days = 2592000 seconds)
curl -X POST http://localhost:8000/v1/budgets \
  -H "X-AnyLLM-Key: Bearer ${GATEWAY_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "max_budget": 10.0,
    "budget_duration_sec": 2592000
  }'
```

--------------------------------

### Create Batch Job

Source: https://mozilla-ai.github.io/any-llm/api/any-llm

Initiate a batch job using `create_batch()` or `acreate_batch()`. Consult the Batch reference for the complete parameter list.

```python
def create_batch(self, **kwargs) -> Batch
```

```python
async def acreate_batch(self, input_file_path, endpoint, completion_window="24h", metadata=None, **kwargs) -> Batch
```

--------------------------------

### Configure gateway settings

Source: https://mozilla-ai.github.io/any-llm/gateway/quickstart

Define the database URL, master key, provider credentials, and model pricing in config.yml.

```yaml
database_url: "postgresql://gateway:gateway@postgres:5432/gateway"


master_key: 09kS0xTiz6JqO....


providers:
  openai:
    api_key: YOUR_OPENAI_API_KEY_HERE
    api_base: "https://api.openai.com/v1"  # optional, useful when you want to use a specific version of the API


models:
  openai:gpt-4:
    input_price_per_million: 0.15
    output_price_per_million: 0.60
```

--------------------------------

### Run Standalone Any-LLM Gateway

Source: https://mozilla-ai.github.io/any-llm/gateway/docker-deployment

Runs the any-llm-gateway as a standalone Docker container, mapping port 8000 and mounting a configuration file. Replace DATABASE_URL with your PostgreSQL connection string.

```bash
docker run -d \
  --name any-llm-gateway \
  -p 8000:8000 \
  -v $(pwd)/config.yml:/app/config.yml \
  -e DATABASE_URL="postgresql://user:pass@host:5432/dbname" \
  ghcr.io/mozilla-ai/any-llm/gateway:latest \
  any-llm-gateway serve --config /app/config.yml
```

--------------------------------

### Configure Docker services

Source: https://mozilla-ai.github.io/any-llm/gateway/quickstart

Define the gateway and PostgreSQL services in docker-compose.yml.

```yaml
services:
  gateway:
    # Use the official production image
    image: ghcr.io/mozilla-ai/any-llm/gateway:latest
    ports:
      - "8000:8000"
    volumes:
      - ./config.yml:/app/config.yml
      # UNCOMMENT the next line ONLY if using Google Vertex AI (requires service_account.json)
      # - ./service_account.json:/app/service_account.json
    command: ["any-llm-gateway", "serve", "--config", "/app/config.yml"]
    depends_on:
      postgres:
        condition: service_healthy
    restart: unless-stopped


  postgres:
    image: postgres:16-alpine
    environment:
      - POSTGRES_USER=gateway
      - POSTGRES_PASSWORD=gateway
      - POSTGRES_DB=gateway
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U gateway"]
      interval: 10s
      timeout: 5s
      retries: 5
    restart: unless-stopped


volumes:
  postgres_data:
```

--------------------------------

### List models using the asynchronous alist_models function

Source: https://mozilla-ai.github.io/any-llm/api/list-models

Shows how to use the `alist_models` function within an async context to retrieve and print model IDs for a provider like 'mistral'. Requires `asyncio` to run.

```python
import asyncio
from any_llm import alist_models


async def main():
    models = await alist_models("mistral")
    for model in models:
        print(model.id)


asyncio.run(main())

```

--------------------------------

### Manage Database Backups and Restores

Source: https://mozilla-ai.github.io/any-llm/gateway/docker-deployment

Perform database operations using docker-compose exec commands.

```bash
# Backup
docker-compose -f docker/docker-compose.yml exec postgres \
  pg_dump -U gateway gateway > backup.sql


# Restore
docker-compose -f docker/docker-compose.yml exec -T postgres \
  psql -U gateway gateway < backup.sql
```

--------------------------------

### Send Message with System Prompt

Source: https://mozilla-ai.github.io/any-llm/api/messages

Include a `system` prompt to guide the LLM's behavior, such as setting its persona or task. This is useful for tasks like translation or role-playing.

```python
response = messages(
    model="claude-sonnet-4-20250514",
    provider="anthropic",
    messages=[{"role": "user", "content": "Translate to French: Hello"}],
    max_tokens=1024,
    system="You are a professional translator.",
)
```

--------------------------------

### AnyLLM.create()

Source: https://mozilla-ai.github.io/any-llm/api/any-llm

Factory method to create a configured AnyLLM instance for a specific provider.

```APIDOC
## AnyLLM.create()

### Description
Factory method that returns a configured AnyLLM instance for the given provider.

### Parameters
#### Request Body
- **provider** (str | LLMProvider) - Required - The provider name (e.g., 'openai', 'anthropic')
- **api_key** (str | None) - Optional - API key for the provider
- **api_base** (str | None) - Optional - Base URL for the provider API
- **kwargs** (Any) - Required - Additional provider-specific arguments

### Response
- **Returns** (AnyLLM) - An AnyLLM instance bound to the specified provider.
```

--------------------------------

### Create a User with Any-LLM Gateway

Source: https://mozilla-ai.github.io/any-llm/gateway/quickstart

Use this cURL command to create a new user for tracking API requests. Associate a user ID and an optional alias.

```bash
curl -s -X POST http://localhost:8000/v1/users \
  -H "X-AnyLLM-Key: Bearer ${GATEWAY_MASTER_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"user_id": "user-123", "alias": "Bob"}'
```

--------------------------------

### View User Metrics with Any-LLM Gateway

Source: https://mozilla-ai.github.io/any-llm/gateway/quickstart

Retrieve usage metrics for a specific user by sending a GET request to the users endpoint with the user ID. This requires authentication with the master key.

```bash
curl -s http://localhost:8000/v1/users/user-123 \
  -H "X-AnyLLM-Key: Bearer ${GATEWAY_MASTER_KEY}" \
  -H "Content-Type: application/json"
```

--------------------------------

### Get Single Provider Metadata

Source: https://mozilla-ai.github.io/any-llm/api/types/provider

Retrieves metadata for a specific LLM provider, including its name, configuration environment variables, supported features like streaming, embedding, and response generation.

```APIDOC
## GET /api/providers/{provider_name}

### Description
Retrieves metadata for a specific LLM provider.

### Method
GET

### Endpoint
`/api/providers/{provider_name}`

### Parameters
#### Path Parameters
- **provider_name** (str) - Required - The name of the LLM provider.

### Response
#### Success Response (200)
- **name** (str) - The name of the provider.
- **env_key** (str) - The environment variable key for the API key.
- **env_api_base** (str | None) - The environment variable for the API base URL, if applicable.
- **doc_url** (str) - URL to the provider's documentation.
- **streaming** (bool) - Whether the provider supports streaming responses.
- **reasoning** (bool) - Whether the provider supports reasoning capabilities.
- **completion** (bool) - Whether the provider supports text completion.
- **embedding** (bool) - Whether the provider supports embedding generation.
- **responses** (bool) - Whether the provider supports generating structured responses.
- **image** (bool) - Whether the provider supports image generation.
- **pdf** (bool) - Whether the provider supports PDF processing.
- **class_name** (str) - The class name of the provider implementation.
- **list_models** (bool) - Whether the provider supports listing available models.
- **messages** (bool) - Whether the provider supports message-based interactions.
- **batch_completion** (bool) - Whether the provider supports batch completion requests.

### Request Example
```python
from any_llm import AnyLLM

llm = AnyLLM.create("openai")
meta = llm.get_provider_metadata()

print(f"Provider: {meta.name}")
print(f"API key env var: {meta.env_key}")
print(f"Supports streaming: {meta.streaming}")
print(f"Supports embedding: {meta.embedding}")
print(f"Supports responses: {meta.responses}")
```

### Response Example
```json
{
  "name": "openai",
  "env_key": "OPENAI_API_KEY",
  "env_api_base": null,
  "doc_url": "https://platform.openai.com/docs/models",
  "streaming": true,
  "reasoning": true,
  "completion": true,
  "embedding": true,
  "responses": true,
  "image": true,
  "pdf": false,
  "class_name": "any_llm.providers.openai.OpenAIProvider",
  "list_models": true,
  "messages": true,
  "batch_completion": true
}
```
```

--------------------------------

### List models using the synchronous list_models function

Source: https://mozilla-ai.github.io/any-llm/api/list-models

Demonstrates how to use the `list_models` function to retrieve and print model IDs for a specified provider, such as 'openai'.

```python
from any_llm import list_models


models = list_models("openai")
for model in models:
    print(model.id)

```

--------------------------------

### Use AnyLLM Class for Multiple Requests

Source: https://mozilla-ai.github.io/any-llm/quickstart

Instantiate the AnyLLM class to manage provider state across multiple requests.

```python
import os


from any_llm import AnyLLM


# Make sure you have the appropriate API key set
api_key = os.environ.get('MISTRAL_API_KEY')
if not api_key:
    raise ValueError("Please set MISTRAL_API_KEY environment variable")


llm = AnyLLM.create("mistral")


response = llm.completion(
    model="mistral-small-latest",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message.content)


metadata = llm.get_provider_metadata()
print(f"Supports streaming: {metadata.streaming}")
print(f"Supports tools: {metadata.completion}")
```
Tokens:
Raw