LiteLLM

https://docs.litellm.ai/docs/learn
Admin
LiteLLM provides a common interface and proxy for calling models from multiple language model...
Tokens:6,735
Snippets:82
Trust Score:9
Update:4 months ago
Show doc for...
Tokens:
Raw
### Docker Compose: Start Deployment

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Start the LiteLLM deployment using Docker Compose. This command will create and start the containers defined in your docker-compose.yml file.

```bash
docker compose up
```

--------------------------------

### Start LiteLLM Gateway

Source: https://docs.litellm.ai/docs/learn/gateway_quickstart

Start the LiteLLM Gateway by running the `litellm` command with your configuration file. The proxy will be available at `http://0.0.0.0:4000`.

```bash
litellm --config config.yaml
```

--------------------------------

### Install LiteLLM Gateway

Source: https://docs.litellm.ai/docs/learn/gateway_quickstart

Install the LiteLLM Gateway with proxy support using the uv tool.

```bash
uv tool install 'litellm[proxy]'
```

--------------------------------

### Install LiteLLM

Source: https://docs.litellm.ai/docs/learn/sdk_quickstart

Install the LiteLLM library using pip. Ensure you specify the version for reproducibility.

```bash
uv add 'litellm==1.82.6'
```

--------------------------------

### Example Successful Gateway Response

Source: https://docs.litellm.ai/docs/learn/gateway_quickstart

A successful response from the LiteLLM Gateway will be in an OpenAI-style format. The assistant's reply is found in `choices[0].message.content`.

```json
{
  "id": "chatcmpl-abc123",
  "created": 1677858242,
  "model": "gpt-4o-mini-2024-07-18",
  "object": "chat.completion",
  "system_fingerprint": "fp_406d6473f8",
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! How can I assist you today?",
        "tool_calls": null,
        "function_call": null,
        "annotations": []
      }
    }
  ],
  "usage": {
    "completion_tokens": 9,
    "prompt_tokens": 13,
    "total_tokens": 22,
    "completion_tokens_details": {
      "accepted_prediction_tokens": 0,
      "audio_tokens": 0,
      "reasoning_tokens": 0,
      "rejected_prediction_tokens": 0
    },
    "prompt_tokens_details": {
      "audio_tokens": 0,
      "cached_tokens": 0
    }
  },
  "service_tier": "default"
}
```

--------------------------------

### Example OpenAI ModelResponse Object

Source: https://docs.litellm.ai/docs/learn/sdk_quickstart

An example of the detailed ModelResponse object returned for an OpenAI-backed model, including choices, usage, and provider-specific fields.

```python
ModelResponse(
    id='chatcmpl-abc123',
    created=1773782130,
    model='gpt-4o-2024-08-06',
    object='chat.completion',
    system_fingerprint='fp_4ff89bf575',
    choices=[
        Choices(
            finish_reason='stop',
            index=0,
            message=Message(
                content="Hello! I'm just a program, but I'm here to help you. How can I assist you today?",
                role='assistant',
                tool_calls=None,
                function_call=None,
                provider_specific_fields={'refusal': None},
                annotations=[]
            ),
            provider_specific_fields={}
        )
    ],
    usage=Usage(
        completion_tokens=21,
        prompt_tokens=13,
        total_tokens=34,
        completion_tokens_details=CompletionTokensDetailsWrapper(...),
        prompt_tokens_details=PromptTokensDetailsWrapper(...)
    ),
    service_tier='default'
)
```

--------------------------------

### Example JSON Completion Output

Source: https://docs.litellm.ai/docs/learn/sdk_quickstart

A simplified JSON representation of a completion response, similar to what OpenAI provides. Note that fields like 'id' and 'created' will vary.

```json
{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1677858242,
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "Hello! I'm doing well, thanks for asking."
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 13,
    "completion_tokens": 12,
    "total_tokens": 25
  }
}
```

--------------------------------

### List Agents via API

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Use curl to send a GET request to the /v1/agents endpoint to retrieve a list of available agents. Requires an Authorization header.

```bash
curl -H 'Authorization: Bearer sk-team-key' \
  'http://localhost:4000/v1/agents'

```

--------------------------------

### Query Tag Spend

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Use a GET request to the /spend/tags endpoint to retrieve the total spend and log count for specific tags. This helps in monitoring budget utilization.

```bash
curl -X GET 'http://localhost:4000/spend/tags' \
  -H 'Authorization: Bearer sk-1234'

```

--------------------------------

### Gateway Configuration File

Source: https://docs.litellm.ai/docs/learn/gateway_quickstart

Create a `config.yaml` file to define model lists and general settings, including master key and database URL.

```yaml
model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  master_key: sk-1234
  database_url: postgresql://llmproxy:dbpassword9090@db:5432/litellm
```

--------------------------------

### Deploy LiteLLM Enterprise with Helm

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Deploy the LiteLLM Enterprise Gateway using the Helm chart and your custom `values-enterprise.yaml` file. Run this command from the root of the cloned `litellm` repository.

```bash
helm install \
  -f values-enterprise.yaml \
  mydeploy \
  deploy/charts/litellm-helm  
```

--------------------------------

### Configure Google SSO

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Set up environment variables for Google SSO integration, including client ID, client secret, and your proxy base URL. This allows Google accounts to log in to the Admin UI.

```bash
GOOGLE_CLIENT_ID="<your-client-id>"
GOOGLE_CLIENT_SECRET="<your-client-secret>"
PROXY_BASE_URL="https://<your-proxy-base-url>"

```

--------------------------------

### Make First LiteLLM Call

Source: https://docs.litellm.ai/docs/learn/sdk_quickstart

Make a basic completion call to an OpenAI model using LiteLLM. Set your API key as an environment variable before making the call.

```python
from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-api-key"

response = completion(
    model="openai/gpt-4o",
    messages=[{"role": "user", "content": "Hello, how are you?"}],
)

print(response.choices[0].message.content)
```

--------------------------------

### Docker Compose: Create .env File

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Create a .env file to configure environment variables for LiteLLM, including master key, salt key, license, and provider API keys. Ensure sensitive keys are kept secure.

```bash
LITELLM_MASTER_KEY="sk-1234"
LITELLM_SALT_KEY="sk-salt-change-me"
LITELLM_LICENSE="eyJ..."
OPENAI_API_KEY="your-api-key"
```

--------------------------------

### Kubernetes: Apply Deployment

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Apply the LiteLLM deployment configuration to your Kubernetes cluster. This will create the necessary pods and manage their lifecycle.

```bash
kubectl apply -f deployment.yaml
```

--------------------------------

### Configure Microsoft SSO

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Set up environment variables for Microsoft SSO integration, including client ID, client secret, tenant ID, and your proxy base URL. This enables Microsoft accounts for Admin UI login.

```bash
MICROSOFT_CLIENT_ID="<your-client-id>"
MICROSOFT_CLIENT_SECRET="<your-client-secret>"
MICROSOFT_TENANT="<your-tenant-id>"
PROXY_BASE_URL="https://<your-proxy-base-url>"

```

--------------------------------

### Docker Compose: Pull Image and Download Compose File

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Pull the LiteLLM database image and download the Docker Compose configuration file. This is the first step for deploying with Docker Compose.

```bash
docker pull ghcr.io/berriai/litellm-database:main-latest
curl -O https://raw.githubusercontent.com/BerriAI/litellm/main/docker-compose.yml
```

--------------------------------

### Clone LiteLLM Repository

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Clone the LiteLLM repository to your local machine to access the Helm chart and configuration files.

```bash
git clone https://github.com/BerriAI/litellm.git  
```

--------------------------------

### Generate Key with Budget and Rate Limits

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Use curl to generate a new virtual key with a specified maximum budget and RPM limit. This is useful for testing budget enforcement.

```bash
curl -X POST 'http://localhost:4000/key/generate' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "max_budget": 0.01,
    "rpm_limit": 1,
    "team_id": "<your-team-id>"
  }'

```

--------------------------------

### Kubernetes: Create ConfigMap for config.yaml

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Create a Kubernetes ConfigMap to store the LiteLLM configuration file. This allows the deployment to access the configuration at runtime.

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: litellm-config
data:
  config.yaml: |
    model_list:
      - model_name: gpt-5.5
        litellm_params:
          model: openai/gpt-5.5
          api_key: os.environ/OPENAI_API_KEY

    litellm_settings:
      callbacks: ["prometheus"]

    general_settings:
      master_key: os.environ/LITELLM_MASTER_KEY
      store_model_in_db: true
```

--------------------------------

### Docker Compose: Configuration File

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Define the LiteLLM configuration in a config.yaml file. This includes model lists, LiteLLM settings like callbacks, and general settings such as master key and database URL.

```yaml
model_list:
  - model_name: gpt-5.5
    litellm_params:
      model: openai/gpt-5.5
      api_key: os.environ/OPENAI_API_KEY

litellm_settings:
  callbacks: ["prometheus"]

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: "postgresql://llmproxy:dbpassword9090@db:5432/litellm"
  store_model_in_db: true
```

--------------------------------

### Port-Forward LiteLLM Service to Localhost

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Expose the deployed LiteLLM service to your local machine by port-forwarding. This allows you to access the gateway at `http://127.0.0.1:4000`.

```bash
kubectl port-forward service/mydeploy-litellm-helm 4000:4000  
```

--------------------------------

### Configure Tag Budget

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Add tag budget configuration to your config.yaml file to set spending limits for specific tags. Restart the proxy after updating the configuration.

```yaml
litellm_settings:
  tag_budget_config:
    poc:chat-app:
      max_budget: 0.000000000001
      budget_duration: 1d

```

--------------------------------

### Configure MCP Server in config.yaml

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Add MCP server details to your LiteLLM configuration file. Ensure the server name, URL, and transport are correctly specified.

```yaml
mcp_servers:
  - server_name: deepwiki
    url: https://mcp.deepwiki.com/mcp
    transport: http
    available_on_public_internet: true

```

--------------------------------

### Configure SSO Callback URL

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Register this redirect URI in your Identity Provider (IdP) for SSO integration with the LiteLLM Admin UI. This URL is specific to your proxy base URL.

```text
https://<your-proxy-base-url>/sso/callback

```

--------------------------------

### Configure Generic OIDC SSO

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Configure environment variables for generic OIDC SSO, including client ID, client secret, and endpoints for authorization, token, and user info. Also set your proxy base URL.

```bash
GENERIC_CLIENT_ID="<your-client-id>"
GENERIC_CLIENT_SECRET="<your-client-secret>"
GENERIC_AUTHORIZATION_ENDPOINT="https://<your-idp>/oauth2/v1/authorize"
GENERIC_TOKEN_ENDPOINT="https://<your-idp>/oauth2/v1/token"
GENERIC_USERINFO_ENDPOINT="https://<your-idp>/oauth2/v1/userinfo"
PROXY_BASE_URL="https://<your-proxy-base-url>"

```

--------------------------------

### Kubernetes: Apply ConfigMap

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Apply the created ConfigMap to your Kubernetes cluster. This makes the configuration available for use by LiteLLM pods.

```bash
kubectl apply -f litellm-config.yaml
```

--------------------------------

### Kubernetes: Apply Service

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Apply the LiteLLM service configuration to your Kubernetes cluster. This makes the LiteLLM application accessible within and potentially outside the cluster.

```bash
kubectl apply -f service.yaml
```

--------------------------------

### Test LiteLLM Enterprise Gateway with Master Key

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Send a test request to the LiteLLM Enterprise Gateway using your master key to verify routing, model access, and basic functionality.

```bash
curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{  
    "model": "gpt-5.5",  
    "messages": [{"role": "user", "content": "Hello from LiteLLM Enterprise Gateway"}]
  }'  
```

--------------------------------

### Kubernetes: Create Secret for Keys

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Create a Kubernetes Secret to securely store sensitive information like API keys, master keys, and license keys. These are then injected as environment variables into the LiteLLM pod.

```bash
kubectl create secret generic litellm-secrets \
  --from-literal=LITELLM_MASTER_KEY="sk-1234" \
  --from-literal=LITELLM_SALT_KEY="sk-salt-change-me" \
  --from-literal=LITELLM_LICENSE="eyJ..." \
  --from-literal=OPENAI_API_KEY="your-api-key" \
  --from-literal=DATABASE_URL="postgresql://user:pass@host:5432/litellm"
```

--------------------------------

### Set OpenAI API Key

Source: https://docs.litellm.ai/docs/learn/gateway_quickstart

Set your OpenAI API key as an environment variable. This is required for the gateway to authenticate with OpenAI.

```bash
export OPENAI_API_KEY="your-api-key"
```

--------------------------------

### Kubernetes: Deployment Configuration

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Define the Kubernetes Deployment for LiteLLM. This includes the container image, ports, environment variables sourced from secrets, configuration volume mounts, and liveness/readiness probes.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: litellm-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: litellm
  template:
    metadata:
      labels:
        app: litellm
    spec:
      containers:
        - name: litellm
          image: docker.litellm.ai/berriai/litellm-database:main-stable
          imagePullPolicy: Always
          ports:
            - containerPort: 4000
          envFrom:
            - secretRef:
                name: litellm-secrets
          args:
            - "--config"
            - "/app/proxy_config.yaml"
          volumeMounts:
            - name: config-volume
              mountPath: /app/proxy_config.yaml
              subPath: config.yaml
              readOnly: true
          livenessProbe:
            httpGet:
              path: /health/liveliness
              port: 4000
            initialDelaySeconds: 120
            periodSeconds: 15
          readinessProbe:
            httpGet:
              path: /health/readiness
              port: 4000
            initialDelaySeconds: 120
            periodSeconds: 15
      volumes:
        - name: config-volume
          configMap:
            name: litellm-config
```

--------------------------------

### Configure Helm Chart to Use Existing Database

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Modify the `values-enterprise.yaml` to point to an existing PostgreSQL database instead of letting the Helm chart provision one. Specify connection details and secret references.

```yaml
db:  
  useExisting: true  
  endpoint: my-postgres.default.svc.cluster.local  
  database: litellm  
  url: postgresql://user:pass@my-postgres:5432/litellm  
  secret:  
    name: litellm-db-secret  
    usePasswordSecret: true  
```

--------------------------------

### Make a Tagged API Request

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Send a POST request to the chat completions endpoint with a 'tags' metadata field to associate the request with a budget. The first call may succeed, but subsequent calls with the same tag might fail if the budget is exceeded.

```bash
curl -X POST 'http://localhost:4000/chat/completions' \
  -H 'Authorization: Bearer sk-team-key' \
  -H 'Content-Type: application/json' \
  -d '{ \
    "model": "gpt-5.5", \
    "messages": [{"role": "user", "content": "Hello"}], \
    "metadata": {"tags": ["poc:chat-app"]} \
  }'

```

--------------------------------

### Print Full LiteLLM Response Object

Source: https://docs.litellm.ai/docs/learn/sdk_quickstart

Print the entire ModelResponse object to inspect all details, including metadata and usage statistics.

```python
print(response)
```

--------------------------------

### Send First Request to Gateway

Source: https://docs.litellm.ai/docs/learn/gateway_quickstart

Send a cURL request to the gateway's chat completions endpoint. Ensure to include the `Content-Type` and `Authorization` headers.

```bash
curl -X POST 'http://0.0.0.0:4000/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer sk-1234' \
  -d '{ \
    "model": "gpt-4o-mini", \
    "messages": [ \
      {"role": "user", "content": "Hello from LiteLLM Gateway"} \
    ] \
  }'
```

--------------------------------

### Create Kubernetes Secret for Environment Variables

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Create a Kubernetes secret to store sensitive information like license keys and provider API keys. This secret will be injected as environment variables.

```bash
kubectl create secret generic litellm-env-secret \
  --from-literal=LITELLM_LICENSE="eyJ..." \
  --from-literal=OPENAI_API_KEY="your-api-key"  
  --from-literal=DATABASE_URL="postgres://user@password:5432"  
```

--------------------------------

### Enterprise Values Configuration for Helm Chart

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Define enterprise-specific settings in `values-enterprise.yaml`. This includes master key, environment secrets, database deployment, and proxy configurations.

```yaml
masterkey: sk-1234  
  
environmentSecrets:  
  - litellm-env-secret  
  
db:  
  deployStandalone: true  
  
proxyConfigMap:  
  create: true  
  
proxy_config:  
  model_list:  
    - model_name: gpt-5.5  
      litellm_params:  
        model: openai/gpt-5.5  
        api_key: os.environ/OPENAI_API_KEY  
  litellm_settings:  
    callbacks: ["prometheus"]  
  general_settings:  
    store_model_in_db: true  
```

--------------------------------

### Invoke MCP Tool via API

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Use curl to send a POST request to the /v1/chat/completions endpoint, specifying an MCP tool for processing. The request includes messages and tool configuration.

```bash
curl -X POST 'http://localhost:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-team-key' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-5.5",
    "messages": [{"role": "user", "content": "TLDR of BerriAI/litellm repo"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/deepwiki",
      "server_label": "deepwiki",
      "require_approval": "never"
    }]
  }'

```

--------------------------------

### Kubernetes: Service Configuration

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Define the Kubernetes Service for LiteLLM. This exposes the LiteLLM deployment to network traffic, typically on a NodePort for external access.

```yaml
apiVersion: v1
kind: Service
metadata:
  name: litellm-service
spec:
  selector:
    app: litellm
  ports:
    - protocol: TCP
      port: 4000
      targetPort: 4000
  type: NodePort
```

--------------------------------

### Print Assistant Message Content

Source: https://docs.litellm.ai/docs/learn/sdk_quickstart

Extract and print the content of the assistant's message from the LiteLLM completion response.

```python
print(response.choices[0].message.content)
```

--------------------------------

### Invoke Agent via A2A SDK

Source: https://docs.litellm.ai/docs/learn/enterprise_quickstart

Python script using the A2A SDK to interact with a registered agent. It resolves the agent card, sends a message, and prints the response.

```python
import httpx, asyncio
from uuid import uuid4
from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest

LITELLM_BASE_URL = "http://localhost:4000"
LITELLM_VIRTUAL_KEY = "sk-team-key"

async def main():
    headers = {"Authorization": f"Bearer {LITELLM_VIRTUAL_KEY}"}
    async with httpx.AsyncClient(headers=headers) as client:
        agents = (await client.get(f"{LITELLM_BASE_URL}/v1/agents")).json()
        agent_id = agents[0]["agent_id"]
        base_url = f"{LITELLM_BASE_URL}/a2a/{agent_id}"
        resolver = A2ACardResolver(httpx_client=client, base_url=base_url)
        a2a_client = A2AClient(
            httpx_client=client,
            agent_card=await resolver.get_agent_card(),
        )
        response = await a2a_client.send_message(
            SendMessageRequest(
                id=str(uuid4()),
                params=MessageSendParams(
                    message={
                        "role": "user",
                        "parts": [{"kind": "text", "text": "Hello, what can you do?"}],
                        "messageId": uuid4().hex,
                    }
                ),
            )
        )
        print(response.model_dump(mode="json", exclude_none=True, indent=2))

asyncio.run(main())

```
Tokens:
Raw