anyllm-proxy

https://github.com/whit3rabbit/anyllm-proxy
Admin
An API translation proxy that lets Anthropic-based tools talk to any OpenAI-compatible backend,...
Tokens:339,353
Snippets:3,118
Trust Score:9.1
License:MIT
Update:3 months ago
Show doc for...
Tokens:
Raw
### Minimal Client Setup with Builder Shorthand

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/crates/client/README.md

Use `Client::builder()` for a quick setup with default authentication and translation configurations. This example shows how to build a client for an OpenAI-compatible backend.

```rust
use anyllm_client::Client;
use anyllm_translate::anthropic::MessageCreateRequest;

let client = Client::builder()
    .base_url("https://api.openai.com/v1/chat/completions")
    .api_key(&std::env::var("OPENAI_API_KEY")?)
    .build()?;

let req: MessageCreateRequest = serde_json::from_str(r#"{{
    "model": "gpt-4o-mini",
    "max_tokens": 256,
    "messages": [{{"role": "user", "content": "Summarize Rust's borrow checker"}}]
}}"#)?;

let resp = client.messages(&req).await?;
println!("{:#?}", resp.content);
```

--------------------------------

### Run anyllm-proxy with Azure OpenAI

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/ENV.md

Example command to start the proxy configured for the Azure OpenAI backend.

```bash
BACKEND=azure \
AZURE_OPENAI_API_KEY=abc123 \
AZURE_OPENAI_ENDPOINT=https://my-resource.openai.azure.com \
AZURE_OPENAI_DEPLOYMENT=gpt-4o \
cargo run -p anyllm_proxy
```

--------------------------------

### Start proxy with client credential forwarding

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/ENV.md

Example command to launch the proxy with client credential forwarding enabled and open relay configured.

```bash
BACKEND=anthropic \
ANTHROPIC_AUTH_TOKEN=$(claude setup-token) \
ANTHROPIC_FORWARD_CLIENT_AUTH=true \
PROXY_OPEN_RELAY=true \
anyllm_proxy
```

--------------------------------

### Run OpenAI Backend with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/openai.md

Starts the anyllm-proxy with the OpenAI backend using environment variables for API key and backend selection. This is a quick way to get started.

```bash
BACKEND=openai OPENAI_API_KEY=sk-... cargo run -p anyllm_proxy
```

--------------------------------

### Run AnyLLM Proxy with Replicate Backend (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/replicate.md

Starts the AnyLLM Proxy using Replicate as the backend, configured via environment variables. This is a quick way to get started.

```bash
BACKEND=replicate REPLICATE_API_KEY=your-key cargo run -p anyllm_proxy
```

--------------------------------

### Run proxy with mTLS configuration

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/ENV.md

Example command to start the proxy with client certificate and CA certificate paths provided via environment variables.

```bash
OPENAI_API_KEY=sk-... \
OPENAI_BASE_URL=https://internal-llm.corp.example.com \
TLS_CLIENT_CERT_P12=/etc/proxy/client.p12 \
TLS_CLIENT_CERT_PASSWORD=changeit \
TLS_CA_CERT=/etc/proxy/corp-ca.pem \
cargo run -p anyllm_proxy
```

--------------------------------

### Alternative Installation Methods

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/README.md

Additional ways to install or run the proxy using Cargo or Docker.

```bash
# Cargo
cargo install anyllm_proxy

# Build from source
cargo build -p anyllm_proxy --release

# Docker
docker run -e OPENAI_API_KEY=sk-... -p 3000:3000 followthewhit3rabbit/anyllm-proxy:latest
```

--------------------------------

### BatchEngine Facade Example

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-03-30-batch-engine-phase1.md

Demonstrates the usage of the BatchEngine facade for submitting and cancelling a job. Assumes prior setup and job creation.

```rust
metadata: None,
                priority: 0,
            })
            .await
            .unwrap();

        let status = engine.cancel(&job.id).await.unwrap();
        assert_eq!(status, BatchStatus::Cancelled);
    }
}
```

--------------------------------

### Run SambaNova Proxy with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/sambanova.md

Starts the anyllm-proxy with the SambaNova backend using environment variables for configuration. This is a quick way to get started.

```bash
BACKEND=sambanova SAMBANOVA_API_KEY=your-key cargo run -p anyllm_proxy
```

--------------------------------

### Nscale Quick Start with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/nscale.md

Configure the proxy to use Nscale as the backend by setting the BACKEND and NSCALE_API_KEY environment variables. This example shows both direct cargo run and Docker deployment.

```bash
BACKEND=nscale NSCALE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=nscale -e NSCALE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Run Proxy with Novita AI (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/novita.md

Starts the anyllm-proxy using environment variables to specify Novita AI as the backend and provide the API key. This is a quick way to get started.

```bash
BACKEND=novita NOVITA_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=novita -e NOVITA_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Start vLLM Server with API Key

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/hosted_vllm.md

Demonstrates how to start the vLLM serving process with an API key for authentication. This key can then be used by the LiteLLM proxy.

```bash
vllm serve meta-llama/Meta-Llama-3.1-8B-Instruct --api-key secret
```

--------------------------------

### Enable and Start Debian Service

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/AGENTS.md

Systemd commands to enable and start the anyllm-proxy service after installation.

```bash
sudo systemctl enable --now anyllm-proxy
```

--------------------------------

### Start the proxy server

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/codedocs/index.md

Launch the anyllm_proxy binary after configuration.

```bash
anyllm_proxy
```

--------------------------------

### Run Mistral Proxy with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/mistral.md

Starts the anyllm-proxy using environment variables for backend selection and API key. This is a quick way to get started.

```bash
BACKEND=mistral MISTRAL_API_KEY=your-key cargo run -p anyllm_proxy
```

--------------------------------

### Install and Run Petals Server

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/petals.md

Install the Petals library and start a local inference server for a specified model. The server defaults to listening on port 8080.

```bash
pip install petals
python -m petals.cli.run_server petals-team/StableBeluga2
```

--------------------------------

### Dashscope Quick Start with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/dashscope.md

Set the BACKEND and DASHSCOPE_API_KEY environment variables to run the anyllm-proxy with Dashscope. This example shows command-line execution and Docker deployment.

```bash
BACKEND=dashscope DASHSCOPE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=dashscope -e DASHSCOPE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Start the proxy with the admin UI

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/codedocs/guides/admin-and-env-imports.md

Launch the proxy with the web interface enabled.

```bash
anyllm_proxy --webui
```

--------------------------------

### Run anyllm-proxy with Ollama (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/ollama.md

Starts the anyllm-proxy service using Ollama as the backend via environment variables. This is a quick way to get started if you prefer environment variable configuration.

```bash
BACKEND=ollama PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
```

```bash
docker run -e BACKEND=ollama -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Start the Proxy

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/codedocs/guides/claude-code-with-local-models.md

Command to launch the proxy and the expected output confirming successful initialization.

```bash
anyllm_proxy
```

```text
anyllm_proxy: data directory: /home/you/.anyllm
anyllm_proxy: loaded 5 variable(s) from env file
```

--------------------------------

### Install via Homebrew

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/README.md

Use this command to install the proxy on macOS systems.

```bash
brew install whit3rabbit/tap/anyllm-proxy
```

--------------------------------

### Usage Example

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/codedocs/api-reference/anyllm-proxy-runtime.md

Demonstrates parsing environment content and calculating usage costs.

```rust
use anyllm_proxy::env_parser::parse_env_content;
use anyllm_proxy::cost::ModelPricing;

let parsed = parse_env_content("OPENAI_API_KEY=sk-...
BACKEND=groq");
assert!(parsed.hard_errors.is_empty());

let pricing = ModelPricing::load();
let usd = pricing.cost_for_usage("gpt-4o-mini", 1200, 300);
println!("{usd}");
```

--------------------------------

### Install via Debian Package

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/README.md

Commands for downloading and installing the proxy on Debian or Ubuntu systems.

```bash
# Check https://github.com/whit3rabbit/anyllm-proxy/releases for the current filename
curl -LO https://github.com/whit3rabbit/anyllm-proxy/releases/latest/download/anyllm-proxy_0.9.0-1_amd64.deb
# arm64: replace amd64 with arm64
sudo dpkg -i anyllm-proxy_*.deb
sudo systemctl enable --now anyllm-proxy
# Configure: edit /etc/default/anyllm-proxy
```

--------------------------------

### Initialize and Start Batch Engine

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-03-30-batch-engine-phase1.md

Initializes batch engine tables, sets up the BatchEngine struct with necessary components like queues and file storage, and starts the webhook dispatcher. Ensure SQLite connection is already established.

```rust
// Initialize batch engine tables.
anyllm_batch_engine::db::init_batch_engine_tables(&conn)?;
anyllm_batch_engine::db::migrate_old_tables(&conn)?;

let db = Arc::new(Mutex::new(conn));
let batch_engine = Arc::new(BatchEngine {
    queue: Arc::new(SqliteQueue::new(db.clone())),
    file_store: FileStore::new(db.clone()),
    webhook_queue: Arc::new(SqliteWebhookQueue::new(db.clone())),
    global_webhook_urls: parse_webhook_urls(),
    webhook_signing_secret: std::env::var("BATCH_WEBHOOK_SIGNING_SECRET").ok(),
});

// Start webhook dispatcher.
let webhook_handle = anyllm_batch_engine::webhook::dispatcher::start_dispatcher(
    batch_engine.webhook_queue.clone(),
    reqwest::Client::new(),
    WebhookConfig::default(),
);
```

--------------------------------

### AnyLLM Proxy Startup Options

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/CONFIG.md

Command-line arguments for launching the proxy with specific configurations, UI, or tool wrapping.

```bash
# Auto-loads ~/.anyllm/.anyllm.env if present
anyllm-proxy

# Explicit env file
anyllm-proxy --env-file /path/to/my.env

# With admin UI
anyllm-proxy --webui

# Both
anyllm-proxy --webui --env-file /path/to/my.env

# Redact detected secrets before forwarding JSON/text requests upstream
anyllm-proxy --redact-secrets

# Wrap a tool (starts proxy, sets ANTHROPIC_* env vars, runs command)
anyllm-proxy run claude
```

--------------------------------

### MiniMax Quick Start with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/minimax.md

Set up MiniMax as the backend using environment variables for direct execution or Docker deployment.

```bash
BACKEND=minimax MINIMAX_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=minimax -e MINIMAX_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### NanoGPT Quick Start with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/nanogpt.md

Run the proxy with NanoGPT as a single backend using environment variables. This is useful for quick local testing.

```bash
BACKEND=nanogpt NANOGPT_API_KEY=your-key cargo run -p anyllm_proxy
```

```bash
# Docker:
docker run -e BACKEND=nanogpt -e NANOGPT_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Run Volcano Engine Provider with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/volcengine.md

Start the LiteLLM Proxy with the Volcano Engine backend configured using environment variables. This is a quick way to get started.

```bash
BACKEND=volcengine VOLCENGINE_API_KEY=your-key cargo run -p anyllm_proxy
```

--------------------------------

### Install and Run Infinity Embedding Server

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/infinity.md

Installs the Infinity embedding server using pip and starts it with a specified Hugging Face model ID. The server defaults to port 7997.

```bash
pip install infinity-emb[all]
infinity_emb v2 --model-id BAAI/bge-small-en-v1.5
```

--------------------------------

### Anyscale Quick Start with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/anyscale.md

Configure and run the anyllm-proxy using Anyscale as the backend via environment variables. This is useful for quick local testing.

```bash
BACKEND=anyscale ANYSCALE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=anyscale -e ANYSCALE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Run AnyLLM Proxy with Pollinations Backend

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/pollinations.md

This command starts the anyllm-proxy with Pollinations as the backend and enables open relay mode. It's the quickest way to get started using Pollinations.

```bash
BACKEND=pollinations PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
```

--------------------------------

### Startup Log Output

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/CONFIG.md

Example of the log output indicating the data directory path at startup.

```text
anyllm_proxy: data directory: /home/user/.anyllm
```

--------------------------------

### Initialize Proxy with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/codedocs/guides/multi-backend-routing.md

Set required API keys and configuration path before launching the proxy service.

```bash
export OPENAI_API_KEY=sk-...
export GROQ_API_KEY=gsk_...
export PROXY_API_KEYS=proxy-user
export PROXY_CONFIG=$HOME/.anyllm/config.yaml

anyllm_proxy
```

--------------------------------

### Main Entry Point (main.tsx)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-04-05-react-admin-ui.md

Sets up the React application, configures the QueryClient for data fetching with default options, and renders the main App component within a QueryClientProvider.

```tsx
import { StrictMode } from 'react'
import { createRoot } from 'react-dom/client'
import { QueryClient, QueryClientProvider } from '@tanstack/react-query'
import App from './App'
import './styles/globals.css'

const queryClient = new QueryClient({
  defaultOptions: {
    queries: {
      retry: 1,
      refetchOnWindowFocus: false,
    },
  },
})

createRoot(document.getElementById('root')!).render(
  <StrictMode>
    <QueryClientProvider client={queryClient}>
      <App />
    </QueryClientProvider>
  </StrictMode>,
)
```

--------------------------------

### Quick Start: Single-Backend (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/gemini.md

Set the BACKEND and GEMINI_API_KEY environment variables to run the proxy with Gemini. This example shows usage with `cargo run` and Docker.

```bash
BACKEND=gemini GEMINI_API_KEY=AIza... cargo run -p anyllm_proxy
```

```bash
docker run -e BACKEND=gemini -e GEMINI_API_KEY=AIza... -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Uptime API Response Shape

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/specs/2026-04-05-react-admin-ui-design.md

Defines the structure of the JSON response for the GET /admin/api/uptime endpoint, providing proxy start time and backend health check history.

```json
{
  "proxy": {
    "started_at": 1234567890,
    "uptime_pct_30d": 99.98,
    "history": [{ "date": "2026-04-05", "status": "up" }]
  },
  "backends": [
    {
      "name": "openai",
      "status": "up",
      "last_checked_at": 1234567890,
      "last_latency_ms": 210,
      "uptime_pct_30d": 99.9,
      "history": [{ "date": "2026-04-05", "status": "up" }]
    }
  ]
}
```

--------------------------------

### Start Ollama Backend

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/codedocs/guides/claude-code-with-local-models.md

Commands to initialize the Ollama server and pull a compatible model.

```bash
ollama serve
ollama pull qwen2.5-coder:32b
```

--------------------------------

### SageMaker Single-Backend Configuration (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/sagemaker.md

Configure the proxy to use a SageMaker backend via environment variables. This example shows the intended setup, but the backend is not yet implemented.

```bash
BACKEND=sagemaker \
  AWS_ACCESS_KEY_ID=AKIA... \
  AWS_SECRET_ACCESS_KEY=... \
  AWS_REGION_NAME=us-east-1 \
  PROXY_OPEN_RELAY=true \
  cargo run -p anyllm_proxy
```

--------------------------------

### Run Bytez Proxy with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/bytez.md

Start the anyllm_proxy with Bytez as the backend using environment variables for configuration. This is useful for quick local testing.

```bash
BACKEND=bytez BYTEZ_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=bytez -e BYTEZ_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Implement GET /admin/api/uptime Route in Rust

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-04-05-react-admin-ui.md

This Rust code defines the handler for the GET /admin/api/uptime endpoint. It retrieves proxy start time and per-backend health check history from the database, structures it into a `UptimeResponse` object, and returns it as JSON. It uses Axum for routing and extracts shared state including the database connection and runtime configuration.

```rust
use crate::admin::db::{backend_history_30d, backend_uptime_pct};
use crate::admin::state::{with_db, SharedState};
use axum::{extract::State, Json};
use serde::Serialize;

#[derive(Serialize)]
pub struct HistoryDay {
    date: String,
    status: String,
}

#[derive(Serialize)]
pub struct ProxyUptimeInfo {
    started_at: u64,
    uptime_pct_30d: f64,
    history: Vec<HistoryDay>,
}

#[derive(Serialize)]
pub struct BackendUptimeInfo {
    name: String,
    status: String,
    last_checked_at: Option<i64>,
    last_latency_ms: Option<i64>,
    uptime_pct_30d: f64,
    history: Vec<HistoryDay>,
}

#[derive(Serialize)]
pub struct UptimeResponse {
    proxy: ProxyUptimeInfo,
    backends: Vec<BackendUptimeInfo>,
}

pub(super) async fn get_uptime(
    State(shared): State<SharedState>,
) -> Json<UptimeResponse> {
    let started_at = shared
        .started_at
        .duration_since(std::time::UNIX_EPOCH)
        .unwrap_or_default()
        .as_secs();

    // Collect backend names from runtime config.
    let backend_names: Vec<String> = {
        let cfg = shared.runtime_config.read().unwrap_or_else(|e| e.into_inner());
        cfg.model_mappings.keys().cloned().collect()
    };

    let response = with_db(&shared.db, move |conn| {
        // Proxy uptime: use process start time; all days are "up" unless we tracked outages.
        // We approximate with the health_checks records for all backends combined.
        let proxy_history: Vec<HistoryDay> = {
            if let Ok(mut stmt) = conn.prepare(
                "SELECT date(checked_at, 'unixepoch') AS day,
                        MIN(CASE WHEN status='down' THEN 0 ELSE 1 END) AS all_up
                 FROM health_checks
                 WHERE checked_at >= strftime('%s','now') - 30*86400
                 GROUP BY day
                 ORDER BY day ASC",
            ) {
                stmt.query_map([], |r| {
                    let day: String = r.get(0)?;
                    let all_up: i64 = r.get(1)?;
                    Ok(HistoryDay {
                        date: day,
                        status: if all_up == 1 { "up".to_string() } else { "down".to_string() },
                    })
                })
                .unwrap_or_else(|_| Box::new(std::iter::empty()) as Box<dyn Iterator<Item=_>>)
                .filter_map(|r| r.ok())
                .collect()
            } else {
                vec![]
            }
        };

        // Collect per-backend data.
        let backends: Vec<BackendUptimeInfo> = backend_names
            .iter()
            .map(|name| {
                let uptime_pct = backend_uptime_pct(conn, name).unwrap_or(100.0);
                let history = backend_history_30d(conn, name)
                    .unwrap_or_default()
                    .into_iter()
                    .map(|(date, status)| HistoryDay { date, status })
                    .collect();

                // Last check for this backend.
                let (last_checked_at, last_latency_ms, current_status) = conn
                    .query_row(
                        "SELECT checked_at, latency_ms, status FROM health_checks
                         WHERE backend = ?1 ORDER BY checked_at DESC LIMIT 1",
                        [name],
                        |r| Ok((r.get::<_, i64>(0)?, r.get::<_, Option<i64>>(1)?, r.get::<_, String>(2)?)),
                    )
                    .map(|(ts, lat, st)| (Some(ts), lat, st))
                    .unwrap_or((None, None, "unknown".to_string()));

                BackendUptimeInfo {
                    name: name.clone(),
                    status: current_status,
                    last_checked_at,
                    last_latency_ms,
                    uptime_pct_30d: uptime_pct,
                    history,
                }
            })
            .collect();

        UptimeResponse {
            proxy: ProxyUptimeInfo {
                started_at,
                uptime_pct_30d: 100.0, // proxy process doesn't self-report downtime
                history: proxy_history,
            },
            backends,
        }
    })
    .await;

    Json(response.unwrap_or_else(|| UptimeResponse {
        proxy: ProxyUptimeInfo { started_at, uptime_pct_30d: 100.0, history: vec![] },
        backends: vec![],
    }))
}

```

--------------------------------

### Generate Embeddings using Voyage AI via Proxy

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/voyage.md

Example of how to call the /v1/embeddings endpoint through the LiteLLM proxy to get embeddings from a Voyage AI model. Ensure the proxy is running and configured.

```bash
curl http://localhost:3000/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $PROXY_API_KEYS" \
  -d '{"model": "voyage-3", "input": "The quick brown fox"}'
```

--------------------------------

### Call Codestral via OpenAI Chat Completions API

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/codestral.md

Example cURL command to interact with the Codestral model using the OpenAI Chat Completions API format through the proxy. This shows how to send a user message and get a model completion.

```bash
curl http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer $PROXY_API_KEYS" \
  -H "Content-Type: application/json" \
  -d '{"model": "codestral-latest", "messages": [{"role": "user", "content": "Write a Python function to reverse a string"}]}'
```

--------------------------------

### Run anyllm-proxy with LM Studio Backend (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/lm_studio.md

Starts the anyllm-proxy with LM Studio as the backend using environment variables. This is useful for quick local testing.

```bash
BACKEND=lm_studio PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
```

```bash
docker run -e BACKEND=lm_studio -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Replace Async Main with Sync Main and Async Run

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-03-28-security-audit-all-fixes.md

This snippet demonstrates how to replace the `#[tokio::main]` attribute with a synchronous `main` function that initializes the Tokio runtime and then calls an asynchronous `run` function. This separation ensures environment setup occurs before the runtime starts.

```rust
fn main() {
    let args: Vec<String> = std::env::args().collect();
    let (env_file_count, multi_config, model_router) = setup_env(&args);

    tokio::runtime::Builder::new_multi_thread()
        .enable_all()
        .build()
        .expect("failed to build tokio runtime")
        .block_on(run(args, env_file_count, multi_config, model_router))
}
```

```rust
async fn run(
    args: Vec<String>,
    _env_file_count: usize,
    multi_config: config::MultiConfig,
    model_router: Option<Arc<std::sync::RwLock<crate::config::model_router::ModelRouter>>>
) {

```

--------------------------------

### Hyperbolic Quick Start with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/hyperbolic.md

Set up Hyperbolic as a backend using environment variables for direct execution or Docker deployment.

```bash
BACKEND=hyperbolic HYPERBOLIC_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=hyperbolic -e HYPERBOLIC_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Add Synchronous Environment Setup Function

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-03-28-security-audit-all-fixes.md

This helper function parses environment files, applies aliases, and loads configuration synchronously. It must be called before the Tokio runtime starts to ensure `set_var` is safe. It returns pre-runtime data for use within the async runtime.

```rust
/// Synchronous env setup: parse env file, apply aliases, load config.
/// MUST be called before the tokio runtime starts so `set_var` is safe.
/// Returns (env_file_count, load_result) for use inside the async runtime.
fn setup_env(args: &[String]) -> (usize, config::MultiConfig, Option<crate::config::model_router::ModelRouter>) {
    // Determine env file path from --env-file flag or .anyllm.env convention
    let env_file_path = args
        .windows(2)
        .find(|w| w[0] == "--env-file")
        .map(|w| w[1].as_str())
        .or_else(|| {
            if std::path::Path::new(".anyllm.env").exists() {
                Some(".anyllm.env")
            } else {
                None
            }
        });
    let env_file_vars = env_file_path.map(parse_env_file).unwrap_or_default();

    // SAFETY: No other threads exist yet; tokio runtime has not started.
    for (key, val) in &env_file_vars {
        unsafe { std::env::set_var(key, val); }
    }
    if !env_file_vars.is_empty() {
        eprintln!(
            "anyllm_proxy: loaded {} variable(s) from env file",
            env_file_vars.len()
        );
    }

    // Compute and apply LiteLLM env var aliases (e.g. LITELLM_MASTER_KEY -> PROXY_API_KEYS)
    let aliases = config::env_aliases::compute_env_aliases();
    // SAFETY: No other threads exist yet.
    for (key, val) in &aliases {
        unsafe { std::env::set_var(key, val); }
    }

    let load_result = config::MultiConfig::load();

    // Apply litellm master_key if PROXY_API_KEYS is still unset
    if let Some(ref mk) = load_result.litellm_master_key {
        if std::env::var("PROXY_API_KEYS").is_err() {
            // SAFETY: No other threads exist yet.
            unsafe { std::env::set_var("PROXY_API_KEYS", mk); }
            eprintln!("anyllm_proxy: applied general_settings.master_key as PROXY_API_KEYS");
        }
    }

    let env_file_count = env_file_vars.len();
    let model_router = load_result.model_router;
    let multi_config = load_result.multi_config;
    (env_file_count, multi_config, model_router)
}
```

--------------------------------

### Run anyllm-proxy with AWS Bedrock

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/ENV.md

Example command to launch the proxy using the Bedrock backend with required AWS credentials and region.

```bash
BACKEND=bedrock \
AWS_REGION=us-east-1 \
AWS_ACCESS_KEY_ID=AKIA... \
AWS_SECRET_ACCESS_KEY=wJalr... \
cargo run -p anyllm_proxy
```

--------------------------------

### Start anyllm-proxy with Vertex AI

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/vertex_ai.md

Run the proxy using environment variables or Docker with the required GCP project, region, and API key.

```bash
BACKEND=vertex_ai \
  VERTEX_PROJECT=my-project-123 \
  VERTEX_REGION=us-central1 \
  VERTEX_API_KEY=AIza... \
  cargo run -p anyllm_proxy
# or with Docker:
docker run \
  -e BACKEND=vertex_ai \
  -e VERTEX_PROJECT=my-project-123 \
  -e VERTEX_REGION=us-central1 \
  -e VERTEX_API_KEY=AIza... \
  -e PROXY_OPEN_RELAY=true \
  -p 3000:3000 \
  followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### GET /admin/api/uptime

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-04-05-react-admin-ui.md

Retrieves the proxy's start time, 30-day uptime percentage, and historical daily status, along with detailed uptime information for each configured backend. This includes their current status, last check time, latency, 30-day uptime percentage, and historical daily status.

```APIDOC
## GET /admin/api/uptime

### Description
Retrieves uptime statistics for the proxy and its configured backends. This includes the proxy's start time, overall uptime percentage over the last 30 days, and a history of daily status. For each backend, it provides its current status, the timestamp of the last health check, the latency of that check, its 30-day uptime percentage, and a history of its daily status.

### Method
GET

### Endpoint
/admin/api/uptime

### Parameters
This endpoint does not accept any path or query parameters.

### Request Body
This endpoint does not accept a request body.

### Response
#### Success Response (200 OK)
Returns a JSON object containing uptime information for the proxy and its backends.

- **proxy** (object) - Uptime information for the proxy.
  - **started_at** (number) - The Unix timestamp when the proxy started.
  - **uptime_pct_30d** (number) - The proxy's uptime percentage over the last 30 days (always 100.0 as it doesn't self-report downtime).
  - **history** (array) - An array of objects, each representing a day's status.
    - **date** (string) - The date in 'YYYY-MM-DD' format.
    - **status** (string) - The status for the day ('up' or 'down').
- **backends** (array) - An array of objects, each representing a backend's uptime information.
  - **name** (string) - The name of the backend.
  - **status** (string) - The current health status of the backend ('up', 'down', or 'unknown').
  - **last_checked_at** (number | null) - The Unix timestamp of the last health check, or null if no checks have been recorded.
  - **last_latency_ms** (number | null) - The latency in milliseconds of the last health check, or null if not available.
  - **uptime_pct_30d** (number) - The backend's uptime percentage over the last 30 days.
  - **history** (array) - An array of objects, each representing a day's status for the backend.
    - **date** (string) - The date in 'YYYY-MM-DD' format.
    - **status** (string) - The status for the day ('up' or 'down').

### Response Example
```json
{
  "proxy": {
    "started_at": 1678886400,
    "uptime_pct_30d": 100.0,
    "history": [
      {"date": "2023-03-15", "status": "up"},
      {"date": "2023-03-16", "status": "up"}
    ]
  },
  "backends": [
    {
      "name": "backend-1",
      "status": "up",
      "last_checked_at": 1678972800,
      "last_latency_ms": 50,
      "uptime_pct_30d": 99.5,
      "history": [
        {"date": "2023-03-15", "status": "up"},
        {"date": "2023-03-16", "status": "up"}
      ]
    }
  ]
}
```
```

--------------------------------

### Add Frontend Job to CI Workflow

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-04-05-react-admin-ui.md

Integrates a new 'frontend' job into the GitHub Actions CI workflow (`.github/workflows/ci.yml`). This job handles Node.js setup, dependency installation, TypeScript checks, linting, and building the admin UI. The 'test' job now depends on 'frontend' to ensure frontend checks pass before Rust tests run.

```yaml
jobs:
  frontend:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'
          cache-dependency-path: crates/proxy/admin-ui/package-lock.json
      - name: Install dependencies
        working-directory: crates/proxy/admin-ui
        run: npm ci
      - name: TypeScript check
        working-directory: crates/proxy/admin-ui
        run: npx tsc --noEmit
      - name: Lint
        working-directory: crates/proxy/admin-ui
        run: npm run lint
      - name: Build
        working-directory: crates/proxy/admin-ui
        run: npm run build
      - name: Upload dist
        uses: actions/upload-artifact@v4
        with:
          name: admin-ui-dist
          path: crates/proxy/admin-ui/dist/

  test:
    needs: frontend
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Download dist
        uses: actions/download-artifact@v4
        with:
          name: admin-ui-dist
          path: crates/proxy/admin-ui/dist/
      - uses: dtolnay/rust-toolchain@stable
        with:
          components: clippy, rustfmt
      - uses: Swatinem/rust-cache@v2
      # ... rest of existing steps unchanged ...

```

--------------------------------

### Lambda AI Quick Start with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/lambda_ai.md

Set up Lambda AI integration using environment variables for the backend and API key. This is useful for quick local testing or deployment.

```bash
BACKEND=lambda_ai LAMBDA_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=lambda_ai -e LAMBDA_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Install and Verify Debian Package

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/specs/2026-04-06-deb-packaging-design.md

Commands to install a Debian package, resolve dependencies, and perform basic verification checks on the installed anyllm-proxy.

```bash
sudo dpkg -i *.deb && sudo apt-get install -f -y
anyllm-proxy --help
systemd-analyze verify anyllm-proxy.service
```

--------------------------------

### Run Infinity Proxy with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/infinity.md

Starts the anyllm proxy with Infinity as the backend using environment variables. This is useful for quick local testing.

```bash
BACKEND=infinity PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
```

```bash
docker run -e BACKEND=infinity -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Featherless AI Single-Backend Quick Start (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/featherless_ai.md

Configure the Featherless AI backend using environment variables for local execution or Docker deployment.

```bash
BACKEND=featherless_ai FEATHERLESS_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=featherless_ai -e FEATHERLESS_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Client Usage Example

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/codedocs/api-reference/anyllm-client.md

Example of initializing the AnyLLM client and executing a message request.

```APIDOC
## Client Usage

### Description
Demonstrates how to configure the client with a base URL and API key, and perform a message creation request.

### Example
```rust
let client = Client::builder()
    .base_url("https://api.openai.com/v1/chat/completions")
    .api_key("sk-...")
    .build()?;

let req: MessageCreateRequest = ...;
let resp = client.messages(&req).await?;
```
```

--------------------------------

### Run Proxy with xAI Backend (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/xai.md

Starts the anyllm-proxy using the xAI backend via environment variables. Ensure you have the xAI API key set.

```bash
BACKEND=xai XAI_API_KEY=your-key cargo run -p anyllm_proxy
```

--------------------------------

### Run anyllm-proxy with llamafile Backend

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/llamafile.md

Start the anyllm-proxy with llamafile as the backend using environment variables. This is useful for quick local testing.

```bash
BACKEND=llamafile PROXY_OPEN_RELAY=true cargo run -p anyllm-proxy
```

```bash
docker run -e BACKEND=llamafile -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Download and Run a llamafile

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/llamafile.md

Instructions for downloading a specific llamafile model, making it executable, and starting the local inference server.

```bash
wget https://huggingface.co/Mozilla/Meta-Llama-3.1-8B-Instruct-llamafile/resolve/main/Meta-Llama-3.1-8B-Instruct.Q6_K.llamafile
chmod +x Meta-Llama-3.1-8B-Instruct.Q6_K.llamafile
./Meta-Llama-3.1-8B-Instruct.Q6_K.llamafile --server --port 8080
```

--------------------------------

### Git Commit Example

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-03-30-model-routing-ux.md

Example of a git commit command used after updating documentation files.

```bash
git add CLAUDE.md
git commit -m "docs: document simple YAML config format in CLAUDE.md"

```

--------------------------------

### Install Admin UI Dependencies

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-04-05-react-admin-ui.md

Navigate to the admin UI directory and install project dependencies using npm.

```bash
cd crates/proxy/admin-ui
npm install
```

--------------------------------

### Run Predibase Backend with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/predibase.md

Configure and run the anyllm-proxy with Predibase as the backend using environment variables. This is useful for quick local testing.

```bash
BACKEND=predibase PREDIBASE_API_KEY=your-key cargo run -p anyllm_proxy
```

--------------------------------

### Run Proxy with Cohere (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/cohere_chat.md

Starts the anyllm-proxy using Cohere as the backend via environment variables. Ensure you have the COHERE_API_KEY set.

```bash
BACKEND=cohere_chat COHERE_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=cohere_chat -e COHERE_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Install AnyLLM Proxy

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/crates/proxy/README.md

Command to install the AnyLLM Proxy using Cargo, assuming it's published to crates.io.

```bash
cargo install anyllm_proxy
```

--------------------------------

### Install anyllm-proxy

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/codedocs/index.md

Install the binary via Homebrew or Cargo. Note that it is not published as a package for npm, pnpm, yarn, or bun.

```bash
# anyllm-proxy is not published as a JavaScript package
brew install whit3rabbit/tap/anyllm-proxy
# or
cargo install anyllm_proxy
```

--------------------------------

### Quick Start: Single-Backend (env vars)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/cloudflare.md

Configure and run the proxy using environment variables for Cloudflare Workers AI. Ensure you replace `<your-account-id>` with your actual Cloudflare account ID.

```bash
BACKEND=cloudflare \
  CLOUDFLARE_API_KEY=your-token \
  OPENAI_BASE_URL=https://api.cloudflare.com/client/v4/accounts/<your-account-id>/ai/v1 \
  PROXY_OPEN_RELAY=true \
  cargo run -p anyllm_proxy
```

--------------------------------

### Featherless AI Anthropic Messages API Example

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/featherless_ai.md

Example of how to call the Featherless AI provider using the Anthropic Messages API format.

```bash
curl http://localhost:3000/v1/messages \
  -H "x-api-key: $PROXY_API_KEYS" \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-llama/Meta-Llama-3.1-8B-Instruct", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```

--------------------------------

### Run GMI Cloud Proxy with Environment Variables

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/gmi_cloud.md

Start the anyllm-proxy with GMI Cloud as the backend using environment variables for configuration. This is useful for quick local testing.

```bash
BACKEND=gmi GMI_CLOUD_API_KEY=your-key cargo run -p anyllm_proxy
# Docker:
docker run -e BACKEND=gmi -e GMI_CLOUD_API_KEY=your-key -e PROXY_OPEN_RELAY=true -p 3000:3000 followthewhit3rabbit/anyllm-proxy
```

--------------------------------

### Start FastChat and Proxy

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/lmsys.md

This command starts the FastChat server and then runs the anyllm-proxy with the LMSYS backend enabled. Ensure FastChat is running before executing this command.

```bash
# Start FastChat first, then:
BACKEND=lmsys PROXY_OPEN_RELAY=true cargo run -p anyllm_proxy
```

--------------------------------

### Run anyllm-proxy with DeepSeek (Environment Variables)

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/deepseek.md

Starts the anyllm-proxy using DeepSeek as the backend by setting the BACKEND and DEEPSEEK_API_KEY environment variables. This is useful for quick local testing.

```bash
BACKEND=deepseek DEEPSEEK_API_KEY=your-key cargo run -p anyllm_proxy
```

--------------------------------

### Implement GET /admin/api/uptime Endpoint

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/superpowers/plans/2026-04-05-react-admin-ui.md

This Rust code snippet implements the logic for the GET /admin/api/uptime endpoint, querying historical uptime data and formatting it.

```rust
ORDER BY day ASC",
).ok()
.and_then(|mut stmt| {
    stmt.query_map([], |r| {
        Ok(HistoryDay {
            date: r.get(0)?,
            status: if r.get::<_, i64>(1)? == 1 { "up".to_string() } else { "down".to_string() },
        })
    }).ok().map(|rows| rows.filter_map(|r| r.ok()).collect())
})
.unwrap_or_default();
```

--------------------------------

### NanoGPT Anthropic Messages API Example

Source: https://github.com/whit3rabbit/anyllm-proxy/blob/main/docs/providers/nanogpt.md

Example of how to call the Anthropic Messages API endpoint through the proxy when using NanoGPT. Replace <model-id> with your specific model.

```bash
curl http://localhost:3000/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: $PROXY_API_KEYS" \
  -d '{"model": "<model-id>", "max_tokens": 1024, "messages": [{"role": "user", "content": "Hello"}]}'
```
Tokens:
Raw