Liter LLM

https://github.com/xberg-io/liter-llm
Admin
Liter LLM is a unified client and proxy built on a Rust core that provides a single, consistent API...
Tokens:202,925
Snippets:2,538
Trust Score:8
License:MIT
Update:2 months ago
Show doc for...
Tokens:
Raw
### Set API key and start server

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/mcp-integration.mdx

Environment variable setup and server startup command.

```bash
export OPENAI_API_KEY="sk-..."
liter-llm mcp --transport stdio --config liter-llm-proxy.toml
```

--------------------------------

### Install via go get

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/partials/installation.md

Installation command for Go projects.

```bash
go get {{ package_name }}
```

--------------------------------

### Install via NuGet

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/partials/installation.md

Installation commands for .NET projects.

```bash
dotnet add package {{ package_name }}
```

```text
Install-Package {{ package_name }}
```

--------------------------------

### Install Catalog Overlay from String

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-zig.md

Installs an overlay registry from a raw JSON string, bypassing network and disk cache. Useful for testing overlay installation logic.

```zig
try installCatalogOverlayFromStr("value");
```

--------------------------------

### Install Catalog Overlay from String

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-go.md

Installs an overlay registry from a raw JSON string, bypassing network and disk cache. Useful for testing overlay installation and fallback behavior.

```go
if err := InstallCatalogOverlayFromStr("value"); err != nil {
    return err
}
```

--------------------------------

### Install Catalog Overlay from String

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-swift.md

Installs an overlay registry from a raw JSON string, bypassing network and disk caches. Useful for testing overlay installation and fallback behavior.

```swift
try installCatalogOverlayFromStr("value")
```

--------------------------------

### Install Catalog Overlay from String

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-python.md

Installs an overlay registry from a raw JSON string, bypassing network and disk cache. Useful for testing overlay installation and fallback behavior.

```python
install_catalog_overlay_from_str("value")
```

--------------------------------

### Install Catalog Overlay from String in C

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-c.md

Installs an overlay registry from a raw JSON string, bypassing network and disk cache. Useful for testing overlay installation and fallback behavior.

```c
literllm_install_catalog_overlay_from_str("value");
```

--------------------------------

### Install via pip

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/partials/installation.md

Standard installation command for Python environments.

```bash
pip install {{ package_name }}
```

--------------------------------

### Install via Composer

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/partials/installation.md

Installation command for PHP projects.

```bash
composer require {{ package_name }}
```

--------------------------------

### Install flutter_rust_bridge_codegen

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/dart/BUILDING.md

One-time setup command to install the required codegen tool.

```sh
cargo install flutter_rust_bridge_codegen
```

--------------------------------

### Install via RubyGems

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/partials/installation.md

Installation command and Gemfile configuration for Ruby.

```bash
gem install {{ package_name }}
```

```ruby
gem '{{ package_name }}'
```

--------------------------------

### Initialize Development Environment

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/contributing.md

Run initial setup and build the project in development mode.

```bash
# One-time setup - installs all dependencies
task setup

# Build in dev mode (fast iteration)
task build:dev
```

--------------------------------

### Install via Mix

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/partials/installation.md

Dependency configuration and installation for Elixir.

```elixir
def deps do
  [
    {:{{ package_name }}, "~> {{ version }}"}
  ]
end
```

```bash
mix deps.get
```

--------------------------------

### Start server with config

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/mcp-integration.mdx

Command to start the MCP server using a specific configuration file.

```bash
liter-llm mcp --transport stdio --config liter-llm-proxy.toml
```

--------------------------------

### Verify installation

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Run a quick check to ensure the library is correctly installed and accessible in the environment.

```bash
python -c "from liter_llm import create_client; print('ok')"
```

```bash
node -e "import('@xberg-io/liter-llm').then(m => { m.createClient('test'); console.log('ok') })"
```

```bash
cargo build
```

```bash
go build ./...
```

--------------------------------

### Start the proxy server

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/server/proxy-server.mdx

Launch the server using the default configuration discovery mechanism.

```bash
liter-llm api
```

--------------------------------

### Install the package

Source: https://github.com/xberg-io/liter-llm/blob/main/crates/liter-llm-node/README.md

Commands to install the library using common Node.js package managers.

```bash
npm install @xberg-io/liter-llm
```

```bash
pnpm add @xberg-io/liter-llm
```

```bash
yarn add @xberg-io/liter-llm
```

--------------------------------

### Quick Start Monorepo Build

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/go/README.md

Build steps for development within the liter-llm monorepo.

```bash
# Build the static FFI library
cargo build -p liter-llm-ffi --release

# Go build will automatically link against the static library
cd packages/go
go build -v
```

--------------------------------

### Install via Node.js package managers

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/partials/installation.md

Installation commands for npm, pnpm, and yarn.

```bash
npm install {{ package_name }}
```

```bash
pnpm add {{ package_name }}
```

```bash
yarn add {{ package_name }}
```

--------------------------------

### Install Catalog Overlay

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-kotlin-android.md

Install an overlay registry from a raw JSON string, bypassing network and disk caches. Useful for testing fallback behaviors.

```kotlin
installCatalogOverlayFromStr("value")
```

--------------------------------

### Install Go Module

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/go/README.md

Use the standard Go module command to add the package to your project.

```bash
go get github.com/xberg-io/liter-llm/packages/go@latest
```

--------------------------------

### Install Liter-LLM for WebAssembly

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Install the WASM package via pnpm.

```bash
pnpm add @xberg-io/liter-llm-wasm
```

--------------------------------

### Install and Run CLI Tools

Source: https://github.com/xberg-io/liter-llm/blob/main/README.md

Commands for installing the CLI via various package managers and running the proxy or MCP server.

```sh
brew install xberg-io/tap/liter-llm
cargo install liter-llm-cli                 # from crates.io
cargo binstall liter-llm-cli               # prebuilt binary from GitHub Releases
npx @xberg-io/liter-llm-cli --help         # npm (self-installs the binary)
docker run -p 4000:4000 -e LITER_LLM_MASTER_KEY=sk-your-key ghcr.io/xberg-io/liter-llm
```

```sh
liter-llm api --config liter-llm-proxy.toml   # OpenAI-compatible proxy (22 endpoints)
liter-llm mcp --transport stdio               # MCP tool server (stdio)
liter-llm mcp --transport http --port 3001    # MCP tool server (Streamable HTTP)
```

--------------------------------

### Install Plugin for Gemini CLI

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/guides/ai-coding-assistants.md

Use the gemini extensions command to install the plugin.

```text
gemini extensions install https://github.com/xberg-io/liter-llm
```

--------------------------------

### Install LiterLLM via NuGet

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/csharp/README.md

Commands to install the package using the .NET CLI or NuGet Package Manager.

```bash
dotnet add package XbergIo.LiterLlm
```

```text
Install-Package XbergIo.LiterLlm
```

--------------------------------

### Install C# Package

Source: https://github.com/xberg-io/liter-llm/blob/main/README.md

Use dotnet to add the C# package.

```sh
dotnet add package LiterLlm
```

--------------------------------

### Install dependencies

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/ruby.md

Run this command in your terminal to install the gem after adding it to your Gemfile.

```bash
bundle install
```

--------------------------------

### Install catalog overlay from string

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-rust.md

Updates the registry using a raw JSON string, bypassing network and disk caches. Useful for testing overlay installation and fallback behaviors.

```rust
install_catalog_overlay_from_str("value")?;
```

--------------------------------

### install_catalog_overlay_from_str/1

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-elixir.md

Installs an overlay registry from a raw catalog JSON string.

```APIDOC
## install_catalog_overlay_from_str(catalog_json)

### Description
Install the overlay registry from a raw catalog JSON string, bypassing the network and disk cache entirely.

### Parameters
- **catalog_json** (String.t()) - Required - The catalog json

### Returns
- {:ok, term()} | {:error, term()} - Returns {:error, reason} on parse failure.
```

--------------------------------

### Start Proxy via CLI

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/multi-tenancy.md

Command to launch the standalone proxy with a master key.

```bash
export LITER_LLM_MASTER_KEY="sk-proxy-$(openssl rand -hex 16)"
liter-llm api --config ./liter-llm-proxy.toml --master-key "$LITER_LLM_MASTER_KEY"
```

--------------------------------

### Start and test remote server

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/mcp-integration.mdx

Commands to start the server on all interfaces and verify connectivity via cURL.

```bash
# Start the server
liter-llm mcp --transport http --host 0.0.0.0 --port 3001 &

# Test the connection
curl http://127.0.0.1:3001/mcp \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $LITER_LLM_MASTER_KEY" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'
```

--------------------------------

### Configure Routing Strategies

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/fallback-routing.md

Examples of initializing the Router with different available routing strategies.

```rust
let router = Router::new(deployments, RoutingStrategy::RoundRobin)?;
```

```rust
let router = Router::new(deployments, RoutingStrategy::Fallback)?;
```

```rust
let router = Router::new(deployments, RoutingStrategy::LatencyBased)?;
```

```rust
let router = Router::new(deployments, RoutingStrategy::CostBased)?;
```

```rust
let router = Router::new(
    deployments,
    RoutingStrategy::WeightedRandom {
        weights: vec![3.0, 2.0, 1.0],
    },
)?;
```

--------------------------------

### installCatalogOverlayFromStr(string $catalogJson)

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-php.md

Install the overlay registry from a raw catalog JSON string.

```APIDOC
## installCatalogOverlayFromStr(string $catalogJson)

### Description
Install the overlay registry from a raw catalog JSON string, bypassing the network and disk cache entirely.

### Signature
`public static function installCatalogOverlayFromStr(string $catalogJson): void`

### Parameters
- **catalogJson** (string) - Required - The catalog JSON string.

### Errors
Throws `CatalogRefreshError` on parse failure.

### Example
```php
installCatalogOverlayFromStr("value");
```
```

--------------------------------

### Install Plugin for GitHub Copilot CLI

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/guides/ai-coding-assistants.md

Add and install the plugin using the copilot CLI commands.

```text
copilot plugin marketplace add https://github.com/xberg-io/liter-llm
copilot plugin install liter-llm@liter-llm
```

--------------------------------

### Install liter-llm Go module

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/go.md

Use the standard Go module command to fetch the package.

```bash
go get {{ package_name }}@latest
```

--------------------------------

### Fetch dependencies

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/elixir/README.md

Run this command in your terminal to install the added dependency.

```bash
mix deps.get
```

--------------------------------

### Start the proxy

Source: https://github.com/xberg-io/liter-llm/blob/main/plugin/skills/running-the-proxy/SKILL.md

Command to launch the API gateway using a specific configuration file.

```bash
liter-llm api --config liter-llm-proxy.toml
```

--------------------------------

### Start the proxy with explicit configuration

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/server/proxy-server.mdx

Launch the server by specifying a configuration file and a master key via environment variables.

```bash
export LITER_LLM_MASTER_KEY="sk-proxy-master-$(openssl rand -hex 16)"
liter-llm api --config ./liter-llm-proxy.toml
```

--------------------------------

### Configure liter-llm feature flags

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/concepts/feature-flags.md

Examples of Cargo.toml configurations ranging from minimal setups to full feature sets.

```toml
# Minimal: just the HTTP client, no Tower, no tracing
liter-llm = "..."

# Production server: Tower stack + OTEL + Bedrock
liter-llm = { version = "...", features = ["tower", "otel", "bedrock", "bedrock-auth"] }

# Azure OpenAI + tracing only
liter-llm = { version = "...", features = ["tower", "tracing", "azure-auth"] }

# Everything
liter-llm = { version = "...", features = ["full"] }
```

--------------------------------

### Install via Maven and Gradle

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/partials/installation.md

Configuration for Java build tools.

```xml
<dependency>
    <groupId>{{ maven_parts[0] }}</groupId>
    <artifactId>{{ maven_parts[1] }}</artifactId>
    <version>{{ version }}</version>
</dependency>
```

```gradle
implementation '{{ package_name }}:{{ version }}'
```

--------------------------------

### Proxy Configuration Example

Source: https://github.com/xberg-io/liter-llm/blob/main/llms.txt

A sample TOML configuration file for the Liter-LLM proxy server.

```toml
[server]
host = "0.0.0.0"
port = 4000

[auth]
master_key = "${LITER_LLM_MASTER_KEY}"

[[virtual_keys]]
key = "sk-team-frontend"
models = ["openai/*", "anthropic/*"]
rpm = 60
budget = 50.0

[[providers]]
name = "openai"
api_key = "${OPENAI_API_KEY}"
```

--------------------------------

### Install Task on Linux

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/contributing.md

Install Task using the official installer script or platform-specific package managers.

```bash
# Using the installer script
sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d -b ~/.local/bin
# Or via package managers:
apt install go-task  # Debian/Ubuntu
pacman -S go-task    # Arch
```

--------------------------------

### Start the proxy with distributed configuration

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/server/proxy-server.mdx

Enable configuration reloading via an etcd endpoint.

```bash
liter-llm api --watch --etcd-endpoint http://127.0.0.1:2379 --etcd-key /liter-llm/config
```

--------------------------------

### Build and serve the proxy with custom authentication

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/server/embedding.md

Demonstrates initializing a ProxyServer with a custom InMemoryKeyResolver and serving it.

```rust
use std::sync::Arc;
use liter_llm_proxy::{ProxyServer, config::ProxyConfig};
use liter_llm::tenant::InMemoryKeyResolver;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ProxyConfig::from_file("liter-llm-proxy.toml")?;

    // Create a custom key resolver (in-memory for this example)
    let resolver = Arc::new(InMemoryKeyResolver::with_entries(vec![
        (
            "sk-my-api-key".to_string(),
            liter_llm::tenant::ResolvedKey {
                tenant_id: "tenant-123".into(),
                allowed_models: vec!["openai/gpt-4o".to_string()],
                monthly_budget: None,
                currency: None,
                metadata: std::collections::HashMap::new(),
                active: true,
            },
        ),
    ]));

    // Build and serve with the custom resolver
    ProxyServer::new(config)
        .with_key_resolver(resolver)
        .serve_with_shutdown(None)
        .await?;

    Ok(())
}
```

--------------------------------

### Initialize Router

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/fallback-routing.md

Basic setup for a Router with multiple deployments using a RoundRobin strategy.

```rust
use liter_llm::tower::{LlmService, Router, RoutingStrategy};

let deployments = vec![
    LlmService::new(client_us_east),
    LlmService::new(client_eu_west),
    LlmService::new(client_us_west),
];

let router = Router::new(deployments, RoutingStrategy::RoundRobin)?;
```

--------------------------------

### Download and link pre-built static library

Source: https://github.com/xberg-io/liter-llm/blob/main/templates/readme/go.md

Steps to download the static library and build the Go project using CGO_LDFLAGS.

```bash
# Example: Linux x86_64
curl -LO https://github.com/xberg-io/liter-llm/releases/download/v{{ version }}/go-ffi-linux-x86_64.tar.gz
tar -xzf go-ffi-linux-x86_64.tar.gz

mkdir -p ~/liter-llm/lib
cp liter-llm-ffi/lib/libliter_llm_ffi.a ~/liter-llm/lib/
```

```bash
# Linux/macOS
CGO_LDFLAGS="-L$HOME/liter-llm/lib -lliter_llm_ffi" go build

# Windows (MSVC)
set CGO_LDFLAGS=-L%USERPROFILE%\liter-llm\lib -lliter_llm_ffi
go build
```

--------------------------------

### Install Node.js Package

Source: https://github.com/xberg-io/liter-llm/blob/main/README.md

Use pnpm to add the Node.js package.

```sh
pnpm add @xberg-io/liter-llm
```

--------------------------------

### Install Ollama

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/local-llms.mdx

Commands to install Ollama on macOS or Linux systems.

```bash
# macOS / Linux
curl -fsSL https://ollama.ai/install.sh | sh

# Or via Homebrew
brew install ollama
```

--------------------------------

### self.install_catalog_overlay_from_str(catalog_json)

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-ruby.md

Installs the overlay registry from a raw catalog JSON string, bypassing network and disk cache.

```APIDOC
## self.install_catalog_overlay_from_str(catalog_json)

### Description
Installs the overlay registry from a raw catalog JSON string, bypassing the network and disk cache entirely. It parses and flattens the provided JSON and atomically swaps it as the active overlay.

### Parameters
- **catalog_json** (String) - Required - The catalog json

### Errors
- Raises CatalogRefreshError
```

--------------------------------

### Install Ruby Gem

Source: https://github.com/xberg-io/liter-llm/blob/main/README.md

Use gem to install the Ruby library.

```sh
gem install liter_llm
```

--------------------------------

### Configure vLLM Provider

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/local-llms.mdx

Commands to start the vLLM server and the corresponding configuration for liter-llm.

```bash
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Llama-3-8B \
    --port 8000
```

```toml
# liter-llm.toml
api_key = ""

[[providers]]
name = "vllm"
base_url = "http://localhost:8000/v1"
model_prefixes = ["vllm/"]
```

--------------------------------

### Install Python Package

Source: https://github.com/xberg-io/liter-llm/blob/main/README.md

Use pip to install the Python package.

```sh
pip install liter-llm
```

--------------------------------

### Configure Llama.cpp Provider

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/local-llms.mdx

Commands to start the llama.cpp server and the corresponding configuration for liter-llm.

```bash
./llama-server -m model.gguf --port 8080
```

```toml
# liter-llm.toml
api_key = ""

[[providers]]
name = "llamacpp"
base_url = "http://localhost:8080/v1"
model_prefixes = ["llamacpp/"]
```

--------------------------------

### Build liter-llm from source

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Use these commands to install Rust, clone the repository, and compile the project.

```bash
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
git clone https://github.com/xberg-io/liter-llm.git
cd liter-llm
task build
```

--------------------------------

### Configure SDK via TOML

Source: https://github.com/xberg-io/liter-llm/blob/main/plugin/skills/liter-llm/SKILL.md

Example configuration file for setting API keys, timeouts, caching, budget, and rate limits.

```toml
# liter-llm.toml
api_key = "${OPENAI_API_KEY}"
model_hint = "openai"
timeout_secs = 120
max_retries = 5

[cache]
max_entries = 512
ttl_seconds = 600

[budget]
global_limit = 50.0
enforcement = "hard"

[rate_limit]
rpm = 60
tpm = 100000
```

--------------------------------

### Build Static Library from Source

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/go/README.md

Manual steps to compile the FFI library from the source repository.

```bash
git clone https://github.com/xberg-io/liter-llm.git
cd liter-llm

cargo build -p liter-llm-ffi --release

mkdir -p ~/liter-llm/lib
cp target/release/libliter_llm_ffi.a ~/liter-llm/lib/

cd ~/my-go-project
CGO_LDFLAGS="-L$HOME/liter-llm/lib -lliter_llm_ffi" go build
```

--------------------------------

### literllm_install_catalog_overlay_from_str()

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-c.md

Installs an overlay registry from a raw catalog JSON string, bypassing network and disk cache.

```APIDOC
## void literllm_install_catalog_overlay_from_str(const char* catalog_json)

### Description
Installs the overlay registry from a raw catalog JSON string, bypassing the network and disk cache entirely. This is primarily a testable seam.

### Signature
`void literllm_install_catalog_overlay_from_str(const char* catalog_json);`

### Parameters
- **catalog_json** (const char*) - Required - The catalog json

### Example
```c
literllm_install_catalog_overlay_from_str("value");
```

### Errors
Returns NULL on error.
```

--------------------------------

### Install liter-llm CLI

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/mcp-integration.mdx

Commands to install the CLI via Homebrew or Cargo.

```bash
brew trust xberg-io/tap
brew install xberg-io/tap/liter-llm
# or
cargo install liter-llm-cli
```

--------------------------------

### Initialize BudgetConfig with defaults

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-python.md

Creates a new BudgetConfig instance using default settings.

```python
result = BudgetConfig.default()
```

--------------------------------

### Install Task on macOS

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/contributing.md

Use Homebrew to install the Task runner on macOS.

```bash
brew install go-task
```

--------------------------------

### Initialize LiterLLM Client

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/snippets/java/usage/configuration.md

Demonstrates client instantiation with custom API key, base URL, timeout, retry settings, and provider hint. The client should be used within a try-with-resources block to ensure proper cleanup.

```java
import io.xberg.literllm.*;
import java.util.List;

public class Main {
    public static void main(String[] args) throws Exception {
        // Positional args: apiKey, baseUrl, timeoutSecs, maxRetries, modelHint.
        // Pass null for any optional value to use the built-in default.
        try (var client = LiterLlm.createClient(
                "sk-...",                          // or System.getenv("OPENAI_API_KEY")
                "https://api.openai.com/v1",       // override provider base URL
                60L,                                // request timeout in seconds
                3,                                  // retry on transient failures
                "openai"                            // pre-resolve provider at construction
        )) {
            var response = client.chat(ChatCompletionRequest.builder()
                .withModel("openai/gpt-4o")
                .withMessages(List.of(
                    new Message.User(new UserMessage(UserContent.of("Hello!"), null))
                ))
                .build());
            System.out.println(response.choices().getFirst().message().content());
        }
    }
}
```

--------------------------------

### Install Hermes Plugin

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/guides/ai-coding-assistants.md

Install the plugin for the Hermes runtime via pip.

```bash
pip install liter-llm-hermes-plugin
```

--------------------------------

### Initialize and use the Liter-LLM client

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/go/README.md

Demonstrates creating a client instance and performing a basic chat completion request.

```go
package main

import (
	"encoding/json"
	"fmt"
	"log"
	"os"

	literllm "github.com/xberg-io/liter-llm/packages/go"
)

func main() {
	client, err := literllm.CreateClient(os.Getenv("OPENAI_API_KEY"), nil, nil, nil, nil)
	if err != nil {
		log.Fatal(err)
	}
	defer client.Free()

	var req literllm.ChatCompletionRequest
	if err := json.Unmarshal([]byte(`{
		"model": "openai/gpt-4o-mini",
		"messages": [{"role": "user", "content": "Hello!"}]
	}`), &req); err != nil {
		log.Fatal(err)
	}

	resp, err := client.Chat(req)
	if err != nil {
		log.Fatalf("chat failed: %v", err)
	}

	if len(resp.Choices) > 0 && resp.Choices[0].Message.Content != nil {
		fmt.Println(*resp.Choices[0].Message.Content)
	}
}
```

--------------------------------

### Initialize Python Client

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/configuration.mdx

Create a client using direct arguments or by passing a serialized JSON configuration.

```python
from liter_llm import create_client

client = create_client(
    api_key="sk-...",
    base_url="https://api.openai.com/v1",
    timeout_secs=120,
    max_retries=5,
    model_hint="openai",
)

response = await client.chat(...)
```

```python
from liter_llm import create_client_from_json
import json

client = create_client_from_json(json.dumps({
    "api_key": "sk-...",
    "timeout_secs": 120,
}))
```

--------------------------------

### Install liter-llm CLI

Source: https://github.com/xberg-io/liter-llm/blob/main/plugin/README.md

Various methods to install or run the liter-llm CLI tool.

```bash
brew install xberg-io/tap/liter-llm
# or from crates.io (compiles from source):
cargo install liter-llm-cli
# or run it without a persistent install (the CLI proxy package self-installs the binary):
npx @xberg-io/liter-llm-cli --help
uvx --from liter-llm-cli liter-llm --help
# or download a prebuilt binary from the latest GitHub release:
#   https://github.com/xberg-io/liter-llm/releases/latest
# or install the CLI from crates.io:
cargo install liter-llm-cli
```

--------------------------------

### Install Liter-LLM Packages

Source: https://github.com/xberg-io/liter-llm/blob/main/llms.txt

Commands to install the Liter-LLM library for various programming environments.

```shell
cargo add liter-llm
```

```shell
pip install liter-llm
```

```shell
pnpm add @xberg-io/liter-llm
```

```shell
go get github.com/xberg-io/liter-llm/packages/go
```

```shell
implementation("io.xberg.literllm:liter-llm:1.9.0-rc.1")
```

```shell
dotnet add package LiterLlm
```

```shell
gem install liter_llm
```

```shell
{:liter_llm, "~> 1.7"} in mix.exs
```

```shell
composer require xberg-io/liter-llm
```

```shell
pnpm add @xberg-io/liter-llm-wasm
```

```shell
cargo build --release -p liter-llm-ffi
```

--------------------------------

### Download Pre-built Static Library

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/go/README.md

Commands to download and prepare the static FFI library for external projects.

```bash
# Example: Linux x86_64
curl -LO https://github.com/xberg-io/liter-llm/releases/download/v1.16.0/go-ffi-linux-x86_64.tar.gz
tar -xzf go-ffi-linux-x86_64.tar.gz

mkdir -p ~/liter-llm/lib
cp liter-llm-ffi/lib/libliter_llm_ffi.a ~/liter-llm/lib/
```

--------------------------------

### Install Task on Windows

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/contributing.md

Install Task using Scoop or Chocolatey package managers.

```powershell
# Using Scoop
scoop install task

# Or using Chocolatey
choco install go-task
```

--------------------------------

### Verify CLI installation

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/usage/mcp-integration.mdx

Commands to check if the liter-llm CLI is correctly installed and in the system PATH.

```bash
liter-llm --version
which liter-llm
```

--------------------------------

### ProxyServer::new

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/server/embedding.md

Initializes a new ProxyServer instance using the provided configuration.

```APIDOC
## ProxyServer::new

### Description
Creates a new instance of the ProxyServer using the specified ProxyConfig.

### Signature
`pub fn new(config: ProxyConfig) -> Self`
```

--------------------------------

### Install Liter-LLM for Ruby

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Requires Ruby 3.2+. Install via gem or add to Gemfile.

```bash
gem install liter_llm
```

```ruby
gem "liter_llm"
```

--------------------------------

### Build and run the application

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/go/README.md

Commands to compile the Go application with the required CGO flags.

```bash
CGO_LDFLAGS="-L$HOME/liter-llm/lib -lliter_llm_ffi" go build
./myapp
```

--------------------------------

### Install Liter-LLM for Python

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Requires Python 3.10+. Use pip or uv to manage the installation.

```bash
pip install liter-llm
```

```bash
uv add liter-llm
```

--------------------------------

### Install CLI via Homebrew

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Use Homebrew to install the liter-llm CLI on macOS or Linux.

```bash
brew trust xberg-io/tap
brew install xberg-io/tap/liter-llm
```

--------------------------------

### Start Proxy and MCP Servers

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Commands to launch the proxy server with a configuration file or the MCP server using stdio transport.

```bash
liter-llm api --config liter-llm-proxy.toml
```

```bash
liter-llm mcp --transport stdio
```

--------------------------------

### Install Plugin for Factory Droid

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/guides/ai-coding-assistants.md

Add and install the plugin using the droid CLI commands.

```text
droid plugin marketplace add https://github.com/xberg-io/liter-llm
droid plugin install liter-llm@liter-llm
```

--------------------------------

### Install Plugin for Claude Code

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/guides/ai-coding-assistants.md

Use the marketplace commands to add and install the liter-llm plugin.

```text
/plugin marketplace add xberg-io/liter-llm
/plugin install liter-llm@liter-llm
```

--------------------------------

### Install CLI via Cargo

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Install the CLI using the Rust toolchain or cargo-binstall for prebuilt binaries.

```bash
cargo install liter-llm-cli
```

```bash
cargo binstall liter-llm-cli
```

--------------------------------

### Build and test the project

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/contributing.md

Execute development builds and run the test suite.

```bash
task build:dev
task test
```

--------------------------------

### Initialize Budget Configuration

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-c.md

Retrieves the default configuration for budget enforcement.

```c
LiterllmBudgetConfig *result = literllm_default();
```

--------------------------------

### Install the liter-llm CLI

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/server/mcp-server.mdx

Use these commands to install the liter-llm binary via Homebrew, Cargo, npm, or Docker.

```bash
brew install xberg-io/tap/liter-llm   # Homebrew (macOS/Linux)
cargo install liter-llm-cli                 # from crates.io
npx @xberg-io/liter-llm-cli --help         # npm — self-installs the binary
docker run ghcr.io/xberg-io/liter-llm  # container image
```

--------------------------------

### Install Liter LLM WASM package

Source: https://github.com/xberg-io/liter-llm/blob/main/crates/liter-llm-wasm/README.md

Commands to install the package using common Node.js package managers.

```bash
npm install @xberg-io/liter-llm-wasm
```

```bash
pnpm add @xberg-io/liter-llm-wasm
```

```bash
yarn add @xberg-io/liter-llm-wasm
```

--------------------------------

### Initialize Crypto Provider

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-go.md

Idempotently install the ring crypto provider as the rustls process default.

```go
EnsureCryptoProvider()
```

--------------------------------

### Install Liter-LLM CLI via Package Managers

Source: https://github.com/xberg-io/liter-llm/blob/main/plugin/skills/liter-llm/SKILL.md

Commands to install the Liter-LLM CLI using Homebrew, npx, uvx, cargo, or Docker.

```bash
# Homebrew (macOS / Linux)
brew install xberg-io/tap/liter-llm

# or run it without a persistent install (the CLI proxy package self-installs the binary)
npx @xberg-io/liter-llm-cli --help
uvx --from liter-llm-cli liter-llm --help

# or download a prebuilt binary from the latest GitHub release:
#   https://github.com/xberg-io/liter-llm/releases/latest

# or build from source
cargo install liter-llm-cli

# or Docker (35MB image)
docker pull ghcr.io/xberg-io/liter-llm
```

--------------------------------

### Perform a web search using LiterLLM

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/snippets/csharp/usage/search.md

Initializes the client with an API key and executes a search query. Ensure the BRAVE_API_KEY environment variable is set before execution.

```csharp
using LiterLlm;

using var client = LiterLlmLib.CreateClient(
    apiKey: Environment.GetEnvironmentVariable("BRAVE_API_KEY")!,
    baseUrl: null, timeoutSecs: null, maxRetries: null, modelHint: null);

var response = await client.Search(new SearchRequest
{
    Model = "brave/web-search",
    Query = "What is Rust programming language?",
    MaxResults = 5
});

foreach (var result in response.Results)
{
    Console.WriteLine($"{result.Title}: {result.Url}");
}
```

--------------------------------

### Install Liter-LLM for Zig

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Requires Zig 0.16+.

```bash
zig fetch --save https://github.com/xberg-io/liter-llm/archive/refs/tags/v1.10.0.tar.gz
```

--------------------------------

### Install Liter-LLM for .NET

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Requires .NET 10+.

```bash
dotnet add package XbergIo.LiterLlm
```

--------------------------------

### Initialize and use the Liter-LLM client

Source: https://github.com/xberg-io/liter-llm/blob/main/plugin/skills/liter-llm/SKILL.md

Demonstrates an asynchronous workflow to create a client and execute a chat completion request.

```python
import asyncio
import os
from liter_llm import create_client
from liter_llm._internal_bindings import ChatCompletionRequest

async def main() -> None:
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    request = ChatCompletionRequest.from_json(
        '{"model":"openai/gpt-4o","messages":[{"role":"user","content":"Hello!"}]}'
    )
    response = await client.chat(request)
    print(response.choices[0].message.content)

asyncio.run(main())
```

--------------------------------

### Install Liter-LLM for Go

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Requires Go 1.26+.

```bash
go get github.com/xberg-io/liter-llm/packages/go
```

--------------------------------

### installCatalogOverlayFromStr(String catalogJson)

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/reference/api-java.md

Installs an overlay registry from a raw JSON string, bypassing network and disk cache.

```APIDOC
## public static void installCatalogOverlayFromStr(String catalogJson)

### Description
Parses and flattens a raw catalog JSON string and atomically swaps it as the active overlay. This is primarily used for testing purposes.

### Parameters
- **catalogJson** (String) - Required - The catalog json string to install.

### Errors
- Throws `CatalogRefreshError` if parsing fails.

### Example
```java
installCatalogOverlayFromStr("{\"key\": \"value\"}");
```
```

--------------------------------

### GET /health

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/snippets/curl/server/health.md

Retrieves the full status of the service.

```APIDOC
## GET /health

### Description
Returns the full status of the service, including the list of configured models.

### Method
GET

### Endpoint
/health

### Response
#### Success Response (200)
- **status** (integer) - Service status code
- **models** (array) - List of configured models
```

--------------------------------

### Start CLI Proxy and MCP Server

Source: https://github.com/xberg-io/liter-llm/blob/main/plugin/README.md

Commands to launch the proxy server with a configuration file and the MCP server using stdio transport.

```bash
liter-llm api --config liter-llm-proxy.toml   # start the proxy (22 endpoints)
liter-llm mcp --transport stdio               # start the MCP server
```

--------------------------------

### GET /health/liveness

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/snippets/curl/server/health.md

Checks if the service process is running.

```APIDOC
## GET /health/liveness

### Description
Returns a 200 status code as long as the process is running.

### Method
GET

### Endpoint
/health/liveness

### Response
#### Success Response (200)
- **status** (integer) - Indicates the service is running.
```

--------------------------------

### Connect to a local LLM instance

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/snippets/kotlin/usage/local_llm.md

Demonstrates initializing the LiterLLM client with a local base URL and executing a chat completion request.

```kotlin
import io.xberg.literllm.android.*
import kotlinx.coroutines.runBlocking

fun main() = runBlocking {
    val client = LiterLlm.createClient(apiKey = "", baseUrl = "http://localhost:11434/v1")
    val request = ChatCompletionRequest(
        model = "ollama/qwen2:0.5b",
        messages = listOf(Message.User(UserMessage(content = UserContent.of("Hello!"))))
    )
    val response = client.chat(request)
    println(response.choices[0].message.content)
}
```

--------------------------------

### Install LiterLLM dependency

Source: https://github.com/xberg-io/liter-llm/blob/main/packages/elixir/README.md

Add the package to your mix.exs file.

```elixir
def deps do
  [
    {:liter_llm, "~> 1.16.0"}
  ]
end
```

--------------------------------

### Install Liter-LLM for Dart/Flutter

Source: https://github.com/xberg-io/liter-llm/blob/main/docs-site/src/content/docs/getting-started/installation.mdx

Requires Dart SDK 3.3+.

```bash
dart pub add liter_llm
```
Tokens:
Raw