Ferro Labs AI Gateway

https://docs.ferrolabs.ai
Admin
Ferro Labs AI Gateway is a single Go binary that sits in front of all your LLM traffic. It exposes...
Tokens:71,772
Snippets:909
Trust Score:2.4
Update:3 months ago
Show doc for...
Tokens:
Raw
### Clone and run cookbook example

Source: https://docs.ferrolabs.ai/frameworks/langchain-python

Set up the multi-provider agent cookbook example.

```bash
git clone https://github.com/ferro-labs/ai-gateway-cookbook  
cd ai-gateway-cookbook/python/02-langgraph-multi-provider-agent  
cp .env.example .env  
make run  
```

--------------------------------

### Install langchain-ferrolabsai

Source: https://docs.ferrolabs.ai/frameworks/langchain-python

Install the package via pip to get started.

```bash
pip install langchain-ferrolabsai  
```

--------------------------------

### Install Instructor and OpenAI

Source: https://docs.ferrolabs.ai/frameworks/instructor

Install the necessary dependencies to use Instructor with an OpenAI-compatible client.

```bash
pip install instructor openai  
```

--------------------------------

### Clone and Run Cookbook Example

Source: https://docs.ferrolabs.ai/frameworks/langgraph

Commands to clone the cookbook repository and run the provided Dockerized example.

```bash
git clone https://github.com/ferro-labs/ai-gateway-cookbook  
cd ai-gateway-cookbook/python/02-langgraph-multi-provider-agent  
cp .env.example .env  
make run
```

--------------------------------

### Install Dependencies

Source: https://docs.ferrolabs.ai/frameworks/langgraph

Install the necessary packages for LangChain integration and LangGraph.

```bash
pip install langchain-ferrolabsai "langgraph>=0.2.0,<0.3.0"
```

--------------------------------

### Install Python SDK

Source: https://docs.ferrolabs.ai/integrations/overview

Install the official Python SDK to access typed clients and admin API features.

```bash
pip install ferrolabsai
```

--------------------------------

### Installation and Execution

Source: https://docs.ferrolabs.ai/operations/cli-reference

Build the binary from source or run via Docker container.

```bash
# Build from source  
make build  
./bin/ferrogw serve  
  
# Or run with Docker  
docker run -p 8080:8080 -e OPENAI_API_KEY=sk-... ghcr.io/ferro-labs/ai-gateway serve
```

--------------------------------

### Install DSPy

Source: https://docs.ferrolabs.ai/frameworks/dspy

Install the DSPy library via pip.

```bash
pip install dspy-ai  
```

--------------------------------

### Setup AsyncFerroClient

Source: https://docs.ferrolabs.ai/integrations/sdk/python/async

Initialize the client with an API key and base URL.

```python
from ferrolabsai import AsyncFerroClient  
  
client = AsyncFerroClient(  
    api_key="sk-ferro-...",  
    base_url="http://localhost:8080",  
)  
```

--------------------------------

### Install Ferro Labs SDK

Source: https://docs.ferrolabs.ai/frameworks/vercel-ai-sdk

Install the official Ferro Labs SDK for typed responses and admin APIs.

```bash
pnpm add @ferro-labs-ai/sdk
```

--------------------------------

### Installed plugins response structure

Source: https://docs.ferrolabs.ai/api-reference/admin

The response format for GET /admin/plugins, returning an array of configured plugins.

```json
[
  { "name": "word-filter", "type": "guardrail", "enabled": true },
  { "name": "max-token", "type": "guardrail", "enabled": false }
]
```

--------------------------------

### Install AutoGen

Source: https://docs.ferrolabs.ai/frameworks/autogen

Install the required Python package for AutoGen.

```bash
pip install pyautogen
```

--------------------------------

### Start Gateway Server

Source: https://docs.ferrolabs.ai/operations/cli-reference

Launch the gateway HTTP server using environment variables for configuration.

```bash
ferrogw serve
```

```bash
GATEWAY_CONFIG=config.yaml \
MASTER_KEY=fgw_... \
OPENAI_API_KEY=sk-... \
ferrogw serve
```

--------------------------------

### Start benchmark containers

Source: https://docs.ferrolabs.ai/benchmarks

Launches the mock upstream and gateway services using Docker Compose.

```bash
# Start the mock upstream and gateway containers  
docker compose up -d  
```

--------------------------------

### Install Mastra dependencies

Source: https://docs.ferrolabs.ai/frameworks/mastra

Install the required core Mastra package and the Vercel OpenAI SDK provider.

```bash
pnpm add @mastra/core @ai-sdk/openai
```

--------------------------------

### Install CrewAI

Source: https://docs.ferrolabs.ai/frameworks/crewai

Install the necessary package to begin using CrewAI.

```bash
pip install crewai
```

--------------------------------

### Install Railway CLI

Source: https://docs.ferrolabs.ai/integrations/deployment/railway

Commands to install the Railway CLI and authenticate.

```bash
npm install -g @railway/cli  
railway login
```

--------------------------------

### Install Vercel AI SDK dependencies

Source: https://docs.ferrolabs.ai/frameworks/vercel-ai-sdk

Install the necessary packages to use the Vercel AI SDK with the Ferro gateway.

```bash
pnpm add ai @ai-sdk/openai
```

--------------------------------

### Install LangChain dependencies

Source: https://docs.ferrolabs.ai/frameworks/langchain-js

Install the required LangChain packages for OpenAI and core functionality.

```bash
pnpm add @langchain/openai @langchain/core
```

--------------------------------

### Start the Gateway with Docker

Source: https://docs.ferrolabs.ai/guides/migration-litellm

Run the Ferro Labs AI Gateway container, mounting your local configuration file.

```bash
docker run -d -p 8080:8080 \
  -v $(pwd)/config.yaml:/config.yaml \
  ghcr.io/ferro-labs/ai-gateway:latest
```

--------------------------------

### Start the stack

Source: https://docs.ferrolabs.ai/integrations/deployment/docker-compose

Initializes the entire service stack in detached mode.

```bash
docker compose up -d  
```

--------------------------------

### Set Gateway Configuration Path

Source: https://docs.ferrolabs.ai/getting-started/configuration

Define the configuration file location via environment variable and start the gateway.

```bash
export GATEWAY_CONFIG=./config.yaml  
./ferrogw  
```

--------------------------------

### Implementing a Timing Plugin

Source: https://docs.ferrolabs.ai/integrations/sdk/go

A complete example of a plugin that logs request duration by storing state in the plugin context metadata.

```go
package myplugin  
  
import (  
	"context"  
	"log"  
	"time"  
  
	"github.com/ferro-labs/ai-gateway/plugin"  
)  
  
// TimingPlugin logs the round-trip duration for every request.  
type TimingPlugin struct{}  
  
// Name returns the plugin identifier used in configuration.  
func (p *TimingPlugin) Name() string { return "timing" }  
  
// Type categorizes the plugin.  
func (p *TimingPlugin) Type() plugin.PluginType { return plugin.TypeLogging }  
  
// Init receives the plugin's config block from the gateway configuration.  
func (p *TimingPlugin) Init(config map[string]interface{}) error { return nil }  
  
// Execute runs at whatever stage the plugin is registered for. Stash the start  
// time on pctx.Metadata in the before_request stage and read it back later.  
func (p *TimingPlugin) Execute(ctx context.Context, pctx *plugin.Context) error {  
	if pctx.Response == nil {  
		// before_request stage — record the start time.  
		pctx.Metadata["timing_start"] = time.Now()  
		log.Printf("[timing] request to model=%s started", pctx.Request.Model)  
		return nil  
	}  
  
	// after_request stage — measure elapsed time.  
	start, ok := pctx.Metadata["timing_start"].(time.Time)  
	if !ok {  
		return nil  
	}  
	log.Printf("[timing] model=%s completed in %s (tokens: %d)",  
		pctx.Response.Model, time.Since(start), pctx.Response.Usage.TotalTokens)  
	return nil  
}  
```

--------------------------------

### Run Gateway via Docker

Source: https://docs.ferrolabs.ai/

Quickly start the gateway using Docker with an OpenAI API key.

```bash
docker run -p 8080:8080 -e OPENAI_API_KEY=sk-... ghcr.io/ferro-labs/ai-gateway:latest
```

--------------------------------

### Configure filesystem MCP server

Source: https://docs.ferrolabs.ai/guides/mcp

Start the filesystem server and define the configuration in config.yaml to enable file access tools.

```bash
# 1. Start the MCP filesystem server  
npx @modelcontextprotocol/server-filesystem /path/to/workspace  
  
# 2. config.yaml  
mcp_servers:  
  - name: filesystem  
    url: "http://localhost:3000/mcp"  
    allowed_tools: [read_file, list_directory, search_files]  
    max_call_depth: 4  
```

--------------------------------

### List Plugins via Admin API

Source: https://docs.ferrolabs.ai/integrations/sdk/python/reference

Retrieves a list of installed plugins from the gateway.

```python
plugins = client.admin.plugins.list()  

```

--------------------------------

### Enable Providers via Environment Variables

Source: https://docs.ferrolabs.ai/faq

Set the required API key environment variables before starting the gateway to enable specific providers.

```bash
export ANTHROPIC_API_KEY=sk-ant-...  
export GROQ_API_KEY=gsk_...  
```

--------------------------------

### GET /admin/plugins

Source: https://docs.ferrolabs.ai/api-reference/admin

Returns a list of all configured plugins and their current status.

```APIDOC
## GET /admin/plugins

### Description
Returns a JSON array describing the configured plugins.

### Method
GET

### Endpoint
/admin/plugins

### Response
#### Success Response (200)
- **plugins** (array) - List of objects containing name, type, and enabled status
```

--------------------------------

### Start the AI Gateway container

Source: https://docs.ferrolabs.ai/guides/migration-openrouter

Run the gateway using Docker, providing necessary API keys for providers and mounting the configuration file.

```bash
docker run -d -p 8080:8080 \
  -e OPENAI_API_KEY=sk-your-openai-key \
  -e ANTHROPIC_API_KEY=sk-ant-your-key \
  -v $(pwd)/config.yaml:/config.yaml \
  ghcr.io/ferro-labs/ai-gateway:latest  
```

--------------------------------

### PromQL Query Examples

Source: https://docs.ferrolabs.ai/guides/observability

Common queries for analyzing request rates, latency, error percentages, token throughput, and circuit breaker status.

```promql
# Request rate by provider  
rate(gateway_requests_total[5m])  
  
# P99 request latency  
histogram_quantile(0.99, rate(gateway_request_duration_seconds_bucket[5m]))  
  
# Error rate percentage (failed requests over all requests)  
sum(rate(gateway_requests_total{status="error"}[5m]))  
  / sum(rate(gateway_requests_total[5m])) * 100  
  
# Token throughput per minute (input + output)  
(rate(gateway_tokens_input_total[1m]) + rate(gateway_tokens_output_total[1m])) * 60  
  
# Estimated spend rate (USD/hour) by model  
rate(gateway_request_cost_usd_total[5m]) * 3600  
  
# Open circuit breakers  
gateway_circuit_breaker_state == 1  
```

--------------------------------

### Model Listing Response Format

Source: https://docs.ferrolabs.ai/api-reference/endpoints

Example JSON response for the GET /v1/models endpoint, showing the enriched model catalog data.

```json
{
  "object": "list",
  "data": [
    {
      "id": "gpt-4o",
      "object": "model",
      "owned_by": "openai",
      "mode": "chat",
      "context_window": 128000,
      "max_output_tokens": 16384,
      "capabilities": ["vision", "function_calling", "streaming", "json_mode"],
      "status": "stable",
      "deprecated": false
    }
  ]
}
```

--------------------------------

### Initialize Configuration

Source: https://docs.ferrolabs.ai/operations/cli-reference

Scaffold a minimal configuration file and generate a master key.

```bash
ferrogw init [flags]
```

```bash
ferrogw init
```

```text
  Ferro Labs AI Gateway -- Setup  
  
  ✔ Created config.yaml  
  ✔ Master key: fgw_4f9c2a1b7e3d8650a1c4f0b29d6e7a83  
  
  ⚠ Save this key -- you need it for the dashboard and API.  
    export MASTER_KEY=fgw_4f9c2a1b7e3d8650a1c4f0b29d6e7a83  
  
  Next steps:  
    1. Set provider API keys (e.g. export OPENAI_API_KEY=sk-...)  
    2. Start the gateway:  ferrogw serve  
    3. Open dashboard:     http://localhost:8080/dashboard
```

--------------------------------

### Example SSE event stream output

Source: https://docs.ferrolabs.ai/api-reference/streaming

The gateway emits token chunks as JSON objects prefixed with 'data:', concluding with a '[DONE]' sentinel.

```text
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1718600000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}  
  
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1718600000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":"Cold"},"finish_reason":null}]}  
  
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1718600000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{"content":" iron"},"finish_reason":null}]}  
  
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1718600000,"model":"gpt-4o-mini","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}  
  
data: [DONE]
```

--------------------------------

### Install Pydantic AI

Source: https://docs.ferrolabs.ai/frameworks/pydantic-ai

Install the required package via pip.

```bash
pip install pydantic-ai
```

--------------------------------

### Run a multi-provider agent recipe

Source: https://docs.ferrolabs.ai/frameworks

Commands to initialize and execute a multi-provider agent demo from the ai-gateway-cookbook repository.

```bash
cd python/02-langgraph-multi-provider-agent  
cp .env.example .env       # fill in FERRO_API_KEY  
make run                   # one command, working multi-provider agent
```

--------------------------------

### Implement Multi-provider Agent

Source: https://docs.ferrolabs.ai/frameworks/langgraph

A complete example of a LangGraph agent using different models for planning, coding, and summarization via the Ferro Labs gateway.

```python
import os  
from typing import TypedDict  
from langchain_core.messages import HumanMessage, SystemMessage  
from langchain_ferrolabsai import FerroChatModel  
from langgraph.graph import END, StateGraph  
  
class State(TypedDict):  
    request: str  
    plan: str  
    code: str  
    summary: str  
  
def _chat(model: str) -> FerroChatModel:  
    return FerroChatModel(  
        model=model,  
        base_url=os.environ["FERRO_BASE_URL"],  
        api_key=os.environ["FERRO_API_KEY"],  
    )  
  
PLANNER, CODER, SUMMARIZER = _chat("gpt-4o"), _chat("claude-3-5-sonnet-20241022"), _chat("gemini-1.5-flash")  
  
def plan(s):      r = PLANNER.invoke([SystemMessage(content="Plan in 3-5 steps."), HumanMessage(content=s["request"])]); return {**s, "plan": r.content}  
def code(s):      r = CODER.invoke([SystemMessage(content="Implement the plan as Python."), HumanMessage(content=s["plan"])]); return {**s, "code": r.content}  
def summarize(s): r = SUMMARIZER.invoke([SystemMessage(content="One sentence."), HumanMessage(content=s["code"])]); return {**s, "summary": r.content}  
  
g = StateGraph(State)  
g.add_node("planner", plan); g.add_node("coder", code); g.add_node("summarizer", summarize)  
g.set_entry_point("planner"); g.add_edge("planner", "coder"); g.add_edge("coder", "summarizer"); g.add_edge("summarizer", END)  
app = g.compile()  
print(app.invoke({"request": "Build a UTC-now CLI", "plan": "", "code": "", "summary": ""})["summary"])
```

--------------------------------

### Run AI Gateway with Docker

Source: https://docs.ferrolabs.ai/getting-started/quickstart

Starts the AI Gateway container on port 8080 with the required OpenAI API key.

```bash
docker run --rm -p 8080:8080 \
  -e OPENAI_API_KEY=sk-your-key \
  ghcr.io/ferro-labs/ai-gateway:latest
```

--------------------------------

### Initialize Fly.io Application

Source: https://docs.ferrolabs.ai/integrations/deployment/fly-io

Create a new directory and initialize the Fly.io application configuration without deploying immediately.

```bash
mkdir ferro-gateway-fly && cd ferro-gateway-fly  
fly launch --no-deploy  
```

--------------------------------

### Install Haystack AI

Source: https://docs.ferrolabs.ai/frameworks/haystack

Install the required Haystack package via pip.

```bash
pip install haystack-ai  
```

--------------------------------

### Build AI Gateway from source

Source: https://docs.ferrolabs.ai/getting-started/quickstart

Clones the repository and runs the service locally using make.

```bash
git clone https://github.com/ferro-labs/ai-gateway.git
cd ai-gateway

export OPENAI_API_KEY=sk-your-key
make run
```

--------------------------------

### Clone the benchmark repository

Source: https://docs.ferrolabs.ai/benchmarks

Initializes the local environment by cloning the open-source benchmark suite.

```bash
# Clone the benchmark repository  
git clone https://github.com/ferro-labs/ai-gateway-performance-benchmarks.git  
cd ai-gateway-performance-benchmarks  
```

--------------------------------

### Install LlamaIndex Dependencies

Source: https://docs.ferrolabs.ai/frameworks/llamaindex

Install the necessary LlamaIndex packages to enable OpenAI-compatible LLM support.

```bash
pip install llama-index llama-index-llms-openai
```

--------------------------------

### Manage Runtime Configuration

Source: https://docs.ferrolabs.ai/operations/cli-reference

Apply new configurations or roll back to previous versions.

```bash
ferrogw admin config set --file new-config.json
ferrogw admin config rollback 3
```

--------------------------------

### Configure simple API-key-based providers

Source: https://docs.ferrolabs.ai/guides/providers-config

Set these environment variables to enable standard providers. Most support an optional <PROVIDER>_BASE_URL override.

```bash
export OPENAI_API_KEY=sk-...  
export ANTHROPIC_API_KEY=sk-ant-...  
export GEMINI_API_KEY=...  
export MISTRAL_API_KEY=...  
export GROQ_API_KEY=gsk-...  
export COHERE_API_KEY=...  
export DEEPSEEK_API_KEY=...  
export TOGETHER_API_KEY=...  
export PERPLEXITY_API_KEY=pplx-...  
export FIREWORKS_API_KEY=fw-...  
export AI21_API_KEY=...  
export XAI_API_KEY=xai-...  
export HUGGING_FACE_API_KEY=hf-...  
export CEREBRAS_API_KEY=...  
export NVIDIA_NIM_API_KEY=...  
export NOVITA_API_KEY=...  
export QWEN_API_KEY=...  
export MOONSHOT_API_KEY=...  
export SAMBANOVA_API_KEY=...  
export DEEPINFRA_API_KEY=...  
export OPENROUTER_API_KEY=...  

```

--------------------------------

### Initialize and Use AsyncFerroClient

Source: https://docs.ferrolabs.ai/integrations/sdk/python/reference

Demonstrates the asynchronous client interface for chat completions.

```python
from ferrolabsai import AsyncFerroClient  
  
client = AsyncFerroClient(api_key="sk-ferro-...")  
  
response = await client.chat.completions.create(  
    model="gpt-4o",  
    messages=[{"role": "user", "content": "Hello!"}],  
)  
print(response.content)  

```

--------------------------------

### GET /admin/logs/stats

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves log statistics.

```APIDOC
## GET /admin/logs/stats

### Description
Retrieves log statistics.

### Method
GET

### Endpoint
/admin/logs/stats

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - Top-N buckets, max 100
- **stage** (string) - Optional
- **model** (string) - Optional
- **provider** (string) - Optional
- **since** (string) - Optional - RFC3339 timestamp
```

--------------------------------

### Initialize Railway Project

Source: https://docs.ferrolabs.ai/integrations/deployment/railway

Command to initialize a new Railway project.

```bash
railway init
```

--------------------------------

### GET /admin/logs

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves system logs.

```APIDOC
## GET /admin/logs

### Description
Retrieves system logs.

### Method
GET

### Endpoint
/admin/logs

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - Default 50, max 200
- **offset** (integer) - Optional - Default 0
- **stage** (string) - Optional
- **model** (string) - Optional
- **provider** (string) - Optional
- **since** (string) - Optional - RFC3339 timestamp
```

--------------------------------

### GET /metrics

Source: https://docs.ferrolabs.ai/operations/monitoring

Retrieves Prometheus-formatted metrics for the gateway.

```APIDOC
## GET /metrics

### Description
Returns system metrics in Prometheus text format for scraping.

### Method
GET

### Endpoint
/metrics
```

--------------------------------

### GET /admin/config/history

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves the configuration change history.

```APIDOC
## GET /admin/config/history

### Description
Retrieves the configuration change history.

### Method
GET

### Endpoint
/admin/config/history
```

--------------------------------

### GET /admin/config

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves the current gateway configuration.

```APIDOC
## GET /admin/config

### Description
Retrieves the current gateway configuration.

### Method
GET

### Endpoint
/admin/config
```

--------------------------------

### Check Version Information

Source: https://docs.ferrolabs.ai/operations/cli-reference

Display build, runtime, and environment version details.

```bash
ferrogw version
```

--------------------------------

### GET /admin/health

Source: https://docs.ferrolabs.ai/api-reference/admin

Checks the health status of the gateway.

```APIDOC
## GET /admin/health

### Description
Checks the health status of the gateway.

### Method
GET

### Endpoint
/admin/health
```

--------------------------------

### Deploy and monitor application

Source: https://docs.ferrolabs.ai/integrations/deployment/fly-io

Commands to trigger a deployment and view real-time logs.

```bash
fly deploy
```

```bash
fly logs
```

--------------------------------

### GET /admin/plugins

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves a list of enabled plugins.

```APIDOC
## GET /admin/plugins

### Description
Retrieves a list of enabled plugins.

### Method
GET

### Endpoint
/admin/plugins

### Response
#### Success Response (200)
- **plugins** (array) - List of objects containing name, type, and enabled status
```

--------------------------------

### Configure OpenAI SDK for Go

Source: https://docs.ferrolabs.ai/integrations/sdk/openai-compatible

Use the DefaultConfig to set the BaseURL before initializing the client.

```go
package main  
  
import (  
    "context"  
    "fmt"  
  
    "github.com/sashabaranov/go-openai"  
)  
  
func main() {  
    cfg := openai.DefaultConfig("sk-your-key")  
    cfg.BaseURL = "http://localhost:8080/v1"  
    client := openai.NewClientWithConfig(cfg)  
  
    resp, err := client.CreateChatCompletion(context.Background(),  
        openai.ChatCompletionRequest{  
            Model: "gpt-4o-mini",  
            Messages: []openai.ChatCompletionMessage{  
                {Role: openai.ChatMessageRoleUser, Content: "Hello from the gateway!"},  
            },  
        },  
    )  
    if err != nil {  
        panic(err)  
    }  
    fmt.Println(resp.Choices[0].Message.Content)  
}
```

--------------------------------

### GET /admin/providers

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves a list of configured providers.

```APIDOC
## GET /admin/providers

### Description
Retrieves a list of configured providers.

### Method
GET

### Endpoint
/admin/providers
```

--------------------------------

### GET /admin/keys

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves a list of all API keys.

```APIDOC
## GET /admin/keys

### Description
Retrieves a list of all API keys.

### Method
GET

### Endpoint
/admin/keys
```

--------------------------------

### Configure Instructor with Ferro Labs

Source: https://docs.ferrolabs.ai/frameworks/instructor

Initialize the Instructor client using the Ferro Labs base URL and API key to enable Pydantic model validation for chat completions.

```python
import os  
import instructor  
from openai import OpenAI  
from pydantic import BaseModel  
  
client = instructor.from_openai(OpenAI(  
    base_url=os.environ["FERRO_BASE_URL"] + "/v1",  
    api_key=os.environ["FERRO_API_KEY"],  
))  
  
class User(BaseModel):  
    name: str  
    age: int  
  
user = client.chat.completions.create(  
    model="claude-3-5-sonnet-20241022",     # routed to Anthropic by Ferro  
    response_model=User,  
    messages=[{"role": "user", "content": "Extract: Alice is 30 years old."}],  
)  
print(user)  # User(name='Alice', age=30)  
```

--------------------------------

### GET /admin/dashboard

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves dashboard information for the gateway.

```APIDOC
## GET /admin/dashboard

### Description
Retrieves dashboard information for the gateway.

### Method
GET

### Endpoint
/admin/dashboard
```

--------------------------------

### GET /health

Source: https://docs.ferrolabs.ai/operations/monitoring

Performs a deep health check of the gateway.

```APIDOC
## GET /health

### Description
Performs a deep health check of the gateway, returning status in JSON format. This is suitable for load balancer integration.

### Method
GET

### Endpoint
/health
```

--------------------------------

### Configure Qwen

Source: https://docs.ferrolabs.ai/guides/providers-config

Requires API key.

```bash
export QWEN_API_KEY=...  

```

--------------------------------

### GET /admin/keys/usage

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves usage statistics for API keys.

```APIDOC
## GET /admin/keys/usage

### Description
Retrieves usage statistics for API keys.

### Method
GET

### Endpoint
/admin/keys/usage

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - Default 20, max 100
- **offset** (integer) - Optional - Default 0
- **sort** (string) - Optional - 'usage' or 'last_used', default 'usage'
- **active** (boolean) - Optional - 'true' or 'false'
- **since** (string) - Optional - RFC3339 timestamp
```

--------------------------------

### Compare Configuration Files

Source: https://docs.ferrolabs.ai/guides/migration-litellm

Contrast the LiteLLM model list and routing configuration with the Ferro Labs provider and strategy structure.

```yaml
model_list:  
  - model_name: gpt-4o  
    litellm_params:  
      model: openai/gpt-4o  
      api_key: "os.environ/OPENAI_API_KEY"  
  
  - model_name: claude-3-5-sonnet  
    litellm_params:  
      model: anthropic/claude-3-5-sonnet-20241022  
      api_key: "os.environ/ANTHROPIC_API_KEY"  
  
router_settings:  
  routing_strategy: least-busy  
  fallbacks:  
    - gpt-4o: [claude-3-5-sonnet]
```

```yaml
providers:  
  - key: openai  
    provider: openai  
    api_key: "${OPENAI_API_KEY}"  
  
  - key: anthropic  
    provider: anthropic  
    api_key: "${ANTHROPIC_API_KEY}"  
  
strategy:  
  mode: fallback  
  
targets:  
  - virtual_key: openai  
    retry:  
      attempts: 3  
      on_status_codes: [429, 502, 503, 504]  
  - virtual_key: anthropic
```

--------------------------------

### GET /health

Source: https://docs.ferrolabs.ai/api

Checks the health status of the gateway service.

```APIDOC
## GET /health

### Description
Returns the health status of the Ferro Labs AI Gateway. This is the only public endpoint that does not require authentication.

### Method
GET

### Endpoint
/health
```

--------------------------------

### Configure Authentication via Environment Variables

Source: https://docs.ferrolabs.ai/integrations/sdk/typescript

Set the API key and base URL using environment variables for automatic client detection.

```bash
export FERRO_API_KEY=sk-ferro-...
export FERRO_BASE_URL=http://localhost:8080   # default
```

--------------------------------

### GET /admin/keys/{id}

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves details for a specific API key.

```APIDOC
## GET /admin/keys/{id}

### Description
Retrieves details for a specific API key.

### Method
GET

### Endpoint
/admin/keys/{id}

### Parameters
#### Path Parameters
- **id** (string) - Required - The ID of the API key
```

--------------------------------

### Execute Go benchmarks

Source: https://docs.ferrolabs.ai/integrations/sdk/go

Run the defined benchmark test with specific duration and count parameters.

```bash
go test -bench=BenchmarkChatCompletion -benchtime=30s -count=3 ./...  
```

--------------------------------

### Configure Manual Environment Variables

Source: https://docs.ferrolabs.ai/integrations/deployment/render

Required environment variables for manual Render deployment, pointing to a PostgreSQL database for storage backends.

```text
MASTER_KEY=fgw_your-master-key  
OPENAI_API_KEY=sk-your-key  
PORT=8080  
  
# Storage backends — all point to the Postgres database  
API_KEY_STORE_BACKEND=postgres  
API_KEY_STORE_DSN=postgresql://user:pass@host:5432/dbname  
CONFIG_STORE_BACKEND=postgres  
CONFIG_STORE_DSN=postgresql://user:pass@host:5432/dbname  
REQUEST_LOG_STORE_BACKEND=postgres  
REQUEST_LOG_STORE_DSN=postgresql://user:pass@host:5432/dbname  
```

--------------------------------

### Run Diagnostics

Source: https://docs.ferrolabs.ai/operations/cli-reference

Execute environment, configuration, and connectivity checks.

```bash
ferrogw doctor
```

```bash
ferrogw doctor
```

```text
  Provider API Keys  
    ✔ openai  
    ✔ anthropic  
    - gemini  
    - groq  
    - mistral  
  
    2 found  
  
  Configuration  
    ✔ config.yaml (strategy=fallback, targets=2)  
  
  Auth  
    ✔ MASTER_KEY is set  
  
  Gateway Connectivity  
    ✔ http://localhost:8080 -- healthy (3ms)
```

--------------------------------

### Integrate with FastAPI

Source: https://docs.ferrolabs.ai/integrations/sdk/python/async

Example of using AsyncFerroClient within a FastAPI route handler.

```python
from fastapi import FastAPI  
from ferrolabsai import AsyncFerroClient  
  
app = FastAPI()  
client = AsyncFerroClient()  
  
@app.post("/chat")  
async def chat(message: str):  
    response = await client.chat.completions.create(  
        model="gpt-4o",  
        messages=[{"role": "user", "content": message}],  
    )  
    return {  
        "reply": response.content,  
        "provider": response.provider,  
        "cost": response.usage.cost_usd,  
    }  
```

--------------------------------

### Initialize FerroClient with Constructor Options

Source: https://docs.ferrolabs.ai/integrations/sdk/typescript

Manually provide authentication credentials and gateway URL during client instantiation.

```typescript
import { FerroClient } from "@ferro-labs-ai/sdk";

const client = new FerroClient({
  apiKey: "sk-ferro-...",
  baseUrl: "https://gateway.example.com",
});
```

--------------------------------

### GET /admin/logs

Source: https://docs.ferrolabs.ai/operations/request-logging

Retrieves a list of request logs based on optional filters.

```APIDOC
## GET /admin/logs

### Description
Retrieves request logs from the system.

### Method
GET

### Endpoint
/admin/logs

### Parameters
#### Query Parameters
- **limit** (integer) - Optional - Default 50, max 200
- **offset** (integer) - Optional - Default 0
- **stage** (string) - Optional - Filter by request stage
- **model** (string) - Optional - Filter by model
- **provider** (string) - Optional - Filter by provider
- **since** (string) - Optional - Filter by RFC3339 timestamp
```

--------------------------------

### Configure SQLite storage backends

Source: https://docs.ferrolabs.ai/security/data-handling

Sets environment variables to point the gateway to local SQLite database files for configuration, API keys, and request logs.

```bash
export CONFIG_STORE_BACKEND=sqlite  
export CONFIG_STORE_DSN=./ferrogw-config.db  
  
export API_KEY_STORE_BACKEND=sqlite  
export API_KEY_STORE_DSN=./ferrogw-keys.db  
  
export REQUEST_LOG_STORE_BACKEND=sqlite  
export REQUEST_LOG_STORE_DSN=./ferrogw-requests.db  
```

--------------------------------

### GET /admin/keys/usage

Source: https://docs.ferrolabs.ai/guides/cost-tracking

Retrieves per-key activity, sorted and filterable, with a rolling summary of usage.

```APIDOC
## GET /admin/keys/usage

### Description
The admin usage endpoint returns per-key activity, sorted and filterable, with a rolling summary.

### Method
GET

### Endpoint
/admin/keys/usage

### Parameters
#### Query Parameters
- **sort** (string) - Optional - Sort order, supports 'usage' or 'last_used'.
- **active** (boolean) - Optional - Filter by active status.
- **since** (string) - Optional - Filter by start timestamp.
- **limit** (integer) - Optional - Limit the number of results.
- **offset** (integer) - Optional - Offset for pagination.

### Response
#### Success Response (200)
- **summary** (object) - Contains total_keys, active_keys, total_usage, and returned_keys.
```

--------------------------------

### Generate Embeddings

Source: https://docs.ferrolabs.ai/integrations/sdk/typescript

Example of generating embeddings for a batch of input strings and accessing the resulting dimensions.

```typescript
const response = await client.embeddings.create({  
  model: "text-embedding-3-small",  
  input: ["Ferro routes LLM requests", "across many providers"],  
});  
  
for (const item of response.data) {  
  console.log(`Dimension: ${item.embedding.length}`);  
}  
```

--------------------------------

### Configure Request Logger Plugin

Source: https://docs.ferrolabs.ai/guides/troubleshooting

Example configuration for the request-logger plugin targeting a Postgres backend.

```yaml
plugins:  
  - name: request-logger  
    type: logging  
    stage: before_request  
    enabled: true  
    config:  
      level: info  
      persist: true  
      backend: postgres  
      dsn: postgres://ferro:ferro_secret@postgres:5432/ferro_logs?sslmode=disable
```

--------------------------------

### Provision and attach Postgres

Source: https://docs.ferrolabs.ai/integrations/deployment/fly-io

Commands to create a new Postgres cluster and attach it to the current application.

```bash
fly postgres create --name ferro-gateway-db --region iad --vm-size shared-cpu-1x --volume-size 1
```

```bash
fly postgres attach ferro-gateway-db
```

--------------------------------

### Structured JSON log format

Source: https://docs.ferrolabs.ai/guides/observability

Example of a single log entry emitted to stdout by the gateway.

```json
{
  "time": "2026-03-11T10:23:45Z",
  "level": "info",
  "trace_id": "a3f9b1c2-d4e5-4678-8901-abcdef012345",
  "msg": "request complete",
  "provider": "anthropic",
  "model": "claude-3-5-sonnet-20241022",
  "latency_ms": 412,
  "status": 200,
  "tokens_prompt": 312,
  "tokens_completion": 87
}
```

--------------------------------

### GET /admin/logs/stats

Source: https://docs.ferrolabs.ai/api-reference/admin

Aggregates scanned log entries into count maps by stage, provider, and model.

```APIDOC
## GET /admin/logs/stats

### Description
Aggregates up to 5000 scanned entries into count maps. The limit query parameter trims provider and model maps to the top-N buckets.

### Method
GET

### Endpoint
/admin/logs/stats

### Query Parameters
- **limit** (integer) - Optional - Trims provider and model maps to top-N buckets

### Response
#### Success Response (200)
- **summary** (object) - Statistics including total_entries, error_entries, total_tokens, and truncation status
- **by_stage** (object) - Count map by stage
- **by_provider** (object) - Count map by provider
- **by_model** (object) - Count map by model
```

--------------------------------

### Configure Authentication

Source: https://docs.ferrolabs.ai/integrations/sdk/python/quickstart

Set credentials via environment variables or pass them directly to the client constructor.

```bash
export FERRO_API_KEY=sk-ferro-...  
export FERRO_BASE_URL=http://localhost:8080   # default  
```

```python
from ferrolabsai import FerroClient  
  
client = FerroClient(  
    api_key="sk-ferro-...",  
    base_url="http://localhost:8080",  
)  
```

--------------------------------

### GET /admin/logs

Source: https://docs.ferrolabs.ai/api-reference/admin

Retrieves a list of request-log entries with summary information and current filter settings.

```APIDOC
## GET /admin/logs

### Description
Returns a data envelope containing request-log entries, a summary of entries, and the applied filters.

### Method
GET

### Endpoint
/admin/logs

### Response
#### Success Response (200)
- **data** (array) - List of request log entries
- **summary** (object) - Contains total_entries and returned_entries
- **filters** (object) - Contains limit, offset, stage, model, provider, and since
```

--------------------------------

### Run Gateway Locally

Source: https://docs.ferrolabs.ai/faq

Deploy the gateway using Docker or build directly from the source repository.

```bash
# Docker  
docker run -p 8080:8080 -e OPENAI_API_KEY=sk-... ghcr.io/ferro-labs/ai-gateway:latest  
  
# From source  
git clone https://github.com/ferro-labs/ai-gateway && cd ai-gateway && make run  
```

--------------------------------

### Configure Replicate

Source: https://docs.ferrolabs.ai/guides/providers-config

Requires API token and explicit registration of text and image models.

```bash
export REPLICATE_API_TOKEN=r8_...  
export REPLICATE_TEXT_MODELS=meta/llama-3-8b-instruct,mistralai/mistral-7b-instruct-v0.2  
export REPLICATE_IMAGE_MODELS=stability-ai/sdxl  

```

--------------------------------

### GET /v1/models

Source: https://docs.ferrolabs.ai/api-reference/endpoints

Retrieves an aggregated list of models available through the gateway, enriched with catalog metadata.

```APIDOC
## GET /v1/models

### Description
Returns the standard OpenAI envelope containing a list of models. Each entry includes base model information enriched with gateway-specific catalog fields such as context window, capabilities, and status.

### Method
GET

### Endpoint
/v1/models

### Response
#### Success Response (200)
- **object** (string) - The object type, typically 'list'.
- **data** (array) - A list of model objects containing id, object, owned_by, mode, context_window, max_output_tokens, capabilities, status, and deprecated fields.

#### Response Example
{
  "object": "list",
  "data": [
    {
      "id": "gpt-4o",
      "object": "model",
      "owned_by": "openai",
      "mode": "chat",
      "context_window": 128000,
      "max_output_tokens": 16384,
      "capabilities": ["vision", "function_calling", "streaming", "json_mode"],
      "status": "stable",
      "deprecated": false
    }
  ]
}
```

--------------------------------

### Configure Ollama Cloud

Source: https://docs.ferrolabs.ai/guides/providers-config

Requires an API key; base URL and models are optional.

```bash
export OLLAMA_API_KEY=...  
export OLLAMA_CLOUD_BASE_URL=https://ollama.com  
export OLLAMA_CLOUD_MODELS=gpt-oss:20b  

```

--------------------------------

### Configure DSPy with Ferro Labs

Source: https://docs.ferrolabs.ai/frameworks/dspy

Initialize the dspy.LM client with Ferro Labs gateway credentials and define a simple QA signature.

```python
import os  
import dspy  
  
lm = dspy.LM(  
    model="openai/claude-3-5-sonnet-20241022",        # litellm-style; routed by Ferro  
    api_base=os.environ["FERRO_BASE_URL"] + "/v1",  
    api_key=os.environ["FERRO_API_KEY"],  
)  
dspy.settings.configure(lm=lm)  
  
class QA(dspy.Signature):  
    """Answer the question."""  
    question: str = dspy.InputField()  
    answer: str = dspy.OutputField()  
  
predict = dspy.Predict(QA)  
print(predict(question="What is a gateway?").answer)  
```

--------------------------------

### Implement Next.js Edge route

Source: https://docs.ferrolabs.ai/frameworks/vercel-ai-sdk

Example of a Next.js Edge route that forwards the trace ID to the client.

```typescript
// app/api/chat/route.ts
import { createOpenAI } from "@ai-sdk/openai";
import { streamText } from "ai";

export const runtime = "edge";

const ferro = createOpenAI({
  baseURL: process.env.FERRO_BASE_URL!,
  apiKey: process.env.FERRO_API_KEY!,
});

export async function POST(req: Request) {
  const { messages, model = "gpt-4o-mini" } = await req.json();
  const result = streamText({ model: ferro(model), messages });
  return result.toDataStreamResponse({
    headers: { "x-ferro-trace-id": result.response?.headers?.["x-trace-id"] ?? "" },
  });
}
```

--------------------------------

### Config history response structure

Source: https://docs.ferrolabs.ai/api-reference/admin

The response format for GET /admin/config/history, showing the in-memory version history.

```json
{
  "data": [
    {
      "version": 1,
      "updated_at": "2026-06-17T10:05:00Z",
      "config": { /* full config snapshot */ },
      "rolled_back_from": null
    }
  ],
  "summary": { "total_versions": 1 }
}
```

--------------------------------

### List Gateway Plugins

Source: https://docs.ferrolabs.ai/operations/cli-reference

Display all registered plugins and their current status.

```bash
ferrogw plugins
```

--------------------------------

### Request logs response structure

Source: https://docs.ferrolabs.ai/api-reference/admin

The response format for GET /admin/logs, containing log entries and metadata.

```json
{
  "data": [ /* request log entries */ ],
  "summary": {
    "total_entries": 1500,
    "returned_entries": 50
  },
  "filters": {
    "limit": 50,
    "offset": 0,
    "stage": "",
    "model": "",
    "provider": "",
    "since": ""
  }
}
```

--------------------------------

### GET /health

Source: https://docs.ferrolabs.ai/guides/observability

Returns a deep health check of the gateway, including per-provider availability and latency metrics.

```APIDOC
## GET /health

### Description
Returns a deep health check with per-provider availability and latency. Returns 200 OK if at least one provider is healthy, or 503 Service Unavailable if all providers are down.

### Method
GET

### Endpoint
/health

### Response
#### Success Response (200)
- **status** (string) - Overall health status
- **providers** (object) - Map of provider names to their health status and latency

#### Response Example
{
  "status": "ok",
  "providers": {
    "openai": { "healthy": true, "latency_ms": 245 },
    "anthropic": { "healthy": true, "latency_ms": 312 },
    "groq": { "healthy": false, "error": "connection refused" }
  }
}
```
Tokens:
Raw