### Launch the RelayFreeLLM Server

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Install dependencies and start the server using the provided Python module.

```bash
# Install dependencies
pip install -r requirements.txt

# Verify provider connectivity (optional but recommended)
python -m tests.test_models_availability

# Start the server
python -m src.server

# Server output:
# INFO: RelayFreeLLM starting up
# INFO: Auto-discovered 5 providers: ['Gemini', 'Groq', 'Mistral', 'Cerebras', 'Ollama']
# INFO: Meta model 'meta-model' ready with providers: ['Gemini', 'Groq', 'Mistral', 'Cerebras', 'Ollama']
# INFO: Uvicorn running on http://0.0.0.0:8000
```

--------------------------------

### Install RelayFreeLLM Dependencies

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Clone the repository and install the necessary Python packages using pip. This is the first step to setting up the RelayFreeLLM gateway.

```bash
git clone https://github.com/msmarkgu/RelayFreeLLM.git
cd RelayFreeLLM
pip install -r requirements.txt
```

--------------------------------

### Start RelayFreeLLM Server

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Execute this command to launch the RelayFreeLLM gateway. Once running, it will be accessible at http://localhost:8000.

```bash
python -m src.server
```

--------------------------------

### Environment Configuration Example

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Configure RelayFreeLLM gateway behavior using environment variables in a .env file. This includes API keys, server settings, and model defaults.

```bash
# .env file configuration

# API Keys for providers (obtain from respective provider dashboards)
GEMINI_APIKEY=your-gemini-api-key        # https://ai.google.dev
GROQ_APIKEY=your-groq-api-key            # https://console.groq.com
MISTRAL_APIKEY=your-mistral-api-key      # https://console.mistral.ai
CEREBRAS_APIKEY=your-cerebras-api-key    # https://cloud.cerebras.ai
DEEPSEEK_APIKEY=your-deepseek-api-key    # Optional
CLOUDFLARE_API_TOKEN=your-cf-token       # Optional
CLOUDFLARE_ACCOUNT_ID=your-cf-account    # Optional

# Local Ollama instance (optional)
OLLAMA_BASE_URL=http://localhost:11434

# Server settings
HOST=0.0.0.0
PORT=8000
LOG_LEVEL=INFO

# Model defaults
DEFAULT_TEMPERATURE=0.8
DEFAULT_MAX_TOKENS=4000
META_MODEL_NAME=meta-model

# Selection strategies: "roundrobin" or "random"
PROVIDER_STRATEGY=roundrobin
MODEL_STRATEGY=roundrobin

# Retry and quota settings
MAX_RETRIES=3
WAIT_FOR_QUOTA=True
MAX_QUOTA_WAIT=3600
```

--------------------------------

### Add a New Provider Client

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Example Python code for integrating a new LLM provider. Implement the `ApiInterface` and define `PROVIDER_NAME` for the new provider.

```python
# src/api_clients/my_provider_client.py
class MyProviderClient(ApiInterface):
    PROVIDER_NAME = "myprovider"
    
    async def call_model_api(self, request, stream):
        # Your API logic here
        pass
```

--------------------------------

### LangChain Chain with RelayFreeLLM

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Utilize RelayFreeLLM within LangChain chains for tasks like translation. This example demonstrates a simple prompt-to-output parser chain.

```python
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a translator. Translate to {language}."),
    ("human", "{text}")
])

chain = prompt | llm | StrOutputParser()
result = chain.invoke({"language": "French", "text": "Hello, how are you?"})
```

--------------------------------

### Configure Free API Keys and Strategies

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Create a .env file to store your API keys for various AI providers and define routing strategies for providers and models. This setup is crucial for RelayFreeLLM to access and manage different AI services.

```bash
GEMINI_APIKEY=      # ai.google.dev
GROQ_APIKEY=        # console.groq.com
MISTRAL_APIKEY=     # console.mistral.ai
CEREBRAS_APIKEY=    # cloud.cerebras.ai
# any other providers you have
#ABC_APIKEY=...
#XYZ_APIKEY=...
#Best_APIKEY=...
# OLLMA model you host locally
#OLLAMA_BASE_URL=http://localhost:11434  # optional

# tell RelayFreeLLM how to choose from providers and provided models.
# default strategy is roundrobin for both.
PROVIDER_STRATEGY=roundrobin # pick provider in turn
MODEL_STRATEGY=random # randomly pick a model of the currently selected provider
```

--------------------------------

### Verify Model Availability

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Run this command to check if RelayFreeLLM can successfully connect to and authenticate with the configured AI model providers. This is an optional but recommended step before starting the server.

```bash
python -m tests.test_models_availability
```

--------------------------------

### GET /v1/models

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Lists all available models that RelayFreeLLM can route to, with optional filtering by type and scale.

```APIDOC
## GET /v1/models

### Description
Retrieves a list of all available models accessible through RelayFreeLLM. This endpoint can be filtered by model type and scale to narrow down the results.

### Method
GET

### Endpoint
/v1/models

### Parameters
#### Query Parameters
- **type** (string) - Optional - Filters models by their type (e.g., `coding`, `text`).
- **scale** (string) - Optional - Filters models by their scale (e.g., `large`, `medium`, `small`).

### Request Example
```bash
curl http://localhost:8000/v1/models?type=coding&scale=large
```

### Response
#### Success Response (200)
- **data** (array) - A list of model objects, each containing details like `id`, `object`, `created`, `owned_by`, `permission`, and `root`. 

#### Response Example
```json
{
  "data": [
    {
      "id": "gemini-2.5-pro",
      "object": "model",
      "created": 1677610602,
      "owned_by": "google",
      "permission": [
        {
          "id": "modelperm-abc",
          "object": "model_permission",
          "created": 1677610602,
          "allow_create_engine": false,
          "allow_sampling": true,
          "allow_logprobs": true,
          "allow_search_indices": false,
          "allow_view": true,
          "allow_fine_tuning": false,
          "organization": "*",
          "group": null,
          "is_blocking": false
        }
      ],
      "root": "gemini-2.5-pro",
      "parent": null
    }
  ]
}
```
```

--------------------------------

### Automatic Failover Example

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Illustrates the automatic failover mechanism when a provider encounters an issue like a rate limit. The gateway retries with other providers until success.

```text
Request → Groq (rate limited)
       → Circuit breaker activates
       → Retry → Gemini
       → Retry → Mistral
       → Success ✓
```

--------------------------------

### GET /v1/usage

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Tracks aggregated usage statistics across all models and providers managed by RelayFreeLLM.

```APIDOC
## GET /v1/usage

### Description
Provides insights into the aggregated usage of the RelayFreeLLM gateway. This endpoint helps in monitoring token consumption and request counts across different models and providers.

### Method
GET

### Endpoint
/v1/usage

### Parameters
This endpoint does not accept any parameters.

### Request Example
```bash
curl http://localhost:8000/v1/usage
```

### Response
#### Success Response (200)
- **total_requests** (integer) - The total number of requests processed.
- **total_tokens_consumed** (integer) - The total number of tokens consumed across all requests.
- **provider_usage** (object) - An object detailing usage per provider.

#### Response Example
```json
{
  "total_requests": 1500,
  "total_tokens_consumed": 75000,
  "provider_usage": {
    "gemini": {
      "requests": 800,
      "tokens_consumed": 40000
    },
    "groq": {
      "requests": 500,
      "tokens_consumed": 25000
    },
    "mistral": {
      "requests": 200,
      "tokens_consumed": 10000
    }
  }
}
```
```

--------------------------------

### Detailed Health Check with Providers

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Get a detailed health status of the gateway, including a list of all registered providers. This is useful for monitoring and debugging.

```bash
curl http://localhost:8000/health
```

--------------------------------

### cURL Request to RelayFreeLLM API

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Example of how to send a chat completion request to the RelayFreeLLM API using cURL. This demonstrates the OpenAI-compatible endpoint and authentication method.

```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer relay-free" \
  -H "Content-Type: application/json" \
  -d '{"model": "meta-model", "messages": [{"role": "user", "content": "Hi"}]}'
```

--------------------------------

### Model Filtering by Type and Scale

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Filter available models by type (text, coding, ocr) and scale (large, medium, small) to get the most appropriate model for your use case.

```APIDOC
## POST /v1/chat/completions (Model Filtering)

### Description
Filter models by type and scale when making a chat completion request.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Query Parameters
- **model** (string) - Required - Use 'meta-model' for automatic routing with filtering.
- **model_type** (string) - Optional - Filters models by type (e.g., 'coding', 'text', 'ocr').
- **model_scale** (string) - Optional - Filters models by scale (e.g., 'large', 'medium', 'small').
- **model_name** (string) - Optional - Filters models by a specific model family name (e.g., 'deepseek').
- **messages** (array) - Required - An array of message objects.

### Request Example (Filtering by Type and Scale)
```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{ 
    "model": "meta-model",
    "model_type": "coding",
    "model_scale": "large",
    "messages": [
      {"role": "user", "content": "Write a Python function to sort a list using quicksort"}
    ]
  }'
```

### Request Example (Filtering by Model Name)
```bash
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{ 
    "model": "meta-model",
    "model_name": "deepseek",
    "messages": [
      {"role": "user", "content": "Solve this math problem step by step: 2x + 5 = 15"}
    ]
  }'
```
```

--------------------------------

### Python SDK Usage with RelayFreeLLM

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Demonstrates how to use the OpenAI Python SDK with RelayFreeLLM. It shows how to initialize the client and make requests to the 'meta-model' for automatic routing or to a specific provider's model.

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="relay-free"
)

# Automatic routing - picks the best available
response = client.chat.completions.create(
    model="meta-model",
    messages=[{"role": "user", "content": "Hello!"}]
)

# Or route to specific provider
response = client.chat.completions.create(
    model="groq/llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "Hello!"}]
)
```

--------------------------------

### Initialize OpenAI Client with RelayFreeLLM

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Modify your existing OpenAI client initialization to point to the RelayFreeLLM local endpoint. This requires changing the base URL and using a placeholder API key.

```python
client = OpenAI(base_url="http://localhost:8000/v1", api_key="fake")
```

--------------------------------

### LangChain Integration with RelayFreeLLM

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Shows how to integrate RelayFreeLLM with LangChain by initializing the ChatOpenAI model with the RelayFreeLLM endpoint URL and API key.

```python
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="relay-free",
    model="meta-model"
)
```

--------------------------------

### List Available Models

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Retrieve all available models across providers with their status and rate limit information. Supports filtering by type and scale using query parameters.

```bash
# List all available models
curl http://localhost:8000/v1/models

# Filter by type and scale
curl "http://localhost:8000/v1/models?type=coding&scale=medium"

# Response:
# {
#   "object": "list",
#   "data": [
#     {
#       "id": "meta-model",
#       "object": "model",
#       "owned_by": "relay-llms",
#       "description": "Unified meta model — automatically routes to the best available provider."
#     },
#     {

```

--------------------------------

### Implement a Custom Provider Client

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Create a new provider client by inheriting from ApiInterface and implementing list_models and call_model_api. Ensure the PROVIDER_NAME is unique and handle specific provider errors to trigger appropriate exceptions.

```python
from ..config import settings
from ..exceptions import ProviderError, RateLimitError, AuthenticationError
from ..logging_util import ProjectLogger
from .api_interface import ApiInterface

class MyProviderClient(ApiInterface):
    """Client for MyProvider LLM API."""

    PROVIDER_NAME = "MyProvider"  # Must be unique

    def __init__(self):
        self.api_key = settings.get_api_key("MYPROVIDER_APIKEY")
        self.logger = ProjectLogger.get_logger(__name__)
        # Initialize your SDK/client here

    async def list_models(self) -> list[str]:
        """Return list of available model names."""
        try:
            # Fetch models from your provider's API
            return ["model-small", "model-large", "model-code"]
        except Exception as e:
            self.logger.error(f"Error listing models: {e}")
            return []

    async def call_model_api(
        self,
        user_prompt: str,
        model: str,
        sys_instruct: str,
        temperature: float,
        max_tokens: int,
        stream: bool = False,
    ) -> str | object:
        """Call the provider's chat completion API."""
        try:
            if stream:
                async def generate():
                    # Implement streaming logic
                    async for chunk in your_streaming_call():
                        yield chunk.text
                return generate()

            # Non-streaming call
            response = await your_api_call(
                model=model,
                prompt=user_prompt,
                system=sys_instruct,
                temperature=temperature,
                max_tokens=max_tokens
            )
            return response.text

        except Exception as e:
            error_str = str(e).lower()
            if "429" in error_str or "rate" in error_str:
                raise RateLimitError("MyProvider", str(e)) from e
            if "401" in error_str or "403" in error_str:
                raise AuthenticationError("MyProvider", str(e)) from e
            raise ProviderError("MyProvider", str(e)) from e
```

--------------------------------

### List Available Models

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Use this curl command to list available models, filtering by type and scale. This helps in understanding which models are accessible through the gateway.

```bash
curl http://localhost:8000/v1/models?type=coding&scale=large
```

--------------------------------

### Streaming Responses with OpenAI SDK

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Enable Server-Sent Events (SSE) streaming for real-time token delivery from any supported provider. Set `stream=True` in your client request. The output streams token by token as they are generated.

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="relay-free"
)

# Stream tokens in real-time
stream = client.chat.completions.create(
    model="meta-model",
    messages=[
        {"role": "user", "content": "Write a short poem about programming"}
    ],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
# Output streams token by token as they're generated
```

--------------------------------

### LangChain Integration with ChatOpenAI

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Integrate RelayFreeLLM as a backend for LangChain applications using the ChatOpenAI interface. Ensure the base URL and API key are correctly configured.

```python
from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage

# Initialize LangChain with RelayFreeLLM backend
llm = ChatOpenAI(
    base_url="http://localhost:8000/v1",
    api_key="relay-free",
    model="meta-model",
    temperature=0.7
)

# Use with LangChain messages
messages = [
    SystemMessage(content="You are a helpful coding assistant."),
    HumanMessage(content="Write a Python function to calculate fibonacci numbers")
]

response = llm.invoke(messages)
print(response.content)
```

--------------------------------

### Direct Provider Routing with OpenAI SDK

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Route requests directly to a specific provider and model using the `provider/model` format. This bypasses automatic selection and circuit breaker logic. Ensure the OpenAI client is configured with the gateway's base URL and API key.

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="relay-free"
)

# Route directly to Groq's Llama model
response = client.chat.completions.create(
    model="Groq/llama-3.3-70b-versatile",
    messages=[
        {"role": "user", "content": "Explain quantum computing in simple terms"}
    ],
    temperature=0.8,
    max_tokens=2000
)

print(response.choices[0].message.content)
# The response.model_extra["meta"] contains provider attribution
```

--------------------------------

### Streaming Responses

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Enable Server-Sent Events (SSE) streaming for real-time token delivery from any supported provider.

```APIDOC
## POST /v1/chat/completions (Streaming)

### Description
Enables real-time streaming of response tokens using Server-Sent Events (SSE).

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Query Parameters
- **model** (string) - Required - The model to use (e.g., 'meta-model').
- **messages** (array) - Required - An array of message objects.
- **stream** (boolean) - Required - Set to `true` to enable streaming.
- **temperature** (number) - Optional - Controls randomness.
- **max_tokens** (integer) - Optional - Maximum tokens to generate.

### Request Example (Python SDK)
```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="relay-free"
)

stream = client.chat.completions.create(
    model="meta-model",
    messages=[
        {"role": "user", "content": "Write a short poem about programming"}
    ],
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
# Output streams token by token as they're generated
```
```

--------------------------------

### Basic Chat Completion with Automatic Routing

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Send a chat completion request to the gateway, which automatically routes to the best available provider. Ensure your client is configured to use the gateway's endpoint.

```bash
# Basic chat completion with automatic provider routing
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Authorization: Bearer relay-free" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-model",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is the capital of France?"}
    ],
    "temperature": 0.7,
    "max_tokens": 1000
  }'

# Response:
# {
#   "id": "chatcmpl-a1b2c3d4e5f6",
#   "object": "chat.completion",
#   "created": 1704067200,
#   "model": "gemini-2.5-flash",
#   "choices": [
#     {
#       "index": 0,
#       "message": {"role": "assistant", "content": "The capital of France is Paris."},
#       "finish_reason": "stop"
#     }
#   ],
#   "usage": {"prompt_tokens": 25, "completion_tokens": 10, "total_tokens": 35},
#   "meta": {"provider": "Gemini", "model": "gemini-2.5-flash", "latency_ms": 1234.56, "attempt": 1}
# }
```

--------------------------------

### List Available Models

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Retrieve all available models across providers with their status and rate limit information. Supports filtering by type and scale.

```APIDOC
## GET /v1/models

### Description
Retrieves a list of all available models, including their status and rate limit information. Supports filtering.

### Method
GET

### Endpoint
/v1/models

### Parameters
#### Query Parameters
- **type** (string) - Optional - Filters models by type (e.g., 'coding', 'text', 'ocr').
- **scale** (string) - Optional - Filters models by scale (e.g., 'large', 'medium', 'small').

### Request Example (List All Models)
```bash
curl http://localhost:8000/v1/models
```

### Request Example (Filtered List)
```bash
curl "http://localhost:8000/v1/models?type=coding&scale=medium"
```

### Response
#### Success Response (200)
- **object** (string) - Type of the object, e.g., 'list'.
- **data** (array) - An array of model objects.
  - **id** (string) - The model identifier.
  - **object** (string) - Type of the object, e.g., 'model'.
  - **owned_by** (string) - The owner of the model (e.g., 'relay-llms').
  - **description** (string) - A description of the model.

#### Response Example
```json
{
  "object": "list",
  "data": [
    {
      "id": "meta-model",
      "object": "model",
      "owned_by": "relay-llms",
      "description": "Unified meta model — automatically routes to the best available provider."
    }
  ]
}
```
```

--------------------------------

### Configure Provider Model Limits

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Define rate limits for specific models in provider_model_limits.json to manage request and token quotas.

```json
{
  "providers": [
    {
      "name": "MyProvider",
      "url": "",
      "models": [
        {
          "name": "my-model-large",
          "type": "text",
          "scale": "large",
          "limits": {
            "requests_per_second": 1,
            "requests_per_minute": 30,
            "requests_per_hour": 1000,
            "requests_per_day": 10000,
            "tokens_per_minute": 50000,
            "tokens_per_hour": 1000000,
            "tokens_per_day": -1
          }
        },
        {
          "name": "my-model-code",
          "type": "coding",
          "scale": "medium",
          "limits": {
            "requests_per_second": 2,
            "requests_per_minute": 60,
            "requests_per_hour": 2000,
            "requests_per_day": 20000,
            "tokens_per_minute": 100000,
            "tokens_per_hour": -1,
            "tokens_per_day": -1
          }
        }
      ]
    }
  ]
}
```

--------------------------------

### POST /v1/chat/completions

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Submit chat messages to the LLM gateway for processing. Supports intent-based model selection and automatic failover.

```APIDOC
## POST /v1/chat/completions

### Description
Submits chat messages to the LLM gateway. Allows for intent-based model selection using parameters like `model_type`, `model_scale`, and `model_name`, or direct model specification. It also handles automatic failover between providers if a selected model becomes unavailable or hits rate limits.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Query Parameters
- **stream** (bool) - Optional - Enable SSE streaming (default: false)

#### Request Body
- **model** (string) - Required - Specifies the model to use. Use `"meta-model"` for auto-routing based on intent, or `"provider/model"` for direct selection.
- **messages** (array) - Required - An array of message objects following the standard OpenAI message format.
- **model_type** (string) - Optional - Filters models by type (e.g., `text`, `coding`, `ocr`).
- **model_scale** (string) - Optional - Filters models by scale (e.g., `large`, `medium`, `small`).
- **model_name** (string) - Optional - Filters models by matching a substring of the model name.

### Request Example
```json
{
  "model": "meta-model",
  "model_type": "coding",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."}, 
    {"role": "user", "content": "Write a Python function to calculate factorial."}
  ]
}
```

### Response
#### Success Response (200)
- **id** (string) - Unique identifier for the completion.
- **object** (string) - Type of object returned, e.g., `chat.completion`.
- **created** (integer) - Unix timestamp of creation.
- **model** (string) - The model used for the completion.
- **choices** (array) - An array of completion choices.
- **usage** (object) - Usage statistics for the request.

#### Response Example
```json
{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "created": 1700000000,
  "model": "gemini-2.5-flash",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "def factorial(n):\n    if n == 0:\n        return 1\n    else:\n        return n * factorial(n-1)"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 50,
    "completion_tokens": 30,
    "total_tokens": 80
  }
}
```
```

--------------------------------

### Intent-Based Model Selection

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

Specify your LLM needs using JSON payloads. RelayFreeLLM interprets these to select the most suitable model from available providers.

```json
{"model": "meta-model", "messages": [...]}
```

```json
{"model": "meta-model", "model_type": "coding", "messages": [...]}
```

```json
{"model": "meta-model", "model_scale": "small", "messages": [...]}
```

```json
{"model": "meta-model", "model_scale": "large", "messages": [...]}
```

```json
{"model": "meta-model", "model_name": "deepseek", "messages": [...]}
```

```json
{"model": "Gemini/gemini-2.5-flash", "messages": [...]}
```

--------------------------------

### Track Aggregated Usage

Source: https://github.com/msmarkgu/relayfreellm/blob/main/README.md

This curl command retrieves your aggregated usage statistics. It's useful for monitoring resource consumption across different LLM calls.

```bash
curl http://localhost:8000/v1/usage
```

--------------------------------

### Simple Health Check

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Perform a basic health check on the gateway. This endpoint returns a simple 'Healthy' status if the gateway is operational.

```bash
curl http://localhost:8000/
```

--------------------------------

### Model Filtering by Type and Scale

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Filter available models by type (text, coding, ocr) and scale (large, medium, small) to select the most appropriate model for your use case. This is done by including `model_type` and `model_scale` in the request payload.

```bash
# Request a large coding model
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-model",
    "model_type": "coding",
    "model_scale": "large",
    "messages": [
      {"role": "user", "content": "Write a Python function to sort a list using quicksort"}
    ]
  }'

# Request a specific model family (e.g., DeepSeek)
curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-model",
    "model_name": "deepseek",
    "messages": [
      {"role": "user", "content": "Solve this math problem step by step: 2x + 5 = 15"}
    ]
  }'
```

--------------------------------

### Direct Provider Routing

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

Route requests directly to a specific provider and model by using the `provider/model` format instead of `meta-model`. This bypasses automatic selection and circuit breaker logic.

```APIDOC
## POST /v1/chat/completions (Direct Routing)

### Description
Allows direct routing to a specific LLM provider and model by specifying the model in the format `provider/model`.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Query Parameters
- **model** (string) - Required - The specific model to use, in the format `provider/model` (e.g., `Groq/llama-3.3-70b-versatile`).
- **messages** (array) - Required - An array of message objects.
- **temperature** (number) - Optional - Controls randomness.
- **max_tokens** (integer) - Optional - Maximum tokens to generate.

### Request Example (Python SDK)
```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="relay-free"
)

response = client.chat.completions.create(
    model="Groq/llama-3.3-70b-versatile",
    messages=[
        {"role": "user", "content": "Explain quantum computing in simple terms"}
    ],
    temperature=0.8,
    max_tokens=2000
)

print(response.choices[0].message.content)
# The response.model_extra["meta"] contains provider attribution
```
```

--------------------------------

### Chat Completions API

Source: https://context7.com/msmarkgu/relayfreellm/llms.txt

The primary endpoint for sending chat completion requests. It accepts OpenAI-compatible request format and automatically routes to the best available provider based on configured strategy and quota availability.

```APIDOC
## POST /v1/chat/completions

### Description
Sends chat completion requests to the RelayFreeLLM gateway, which automatically routes to the best available provider.

### Method
POST

### Endpoint
/v1/chat/completions

### Parameters
#### Query Parameters
- **model** (string) - Required - The model to use for the chat completion. Use 'meta-model' for automatic routing.
- **messages** (array) - Required - An array of message objects, each with a 'role' (system, user, assistant) and 'content'.
- **temperature** (number) - Optional - Controls randomness. Lower values make output more focused and deterministic.
- **max_tokens** (integer) - Optional - The maximum number of tokens to generate in the completion.
- **model_type** (string) - Optional - Filters models by type (e.g., 'coding', 'text', 'ocr').
- **model_scale** (string) - Optional - Filters models by scale (e.g., 'large', 'medium', 'small').
- **model_name** (string) - Optional - Filters models by a specific model family name (e.g., 'deepseek').

### Request Example
```json
{
  "model": "meta-model",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"}
  ],
  "temperature": 0.7,
  "max_tokens": 1000
}
```

### Response
#### Success Response (200)
- **id** (string) - Unique identifier for the chat completion.
- **object** (string) - Type of the object, e.g., 'chat.completion'.
- **created** (integer) - Unix timestamp of when the completion was created.
- **model** (string) - The specific model used for the completion.
- **choices** (array) - An array of completion choices.
  - **index** (integer) - Index of the choice.
  - **message** (object) - The message content from the assistant.
    - **role** (string) - Role of the message sender ('assistant').
    - **content** (string) - The generated text.
  - **finish_reason** (string) - The reason the model stopped generating tokens (e.g., 'stop', 'length').
- **usage** (object) - Token usage statistics.
  - **prompt_tokens** (integer) - Number of tokens in the prompt.
  - **completion_tokens** (integer) - Number of tokens in the completion.
  - **total_tokens** (integer) - Total tokens used.
- **meta** (object) - Metadata about the provider and routing.
  - **provider** (string) - The LLM provider used.
  - **model** (string) - The specific model identifier from the provider.
  - **latency_ms** (number) - Latency in milliseconds.
  - **attempt** (integer) - The attempt number for this request.

#### Response Example
```json
{
  "id": "chatcmpl-a1b2c3d4e5f6",
  "object": "chat.completion",
  "created": 1704067200,
  "model": "gemini-2.5-flash",
  "choices": [
    {
      "index": 0,
      "message": {"role": "assistant", "content": "The capital of France is Paris."},
      "finish_reason": "stop"
    }
  ],
  "usage": {"prompt_tokens": 25, "completion_tokens": 10, "total_tokens": 35},
  "meta": {"provider": "Gemini", "model": "gemini-2.5-flash", "latency_ms": 1234.56, "attempt": 1}
}
```
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.