### Configuration File Example

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Example structure of a YAML configuration file for RouteLLM routers.

```yaml
# Example configuration for RouteLLM routers
# This file specifies the configurations for routers trained on Arena data augmented using GPT-4 as a judge.
# The models and datasets used are all hosted on Hugging Face under the RouteLLM and LMSYS organizations.

# Example router configurations (replace with actual configurations)
router_mf:
  model_name: "gpt-4-1106-preview"
  provider: "openai"
  # other initialization arguments...

router_bert:
  model_name: "bert-base-uncased"
  # other initialization arguments...

# Add configurations for other routers as needed
```

--------------------------------

### Install RouteLLM from PyPI

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Installs the RouteLLM package with serving and evaluation capabilities using pip.

```bash
pip install "routellm[serve,eval]"
```

--------------------------------

### Install RouteLLM from Source

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Clones the RouteLLM repository and installs it from source, including serving and evaluation features.

```bash
git clone https://github.com/lm-sys/RouteLLM.git
cd RouteLLM
pip install -e .[serve,eval]
```

--------------------------------

### Martian Integration Example

Source: https://github.com/lm-sys/routellm/blob/main/benchmarks/README.md

Shows how to set up an OpenAI client for Martian, including the API endpoint and key. The example highlights the use of the 'router' model and the 'extra_body' parameter to specify supported models and cost controls for routing.

```python
client = openai.OpenAI(
    base_url="https://withmartian.com/api/openai/v1",
    api_key="MARTIAN_API_KEY",
)
response = client.chat.completions.create(
        model="router",
        extra_body={
            "models": ["gpt-4-turbo-128k", "llama-2-70b-chat"],
            "max_cost_per_million_tokens": 10.45,
        },
		...
    )
```

--------------------------------

### Starting the RouteLLM OpenAI-Compatible Server

Source: https://github.com/lm-sys/routellm/blob/main/README.md

This command starts the RouteLLM server, which is compatible with OpenAI's API. It allows you to specify which routers to make available and the configuration file to use. The server routes requests based on different strategies defined in the configuration.

```python
python -m routellm.openai_server --routers mf --config config.example.yaml
```

--------------------------------

### Run RouteLLM Chat Demo

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Starts a local router chatbot demo to interact with RouteLLM and observe message routing based on a specified router and threshold.

```python
python -m examples.router_chat --router mf --threshold 0.11593
```

--------------------------------

### Unify AI Integration Example

Source: https://github.com/lm-sys/routellm/blob/main/benchmarks/README.md

Demonstrates how to initialize an OpenAI client for Unify AI and make a chat completion request. It specifies the base URL, API key, and a model string that includes routing parameters and supported models.

```python
client = openai.OpenAI(
	base_url="https://api.unify.ai/v0/",
	api_key="UNIFY_API_KEY"
)
response = client.chat.completions.create(
        model="router@q:1|c:1.71e-03|t:1.10e-05|i:1.09e-03|models:gpt-4-turbo,mixtral-8x7b-instruct-v0.1",
		...
    )
```

--------------------------------

### Launch RouteLLM OpenAI-Compatible Server

Source: https://github.com/lm-sys/routellm/blob/main/examples/routing_to_local_models.md

Starts an OpenAI-compatible server for RouteLLM, configured with the 'mf' router and a local Ollama model as the weak model. The server listens on http://0.0.0.0:6060.

```bash
> export OPENAI_API_KEY=sk-...
> python -m routellm.openai_server --routers mf --weak-model ollama_chat/llama3 --config.example.yaml
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:6060 (Press CTRL+C to quit)
```

--------------------------------

### Chat Completions with Controller

Source: https://github.com/lm-sys/routellm/blob/main/examples/python_sdk.md

Shows how to use the Controller's chat completions interface, which mirrors OpenAI's client. It includes an example of creating a completion with a specific router and cost threshold.

```python
response = client.chat.completions.create(
  # This tells RouteLLM to use the MF router with a cost threshold of 0.11593
  model="router-mf-0.11593",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)
print(response.choices[0]["message"]["content"])
```

--------------------------------

### Run Llama 3 8B Locally with Ollama

Source: https://github.com/lm-sys/routellm/blob/main/examples/routing_to_local_models.md

Starts an Ollama server running the Llama 3 model locally. This server will be accessible at http://localhost:11434/v1.

```bash
ollama run llama3
```

--------------------------------

### Calibrate Routing Threshold

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Calibrates the cost threshold for a router to control the trade-off between cost and quality. This example calibrates for 50% strong model calls.

```bash
> python -m routellm.calibrate_threshold --routers mf --strong-model-pct 0.5 --config config.example.yaml
For 50.0% strong model calls for mf, threshold = 0.11593
```

--------------------------------

### Launch OpenAI-Compatible Server

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Launches an OpenAI-compatible server for RouteLLM, allowing integration with existing OpenAI clients. Requires setting API keys.

```bash
> export OPENAI_API_KEY=sk-XXXXXX
> export ANYSCALE_API_KEY=esecret_XXXXXX
> python -m routellm.openai_server --routers mf --strong-model gpt-4-1106-preview --weak-model anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:6060 (Press CTRL+C to quit)
```

--------------------------------

### Initialize RouteLLM Controller

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Initializes the RouteLLM controller, replacing an OpenAI client to route queries between strong and weak models. Requires setting API keys for the models.

```python
import os
from routellm.controller import Controller

os.environ["OPENAI_API_KEY"] = "sk-XXXXXX"
# Replace with your model provider, we use Anyscale's Mixtral here.
os.environ["ANYSCALE_API_KEY"] = "esecret_XXXXXX"

client = Controller(
  routers=["mf"],
  strong_model="gpt-4-1106-preview",
  weak_model="anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1",
)
```

--------------------------------

### RouteLLM Supported Routers

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Lists the available routers in RouteLLM, including their implementation details and recommended usage.

```text
1. `mf`: Uses a matrix factorization model trained on the preference data (recommended).
2. `sw_ranking`: Uses a weighted Elo calculation for routing, where each vote is weighted according to how similar it is to the user's prompt.
3. `bert`: Uses a BERT classifier trained on the preference data.
4. `causal_llm`: Uses a LLM-based classifier tuned on the preference data.
5. `random`: Randomly routes to either model.
```

--------------------------------

### Initialize RouteLLM Controller

Source: https://github.com/lm-sys/routellm/blob/main/examples/python_sdk.md

Demonstrates how to initialize the Controller class with specified routers, models, and configuration. It also shows how to set environment variables for API keys and override API base and key.

```python
import os
from routellm.controller import Controller

os.environ["OPENAI_API_KEY"] = "sk-XXXXXX"
os.environ["ANYSCALE_API_KEY"] = "esecret_XXXXXX"

client = Controller(
  # List of routers to initialize
  routers=["mf"],
  # The pair of strong and weak models to route to
  strong_model="gpt-4-1106-preview",
  weak_model="anyscale/mistralai/Mixtral-8x7B-Instruct-v0.1",
  # The config for the router (best-performing config by default)
  config = {
    "mf": {
      "checkpoint_path": "routellm/mf_gpt4_augmented"
    }
  },
  # Override API base and key for LLM calls
  api_base=None,
  api_key=None,
  # Display a progress bar for operations
  progress_bar=False,
)
```

--------------------------------

### Add a New Benchmark

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Guidelines for incorporating a new benchmark into RouteLLM by implementing the abstract Benchmark class and updating the evaluate.py module.

```python
To add a new benchmark to RouteLLM, implement the abstract `Benchmark` class in `benchmarks.py` and update the `evaluate.py` module to properly initialize the new benchmark class. Ideally, the results for the benchmark should be precomputed to avoid having to regenerate the results for each evaluation run -- see the existing benchmarks for examples on how to do this.
```

--------------------------------

### OpenAI Compatible API Endpoints

Source: https://github.com/lm-sys/routellm/blob/main/benchmarks/README.md

Documentation for interacting with services that provide an OpenAI-compatible API, such as Unify AI and Martian. This includes initialization parameters and request structures.

```APIDOC
UnifyAI:
  Base URL: https://api.unify.ai/v0/
  Authentication: API Key (UNIFY_API_KEY)
  Chat Completions:
    Method: POST
    Endpoint: /chat/completions
    Parameters:
      model: string - Specifies the router and supported models (e.g., "router@q:1|c:1.71e-03|t:1.10e-05|i:1.09e-03|models:gpt-4-turbo,mixtral-8x7b-instruct-v0.1")
      ...

Martian:
  Base URL: https://withmartian.com/api/openai/v1
  Authentication: API Key (MARTIAN_API_KEY)
  Chat Completions:
    Method: POST
    Endpoint: /chat/completions
    Parameters:
      model: string - Typically "router"
      extra_body: object - Contains routing configurations:
        models: array[string] - List of supported models (e.g., ["gpt-4-turbo-128k", "llama-2-70b-chat"])
        max_cost_per_million_tokens: number - Cost control parameter for routing decisions
      ...

```

--------------------------------

### Initialize RouteLLM Controller with Local Model

Source: https://github.com/lm-sys/routellm/blob/main/examples/routing_to_local_models.md

Creates a RouteLLM Controller instance, specifying 'mf' as the router and a local Ollama model ('ollama_chat/llama3') as the weak model. Requires setting the OPENAI_API_KEY environment variable.

```python
import os

os.environ["OPENAI_API_KEY"] = "sk-XXXXXX"

client = Controller(
  routers=["mf"],
  strong_model="gpt-4-1106-preview",
  weak_model="ollama_chat/llama3",
)
```

--------------------------------

### Add a New Router

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Instructions for adding a new router to RouteLLM by implementing the abstract Router class and updating the ROUTER_CLS dictionary.

```python
To add a new router to RouteLLM, implement the abstract `Router` class in `routers.py` and add the new router to the `ROUTER_CLS` dictionary. Then, you can use immediately the new router in the server or evaluation framework.

There is only a single method to implement: `calculate_strong_win_rate`, which takes in the user prompt and returns the win rate for the strong model conditioned on that given prompt - if this win rate is great than user-specified cost threshold, then the request is routed to the strong model. Otherwise, it is routed to the weak model.
```

--------------------------------

### Evaluate Routers on a Benchmark

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Command to evaluate specified routers on a given benchmark using a configuration file. Supports multiple benchmarks and allows cache overwriting.

```python
python -m routellm.evals.evaluate --routers random sw_ranking bert --benchmark gsm8k --config config.example.yaml
```

--------------------------------

### Route Prompt with Controller

Source: https://github.com/lm-sys/routellm/blob/main/examples/python_sdk.md

Demonstrates the `route` method of the Controller, which determines the best model for a given prompt based on a specified router and threshold.

```python
routed_model = client.route(
	prompt="What's the squareroot of 144?",
	router="mf",
	threshold=0.11593,
)
print(f"Prompt should be routed to {routed_model}")
```

--------------------------------

### Generate Completion with Router

Source: https://github.com/lm-sys/routellm/blob/main/README.md

Generates a completion using the RouteLLM client, specifying the router and calibrated threshold in the model name.

```python
response = client.chat.completions.create(
  # This tells RouteLLM to use the MF router with a cost threshold of 0.11593
  model="router-mf-0.11593",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)
```

--------------------------------

### Configure OpenAI Client for RouteLLM Server

Source: https://github.com/lm-sys/routellm/blob/main/examples/routing_to_local_models.md

Configures the OpenAI client to communicate with the RouteLLM server. Sets the base URL to the server's address and uses a placeholder API key.

```python
import openai

client = openai.OpenAI(
  base_url="https://localhost:6060/v1",
  api_key="no_api_key"
)
...
response = client.chat.completions.create(
  model="router-mf-0.11593",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)
```

--------------------------------

### Calibrating RouteLLM Thresholds

Source: https://github.com/lm-sys/routellm/blob/main/README.md

This command calibrates the routing thresholds for specified routers. It helps determine the optimal threshold value to achieve a desired percentage of calls routed to the stronger model, balancing cost and quality. The calibration process uses a dataset to estimate the threshold.

```python
> python -m routellm.calibrate_threshold --task calibrate --routers mf --strong-model-pct 0.5 --config config.example.yaml
For 50.0% strong model calls for mf, threshold = 0.11593
```

--------------------------------

### RouteLLM Research Paper Citation

Source: https://github.com/lm-sys/routellm/blob/main/README.md

This BibTeX entry provides the full citation details for the RouteLLM research paper, which is the basis for the code in this repository. It includes authors, title, year, arXiv ID, and URL.

```bibtex
@misc{ong2024routellmlearningroutellms,
      title={RouteLLM: Learning to Route LLMs with Preference Data},
      author={Isaac Ong and Amjad Almahairi and Vincent Wu and Wei-Lin Chiang and Tianhao Wu and Joseph E. Gonzalez and M Waleed Kadous and Ion Stoica},
      year={2024},
      eprint={2406.18665},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2406.18665},
}
```

--------------------------------

### RouteLLM Server Request Model Format

Source: https://github.com/lm-sys/routellm/blob/main/README.md

When making requests to the RouteLLM server, clients specify the router and cost threshold using a specific format in the 'model' field. This format allows the server to identify which routing strategy and cost-quality tradeoff to apply for each request.

```APIDOC
model: router-[ROUTER NAME]-[THRESHOLD]

Example:
router-mf-0.5
```

--------------------------------

### Generate Completions with RouteLLM Controller

Source: https://github.com/lm-sys/routellm/blob/main/examples/routing_to_local_models.md

Generates a completion using the RouteLLM controller. The model specified ('router-mf-0.11593') is determined by the calibrated threshold for the 'mf' router, directing traffic between strong and weak models.

```python
response = client.chat.completions.create(
  model="router-mf-0.11593",
  messages=[
    {"role": "user", "content": "Hello!"}
  ]
)
```

--------------------------------

### Batch Calculate Win Rate

Source: https://github.com/lm-sys/routellm/blob/main/examples/python_sdk.md

Illustrates the `batch_calculate_win_rate` method for offline evaluations. It takes a pandas Series of prompts and returns the win rate for the strong model, calculated by the specified router.

```python
import pandas as pd

prompts = pd.Series(["What's the squareroot of 144?", "Who's the last president of the US?", "Is the sun a star?"])
win_rates = client.batch_calculate_win_rate(prompts=prompts, router="mf")

print(f"Calculated win rate for prompts:\n{win_rates.describe()}")
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.