Not Diamond

https://docs.notdiamond.ai/docs
Admin
Not Diamond is an intelligent multi-model AI infrastructure platform that adapts prompts across...
Tokens:14,636
Snippets:159
Trust Score:8
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Not Diamond Python SDK

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Install the Not Diamond Python SDK and pandas for prompt optimization tasks.

```shell
pip install notdiamond pandas
```

--------------------------------

### Install Not Diamond Python SDK

Source: https://docs.notdiamond.ai/docs/classification

Installs the Not Diamond Python SDK and the datasets library. Ensure you have Python and pip installed.

```shell
pip install notdiamond datasets==3.6.0
```

--------------------------------

### Format Dataset for Prompt Optimization

Source: https://docs.notdiamond.ai/docs/classification

Prepare your dataset for prompt optimization by ensuring each sample includes prompt template field arguments and the ground truth class name in the 'answer' field. This example demonstrates formatting for classification tasks.

```python
import json

categories = [
    "activate_my_card",
    "age_limit",
    "apple_pay_or_google_pay",
    "atm_support",
    "automatic_top_up",
    "balance_not_updated_after_bank_transfer",
    "balance_not_updated_after_cheque_or_cash_deposit",
    "beneficiary_not_allowed",
    "cancel_transfer",
    "card_about_to_expire",
    "card_acceptance",
    "card_arrival",
    "card_delivery_estimate",
    "card_linking",
    "card_not_working",
    "card_payment_fee_charged",
    "card_payment_not_recognised",
    "card_payment_wrong_exchange_rate",
    "card_swallowed",
    "cash_withdrawal_charge",
    "cash_withdrawal_not_recognised",
    "change_pin",
    "compromised_card",
    "contactless_not_working",
    "country_support",
    "declined_card_payment",
    "declined_cash_withdrawal",
    "declined_transfer",
    "direct_debit_payment_not_recognised",
    "disposable_card_limits",
    "edit_personal_details",
    "exchange_charge",
    "exchange_rate",
    "exchange_via_app",
    "extra_charge_on_statement",
    "failed_transfer",
    "fiat_currency_support",
    "get_disposable_virtual_card",
    "get_physical_card",
    "getting_spare_card",
    "getting_virtual_card",
    "lost_or_stolen_card",
    "lost_or_stolen_phone",
    "order_physical_card",
    "passcode_forgotten",
    "pending_card_payment",
    "pending_cash_withdrawal",
    "pending_top_up",
    "pending_transfer",
    "pin_blocked",
    "receiving_money",
    "refund_not_showing_up",
    "request_refund",
    "reverted_card_payment",
    "supported_cards_and_currencies",
    "terminate_account",
    "top_up_by_bank_transfer_charge",
    "top_up_by_card_charge",
    "top_up_by_cash_or_cheque",
    "top_up_failed",
    "top_up_limits",
    "top_up_reverted",
    "topping_up_by_card",
    "transaction_charged_twice",
    "transfer_fee_charged",
    "transfer_into_account",
    "transfer_not_received_by_recipient",
    "transfer_timing",
    "unable_to_verify_identity",
    "verify_my_identity",
    "verify_source_of_funds",
    "verify_top_up",
    "virtual_card_not_working",
    "visa_or_mastercard",
    "why_verify_identity",
    "wrong_amount_of_cash_received",
    "wrong_exchange_rate_for_cash_withdrawal"
]

fields = ["question", "categories"]

pa_ds = [
    {
        "fields": {
            "question": sample["text"],
            "categories": json.dumps(categories)
        },
        "answer": json.dumps({
            "intent": categories[sample["label"]], 
        })
    }
    for sample in ds
]

print(prompt_template.format(**pa_ds[0]['fields']))
```

--------------------------------

### Request Prompt Optimization with TypeScript SDK

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Initiate a prompt optimization job using the TypeScript SDK. This example demonstrates defining target models and utilizing prototype mode for optimization with a minimal dataset.

```typescript
import NotDiamond from 'notdiamond';

const client = new NotDiamond({api_key: process.env.NOTDIAMOND_API_KEY});

// Define target models to optimize for
const targetModels = [
  { provider: 'openai', model: 'gpt-5-mini-2025-08-07' },
  { provider: 'google', model: 'gemini-2.5-flash' }
];

// Request optimization
const response = await client.promptOptimization.optimize({
  system_prompt: systemPrompt,
  template: promptTemplate,
  fields: fields,
  train_goldens: trainGoldens,
  test_goldens: testGoldens,
  target_models: targetModels,
  evaluation_metric: 'LLMaaJ:Sem_Sim_1',
  prototype_mode: true
});

const optimizationRunId = response.optimization_run_id;
console.log(`Optimization job started: ${optimizationRunId}`);
```

--------------------------------

### Send First Not Diamond API Request (Python)

Source: https://docs.notdiamond.ai/docs/quickstart-routing

Send your first request to Not Diamond to select the best LLM for your task. This example uses Python and demonstrates routing based on cost.

```python
import os

from notdiamond import NotDiamond

# Define the Not Diamond routing client
client = NotDiamond(api_key=os.environ.get("NOTDIAMOND_API_KEY"))

# The best LLM is determined by Not Diamond based on the messages and specified models
result = client.model_router.select_model(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Concisely explain merge sort."}  # Adjust as desired
    ],
    llm_providers=[
        {"provider": "openai", "model": "gpt-5-2025-08-07"},
        {"provider": "openai", "model": "gpt-5-nano-2025-08-07"},
        {"provider": "anthropic", "model": "claude-sonnet-4-20250514"}
    ],
    tradeoff="cost"  # Optional: "cost", "latency", or None (default, optimizes for quality)
)

print("Not Diamond session ID: ", result.session_id)  # A unique ID of Not Diamond's recommendation
print("LLM called: ", result.provider.model)  # The LLM routed to

# Now call the selected model using your preferred SDK (OpenAI, Anthropic, etc.)

```

--------------------------------

### Install Not Diamond with npm

Source: https://docs.notdiamond.ai/docs/quickstart-routing

Install the Not Diamond Node.js package using npm.

```typescript
npm install notdiamond
```

--------------------------------

### Install Not Diamond with pip

Source: https://docs.notdiamond.ai/docs/quickstart-routing

Install the Not Diamond Python package using pip.

```shell
pip install notdiamond
```

--------------------------------

### Golden Example Structure

Source: https://docs.notdiamond.ai/docs/requirements-and-concepts

Defines the structure of a single golden example, including input fields and the expected answer. This format is crucial for training and evaluating prompt performance.

```python
{
    "fields": {
        "question": "What is the capital of France?",
        "context": "France is a country in Western Europe..."
    },
    "answer": "Paris"
}
```

--------------------------------

### Send First Not Diamond API Request (TypeScript)

Source: https://docs.notdiamond.ai/docs/quickstart-routing

Send your first request to Not Diamond to select the best LLM for your task. This example uses TypeScript and demonstrates routing based on cost.

```typescript
import NotDiamond from 'notdiamond';

// Initialize the Not Diamond client
const client = new NotDiamond({api_key: process.env.NOTDIAMOND_API_KEY});

// The best LLM is determined by Not Diamond based on the messages and specified models
const result = await client.modelRouter.selectModel({
  messages: [
    { role: 'system', content: 'You are a world class programmer.' },
    { role: 'user', content: 'Concisely explain merge sort.' }  // Adjust as desired
  ],
  llmProviders: [
    { provider: 'openai', model: 'gpt-5-2025-08-07' },
    { provider: 'openai', model: 'gpt-5-nano-2025-08-07' },
    { provider: 'anthropic', model: 'claude-sonnet-4-20250514' }
  ],
  tradeoff: 'cost'  // Optional: 'cost', 'latency', or omit (default, optimizes for quality)
});

console.log('Not Diamond session ID:', result.sessionId);  // A unique ID of Not Diamond's recommendation
console.log('LLM called:', result.provider.model);  // The LLM routed to

// Now call the selected model using your preferred SDK (OpenAI, Anthropic, etc.)

```

--------------------------------

### Define System Prompt and Prompt Template (TypeScript)

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Defines the system prompt for answering questions based on context and a user message template in TypeScript. The system prompt guides the AI's response generation, and the template structures the input.

```TypeScript
const systemPrompt = `I'd like for you to answer questions about a context text that will be provided. I'll give you a pair
with the form:
Context: "context text"
Question: "a question about the context"
Generate an explicit answer to the question that will be output. Make sure that the answer is the only output you provide,
and the analysis of the context should be kept to yourself. Answer directly and do not prefix the answer with anything such as
"Answer:" nor "The answer is:". The answer has to be the only output you explicitly provide.

The answer has to be as short, direct, and concise as possible. If the answer to the question can not be obtained from the provided
context paragraph, output "UNANSWERABLE". Here's the context and question for you to reason about and answer.
`;

const promptTemplate = `
Context: {context}

Question: {question}
`;

const fields = ['question', 'context'];
```

--------------------------------

### Predefined Evaluation Metric

Source: https://docs.notdiamond.ai/docs/requirements-and-concepts

Example of using a predefined evaluation metric for prompt quality assessment. 'LLMaaJ:Sem_Sim_1' is the default for semantic similarity.

```python
"evaluation_metric": "LLMaaJ:Sem_Sim_1"  # Default semantic similarity
```

--------------------------------

### RAGAS_FAITHFULNESS Example Golden Format

Source: https://docs.notdiamond.ai/docs/evaluation-metrics

Example of how to format data for the RAGAS_FAITHFULNESS metric. This metric ensures the response is faithful to the provided context. The 'answer' field is not used by this metric.

```Python
fields = ["question", "context"]

pa_ds = [
    {
        "fields": {
            "question": "User input",
            "context": "Context retrieved based on user input"
        },
        "answer": "Reference answer. This is not used in RAGAS_FAITHFULNESS"
    }
    for sample in ds
]
```

--------------------------------

### Get Session ID for Model Selection

Source: https://docs.notdiamond.ai/docs/key-concepts

Retrieve the unique session ID after selecting a model for a query. This ID is useful for tracking and debugging specific routing decisions.

```python
result = client.model_router.select_model(
    messages=[{"role": "user", "content": "Hello"}],
    llm_providers=[{"provider": "openai", "model": "gpt-5-2025-08-07"}]
)

print(result.session_id)  # e.g., "550e8400-e29b-41d4-a716-446655440000"
```

--------------------------------

### Download HotpotQA Dataset

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Use this command to download the HotpotQA dataset for prompt optimization.

```shell
wget "https://drive.google.com/uc?export=download&id=1TeXM_Z-F3-o6ouooigaEWT3axUJA65kv" -O hotpotqa.jsonl
```

--------------------------------

### Initialize NotDiamond Client (Python)

Source: https://docs.notdiamond.ai/docs/quickstart-routing

Initialize the NotDiamond client with your API key. Ensure the API key is set as an environment variable.

```python
client = NotDiamond(api_key=os.environ.get("NOTDIAMOND_API_KEY"))
```

--------------------------------

### Request Prompt Optimization with Python SDK

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Use this snippet to request prompt optimization by defining target models and providing training/testing data. Prototype mode can be enabled for projects with fewer than 25 training samples.

```python
import os

from notdiamond import NotDiamond

client = NotDiamond(api_key=os.environ.get("NOTDIAMOND_API_KEY"))

# Define target models to optimize for
target_models = [
    {"provider": "openai", "model": "gpt-5-mini-2025-08-07"},
    {"provider": "google", "model": "gemini-2.5-flash"},
]

# Request optimization
response = client.prompt_optimization.optimize(
    system_prompt=system_prompt,
    template=prompt_template,
    fields=fields,
    train_goldens=train_goldens,
    test_goldens=test_goldens,
    target_models=target_models,
    evaluation_metric="LLMaaJ:Sem_Sim_1",
    prototype_mode=True
)

optimization_run_id = response.optimization_run_id
print(f"Optimization job started: {optimization_run_id}")
```

--------------------------------

### Prompt Template and Fields Definition

Source: https://docs.notdiamond.ai/docs/requirements-and-concepts

Illustrates how to define a prompt template with placeholders and a corresponding list of fields. Ensures consistency between the template and the golden data.

```python
# Your template has two fields: {context} and {question}
prompt_template = """
Context: {context}
Question: {question}
"""

# Your fields list must match
fields = ["context", "question"]

# Each golden example must provide values for these fields
golden_data = [
    {
        "fields": {
            "context": "...",
            "question": "..."
        },
        "answer": "..."
    }
]
```

--------------------------------

### Import Dependencies for Python Training

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Import necessary libraries for router training in Python.

```python
import pandas as pd
from notdiamond import NotDiamond
```

--------------------------------

### Submit Prompt Optimization Request (TypeScript SDK)

Source: https://docs.notdiamond.ai/docs/classification

Utilize the Not Diamond TypeScript SDK to initiate a prompt optimization job. Make sure your API key is configured in the environment. The response includes an optimization run ID.

```typescript
import NotDiamond from 'notdiamond';

const client = new NotDiamond({api_key: process.env.NOTDIAMOND_API_KEY});

// Note: Prepare your dataset similar to the Python example above
// Define systemPrompt, promptTemplate, fields, goldenData, originModel, and targetModels

const response = await client.promptOptimization.optimize({
  systemPrompt: systemPrompt,
  template: promptTemplate,
  fields: fields,
  goldens: goldenData,  // Your prepared dataset
  originModel: originModel,
  targetModels: targetModels,
  evaluationMetric: 'LLMaaJ:Sem_Sim_1'
});

const optimizationRunId = response.optimizationRunId;
console.log(`Optimization job started: ${optimizationRunId}`);
```

--------------------------------

### Train Custom Router with Custom Model (Python)

Source: https://docs.notdiamond.ai/docs/routing-between-custom-models

Train a custom router by including a custom model configuration in the `llm_providers` list. Ensure the custom model has unique provider/model names and defined quality/cost/latency parameters.

```python
import os
import json
from notdiamond import NotDiamond

# Initialize ND client
client = NotDiamond(api_key=os.environ.get("NOTDIAMOND_API_KEY"))

# Define your custom model
custom_model = {
    "provider": "custom",
    "model": "my-finetuned-model",
    "is_custom": True,
    "context_length": 200000,
    "input_price": 0.1,   # USD per million input tokens
    "output_price": 0.2,  # USD per million output tokens
    "latency": 0.01       # time to first token (seconds)
}

# LLMs to train router on (mix of standard and custom models)
llm_providers = [
    {"provider": "openai", "model": "gpt-4o"},
    {"provider": "openai", "model": "gpt-4o-mini"},
    {"provider": "google", "model": "gemini-2.5-pro"},
    {"provider": "anthropic", "model": "claude-opus-4-20250514"},
    custom_model,  # Add your custom model to the list
]

# Train the custom router (assuming the evaluation_data.csv exists)
with open("evaluation_data.csv", "rb") as f:
    response = client.custom_router.train_custom_router(
        dataset_file=f,
        language="english",
        llm_providers=json.dumps(llm_providers),
        maximize=True,
        prompt_column="prompt",
    )

preference_id = response.preference_id
print("Custom router preference ID:", preference_id)
```

--------------------------------

### Select Model using Model Router (Python)

Source: https://docs.notdiamond.ai/docs/quickstart-routing

Use the model router to select the best LLM provider for a query. You can specify messages, LLM providers to choose from, and a tradeoff preference (cost, latency, or quality).

```python
result = client.model_router.select_model(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Concisely explain merge sort."}  # Adjust as desired
    ],
    llm_providers=[
        {"provider": "openai", "model": "gpt-5-2025-08-07"},
        {"provider": "openai", "model": "gpt-5-nano-2025-08-07"},
        {"provider": "anthropic", "model": "claude-sonnet-4-20250514"}
    ],
    tradeoff="cost"  # Optional: "cost", "latency", or None (default, optimizes for quality)
)
```

--------------------------------

### Specify Origin and Target Models

Source: https://docs.notdiamond.ai/docs/classification

Define the origin model for querying with the system prompt and template, and list the target models for optimized prompts. Ensure the model identifiers are correctly formatted.

```python
origin_model = {"provider": "openai", "model": "gpt-4o-2024-08-06"}
target_models = [
    {"provider": "anthropic", "model": "claude-sonnet-4-20250514"},
]
```

--------------------------------

### Define System Prompt and Prompt Template (Python)

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Defines the system prompt for answering questions based on context and a user message template. The system prompt specifies the desired output format and behavior, including handling unanswerable questions. The prompt template uses placeholders for context and question.

```Python
system_prompt = """I'd like for you to answer questions about a context text that will be provided. I'll give you a pair
with the form:
Context: "context text"
Question: "a question about the context"
Generate an explicit answer to the question that will be output. Make sure that the answer is the only output you provide,
and the analysis of the context should be kept to yourself. Answer directly and do not prefix the answer with anything such as
"Answer:" nor "The answer is:". The answer has to be the only output you explicitly provide.

The answer has to be as short, direct, and concise as possible. If the answer to the question can not be obtained from the provided
context paragraph, output "UNANSWERABLE". Here's the context and question for you to reason about and answer.
"

prompt_template = """
Context: {context}

Question: {question}
"""

fields = ["question", "context"]
```

--------------------------------

### Initialize NotDiamond Client (TypeScript)

Source: https://docs.notdiamond.ai/docs/quickstart-routing

Initialize the NotDiamond client with your API key. Ensure the API key is set as an environment variable.

```typescript
const client = new NotDiamond({api_key: process.env.NOTDIAMOND_API_KEY});
```

--------------------------------

### Load and Format JSON Dataset (Python)

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Loads a JSON dataset from a specified path, formats it into samples with 'fields' and 'answer', and splits it into training and testing sets. This function is useful for preparing data for machine learning models.

```Python
import pandas as pd

def load_json_dataset(dataset_path: str, n_samples: int = 200) -> list[dict]:
    df = pd.read_json(dataset_path, lines=True)
    n_samples = min(n_samples, len(df))
    if n_samples:
        df = df.iloc[[v for v in range(n_samples)]]

    golden_dataset = []
    for idx, row in df.iterrows():
        sample_fields = {"question": row["question"]}
        sample_fields["context"] = "\n\n".join(row["documents"])
        answer = row["response"]

        data_sample = dict(
            fields=sample_fields,
            answer=answer,
        )
        golden_dataset.append(data_sample)
    return golden_dataset
  
# Define the train and test set sizes
all_goldens = load_json_dataset("hotpotqa.jsonl", 20)
train_goldens = all_goldens[:10]
test_goldens = all_goldens[10:]

print(f"Loaded {len(train_goldens)} train samples and {len(test_goldens)} test samples")
print(f"\nExample prompt:\n{prompt_template.format(**train_goldens[0]['fields'])[:500]}...")
```

--------------------------------

### Import Dependencies for TypeScript Training

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Import necessary libraries for router training in TypeScript.

```typescript
import NotDiamond from 'notdiamond';
import * as fs from 'fs';
```

--------------------------------

### Optimize Model Selection for Latency

Source: https://docs.notdiamond.ai/docs/key-concepts

Select the best LLM by prioritizing faster response times. This tradeoff favors models that can deliver results more quickly.

```python
# Optimize for latency
result = client.model_router.select_model(
    messages=[{"role": "user", "content": "Hello"}],
    llm_providers=[
        {"provider": "openai", "model": "gpt-5-2025-08-07"},
        {"provider": "openai", "model": "gpt-5-mini-2025-08-07"}
    ],
    tradeoff="latency"  # Options: "cost", "latency", or None (quality)
```

--------------------------------

### Retrieve Prompt Optimization Results (Python SDK)

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Use the Python SDK to check the status of an optimization run and retrieve the final results, including optimized prompts and scores for each target model.

```python
# Check status
status = client.prompt_optimization.get_optimization_results(optimization_run_id)
print(f"Status: {status}")

# Get results when complete
results = client.prompt_optimization.get_optimization_results(optimization_run_id)

# View optimized prompts for each target model
for target in results.target_models:
    print(f"\nModel: {target.model_name}")
    print(f"Pre-optimization score: {target.pre_optimization_score}")
    print(f"Post-optimization score: {target.post_optimization_score}")
    print(f"\nOptimized system prompt:\n{target.system_prompt}")
    print(f"\nOptimized template:\n{target.user_message_template}")
```

--------------------------------

### Train Custom Router with Python SDK

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Trains a custom router using the NotDiamond Python SDK. Requires a CSV dataset and specifies LLM providers. The dataset must contain at least 15 samples and can be up to 5MB or 10,000 samples.

```Python
import os
import json
import pandas as pd
from notdiamond import NotDiamond

# Load dataset
df = pd.read_csv("humaneval.csv")
print("Original columns:", df.columns.tolist())

# Models included in the dataset
models_in_csv = [
    "anthropic/claude-sonnet-4-5-20250929",
    "openai/gpt-5-2025-08-07",
    "google/gemini-2.5-pro",
    "openai/gpt-5-mini-2025-08-07",
    "anthropic/claude-opus-4-20250514",
]

# Save normalized CSV
df.to_csv("humaneval.csv", index=False)

# Initialize ND client
client = NotDiamond(api_key=os.environ.get("NOTDIAMOND_API_KEY"))

# LLMs to train router on
llm_providers = [
    {"provider": "anthropic", "model": "claude-sonnet-4-5-20250929"},
    {"provider": "openai", "model": "gpt-5-2025-08-07"},
    {"provider": "google", "model": "gemini-2.5-pro"},
    {"provider": "openai", "model": "gpt-5-mini-2025-08-07"},
    {"provider": "anthropic", "model": "claude-opus-4-20250514"},
]

# Train the custom router
with open("humaneval.csv", "rb") as f:
    response = client.custom_router.train_custom_router(
        dataset_file=f,
        language="english",
        llm_providers=json.dumps(llm_providers),
        maximize=True,
        prompt_column="Input",
    )

preference_id = response.preference_id
print("Custom router preference ID:", preference_id)
```

--------------------------------

### Origin and Target Model Configuration

Source: https://docs.notdiamond.ai/docs/requirements-and-concepts

Specifies the origin model for baseline comparison and a list of target models for migration. Supports up to four target models per request.

```python
origin_model = {"provider": "google", "model": "gemini-2.5-pro"}

target_models = [
    {"provider": "anthropic", "model": "claude-sonnet-4-5-20250929"},
    {"provider": "openai", "model": "gpt-5-2025-08-07"}
]
```

--------------------------------

### Blend Cost and Quality for Model Selection

Source: https://docs.notdiamond.ai/docs/key-concepts

Achieve a balance between cost and quality when selecting an LLM using a continuous scale. A value of 0 prioritizes quality, while 10 strictly selects the cheapest model.

```python
# Blend cost and quality (5 = balanced)
result = client.model_router.select_model(
    messages=[{"role": "user", "content": "Hello"}],
    llm_providers=[
        {"provider": "openai", "model": "gpt-5-2025-08-07"},
        {"provider": "openai", "model": "gpt-5-mini-2025-08-07"}
    ],
    cost_quality_tradeoff=5  # 0 = quality-first, 10 = cheapest model
)
```

--------------------------------

### Python SDK - Using EXACT_MATCH

Source: https://docs.notdiamond.ai/docs/evaluation-metrics

Use the EXACT_MATCH metric for classification tasks in the Python SDK. Ensure the API key is set.

```python
import os

from notdiamond import NotDiamond

client = NotDiamond(api_key=os.environ.get("NOTDIAMOND_API_KEY"))

response = client.prompt_optimization.optimize(
    system_prompt=system_prompt,
    template=prompt_template,
    fields=fields,
    goldens=golden_data,
    origin_model=origin_model,
    target_models=target_models,
    evaluation_metric="EXACT_MATCH"  # Use exact match for classification tasks
)
```

--------------------------------

### Retrieve Optimization Results (Python SDK)

Source: https://docs.notdiamond.ai/docs/classification

Use the Python SDK to check the status of an optimization job and retrieve the optimized prompts and evaluation metrics once the job is completed.

```python
# Check job status
status = client.prompt_optimization.get_optimization_results(optimization_run_id)
print(f"Status: {status}")

# When status is 'completed', fetch results
results = client.prompt_optimization.get_optimization_results(optimization_run_id)

# View optimized prompts
for target in results.target_models:
    print(f"\nModel: {target.model_name}")
    print(f"Pre-optimization score: {target.pre_optimization_score}")
    print(f"Post-optimization score: {target.post_optimization_score}")
    print(f"\nOptimized system prompt:\n{target.system_prompt}")
    print(f"\nOptimized template:\n{target.user_message_template}")
```

--------------------------------

### Select Model using Trained Router with Python SDK

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Selects an LLM using a previously trained custom router. Pass the `preference_id` obtained from training to route messages effectively. Training must be complete before this can be used.

```Python
from notdiamond import NotDiamond

client = NotDiamond()

llm_providers = [
    {"provider": "anthropic", "model": "claude-sonnet-4-5-20250929"},
    {"provider": "openai", "model": "gpt-5-2025-08-07"},
    {"provider": "google", "model": "gemini-2.5-pro"},
    {"provider": "openai", "model": "gpt-5-mini-2025-08-07"},
    {"provider": "anthropic", "model": "claude-opus-4-20250514"},
]

messages = [
    {"role": "user", "content": "Write merge sort in 3 lines."}
]

result = client.model_router.select_model(
    messages=messages,
    llm_providers=llm_providers,
    preference_id=preference_id,
)

print("ND session ID:", result.session_id)
print("LLM selected by router:", result)
```

--------------------------------

### Submit Prompt Optimization Request (Python SDK)

Source: https://docs.notdiamond.ai/docs/classification

Use the Not Diamond Python SDK to submit a prompt optimization request. Ensure your API key is set as an environment variable. This function returns an optimization run ID.

```python
import os

from notdiamond import NotDiamond

client = NotDiamond(api_key=os.environ.get("NOTDIAMOND_API_KEY"))

response = client.prompt_optimization.optimize(
    system_prompt=system_prompt,
    template=prompt_template,
    fields=fields,
    goldens=pa_ds,
    origin_model=origin_model,
    target_models=target_models,
    evaluation_metric="LLMaaJ:Sem_Sim_1"
)

optimization_run_id = response.optimization_run_id
print(f"Optimization job started: {optimization_run_id}")
```

--------------------------------

### Route with Custom Model (Python)

Source: https://docs.notdiamond.ai/docs/routing-between-custom-models

Use a trained custom router for model selection by passing the `preference_id` and the `llm_providers` list (including custom models) to `model_router.select_model()`.

```python
messages = [
    {"role": "user", "content": "Write merge sort in 3 lines."}
]

result = client.model_router.select_model(
    messages=messages,
    llm_providers=llm_providers,
    preference_id=preference_id,  # Preference ID from training
)

print("LLM selected:", result)
```

--------------------------------

### Define System Prompt

Source: https://docs.notdiamond.ai/docs/classification

Defines the system prompt for the assistant, instructing it to categorize banking-related questions.

```python
system_prompt = """
You are a helpful assistant that categorizes banking-related questions provided by the user.
"""

```

--------------------------------

### Optimize Model Selection for Cost

Source: https://docs.notdiamond.ai/docs/key-concepts

Select the best LLM by prioritizing cost-effectiveness. This tradeoff favors cheaper models when their quality is comparable to more expensive options.

```python
# Optimize for cost
result = client.model_router.select_model(
    messages=[{"role": "user", "content": "Hello"}],
    llm_providers=[
        {"provider": "openai", "model": "gpt-5-2025-08-07"},
        {"provider": "openai", "model": "gpt-5-mini-2025-08-07"}
    ],
    tradeoff="cost"  # Options: "cost", "latency", or None (quality)
```

--------------------------------

### Custom LLM-as-a-judge Configuration

Source: https://docs.notdiamond.ai/docs/evaluation-metrics

Configuration for a custom LLM-as-a-judge metric. It includes the prompt for the LLM judge, the model to use, and the cutoff score for correctness.

```Python
evaluation_config = {
    "llm_judging_prompt": (
        "Does the assistant's answer properly answer the user's question? question: {question} answer: {answer} "
        "Score a 1 if the answer is correct, 0 otherwise. Do not output any other values or text - only the score."
        ),
    "llm_judge": "openai/gpt-5-2025-08-07",
    "correctness_cutoff": 0,
}
```

--------------------------------

### Train Custom Router with Custom Model (TypeScript)

Source: https://docs.notdiamond.ai/docs/routing-between-custom-models

Train a custom router by including a custom model configuration in the `llm_providers` list. Ensure the custom model has unique provider/model names and defined quality/cost/latency parameters.

```typescript
import NotDiamond from 'notdiamond';
import * as fs from 'fs';

// Initialize ND client
const client = new NotDiamond({api_key: process.env.NOTDIAMOND_API_KEY});

// Define your custom model
const customModel = {
  provider: 'custom',
  model: 'my-finetuned-model',
  is_custom: true,
  context_length: 200000,
  input_price: 0.1,   // USD per million input tokens
  output_price: 0.2,  // USD per million output tokens
  latency: 0.01       // time to first token (seconds)
};

// LLMs to train router on (mix of standard and custom models)
const llmProviders = [
  { provider: 'openai', model: 'gpt-4o' },
  { provider: 'openai', model: 'gpt-4o-mini' },
  { provider: 'google', model: 'gemini-2.5-pro' },
  { provider: 'anthropic', model: 'claude-opus-4-20250514' },
  customModel,  // Add your custom model to the list
];

// Train the custom router (assuming the evaluation_data.csv exists)
const datasetFile = fs.createReadStream('evaluation_data.csv');

const response = await client.customRouter.trainCustomRouter({
  dataset_file: datasetFile,
  language: 'english',
  llm_providers: JSON.stringify(llmProviders),
  maximize: true,
  prompt_column: 'prompt',
});

const preferenceId = response.preference_id;
console.log('Custom router preference ID:', preferenceId);
```

--------------------------------

### Load JSON Dataset (TypeScript)

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Asynchronously loads a JSON dataset line by line from a file, formats each entry into a DataSample object, and returns a promise resolving to an array of these samples. It limits the number of samples loaded.

```TypeScript
import * as fs from 'fs';
import * as readline from 'readline';

interface DataSample {
  fields: { question: string; context: string };
  answer: string;
}

async function loadJsonDataset(datasetPath: string, nSamples: number = 200): Promise<DataSample[]> {
  const fileStream = fs.createReadStream(datasetPath);
  const rl = readline.createInterface({ input: fileStream, crlfDelay: Infinity });

  const goldenDataset: DataSample[] = [];

  for await (const line of rl) {
    if (goldenDataset.length >= nSamples) break;
    
    const row = JSON.parse(line);
    goldenDataset.push({
      fields: {
        question: row.question,
        context: row.documents.join('\n\n')
      },
      answer: row.response
    });
  }

  return goldenDataset;
}

// Define the train and test set sizes
const allGoldens = await loadJsonDataset('hotpotqa.jsonl', 20);
const trainGoldens = allGoldens.slice(0, 10);
const testGoldens = allGoldens.slice(10);

console.log(`Loaded ${trainGoldens.length} train samples and ${testGoldens.length} test samples`);
```

--------------------------------

### Train Custom Router with TypeScript SDK

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Trains a custom router using the NotDiamond TypeScript SDK. Requires a CSV dataset and specifies LLM providers. The dataset must contain at least 15 samples and can be up to 5MB or 10,000 samples.

```TypeScript
import NotDiamond from 'notdiamond';
import * as fs from 'fs';

// Initialize ND client
const client = new NotDiamond({api_key: process.env.NOTDIAMOND_API_KEY});

// LLMs to train router on
const llmProviders = [
  { provider: 'anthropic', model: 'claude-sonnet-4-5-20250929' },
  { provider: 'openai', model: 'gpt-5-2025-08-07' },
  { provider: 'google', model: 'gemini-2.5-pro' },
  { provider: 'openai', model: 'gpt-5-mini-2025-08-07' },
  { provider: 'anthropic', model: 'claude-opus-4-20250514' },
];

// Train the custom router
const datasetFile = fs.createReadStream('humaneval.csv');

const response = await client.customRouter.trainCustomRouter({
  dataset_file: datasetFile,
  language: 'english',
  llm_providers: JSON.stringify(llmProviders),
  maximize: true,
  prompt_column: 'Input',
});

const preferenceId = response.preference_id;
console.log('✅ Custom router preference ID:', preferenceId);
```

--------------------------------

### Retrieve Prompt Optimization Results (TypeScript SDK)

Source: https://docs.notdiamond.ai/docs/quickstart-prompt-optimization

Utilize the TypeScript SDK to monitor optimization job status and fetch detailed results, such as optimized prompts and performance scores for various target models.

```typescript
// Check status
const status = await client.promptOptimization.getOptimziationStatus(optimizationRunId);
console.log(`Status: ${status}`);

// Get results when complete
const results = await client.promptOptimization.getOptimziationStatus(optimizationRunId);

// View optimized prompts for each target model
for (const target of results.target_models) {
  console.log(`\nModel: ${target.model_name}`);
  console.log(`Pre-optimization score: ${target.pre_optimization_score}`);
  console.log(`\nOptimized system prompt:\n${target.system_prompt}`);
  console.log(`\nOptimized template:\n${target.user_message_template}`);
}
```

--------------------------------

### Select Model using Model Router (TypeScript)

Source: https://docs.notdiamond.ai/docs/quickstart-routing

Use the model router to select the best LLM provider for a query. You can specify messages, LLM providers to choose from, and a tradeoff preference (cost, latency, or quality).

```typescript
const result = await client.modelRouter.selectModel({
  messages: [
    { role: 'system', content: 'You are a world class programmer.' },
    { role: 'user', content: 'Concisely explain merge sort.' }  // Adjust as desired
  ],
  llmProviders: [
    { provider: 'openai', model: 'gpt-5-2025-08-07' },
    { provider: 'openai', model: 'gpt-5-nano-2025-08-07' },
    { provider: 'anthropic', model: 'claude-sonnet-4-20250514' }
  ],
  tradeoff: 'cost'  // Optional: 'cost', 'latency', or omit (default, optimizes for quality)
});
```

--------------------------------

### Retrieve Optimization Results (TypeScript SDK)

Source: https://docs.notdiamond.ai/docs/classification

Utilize the TypeScript SDK to monitor the status of an optimization run and access the optimized prompts and evaluation metrics upon completion.

```typescript
// Check job status
const status = await client.promptOptimization.getOptimizationStatus(optimizationRunId);
console.log(`Status: ${status}`);

// When status is 'completed', fetch results
const results = await client.promptOptimization.getOptimizationStatus(optimizationRunId);

// View optimized prompts
for (const target of results.targetModels) {
  console.log(`\nModel: ${target.modelName}`);
  console.log(`Pre-optimization score: ${target.preOptimizationScore}`);
  console.log(`Post-optimization score: ${target.postOptimizationScore}`);
  console.log(`\nOptimized system prompt:\n${target.systemPrompt}`);
  console.log(`\nOptimized template:\n${target.userMessageTemplate}`);
}
```

--------------------------------

### Download Evaluation Data

Source: https://docs.notdiamond.ai/docs/routing-between-custom-models

Download the evaluation data CSV file required for training custom routers.

```shell
curl -L "https://drive.google.com/uc?export=download&id=16gZsbU71HQue59-A4hFK_XwPIebR-CnF" -o evaluation_data.csv
```

--------------------------------

### Download Banking77 Dataset

Source: https://docs.notdiamond.ai/docs/classification

Downloads a specified number of samples from the PolyAI/banking77 dataset using the Hugging Face datasets library.

```python
from datasets import load_dataset

n_samples = 25

ds = load_dataset("PolyAI/banking77")["train"].select(list(range(n_samples)))
```

--------------------------------

### Define Prompt Template

Source: https://docs.notdiamond.ai/docs/classification

Defines the prompt template for the classification task, including placeholders for categories and the text to classify.

```python
prompt_template = """
Sharing a text from banking domain which needs to be classified into one of the 77 classes mentioned below
The text is customer support query.

Categories to classify the data :

{categories}

Text To classify : {question}
"""

```

--------------------------------

### Select Model using Trained Router with TypeScript SDK

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Selects an LLM using a previously trained custom router. Pass the `preferenceId` obtained from training to route messages effectively. Training must be complete before this can be used.

```TypeScript
import NotDiamond from 'notdiamond';

const client = new NotDiamond();

const llmProviders = [
  { provider: 'anthropic', model: 'claude-sonnet-4-5-20250929' },
  { provider: 'openai', model: 'gpt-5-2025-08-07' },
  { provider: 'google', model: 'gemini-2.5-pro' },
  { provider: 'openai', model: 'gpt-5-mini-2025-08-07' },
  { provider: 'anthropic', model: 'claude-opus-4-20250514' },
];

const messages = [
  { role: 'user', content: 'Write merge sort in 3 lines.' }
];

const result = await client.modelRouter.selectModel({
  messages: messages,
  llmProviders: llmProviders,
  preferenceId: preferenceId,
});

console.log('ND session ID:', result.sessionId);
console.log('LLM selected by router:', result);
```

--------------------------------

### Download Dataset

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Download the humaneval.csv dataset using curl.

```shell
curl -L "https://drive.google.com/uc?export=download&id=17hJS_-ecUnMPmRYjQCQmdK5HKP974QQW" -o humaneval.csv
```

--------------------------------

### TypeScript SDK - Using EXACT_MATCH

Source: https://docs.notdiamond.ai/docs/evaluation-metrics

Use the EXACT_MATCH metric for classification tasks in the TypeScript SDK. Ensure the API key is set.

```typescript
import NotDiamond from 'notdiamond';

const client = new NotDiamond({api_key: process.env.NOTDIAMOND_API_KEY});

const response = await client.promptOptimization.optimize({
  systemPrompt: systemPrompt,
  template: promptTemplate,
  fields: fields,
  goldens: goldenData,
  originModel: originModel,
  targetModels: targetModels,
  evaluationMetric: 'EXACT_MATCH'  // Use exact match for classification tasks
});
```

--------------------------------

### Set API Key

Source: https://docs.notdiamond.ai/docs/evaluation-metrics

Set your NotDiamond API key as an environment variable.

```shell
export NOTDIAMOND_API_KEY=YOUR_NOTDIAMOND_API_KEY
```

--------------------------------

### Update Custom Router with Python SDK

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Use this snippet to update an existing custom router by providing a new dataset file and setting override to True. Ensure the preference ID is reused to update the correct router.

```Python
with open("humaneval.csv", "rb") as f:
    response = client.custom_router.train_custom_router(
        dataset_file=f,
        language="english",
        llm_providers=json.dumps(llm_providers),
        maximize=True,
        prompt_column="Input",
        preference_id=preference_id,  # reuse the existing router
        override=True,                # overwrite the previous router
    )

print("Router updated:", response.preference_id)
```

--------------------------------

### Custom Evaluation Metric Configuration

Source: https://docs.notdiamond.ai/docs/requirements-and-concepts

Defines a custom evaluation metric using an LLM judge and a specific prompt. Allows for tailored quality assessment with a configurable correctness cutoff.

```python
"evaluation_config": {
    "llm_judging_prompt": "Your custom judging prompt with {question} and {answer}",
    "llm_judge": "openai/gpt-5-2025-08-07",
    "correctness_cutoff": 0
}
```

--------------------------------

### Update Custom Router with TypeScript SDK

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Use this snippet to update an existing custom router using the TypeScript SDK. Provide the new dataset file and set override to true. Reusing the preference ID ensures the correct router is updated.

```TypeScript
const datasetFile = fs.createReadStream('humaneval.csv');

const response = await client.customRouter.trainCustomRouter({
  datasetFile: datasetFile,
  language: 'english',
  llmProviders: JSON.stringify(llmProviders),
  maximize: true,
  promptColumn: 'Input',
  preferenceId: preferenceId,  // reuse the existing router
  override: true,              // overwrite the previous router
});

console.log('Router updated:', response.preferenceId);
```

--------------------------------

### Route with Custom Model (TypeScript)

Source: https://docs.notdiamond.ai/docs/routing-between-custom-models

Use a trained custom router for model selection by passing the `preferenceId` and the `llmProviders` list (including custom models) to `modelRouter.selectModel()`.

```typescript
const messages = [
  { role: 'user', content: 'Write merge sort in 3 lines.' }
];

const result = await client.modelRouter.selectModel({
  messages: messages,
  llmProviders: llmProviders,
  preferenceId: preferenceId,  // Preference ID from training
});

console.log('LLM selected:', result);
```

--------------------------------

### Semantic Similarity Prompt (LLM-as-a-judge)

Source: https://docs.notdiamond.ai/docs/evaluation-metrics

This prompt is used to evaluate the semantic similarity between a predicted answer and a reference answer using an LLM judge. It outputs a binary score (0 or 1).

```Text
Given the predicted answer and reference answer, compare them and check whether they mean the same.

Following are the given answers:

Predicted Answer: {predicted_answer}
Reference Answer: {gt_answer}

On a NEW LINE, give a score of 1 if the answers mean the same, and 0 if they differ, and nothing more.
```

--------------------------------

### Rename Dataset Fields (macOS/Linux)

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Rename the 'final_score' column to 'score' in humaneval.csv using sed.

```shell
sed -i '' 's/final_score/score/g' humaneval.csv
```

--------------------------------

### Rename Dataset Fields (Colab)

Source: https://docs.notdiamond.ai/docs/router-training-quickstart

Rename the 'final_score' column to 'score' in humaneval.csv using sed for Colab environments.

```shell
sed -i 's/final_score/score/g' humaneval.csv
```
Tokens:
Raw