{original_truncated.replace('<', '<').replace('>', '>')}
{optimized_truncated.replace('<', '<').replace('>', '>')}
"""
display(HTML(html_output))
else:
print("No result files found. Make sure the optimization process completed successfully.")
```
--------------------------------
### Create a New Prompt Optimization Project
Source: https://context7.com/meta-llama/prompt-ops/llms.txt
Scaffolds a new project directory with essential files for prompt optimization. Use the default model or specify a different provider and model.
```bash
# Create a project using the default OpenRouter Llama model
prompt-ops create my-project
```
```bash
# Create with a different model provider (Groq)
prompt-ops create my-project --model groq/meta-llama/llama-4-maverick-17b-128e-instruct
```
```text
# Resulting project layout:
# my-project/
# ├── .env ← add your API key here
# ├── README.md
# ├── config.yaml
# ├── data/
# │ └── dataset.json ← sample QA pairs
# ├── prompts/
# │ └── prompt.txt ← original system prompt
# └── results/ ← created on first run
```
--------------------------------
### Clone prompt-ops repository
Source: https://github.com/meta-llama/prompt-ops/blob/main/notebook/prompt_ops_101_cerebras.ipynb
Use this command to clone the prompt-ops repository from GitHub. Ensure you have git installed and configured.
```bash
git clone https://github.com/meta-llama/prompt-ops.git
```
--------------------------------
### Initialize and Use PromptMigrator in Python
Source: https://context7.com/meta-llama/prompt-ops/llms.txt
Set up models, metrics, and datasets to optimize prompts using the PromptMigrator. This involves configuring DSPy adapters and loading datasets before running the optimization and evaluation.
```python
import dspy
from prompt_ops.core.migrator import PromptMigrator
from prompt_ops.core.prompt_strategies import BasicOptimizationStrategy
from prompt_ops.core.datasets import ConfigurableJSONAdapter, load_dataset
from prompt_ops.core.metrics import FacilityMetric
from prompt_ops.core.model import setup_model
# 1. Set up model (DSPy adapter for BasicOptimizationStrategy)
task_model = setup_model(
model_name="openrouter/meta-llama/llama-3.3-70b-instruct",
adapter_type="dspy",
api_key="sk-or-...",
max_tokens=2048,
temperature=0.0,
)
# 2. Set up metric and dataset
metric = FacilityMetric(output_field="answer", strict_json=False)
adapter = ConfigurableJSONAdapter(
dataset_path="data/dataset.json",
input_field=["fields", "input"],
golden_output_field="answer",
)
# 3. Build strategy and migrator
strategy = BasicOptimizationStrategy(
model_name="llama-3.3-70b-instruct",
metric=metric,
task_model=task_model,
prompt_model=task_model,
auto="basic", # "basic" | "intermediate" | "advanced"
num_threads=4,
compute_baseline=True,
)
migrator = PromptMigrator(
strategy=strategy,
task_model=task_model,
prompt_model=task_model,
)
# 4. Load dataset with train/val/test splits
trainset, valset, testset = migrator.load_dataset_with_adapter(
adapter,
train_size=0.25,
validation_size=0.25,
)
# 5. Optimize
original_prompt = open("prompts/prompt.txt").read()
optimized_program = migrator.optimize(
prompt_data={
"text": original_prompt,
"inputs": ["question"],
"outputs": ["answer"],
},
trainset=trainset,
valset=valset,
testset=testset,
save_to_file=True,
file_path="results/optimized.json",
save_yaml=True,
)
print(optimized_program.signature.instructions)
# → "You are an expert in analyzing customer service messages. Your task is..."
# 6. Evaluate on test set
score = migrator.evaluate(program=optimized_program, devset=testset)
print(f"Test score: {score:.3f}")
```
--------------------------------
### Display Sample System Prompt
Source: https://github.com/meta-llama/prompt-ops/blob/main/notebook/prompt_ops_101_cerebras.ipynb
Use the `cat` command to display the content of the `prompt.txt` file. This file contains the system prompt that will be used for optimization and evaluation.
```python
!cat my-cerebras-project/prompts/prompt.txt
```
--------------------------------
### Troubleshoot Import Errors
Source: https://github.com/meta-llama/prompt-ops/blob/main/frontend/backend/tests/README.md
If tests fail with import errors, ensure you are in the correct directory (`frontend/backend`) and have installed all necessary dependencies.
```bash
cd frontend/backend
pip install -r requirements.txt
```
--------------------------------
### Unit Test Template
Source: https://github.com/meta-llama/prompt-ops/blob/main/frontend/backend/tests/README.md
A template for writing unit tests, including setup for testing a specific function and handling expected exceptions.
```python
"""Unit tests for new_module.py"""
import pytest
from new_module import function_to_test
class TestNewFeature:
"""Tests for new feature."""
def test_basic_functionality(self):
"""Test basic use case."""
result = function_to_test("input")
assert result == "expected"
def test_error_handling(self):
"""Test error conditions."""
with pytest.raises(ValueError):
function_to_test("invalid")
```
--------------------------------
### Configure Environment Variables
Source: https://github.com/meta-llama/prompt-ops/blob/main/frontend/README.md
Create a `.env` file in the `frontend/backend` directory to store API keys. Ensure the `OPENROUTER_API_KEY` is provided.
```bash
# In frontend/backend/.env
OPENROUTER_API_KEY=your_openrouter_api_key_here
OPENAI_API_KEY=your_openai_api_key_here # Optional: for fallback enhance feature
```
--------------------------------
### Configure BasicOptimizationStrategy in Python
Source: https://context7.com/meta-llama/prompt-ops/llms.txt
Instantiate BasicOptimizationStrategy with a specific metric and model configuration for prompt optimization. This strategy uses DSPy's MIPROv2 optimizer for bootstrapped demo generation and Bayesian instruction search.
```python
from prompt_ops.core.prompt_strategies import BasicOptimizationStrategy
from prompt_ops.core.metrics import StandardJSONMetric
metric = StandardJSONMetric(
output_fields=["urgency", "sentiment"],
required_fields=["urgency", "sentiment", "categories"],
nested_fields={"categories": ["emergency_repair", "routine_maintenance"]},
strict_json=False,
)
strategy = BasicOptimizationStrategy(
model_name="llama-3.3-70b-instruct",
metric=metric,
task_model=task_model,
prompt_model=task_model,
# MIPROv2 parameters
auto="basic", # maps to DSPy "light" mode (~10 trials)
max_bootstrapped_demos=4,
max_labeled_demos=5,
num_candidates=10,
minibatch=True,
minibatch_size=25,
compute_baseline=True, # evaluate original prompt first
num_threads=4,
seed=42,
)
# strategy.run() is called internally by PromptMigrator.optimize()
optimized = strategy.run({
"text": "You are a helpful assistant. Extract urgency, sentiment, and categories as JSON.",
"inputs": ["question"],
"outputs": ["answer"],
})
print(optimized.signature.instructions)
```
--------------------------------
### Store Optimized Prompt
Source: https://github.com/meta-llama/prompt-ops/blob/main/use-cases/ms-marco-pdo/MSMARCO_PDO_eval.ipynb
This variable stores the final optimized prompt string after the PDO process. It is designed to guide the model in providing clear, step-by-step explanations for user questions.
```python
OPTIMIZED_PROMPT = (
"""Embody the role of a knowledgeable researcher and provide a clear,
step-by-step explanation to answer the user's question, utilizing the provided
context and avoiding technical jargon unless absolutely necessary, to ensure a
concise and accurate response."""
)
```
--------------------------------
### Configure Prompt with Inline Text or File
Source: https://github.com/meta-llama/prompt-ops/blob/main/docs/intermediate/readme.md
Set up your prompt using either inline text with placeholders or by referencing an external file. Specify which dataset fields map to prompt inputs and which model response fields to capture.
```yaml
prompt:
# Option 1: Inline Text
text: |
Giving the following message:
---
{{question}}
---
Extract and return a json with the following keys and values:
- "urgency" as one of `high`, `medium`, `low`
- "sentiment" as one of `negative`, `neutral`, `positive`
- "categories" Create a dictionary with categories as keys and boolean values...
# Option 2: File Path
file: "../use-cases/facility-support-analyzer/facility_prompt_sys.txt"
inputs: ["question"]
outputs: ["answer"]
```
--------------------------------
### Set up Training Arguments for Fine-tuning
Source: https://github.com/meta-llama/prompt-ops/blob/main/notebook/prompt_ops_101_cerebras.ipynb
Configures the training arguments for fine-tuning the model. This includes setting the output directory, learning rate, and number of epochs.
```python
from transformers import TrainingArguments, Trainer
output_dir = "./cerebras-gpt-111m-finetuned"
training_args = TrainingArguments(
output_dir=output_dir,
overwrite_output_dir=True,
num_train_epochs=1,
per_device_train_batch_size=4,
save_steps=10_000,
save_total_limit=2,
learning_rate=2e-5,
warmup_steps=500,
logging_dir=f"{output_dir}/logs",
logging_steps=50,
fp16=True # Use mixed precision if supported
)
```
--------------------------------
### Run PDO Migration
Source: https://github.com/meta-llama/prompt-ops/blob/main/use-cases/ms-marco-pdo/MSMARCO_PDO_eval.ipynb
This code snippet loads environment variables, including the OPENROUTER_API_KEY, and then executes the prompt-ops migrate command using a specified configuration file. Ensure your .env file and config.yaml are correctly set up.
```python
from dotenv import load_dotenv
load_dotenv("/Users/yuanchenwu/Downloads/prompt-ops/local.env")
!prompt-ops migrate --config config.yaml --log-level INFO
```
--------------------------------
### Run Migration with Configuration
Source: https://github.com/meta-llama/prompt-ops/blob/main/docs/inference_providers.md
Use this command to run a migration with a specified configuration file. Ensure the config file path is correct.
```bash
prompt-ops migrate --config configs/your_config.yaml
```
--------------------------------
### Evaluate Optimized Prompt with LLM Judge
Source: https://github.com/meta-llama/prompt-ops/blob/main/use-cases/ms-marco-pdo/MSMARCO_PDO_eval.ipynb
This function scores predictions against ground truth using a judge prompt and guided JSON decoding. It falls back to free-form decoding if JSON schema is unsupported.
```python
def judge_scores(preds: list[str]):
"""Judge with guided JSON decoding (fallback to free-form if unsupported)."""
import json as _json
import numpy as _np
scores = []
explanations = []
judge_prompts = []
for pred, r in zip(preds, rows):
judge_prompts.append(
JUDGE_PROMPT.format(
question=r["question"],
ground_truth=r["answer"],
prediction=pred,
)
)
response_format = {
"type": "json_schema",
"json_schema": {
"name": "judge_score",
"schema": {
"type": "object",
"properties": {
"explanation": {
"type": "string",
"description": "Detailed reasoning based on the comparison",
},
"score": {
"type": "integer",
"description": "Score from 1 to 5",
"enum": [1, 2, 3, 4, 5],
},
},
"required": ["explanation", "score"],
"additionalProperties": False,
},
"strict": True,
},
}
try:
outs = model.generate_batch(
judge_prompts, max_threads=10, temperature=0.0, response_format=response_format
)
except Exception:
outs = model.generate_batch(
judge_prompts, max_threads=10, temperature=0.0
)
for o in outs:
obj = _try_json_parse(o)
score = obj.get("score", 0)
exp = obj.get("explanation", "")
try:
score = float(score)
except Exception:
score = 0.0
scores.append(score)
explanations.append(exp)
return _np.array(scores), explanations
```
--------------------------------
### Define Optimized Prompt String
Source: https://github.com/meta-llama/prompt-ops/blob/main/use-cases/web-of-lies-pdo/PDO_web_of_lies_eval.ipynb
Stores a detailed, optimized prompt string designed for logical reasoning tasks like Web of Lies. The prompt guides the model through analyzing statements, making assumptions, and determining truthfulness.
```python
optimized_prompt='''Analyze the given statements about individuals telling the truth or lying, state assumptions about the truthfulness of each person based on the chain of statements, and determine the truthfulness of the last person mentioned in the question. For example, if Person A says Person B is lying and Person B says Person C is telling the truth, and it's given that Person A is lying, then Person B must be telling the truth, which means Person C is also telling the truth. Another example: if Person X claims Person Y is lying and Person Y says Person Z is truthful, but it's known that Person X is truthful, then Person Y is lying, and thus Person Z's truthfulness depends on the statement Person Y made about them. Apply this reasoning process to the given statements, as in the case where Person P says Person Q is lying, Person Q says Person R is telling the truth, and Person R says Person S is lying - determine the truthfulness of Person S based on the assumptions made about each person's statement.'''
```
--------------------------------
### Load and Display Optimized Prompt
Source: https://github.com/meta-llama/prompt-ops/blob/main/notebook/prompt_ops_101_cerebras.ipynb
This Python code loads the latest optimized prompt from the results directory, extracts the system prompt, truncates few-shot examples for readability, and prepares it for HTML display. It requires glob, os, re, and IPython.display.
```python
import glob
import os
import re
from IPython.display import display, HTML
def truncate_examples(text, max_question_lines=3, max_answer_chars=100):
"""
Truncates few-shot examples to make them more readable.
Shows only the first few lines of questions and truncates answers.
"""
def truncate_section(match):
full_text = match.group(0)
# Extract question part
question_match = re.search(r'Question:(.*?)Answer:', full_text, re.DOTALL)
answer_match = re.search(r'Answer:(.*?)(?=Example \d+:|$)', full_text, re.DOTALL)
result = match.group(1) # Keep "Example N:"
if question_match:
question = question_match.group(1).strip()
lines = question.split('\n')
if len(lines) > max_question_lines:
truncated = '\n'.join(lines[:max_question_lines])
result += f"\n Question: {truncated}\n [...truncated {len(lines) - max_question_lines} more lines...]"
else:
result += f"\n Question: {question}"
if answer_match:
answer = answer_match.group(1).strip()
if len(answer) > max_answer_chars:
result += f"\n Answer: {answer[:max_answer_chars]}... [truncated]"
else:
result += f"\n Answer: {answer}"
return result + "\n "
# Find and truncate each example
pattern = r'(Example \d+:)(.*?)(?=Example \d+:|$)'
truncated = re.sub(pattern, truncate_section, text, flags=re.DOTALL)
return truncated
result_files = glob.glob('my-cerebras-project/results/*.yaml')
if result_files:
latest_result = max(result_files, key=os.path.getctime)
# Simple approach: just read and split
with open(latest_result, 'r') as f:
content = f.read()
# Split on 'config:' and take the first part (the system prompt)
system_section = content.split('config:')[0]
system_prompt = system_section.replace('system: |-', '').strip()
# Truncate the examples
system_prompt_truncated = truncate_examples(system_prompt)
# Create HTML output with word wrapping (dark theme)
html_output = f"""